gstreamer module

Introduction

The gstreamer module is the output layer responsible for taking in-memory GEMC data collections and serializing them through a plugin architecture. The module supports multiple output backends, allowing the same event or frame production flow to be directed to different storage formats without changing the event-building code.

In practice, a user configures one or more outputs through the -gstreamer option, and the module loads the corresponding plugin libraries at runtime. Each plugin then receives a high-level publish sequence from the common GStreamer interface.

Supported built-in format tokens currently include:

• root
• ascii
• csv
• jlabsro
• json

These map to plugin names using the standard naming convention:

• gstreamer_root_plugin
• gstreamer_ascii_plugin
• gstreamer_csv_plugin
• gstreamer_jlabsro_plugin
• gstreamer_json_plugin

Ownership and lifecycle

The module is designed around explicit ownership boundaries:

• Configuration is provided through GOptions.
• Streamer definitions are represented by GStreamerDefinition values.
• Concrete streamer instances are created dynamically by the plugin manager.
• In multithreaded usage, each worker thread typically owns its own streamer map and therefore its own plugin instances.

Event data ownership remains outside the streamer. The base streamer only stores std::shared_ptr<GEventDataCollection> objects in its buffer so that all event-owned hit data remain valid until the flush is complete. Concrete plugins receive read-only views during publication and do not take ownership of the source event content.

The normal lifecycle is:

• define options
• parse user configuration
• create per-thread streamer instances
• call define_gstreamer()
• call set_loggers()
• open connections
• publish run, event, or frame data
• close connections

Architecture

The architecture is intentionally layered:

Core abstractions

• GStreamer : abstract base class implementing buffering and the publish flow.
• GStreamerDefinition : value type describing one configured output.
• gstreamer namespace helpers : parse option nodes and instantiate per-thread streamer maps.

Plugin layer

Each concrete plugin derives from GStreamer and overrides only the hooks relevant to its output model. For event publication, the base class drives a fixed sequence:

• startEvent()
• publishEventHeader()
• publishEventGeneratedParticles()
• publishEventTrueInfoData()
• publishEventDigitizedData()
• endEvent()

Similar hook sequences exist for run data and frame streams.

Generated-particle banks

Event output includes two generated-particle banks when generator metadata is available:

• generated : runtime records for particles propagated in Geant4 plus source-only file rows that are not propagated.
• generated_tracked : runtime records for particles propagated in Geant4. For Lund input, this corresponds to rows with type == 1.

Both banks publish the same columns: particle name, pid, source type, multiplicity, momentum, theta, phi, and vertex components. The concrete representation is backend-specific:

• ASCII writes named generated-particle bank blocks.
• CSV writes <rootname>_generated.csv and <rootname>_generated_tracked.csv.
• JSON writes both banks under the event-level generated object.
• ROOT writes TTrees named generated and generated_tracked.

Design notes

Design choices in this module include:

• buffering is centralized in the base class so plugins stay focused on serialization
• format discovery is runtime-based through plugin naming conventions
• streamer instances are expected to be thread-local in normal operation
• frame streaming flushes pending event buffers first so event and frame outputs are not mixed unintentionally in the same stream

The list of Available Options and their usage

The module contributes the following user-facing options through gstreamer::defineOptions() :

Primary gstreamer options

• ebuffer
  • Meaning : number of events buffered in memory before automatic flush
  • Usage : tune batching behavior for throughput versus memory usage
  • Default : DEFAULT_GSTREAMER_BUFFER_FLUSH_LIMIT
• -gstreamer
  • Meaning : list of output definitions
  • Required fields per entry :
    • format
    • filename
  • Optional field per entry :
    • type, defaulting to "event"

Example:

-gstreamer="[{format: root, filename: out_root}, {format: ascii, filename: out_txt, type: event}]"

Options from dependent modules

The module also aggregates options defined by the gdynamicdigitization module through gdynamicdigitization::defineOptions(). Those options apply when the example or application simultaneously uses dynamic digitization and streaming.

Module verbosity

Classes in this module log through the module logger category "gstreamer".

Typical verbosity interpretation is:

• level 0 : high-level operational progress such as worker startup, output opening, and summary actions
• level 1 : plugin and connection management details such as file open and close messages
• level 2 : detailed publish flow, including event flush activity, per-detector output actions, and tree creation in plugins such as the ROOT backend

Debug output is intended for developers. It prints function-level traces, lifecycle transitions, and internal state information that help diagnose ordering, buffering, and plugin behavior.

Examples

The following example is documented as part of this module:

• gstreamer_example : multithreaded synthetic event publication using per-thread streamer maps

Example snippet:

auto gopts = std::make_shared<GOptions>(argc, argv, gstreamer::defineOptions());
auto gstreamer_map = gstreamer::gstreamersMapPtr(gopts, tid);
 
for (const auto& [name, streamer] : *gstreamer_map) {
    streamer->openConnection();
    streamer->publishEventData(eventData);
    streamer->closeConnection();
}