gstreamer module

Table of Contents

• Introduction
• Streaming readout (upcoming in the next release)
• Ownership and lifecycle
• Architecture
  • Core abstractions
  • Plugin layer
  • Generated-particle banks
  • Design notes
• The list of Available Options and their usage
  • Primary gstreamer options
  • Options from dependent modules
• Module verbosity
• Examples

Plugin-based data publication

Introduction

The gstreamer module is the output layer responsible for taking in-memory GEMC data collections and serializing them through a plugin architecture. The module supports multiple output backends, allowing the same event or frame production flow to be directed to different storage formats without changing the event-building code.

In practice, a user configures one or more outputs through the -gstreamer option, and the module loads the corresponding plugin libraries at runtime. Each plugin then receives a high-level publish sequence from the common GStreamer interface.

Supported built-in format tokens currently include:

• root
• ascii
• csv
• sro (upcoming in the next release)
• json

The ordinary event/run formats map to plugins using the standard naming convention:

• gstreamer_root_plugin
• gstreamer_ascii_plugin
• gstreamer_csv_plugin
• gstreamer_json_plugin

Streaming readout (upcoming in the next release)

Configure one shared SRO output alongside any ordinary event outputs:

gstreamer:
  - format: sro
    filename: output
    implementation: experiment_sro

The core GSROFactory selects this output and loads experiment_sro.gplugin through the normal plugin search path. The library exports GSROImplementationFactory and derives from GSROImplementation. Its configure_run supplies the timing model; create_crate supplies a frame sink and crate plugin. Filename suffixes, encoding, acquisition limits, overlap response, and incomplete-frame policy belong to that implementation. It must give each crate/run a distinct output filename.

Worker digitizers override GDynamicDigitization::stream_hit and invoke GSROEmit immediately for each owned GSROData contribution. The framework assigns event/sequence IDs, creates crates lazily, and calls complete_event even for empty events. An aborted event holds back the safe-time prefix. At run end the master (or sequential run owner) waits for delivery acknowledgments and joins all crate threads. A background or worker error cancels SRO input; checked shutdown reports it after joining.

SRO has no per-worker event buffer. The existing ebuffer option applies only to ordinary streamers. Crate queues apply backpressure; ordering buffers and event bookkeeping have separate failure limits. The old jlabsro factory and its fixed binary layout are replaced by this implementation API.

Ownership and lifecycle

The module is designed around explicit ownership boundaries:

• Configuration is provided through GOptions.
• Streamer definitions are represented by GStreamerDefinition values.
• Concrete streamer instances are created dynamically by the plugin manager.
• In multithreaded usage, each worker thread typically owns its own streamer map and therefore its own plugin instances.

Event data ownership remains outside the streamer. The base streamer only stores std::shared_ptr<GEventDataCollection> objects in its buffer so that all event-owned hit data remain valid until the flush is complete. Concrete plugins receive read-only views during publication and do not take ownership of the source event content.

The normal lifecycle is:

• define options
• parse user configuration
• create per-thread streamer instances
• call define_gstreamer()
• call set_loggers()
• open connections
• publish run, event, or frame data
• close connections

Architecture

The architecture is intentionally layered:

Core abstractions

• GStreamer : abstract base class implementing buffering and the publish flow.
• GStreamerDefinition : value type describing one configured output.
• gstreamer namespace helpers : parse option nodes and instantiate per-thread streamer maps.

Plugin layer

Each concrete plugin derives from GStreamer and overrides only the hooks relevant to its output model. For event publication, the base class drives a fixed sequence:

• startEvent()
• publishEventHeader()
• publishEventGeneratedParticles()
• publishEventTrueInfoData()
• publishEventDigitizedData()
• endEvent()

Similar hook sequences exist for run data and frame streams.

Buffered event publication and plugin hook order

Generated-particle banks

Event output includes two generated-particle banks when generator metadata is available:

• generated : runtime records for particles propagated in Geant4 plus source-only file rows that are not propagated.
• generated_tracked : runtime records for particles propagated in Geant4. For Lund input, this corresponds to rows with type == 1.

Both banks publish the same columns: particle name, pid, source type, multiplicity, momentum, theta, phi, and vertex components. The concrete representation is backend-specific:

• ASCII writes named generated-particle bank blocks.
• CSV appends _generated.csv and _generated_tracked.csv to the configured root filename.
• JSON writes both banks under the event-level generated object.
• ROOT writes TTrees named generated and generated_tracked.

Design notes

Design choices in this module include:

• buffering is centralized in the base class so plugins stay focused on serialization
• format discovery is runtime-based through plugin naming conventions
• streamer instances are expected to be thread-local in normal operation
• frame streaming flushes pending event buffers first so event and frame outputs are not mixed unintentionally in the same stream

The list of Available Options and their usage

The module contributes the following user-facing options through gstreamer::defineOptions() :

Primary gstreamer options

• ebuffer
  • Meaning : number of events buffered in memory before automatic flush
  • Usage : tune batching behavior for throughput versus memory usage
  • Default : gstreamer::DEFAULT_GSTREAMER_BUFFER_FLUSH_LIMIT
• -gstreamer
  • Meaning : list of output definitions
  • Required fields per entry :
    • format
    • filename
  • Optional field per entry :
    • type, defaulting to "event"

Example:

-gstreamer="[{format: root, filename: out_root}, {format: ascii, filename: out_txt, type: event}]"

Options from dependent modules

The module also aggregates options defined by the gdynamicdigitization module through gdynamicdigitization::defineOptions(). Those options apply when the example or application simultaneously uses dynamic digitization and streaming.

Module verbosity

Classes in this module log through the module logger category "gstreamer".

Typical verbosity interpretation is:

• level 0 : high-level operational progress such as worker startup, output opening, and summary actions
• level 1 : plugin and connection management details such as file open and close messages
• level 2 : detailed publish flow, including event flush activity, per-detector output actions, and tree creation in plugins such as the ROOT backend

Debug output is intended for developers. It prints function-level traces, lifecycle transitions, and internal state information that help diagnose ordering, buffering, and plugin behavior.

Examples

The following example is documented as part of this module:

• gstreamer_example : multithreaded synthetic event publication using per-thread streamer maps

Example snippet:

auto gopts = std::make_shared<GOptions>(argc, argv, gstreamer::defineOptions());
auto gstreamer_map = gstreamer::gstreamersMapPtr(gopts, tid);
 
for (const auto& [name, streamer] : *gstreamer_map) {
    streamer->openConnection();
    streamer->publishEventData(eventData);
    streamer->closeConnection();
}