deploy: 991eb75

commands/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 1b1668d679e29e7450236151be693e06
+tags: 645f666f9bcd5a90fca523b33c5a78b7

commands/_sources/developer/explanations/design-decisions.md.txt

@@ -0,0 +1,31 @@
+# Design Decisions
+
+The frameReceiver and frameProcessor are implemented as two separate applications for a
+few key reasons:
+
+## Separation of concerns
+
+There are fundamentally two separate problem domains: data capture - receiving data from
+the network interface onto the system - and data processing. These two operations can be
+almost completely isolated with no coordination of the two processes other than to pass
+pointers to data frames in memory. By splitting the data acquisition process into these
+two parts they can be developed and debugged in isolation rather than within in a single
+process with multiple threads.
+
+By focusing one process on data capture, it is possible to optimise as required, such
+as by allowing it higher priority to ensure sufficient resources. Once data is on the
+system, it enters the domain of procesing and writing to persistent storage. In this
+domain bottlenecks in the processing can be overcome by increasing the resources of the
+processing server, such as by adding more memory, or by optimising the logic of an
+isolated FrameProcessorPlugin.
+
+## Monitoring
+
+The first failure point in the data acquisition pipeline is receiving data from the
+network interface into system memory. If this is not done quickly enough, packets will
+be dropped at the kernel layer with no indication of what went wrong. Allowing the
+frameReceiver to do the minimum work to gather data into memory means this usage can be
+monitored and when it is exhausted it will be clear where data was lost. This monitoring
+extends throughout the frameProcessor through processing time metrics for each plugin
+and watchdog timers around certain I/O to show when a call is blocking for longer than
+expected.

commands/_sources/developer/explanations/index.md.txt

@@ -0,0 +1,4 @@
+# **Explanations**
+
+```{tableofcontents}
+```

commands/_sources/developer/explanations/overview.md.txt

@@ -0,0 +1,39 @@
+# Overview
+
+The design goal for odin-data is to provide a generic, high-performance, scalable data
+capture and processing framework upon which specific detector data acquisition systems
+can be built with dynamically loaded plugins.
+
+The framework consists of two separate C++ applications; the frameReceiver and
+frameProcessor. The frameReceiver focuses on data capture - receiving data from a
+network interface into system memory buffers. The frameProcessor listens on a ZeroMQ
+channel for messages from the frameReceiver containing a pointer to a data buffer in
+shared memory, which it will then pass through a set of plugins, before notifying the
+frameReceiver that it can re-use the shared memory.
+
+The intended use case is to run multiple frameReceiver-frameProcessor pairs (nodes),
+possibly across multiple servers, depending on the data throughput and procesing
+requirements. However, an individual node does not communicate with other nodes - it is
+simply made aware of the total number of nodes and its own rank in the overall system.
+This lets each node verify that it is receiving the correct frame numbers (in temporal
+mode), or the correct data channels (in geographical mode).
+
+## IPC Interface
+
+The `--ready` and `--release` arguments of the frameReceiver and frameProcessor
+configure the ZeroMQ channels used to send and receive messages to manage access to
+shared memory buffers.
+
+Without an application to listen to the frameReceiver ready messages and send release
+messages in response, it will exhaust the shared memory buffers and then ignore all
+further data. However, the frameProcessor can be used as a diagnostic without doing
+anything with the shared buffers: if no plugins are configured, the application will
+still communicate with the frameReceiver and immediately releasing the shared buffers it
+receives.
+
+Where possible, the frame data transferred through a shared memory buffer is processed
+in place to minimise the number of copies. However some processing requires a new memory
+buffer to output to. This is a decision to be made for each individual process plugin.
+See [Types of Frame] for more details.
+
+[Types of Frame]: ../how-to/frame-processor-plugin.md#types-of-frame

commands/_sources/developer/how-to/common.md.txt

@@ -0,0 +1,49 @@
+# Common API
+
+The frameProcessor and frameReceiver share some common code, mainly relating to the API
+between them.
+
+## IPC Interface
+
+The [IpcChannel] and [IpcMessage] classes are used for all interprocess communications -
+both between frameReceiver and frameProcesor as well as for the control interface for
+client applications. The [IpcMessage] class has some enumerated message types for common
+functions.
+
+## SharedBufferManager
+
+The SharedBufferManager is the shared code interface used by both applications to
+access the shared memory buffers. This class allows accessing buffers by a simple index,
+which simplifies the communication over the ready and release channels required
+manage access to the shared memory.
+
+## Detector Constants and Buffer Header
+
+Detector implementations should create a definitions header containing any constants,
+utilities and most importantly a common frame header struct that can be used in both the
+[FrameDecoder] and a [FrameProcessorPlugin] to read and write to buffers in shared
+memory. With this common API, the decoder can insert the header in front of the data blob
+in shared memory and the processor plugin can simply cast the void pointer into a pointer
+to a detector specific header to access the members and offset that pointer by the size
+of the header to access the data.
+
+## Dynamic Class Versioning
+
+As dynamically loadable classes, both the [FrameDecoder] and [FrameProcessorPlugin]
+inherit [IVersionedObject], so must implement the following methods:
+
+- `get_version_major`
+- `get_version_minor`
+- `get_version_patch`
+- `get_version_short`
+- `get_version_long`
+
+These methods are used to provide version information to clients about the exact classes
+that have been loaded into the applications.
+
+[IpcChannel]: OdinData::IpcChannel
+[IpcMessage]: OdinData::IpcMessage
+[FrameDecoder]: FrameReceiver::FrameDecoder
+[FrameProcessorPlugin]: FrameProcessor::FrameProcessorPlugin
+[IVersionedObject]: OdinData::IVersionedObject
+

commands/_sources/developer/how-to/docs.md.txt

@@ -0,0 +1,46 @@
+# Docs
+
+The documentation for odin-data uses
+[sphinx](https://www.sphinx-doc.org/en/master/index.html) and [MyST](https://myst-parser.readthedocs.io/en/latest/index.html)
+for written documentation along with
+[doxygen](https://www.doxygen.nl/index.html) and [breathe](https://breathe.readthedocs.io/en/latest/)
+to generate reference API docs and integrate them into the docs.
+
+
+## Building Docs
+
+To generate the API reference docs using doxygen (requires `dot` for UML diagrams):
+
+    $ docs/doxygen.sh
+
+To build and view the documentation:
+
+```bash
+$ python -m venv venv
+$ source venv/bin/activate
+(venv) $ pip install ./python[dev]
+(venv) $ sphinx-build -b html docs docs/build/html
+```
+
+This will build the docs into `docs/build/html`. They can be viewed by opening
+`index.html` in a web browser. When editing the docs it is useful to get a live preview.
+In this case, use `sphinx-autobuild`:
+
+```bash
+(venv) $ sphinx-autobuild docs docs/build/html/
+...
+The HTML pages are in docs/_build/html.
+[sphinx-autobuild] Serving on http://127.0.0.1:8000
+```
+
+This will build the docs and host them on local web server to view them live
+in a web browser. You can edit the files and your browser will automatically
+reload with the live changes.
+
+## Writing Docs
+
+The source files are organised under `docs/` in the same structure they appear in the
+toctree, which is defined in `docs/sphinx.yaml`. To add content to an existing page,
+simply navigate to the markdown file for the page and edit it. To add a new page, create
+a markdown file in the appropriate place in the hierarchy and then add it to
+`sphinx.yaml`.

commands/_sources/developer/how-to/excalidraw.md.txt

@@ -0,0 +1,21 @@
+# How to embed Excalidraw diagrams
+
+Start off by creating your diagram in <https://excalidraw.com>
+
+```{raw} html
+:file: ../../images/excalidraw-example.svg
+```
+
+Click 'Save as image' and make sure the 'Embed scene' checkbox is enabled. This is required for loading your image back into Excalidraw should you wish to make changes later on. Name your file and export to SVG, saving it inside `doc/images`.
+
+Add the following to embed it inside your documentation:
+
+``````
+```{raw} html
+:file: ../../images/excalidraw-example.svg
+```
+``````
+
+It is preferred to use the above convention over `[]()` in order to retain the font used by Excalidraw.
+
+Rebuild the docs and open the resulting html inside a browser.

commands/_sources/developer/how-to/frame-decoder.md.txt

@@ -0,0 +1,40 @@
+# FrameDecoder
+
+The purpose of the [FrameDecoder] is to gather incoming packets or messages from a data
+stream, perform minimal work to construct a single data blob in shared memory for the
+FrameProcessor to handle further processing.
+
+This class defines a few virtual methods to be implemented by concrete implementations.
+Some are pure virtual - annotated with `*` - and some are optional. These are:
+
+- [get_frame_buffer_size](FrameReceiver::FrameDecoder::get_frame_buffer_size)*
+- [get_frame_header_size](FrameReceiver::FrameDecoder::get_frame_header_size)*
+- [monitor_buffers](FrameReceiver::FrameDecoder::monitor_buffers)*
+- [get_status](FrameReceiver::FrameDecoder::get_status)
+- [request_configuration](FrameReceiver::FrameDecoder::request_configuration)
+- [reset_statistics](FrameReceiver::FrameDecoder::reset_statistics)
+
+There is generic support for UDP and ZMQ data streams that a decoder plugin can be built
+on top of, depending on the use case. These are [FrameDecoderUDP] and [FrameDecoderZMQ],
+each a child of [FrameDecoder]. They must be paired with the [FrameReceiverUDPRxThread]
+and [FrameReceiverUDPRxThread] respectively (e.g. setting the `rx_type` config to `udp`).
+Both extend the base class and add further virtual methods to be implemented in a
+concrete decoder.
+
+## FrameDecoderUDP
+- [requires_header_peek](FrameReceiver::FrameDecoderUDP::requires_header_peek)*
+- [get_packet_header_size](FrameReceiver::FrameDecoderUDP::get_packet_header_size)*
+- [get_next_payload_buffer](FrameReceiver::FrameDecoderUDP::get_next_payload_buffer)*
+- [get_next_payload_size](FrameReceiver::FrameDecoderUDP::get_next_payload_size)*
+
+## FrameDecoderZMQ
+- [get_next_message_buffer](FrameReceiver::FrameDecoderZMQ::get_next_message_buffer)*
+- [process_message](FrameReceiver::FrameDecoderZMQ::process_message)*
+- [frame_meta_data](FrameReceiver::FrameDecoderZMQ::frame_meta_data)*
+
+% Links
+[FrameDecoder]: FrameReceiver::FrameDecoder
+[FrameDecoderUDP]: FrameReceiver::FrameDecoderUDP
+[FrameDecoderZMQ]: FrameReceiver::FrameDecoderZMQ
+[FrameReceiverUDPRxThread]: FrameReceiver::FrameReceiverUDPRxThread
+[FrameRecieverZMQRxThread]: FrameReceiver::FrameRecieverZMQRxThread

commands/_sources/developer/how-to/frame-processor-plugin.md.txt

@@ -0,0 +1,61 @@
+# FrameProcessorPlugin
+
+The [FrameProcessorPlugin] is implemented for various purposes and an arbitrary tree of
+plugins can be loaded and connected within the frameProcessor application. Detector
+plugins are not special in any way from more general plugins, except that they are
+generally the first plugin in the tree.
+
+This class defines a few virtual methods to be implemented by concrete implementations.
+
+- [process_frame]
+- [configure]*
+- [requestConfiguration]*
+- [status]*
+- [process_end_of_acquisition]*
+
+While the [configure], [requestConfiguration] and [status] methods are technically
+optional, they should be implemented if the plugin has parameters to expose.
+
+If the plugin should take some action when an
+[end of acquisition frame](#endofacquisitionframe) is passed, then
+[process_end_of_acquisition] can be implemented to handle this case.
+
+Additionally, the base class provides the [push] method, which should be called by child
+classes at the end of [process_frame] to pass frames on to  any listening plugins in the
+plugin tree. If the plugin is intended to only be at the end of a plugin chain, then this
+does not need to be implemented.
+
+## Types of Frame
+
+The [Frame] class is the container used to pass data between plugins. It is an abstract
+base class with a few concrete implementations.
+
+### SharedBufferFrame
+
+The [SharedBufferFrame] simply stores the pointer to the shared memory buffer as provided
+by the frameReceiver. When possible to perform any processing in place, it is best to
+publish an instance of this class to subsequent plugins to avoid an unecessary copy.
+
+### DataBlockFrame
+
+For cases where it is not possible to operate in place a [DataBlockFrame] can be
+allocated from the DataBlockPool. For example, the BloscPlugin does this because the
+compression algorithm requires input and output data pointers.
+
+### EndOfAcquisitionFrame
+
+The [EndOfAcquisitionFrame] is a meta Frame that is used only to signal the end of an
+acquisition to plugins further down the plugin chain. This is typically produced by
+detector processing plugins that can detect the end of acquisition from the data stream.
+
+[FrameProcessorPlugin]: FrameProcessor::FrameProcessorPlugin
+[process_frame]: FrameProcessor::FrameProcessorPlugin::process_frame
+[configure]: FrameProcessor::FrameProcessorPlugin::configure
+[requestConfiguration]: FrameProcessor::FrameProcessorPlugin::requestConfiguration
+[status]: FrameProcessor::FrameProcessorPlugin::status
+[process_end_of_acquisition]: FrameProcessor::FrameProcessorPlugin::process_end_of_acquisition
+[push]: FrameProcessor::FrameProcessorPlugin::push
+[Frame]: FrameProcessor::Frame
+[SharedBufferFrame]: FrameProcessor::SharedBufferFrame
+[DataBlockFrame]: FrameProcessor::DataBlockFrame
+[EndOfAcquisitionFrame]: FrameProcessor::EndOfAcquisitionFrame

commands/_sources/developer/how-to/frame-simulator-plugin.md.txt

@@ -0,0 +1,30 @@
+# FrameSimulatorPlugin
+
+The [FrameSimulatorPlugin] is the base class for detector specific implementations that
+can be loaded into the frameSimulator application.
+
+This class defines a few virtual methods to be implemented by concrete implementations.
+Some are pure virtual - annotated with `*` - and some are optional. These are:
+
+- [populate_options]
+- [setup]
+- [simulate]*
+
+These methods are used to provide options to `--help` output, parse commandline options
+and run the simulation.
+
+There is also a helper child class [FrameSimulatorPluginUDP] that provides some utility
+for loading pcap files, providing virtual methods to process the file into packets, and
+implements generic [setup] and [simulate] methods. The methods for child classes to
+implement are:
+
+- [extract_frames]*
+- [create_frames]*
+
+[FrameSimulatorPlugin]: FrameSimulator::FrameSimulatorPlugin
+[populate_options]: FrameSimulator::FrameSimulatorPlugin::populate_options
+[setup]: FrameSimulator::FrameSimulatorPlugin::setup
+[simulate]: FrameSimulator::FrameSimulatorPlugin::simulate
+[FrameSimulatorPluginUDP]: FrameSimulator::FrameSimulatorPluginUDP
+[extract_frames]: FrameSimulator::FrameSimulatorPluginUDP::extract_frames
+[create_frames]: FrameSimulator::FrameSimulatorPluginUDP::create_frames

commands/_sources/developer/how-to/index.md.txt

@@ -0,0 +1,4 @@
+# **How-To Guides**
+
+```{tableofcontents}
+```

commands/_sources/developer/index.md.txt

@@ -0,0 +1,59 @@
+# Developer Guide
+
+::::{grid} 2
+:gutter: 4
+
+:::{grid-item-card} {material-regular}`directions_run;3em` [Tutorials](./tutorials/index)
+
+---
+
+- [Write Odin support for a detector](./tutorials/write-detector-support)
+
++++
+
+Tutorials for getting up and running as a developer
+
+:::
+
+:::{grid-item-card} {material-regular}`task;3em` [How-To Guides](./how-to/index)
+
+---
+
+- [Implement a FrameDecoder](./how-to/frame-decoder)
+- [Implement a FrameProcessorPlugin](./how-to/frame-processor-plugin)
+- [Implement a FrameSimulatorPlugin](./how-to/frame-simulator-plugin)
+
++++
+
+Practical step-by-step guides for day-to-day dev tasks
+
+:::
+
+:::{grid-item-card} {material-regular}`apartment;3em` [Explanations](./explanations/index)
+
+---
+
+- [Design Decisions](./explanations/design-decisions)
+
++++
+
+Explanations of how and why the architecture is why it is
+
+:::
+
+:::{grid-item-card} {material-regular}`description;3em` [Reference](./reference/index)
+
+---
+
+- [FrameReceiver API](./reference/api/frame-receiver)
+- [FrameProcessor API](./reference/api/frame-processor)
+- [FrameSimulator API](./reference/api/frame-simulator)
+- [Live View Specification](./reference/design-docs/live-view-spec)
+
++++
+
+Technical reference material, internal API docs and development guidelines
+
+:::
+
+::::