Technical Documentation

SNBmodules

Doxygen's generation (with docs/Doxyfile the doxygen configuration file in snbmodules) :

doxygen docs/Doxyfile

Here is an example design choice for the Supernova Burst dataflow (SNB). In this context, 150 TB of data over 100 seconds of recording will be in the underground. How to chip them out under the different SNB dataflow challenges ?

Requirements

to respect timings of the different scenario of the data taking mode :

• Speed and scalability
• Fully controllable transfers
• Get transfer information
• Modularity, can use multiple transfer implementation

Transfer requirements

• Pause / Resume
• Performance
• Rate limited
• Data integrity checking
• Many to One

Design of SNBmodules

My Design choices

• Client started on each node can create

  • Downloader Session (to receive files)
  • Uploader Session (to send files)

• Bookkeeper

  • Visual interface
  • Track files transfer metadata
    • File full path
    • Hash
    • Bytes size
    • Bytes transfered
    • Actual Transmission speed
    • Status
      • PREPARING
      • ERROR
      • SUCCESS_UPLOAD
      • SUCCESS_DOWNLOAD
      • FINISHED
      • CANCELLED
      • PAUSED
      • WAITING
      • CHECKING
      • HASHING
      • UPLOADING
      • DOWNLOADING
    • Source client
    • Destination client
    • Error code if any
    • Start time of transfer
    • End time of transfer
    • Duration of transfer (useful if ongoing)
    • Group transfer ID

• Uploaders Clients control and initiate transfers that can be on any implemented protocol

  • dummy (debug purposes)
  • SCP (testing purposes)
  • BITTORRENT
  • RCLONE

• Can be modified !

SNBmodules - Client Use Case diagram	SNBmodules - Bookkeeper Use Case diagram

Activity Diagrams

Implementation

• Notification system
  • Using Connectivity Service library (IOManager)
  • Use wrapper interface : interchangeability
• Transfer Interface
  • Fully modular
  • Choice of transfer protocol libraries matures to respect requirements
    • Already existing libraries : unify them in my solution
    • Fast and multi-connection support
    • SCP (for test purposes), one to one
    • LibTorrent : BitTorrent peer to peer protocol
    • RClone : HTTP protocol, one to one with multiple connections
  • Parameters parsed on transmission
• Control aspects
  • Expert Commands send to Uploader client
  • Graphic Interface

Integration

Integration tests

daqconf modifications available in ljoly/snbmodules-config branch in daqconf

• snbmodules_gen.py
• snbmodulesgen.jsonnet

Implemented integration tests

Run with

pytest -s 'test_name.py'

• snbmodules/integtest/snb_1node_1app_system_quick_test.py : one full Node with readout and raw recording activated. Here, one snbmodules application start multiple clients on the same host. Useful for tests purposes.
• snbmodules/integtest/snb_1node_multiapp_system_quick_test.py : Same as before, but all clients are started in a different nanorc application.

The integration tests uses python check scripts and commands :

• Using raw_file_check.py utility functions to test integrity of RAW binary files transferred locally
• Using transfer_check.py utility functions to test errors and check transfers in the Bookkeeper log file
• record-cmd.json : Expert command asking the readout application to record during 1 second
• new-RClone-transfer.json : Initiate a new transfer using RClone protocol (SFTP for test purposes, but can be changed : careful BitTorrent only support transfer to different IP addresses). The list of files and destination clients is automatically filled up by the integration tests.
• start-transfer.json : Simply send the start transfer command.

Systems tests for multi node testing

Available in daq-systemtest Different configuration files with different setup. The standard idea is that each host have one client startup and the bookkeeper is started somewhere else. You can refer to the scale configuration large_scale_system_with_snbmodules.json to see a standard configuration.