Skip to content

Avro Packet Maker

Jump to bottom
Sulekha Kishore edited this page Jul 11, 2022 · 10 revisions

Overview
This processor works with a table (dataframe) of candidates to generate an alert packet for each candidate. The alerts are formatted as Apache Avro, which is a data serialization system that relies on schemas to validate and enforce data types and contents.

Alert Schema

Packets are made based on a nested schema format, similar to that used by ZTF. Schemas are in processors/alert_packets/alert_schema/ and are given as .avsc files. alert.avsc is the top-level schema. candidate.avsc and prv_candidate.avsc are packed into alert.avsc.

Requires python 3 and avro.schema for combining nested schemas. This module specifically works only with avro version 1.10.1 (which you can get with pip install avro==1.10.1). Also uses fastavro library, which is available with pip install fastavro.

Saving Packets Locally
If Avro packets are being saved locally, ensure that the output data directory has been set correctly. Use export OUTPUT_DATA_DIR=/path/to/directory to set.

Clone this wiki locally