network buffering / mlat stagger customisation option (#68)

Increasing the beast-reduce flush interval will reduce network load for aggregators by sending less but larger network packets. mlat-client startup can take quite some time when using several aggregators, add an ENV var so people can reduce it if they prefer. mlathub was using beast_reduce_out to send data, this is local transfer so using beast_reduce is not necessary. (mlat results are not high message rate in any case)
sdr-enthusiasts · Mar 14, 2024 · 01fcc34 · 01fcc34
2 parents 3e08831 + 6f6587f
commit 01fcc34
Show file tree

Hide file tree

Showing 4 changed files with 11 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -203,6 +203,7 @@ The following parameters must be set (mandatory) for the container to function:
 | `READSB_EXTRA_ARGS`           | Optional, allows to specify extra parameters for readsb                                                                          |                                | Unset     |
 | `S6_SERVICES_GRACETIME`       | Optional, set to 30000 when saving traces / globe_history                                                                        |                                | `3000`    |
 | `LOGLEVEL`                    | `verbose` (all messages), `error` (errors only), `none` (minimal)                                                                |                                | `verbose` |
+| `MLAT_STARTUP_STAGGER`        | mlat-client startup staggering for tidy logs (reduce for quicker startup)                                                        |                                | `15`      |
 
 `READSB_EXTRA_ARGS` just passes arguments to the commandline, you can check this file for more options for wiedehopf's readsb fork: <https://github.com/wiedehopf/readsb/blob/dev/help.h>
 
@@ -367,8 +368,9 @@ There are many optional parameters relating to the ingestion of data and the gen
 | `READSB_NET_BEAST_INPUT_PORT`         | TCP Beast input listen ports                                                                                                                                                                                                                                                   | `--net-bi-port=<ports>`                 | `30004,30104` |
 | `READSB_NET_BEAST_OUTPUT_PORT`        | TCP Beast output listen ports                                                                                                                                                                                                                                                  | `--net-bo-port=<ports>`                 | `30005`       |
 | `READSB_NET_BUFFER`                   | TCP buffer size 64Kb \* (2^n)                                                                                                                                                                                                                                                  | `--net-buffer=<n>`                      | `2` (256Kb)   |
-| `READSB_NET_RAW_OUTPUT_INTERVAL`      | TCP output flush interval in seconds (maximum interval between two network writes of accumulated data).                                                                                                                                                                        | `--net-ro-interval=<rate>`              | `0.05`        |
-| `READSB_NET_RAW_OUTPUT_SIZE`          | TCP output flush size (maximum amount of internally buffered data before writing to network).                                                                                                                                                                                  | `--net-ro-size=<size>`                  | `1200`        |
+| `READSB_NET_RAW_OUTPUT_INTERVAL`      | TCP output flush interval in seconds (maximum output buffer delay).                                                                                                                                                                                                            | `--net-ro-interval=<seconds>`           | `0.05`        |
+| `READSB_NET_RO_INTERVAL_BEAST_REDUCE` | TCP output flush interval in seconds for beast reduce outputs (maximum output buffer delay).                                                                                                                                                                                   | `--net-ro-interval-beast-reduce=<secs>  | `0.12`        |
+| `READSB_NET_RAW_OUTPUT_SIZE`          | TCP output flush size (maximum amount of internally buffered data before writing to network).                                                                                                                                                                                  | `--net-ro-size=<size>`                  | `1280`        |
 | `READSB_NET_CONNECTOR_DELAY`          | Outbound re-connection delay.                                                                                                                                                                                                                                                  | `--net-connector-delay=<seconds>`       | `30`          |
 | `READSB_NET_HEARTBEAT`                | TCP heartbeat rate in seconds (0 to disable).                                                                                                                                                                                                                                  | `--net-heartbeat=<rate>`                | `35`          |
 | `READSB_NET_RAW_INPUT_PORT`           | TCP raw input listen ports.                                                                                                                                                                                                                                                    | `--net-ri-port=<ports>`                 | `30001`       |

diff --git a/rootfs/etc/s6-overlay/scripts/mlat-client b/rootfs/etc/s6-overlay/scripts/mlat-client
@@ -205,7 +205,7 @@ do
 
 
     # stagger by 15 second so they don't all start at the same time
-    sleep 15 & wait $!
+    sleep "${MLAT_STARTUP_STAGGER:-15}" & wait $!
 
     # ------------------------------------------------
     # run this Mlat_client instance in the background:

diff --git a/rootfs/etc/s6-overlay/scripts/mlathub b/rootfs/etc/s6-overlay/scripts/mlathub
@@ -27,7 +27,7 @@ MLATHUB_CMD+=("--mlat")
 MLATHUB_CMD+=("--forward-mlat")
 MLATHUB_CMD+=("--forward-mlat-sbs")
 
-MLATHUB_CMD+=("--net-connector=localhost,30004,beast_reduce_out")
+MLATHUB_CMD+=("--net-connector=localhost,30004,beast_out")
 
 # define some more ports:
 MLATHUB_CMD+=("--net-sbs-port=${MLATHUB_SBS_OUT_PORT:-31003}")

diff --git a/rootfs/etc/s6-overlay/scripts/readsb b/rootfs/etc/s6-overlay/scripts/readsb
@@ -207,6 +207,11 @@ if [[ -n "$READSB_NET_RAW_OUTPUT_INTERVAL" ]]; then
     READSB_CMD+=("--net-ro-interval=$READSB_NET_RAW_OUTPUT_INTERVAL")
 fi
 
+# use 0.12s/120ms to reduce number of packets sent to aggregators (compared to ro-interval 50ms)
+# most aggregator feed clients historically use 200ms so this is already a reduction in jitter / latency
+READSB_CMD+=("--net-ro-interval-beast-reduce=${READSB_NET_RO_INTERVAL_BEAST_REDUCE:-0.12}")
+
+
 # Handle "--net-ri-port=<ports>"
 if [[ -n "$READSB_NET_RAW_OUTPUT_PORT" ]]; then
     READSB_CMD+=("--net-ro-port=$READSB_NET_RAW_OUTPUT_PORT")