Handling of sonde data with no GPS lock

[darksidelemm]

Right now we just drop any data with no gps lock on the floor right away. There is some value in recording this data however, and even uploading it to SondeHub - assuming we record the fact that it has no GPS lock appropriately.

Things to check:

• Where in auto_rx we do checks for either sats=0 or lat/lon=0 and work out what the behaviour should be changed to.
  • e.g. the chasemapper output should still drop this data. Logging should continue, as should uploading to SondeHub
• Make sure the web interface map will correctly handle 0/0 positions (e.g. it should drop them)

I would add the following constraints when handling data with no GPS lock:

• Compare the reported time from the Sonde to the system UTC time. If it's more than a few minutes out (refer horusdemodlib for an example of handling this), still drop the packet. This will require that the sonde at least obtained GPS lock once, and still reports the correct time (I think this is the case)

[TheSkorm]

We probably need to confirm/check/design how to handle this in sondehub as well to make sure it doesn't get dropped (zcheck, other filters). We probably want to add a warning system like sondehub amateur and send that back the client and trip the telemetry_hidden (also missing from sondehub pro)

[darksidelemm]

Turns out the checks for sats=0 aren't actually present on SondeHub-Pro tracker, nor any checks for 0.0/0.0. I guess we've gotten lucky in that the clients are discarding data like this.

[j0uni]

I upload the same data I uploaded to rdzsonde issue considering this for future reference or such.

"Here is raw data from V1631304 recorded wih Auto-RX. This sonde had very typical inteference pattern so this is a very good example what is happening."

20231028-053604_RS41_403000000.raw.zip

[rs1729]

7B paket being all zeros happens from time to time, that is not unusual, isolated frames, sometimes every 5-10 seconds for a period of time.

7b15000000000000000000000000000000000000000000d937

Position ECEF(0,0,0) would be rejected by rs41mod, no position output. This would also mean sats: 00.

There are 3 frames with empty 7B packet in the raw data above.

The 7B packet is basically the ublox6 NAV-SOL (0x01 0x06), it contains the ECEF coordinates, and also numSV (numSats).

The frozen 7B packet with position/velocity

lat: 60.70705  lon: 24.22422  alt: 24582.01   vH:  0.0  D:   0.0  vV: 0.0  sats: 00

is

7b150d4d1211c94fae07073c252100000000000000ffff9078

A few frames before this, the 7B packet already shows 00ffffbefore the CRC bytes at the end, and sats drop to zero just before that:

7b1586321211180fae07213b25216902e205150009061579b8
7b15ef341211f914ae07363b25216902e205150000c8ffa25c
7b1558371211db1aae074b3b25216902e205150000ffff0fc4

The last 3 bytes (excluding CRC) in the 7B packet are numSats, sAcc, pDOP.
Entire frames:

8635f44093df1a6092f1b175e21d32233d144ba82cc798b634342f2106bc3b2a3b3af420ea3a85e1e17ec50eaecf15d7ad663547d7d7e25c0f79281c1956313633313330341a000001000013000022000732013331333034f74e000058021205b43ca4b2407a2a642a0209060245f002d68708e38e072593084c0b020a060244f0020000000000000000000000000000000af77c1eed088f045f201cec02b3068109d013ec20cb0cc603ef01b104f011ef1feb5f4e7d5987b82f0107077b370c68a0ff79c1111282c2ffb191b2182dbafe4fdac41f81c2ffbcd8600ccfb1ff397e621951ba007727601928d2ff32000000c69dffa0ec2c0904a000a983700e08c7feeed6d50ea42c008cac1b0deb17ff36627b1586321211180fae07213b25216902e205150009061579b876110000000000000000000000000000000000ecc7 [OK]
8635f44093df1a60f5f8f6c047bcb67bc9636d801206e43db2309c7bb05a26bb8489e10d708166f998651d857b1d456b2a60d5433d1344b10f79281d1956313633313330341a0000010000140000250007320206148732000000ffff00000000031723e9df7a2a6f2a0206060242f002d38708e18e07229308290b0206060241f0020000000000000000000000000000009dc07c1eed0877085f201ceb02d2068009d013eb20aa0cc603ee01d004f011ee1fea1b487d598bb72f01ffe27d370c68a0ffa0e7111269c2ff3bafb11896bafe08ffc41f69c2ff18ed600cfab1ff149b631981ba00815c601991d2ff150000009d9dffd7ee2d09b59f00f2ac6f0ec2c6fe0b66d60ea32c00b2261b0dc417ff29e37b15ef341211f914ae07363b25216902e205150000c8ffa25c76110000000000000000000000000000000000ecc7 [OK]
8635f44093df1a60353b6577a574a00d043f69213c41da95f3bff0fd2870ad75e7685583edc644c1ee65b3e0952fb9366180c1770e38afcb0f79281e1956313633313330341b00000100001500002800073203e8030004000700bf0291b3000600803bee097a2a7f2a0209060246f002d88708e68e072a9308220b020a060244f00200000000000000000000000000000029787c1eed085f0c5f2019951ceb02d211ee09d013eb20c90ca715b503ee1feb04ef86377d5990b62f01ffa4fde3170000009f80370cb2a0ff700b121238c2ff03f5d60eec2c004823c51f39c2ff8701610c62b2ffc6b76419c0ba009091601903d3ff1bfc621800000029000000019effb2a01a0de317ffcad56e0ecac6fea5f17b1558371211db1aae074b3b25216902e205150000ffff0fc476110000000000000000000000000000000000ecc7 [OK]

[ 6428] (V1631304) (2.6 V)  Sat 2023-10-28 06:51:37.999 (W 2285)  lat: 60.70806  lon: 24.22197  alt: 24516.46   vH: 15.1  D: 132.3  vV: 6.0    sats: 09  sAcc:  0.6  pDOP:  2.1  : fw 0x4ef7 
[ 6429] (V1631304) (2.6 V)  Sat 2023-10-28 06:51:38.999 (W 2285)  lat: 60.70797  lon: 24.22217  alt: 24522.42   vH: 15.1  D: 132.3  vV: 6.0    sats: 00  sAcc: 20.0  pDOP: -1.0  : BK 00 
[ 6430] (V1631304) (2.7 V)  Sat 2023-10-28 06:51:39.999 (W 2285)  lat: 60.70788  lon: 24.22238  alt: 24528.38   vH: 15.1  D: 132.3  vV: 6.0    sats: 00  sAcc: -1.0  pDOP: -1.0 

That the other two GPS packets 7C and 7D are zero, is very rare.
7C contains time/date (GPSweek, GPSiTOW).
There are a few frames with 7C zero in the raw data
#831 (comment)
all in the frozen section.

7c1e000000000000000000000000000000000000000000000000000000000000452a

This would give Sun 1980-01-06 00:00:00.000, first day of GPS.

[rs1729]

Additional info:
Date/Time: Sun 1980-01-06 00:00:00.000 would also be in the JSON output, as long as the position is plausible, even if "sats: 00". Though I would assume, 7C: 00 .. 00 would always mean "no GPS lock" and "sats: 00" in 7B, so auto_rx or sondehub would know that the GPS data is unreliable.

ecef(0,0,0) would mean lat: 180.00000 lon: 0.00000 alt: -6378137.00.
That's not a plausible position, middle of earth. This is rejected, i.e. not position output.

[darksidelemm]

SondeHub will block data with a datetime older than 48 hours from 'now' from entering the database, so passing on even this kind of frame shouldn't be a huge issue as it'll just get dropped.
I'd consider doing some checks before uploading to confirm the reported datetime from the sonde is within some reasonable range from the receivers local time before uploading however.