test(scan): Scan blocks containing ZECpages transactions with the ZECpages shielded key

[teor2345]

Motivation

When we have a draft API, let's try scanning some real blocks with a real key.

ZECpages publishes their key here:
https://zecpages.com/boardinfo

Complex Code or Requirements

If we run this in CI we will need cached state. But to start with we can just run it manually and skip the test unless a specific env var is set.

Implementation

To verify the test worked, we might need a method that gets all the sapling results for a key in the database, in a specific height range.

Testing

Unit test:

• scan a real block test vector that contains a zecpages post
• scan one that doesn't

Integration test:

• scan the entire blockchain for zecpages posts - this test currently takes 15 hours, so it needs to be a manual-only test, or a scheduled weekly test, or we should scan batches of 100 blocks in parallel
• check that the results match zecpages posts

Related Work

This is a test for the blockchain scanning work.

[teor2345]

I am doing part of this testing, but I think it would be good to assign multiple people to it.

[teor2345]

I manually tested this over the weekend, here are some results:

The scanner works and there were no panics or hangs.

For a full scan with a synced state using du -shc <db path>
10M .cache/zebra-custom/private-scan/v0/mainnet
245G .cache/zebra-custom/state/v25/mainnet

For an incremental scan as the state is being synced:
12M .cache/zebra-tmp/zebra-private-scan-v0-mainnet-0JYZye
262G .cache/zebra-tmp/zebra-state-v25-mainnet-SLEwyi

The size is acceptable for a few keys, but adding the key to every row might be an issue for a user with lots of keys or shielded transactions. (This is out of scope for the MVP.)

It took about 15 hours to scan the entire chain using 1 thread and 1 key in 3 different variants (full, individual spend, and individual change). Performance improvements are out of scope for the MVP.

This is about 42 blocks per second, which is already fast enough to make some complicated changes unnecessary.

2023-12-01T04:48:03.819888Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(419200), current tip Some((Height(2315738), block::Hash("0000000000773b953a99b4a115b479a31fb674060160faa843a845451c4e4582")))
2023-12-01T04:51:57.867986Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(500000), current tip Some((Height(2315743), block::Hash("0000000000494b6e8dfe5303ff70acdd073455a872d43f76d056bd28d6a7af43")))
2023-12-01T04:57:54.983238Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(600000), current tip Some((Height(2315746), block::Hash("00000000017abab20936b148ac1ed54a38cb87019f7ac489baeab2db256ea59e")))
2023-12-01T05:02:00.310009Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(700000), current tip Some((Height(2315748), block::Hash("00000000016159dc71be487ffd3d19d4f5f191084d61c755e5ed4b0371e751f5")))
2023-12-01T05:05:37.861666Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(800000), current tip Some((Height(2315750), block::Hash("00000000006a7468afaa144227b6f0518a8cd82f21c087def34985bd3453f66a")))
2023-12-01T05:10:00.642118Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(900000), current tip Some((Height(2315753), block::Hash("00000000016fa4f64e409b7ece063a4aad0954fb204ad213c76f37f6a7d64183")))
2023-12-01T05:14:29.599159Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1000000), current tip Some((Height(2315755), block::Hash("000000000053876471a34581d5e315071f0b40ff8363103540235e8825550eff")))
2023-12-01T05:18:50.891287Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1100000), current tip Some((Height(2315764), block::Hash("000000000176e2fdbe18192b120dc794e4f1a52495c7d10fc071145ebc10af0f")))
2023-12-01T05:23:50.533209Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1200000), current tip Some((Height(2315771), block::Hash("00000000010adae131139daccc514e96dfb2fdc681ce8304dc56bab2853a77b4")))
2023-12-01T05:28:08.864162Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1300000), current tip Some((Height(2315775), block::Hash("00000000012eac1f2b09f9eb190ae249fb1c6f19d5942ffa1197bd800e29f47d")))
2023-12-01T05:31:59.213181Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1400000), current tip Some((Height(2315777), block::Hash("0000000001604b01ddee6e8e6a33f0116238ad0774113be000f3ae52c9c72aba")))
2023-12-01T05:38:29.657668Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1500000), current tip Some((Height(2315786), block::Hash("00000000006912a813eb6b67b919f33cd093429f99bbd7b9d08382715068f9b3")))
2023-12-01T05:42:27.258290Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1600000), current tip Some((Height(2315788), block::Hash("0000000000ddeb67c1fae88d883332a36d99000db25697a9a5ea4e78d34d84ab")))
2023-12-01T05:46:53.341565Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1700000), current tip Some((Height(2315792), block::Hash("0000000000e2bac25380123b0ac223eb9a90a7311faba69795520b66a0122d32")))
2023-12-01T14:21:31.680360Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1800000), current tip Some((Height(2316191), block::Hash("0000000000e72ca50ccda461fd33b95659441262f54509bd90d224787d5e50a3")))
2023-12-01T18:01:47.501078Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1900000), current tip Some((Height(2316372), block::Hash("000000000108bf51727d1cb3bb1afd78d5d51e1d849a5a8bc872443512c94aa0")))
2023-12-01T18:27:50.504551Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(2000000), current tip Some((Height(2316395), block::Hash("000000000159e1c4899b138a7a37a9bb5737bbada6bd8ad8ea736d1dfb9d79c2")))
2023-12-01T18:45:20.587818Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(2100000), current tip Some((Height(2316402), block::Hash("00000000012a9c55594cb0ceeb8fddc9597ce84e493b3b06b1f8d63c6fcc9202")))
2023-12-01T19:02:30.382959Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(2200000), current tip Some((Height(2316408), block::Hash("000000000087d8dfd860417855aa2a8da8f95c99a0bb60783242b15d88bda626")))
2023-12-01T19:08:21.861275Z  INFO {zebrad="850bd6a" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(2300000), current tip Some((Height(2316413), block::Hash("00000000014229888b9cbd5417db40c4a2bc1c2e0f1edbac318e43ce2e981ca8")))

[mpguerra]

Hey team! Please add your planning poker estimate with Zenhub @arya2 @oxarbitrage @teor2345 @upbqdn

[teor2345]

Hey @arya2 just letting you know this it blocked by #8050. Doing it now is optional, it must be done after #8050 merges.

[teor2345]

I am checking this now with PR #8062

[teor2345]

I manually tested again after PR #8062, here are some results:

The scanner works and there were no panics or hangs.

For a full scan with a synced state using du -shc <db path>
8.0K ~/Library/Caches/zebra/network
80M ~/Library/Caches/zebra/private-scan
255G ~/Library/Caches/zebra/state

(No incremental scan this time.)

The size is acceptable for a few keys, but adding the key to every row might be an issue for a user with lots of keys or shielded transactions. (This is out of scope for the MVP.)

Scanning time appears to be similar or faster. I'm using different hardware and OS, so the previous time isn't comparable.

2023-12-06T15:29:27.763153Z  INFO {zebrad="17ef519" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(1800000), current tip Some((Height(2321973), block::Hash("00000000016eb2082bccbe52b81e7f8e88ca6dfb3cdb8b73eea795da0025fd4c")))
2023-12-06T15:29:27.763180Z  INFO {zebrad="17ef519" net="Main"}: zebra_scan::scan: Scanning the blockchain for key 0, started at block Height(419200)
...
2023-12-06T19:36:28.823502Z  INFO {zebrad="17ef519" net="Main"}: zebra_scan::scan: Scanning the blockchain: now at block Height(2300000), current tip Some((Height(2322166), block::Hash("0000000000f8b08cf7e181478797602cc9b42a8b0bbeeb8a8c123d08ef3c522b")))
2023-12-06T19:36:28.823519Z  INFO {zebrad="17ef519" net="Main"}: zebra_scan::scan: Scanning the blockchain for key 0, started at block Height(419200)

[teor2345]

I just tested the latest database format using ldb and it appears to work:
#8046 (comment)

I also verified the scanned entries against actual zecpages posts:
#8046 (comment)

[mpguerra]

Are we good to close this one? I feel like we've tested this a number of times already

[upbqdn]

The description mentions unit tests and integration tests. I think we're done with both integration tests, but we don't have a unit test with a real block containing a ZECpages post.

[teor2345]

We talked about having two people do the manual test, which I think has happened, but it's not written on this ticket.

[teor2345]

Ah, we were also waiting for PR #8080 to merge, because that's the last major code change. I guess someone should re-test after that.

[upbqdn]

I'll check if #8080 works as intended with the results reader.

[mpguerra]

I'll check if #8080 works as intended with the results reader.

@upbqdn did you manage to check this? Can this issue be closed now?

[upbqdn]

I checked it today. Scanning works as expected, but a version update broke the reader, so I fixed it in #8157.