Compression #67

LINCKODE · 2024-09-30T13:00:36Z

Implements database compression using ZSTD
These changes drastically decrease the storage size.

Protobuf object to represent new quorum data format, which saves more than 50% the size.
We now save the full quorum tick data only for the last epoch of the tick.
Quorum data endpoint now reconstructs data using information from the next tick. Data of the last epoch tick is returned directly from the DB.

…oint.

# Conflicts: # protobuff/archive.pb.go # protobuff/archive_grpc.pb.go

LINCKODE added 20 commits August 30, 2024 21:57

Test db compression

d8de2da

Use set db to use zstd compression and deploy docker

6e0ada0

Test db compression

f1131d5

Use set db to use zstd compression and deploy docker

d1bfb3d

Merge remote-tracking branch 'origin/compression' into compression

5702f67

Bump to trigger deployment

6db5312

Add missing db.Close() statement.

f78a43e

Implement support for new quorum data format.

8898443

Fix inaccuracies in salted digest calculations.

d2c027e

Update README.md with warning.

45a767d

Add spaces

61a5472

Implement suggested fixes.

7615d71

Implement suggested fixes.

362b41b

Rename new quorum structure.

87004ec

Remove v2 quorum endpoint.

31b756e

Fixed test.

f65e9de

Remove GetTickV2

4638f16

Remove GetLatestTickV2

133029c

Implement handling for last epoch tick edge-case for quorum data endp…

e271222

…oint.

Merge branch 'dev' into compression

6a46dfa

# Conflicts: # protobuff/archive.pb.go # protobuff/archive_grpc.pb.go

LINCKODE merged commit d667021 into dev Oct 2, 2024
2 checks passed

Provide feedback