Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compression #59

Merged
merged 18 commits into from
Sep 10, 2024
Merged

Compression #59

merged 18 commits into from
Sep 10, 2024

Conversation

LINCKODE
Copy link
Member

@LINCKODE LINCKODE commented Sep 9, 2024

Adds support for ZSTD compression and new quorum data format.

  • Added configuration option to choose Compression setting.
  • Added alternative Protobuf object to represent new quorum diff format.
  • Added new endpoint to return the re-assembled quorum data, from the new more minimal format. (/v2/ticks/{tick_number}/quorum)
  • Added methods to read/write new quorum data format to/from db.
  • Updated README.md to warn about potential incompatibilities between archiver versions.

Process of migrating an Archiver instance:

  1. Stop archiver.
  2. Run the archiver-db-migrator tool, specifying the path of the old database, and the path of where to store the new database directory. Wait for the process to stop.
  3. Rename the old database directory to something else, and the new directory to the old name, thus we don't need to modify the archiver configuration.
  4. Update the archiver version in the docker compose file.
  5. Start archiver.

@LINCKODE LINCKODE added the enhancement New feature or request label Sep 9, 2024
@LINCKODE LINCKODE requested a review from 0xluk September 9, 2024 11:53
@LINCKODE LINCKODE self-assigned this Sep 9, 2024
@LINCKODE LINCKODE linked an issue Sep 9, 2024 that may be closed by this pull request
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
main.go Outdated Show resolved Hide resolved
main.go Outdated Show resolved Hide resolved
rpc/v2_endpoints.go Outdated Show resolved Hide resolved
rpc/v2_endpoints.go Outdated Show resolved Hide resolved
@0xluk
Copy link
Contributor

0xluk commented Sep 9, 2024

Process of migrating an Archiver instance: section should reside in a separate md file which should be referenced from the general readme.md

rpc/rpc_server.go Outdated Show resolved Hide resolved
rpc/rpc_server.go Outdated Show resolved Hide resolved
rpc/rpc_server.go Outdated Show resolved Hide resolved
rpc/rpc_server.go Outdated Show resolved Hide resolved
validator/quorum/models.go Outdated Show resolved Hide resolved
validator/quorum/models.go Outdated Show resolved Hide resolved
@@ -116,9 +121,14 @@ message SkippedTicksIntervalList {
repeated SkippedTicksInterval skipped_ticks = 1;
}

message QuorumTickData {
message QuorumTickDataV2 {
Copy link
Contributor

@0xluk 0xluk Sep 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we still have v1 and v2? this version of archive should be used with a clean(migrated) version of storage so we shouldn't keep any reference to the old quorum format, or am I missing something?

@0xluk 0xluk merged commit 5a0df45 into dev Sep 10, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

PebbleDB storage compression for shrinking DB size
2 participants