Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement artifacts cloud synchronization in Incremental Cardano DB with GCP #2211

Open
8 tasks done
Tracked by #2047
jpraynaud opened this issue Jan 13, 2025 · 0 comments · May be fixed by #2233
Open
8 tasks done
Tracked by #2047

Implement artifacts cloud synchronization in Incremental Cardano DB with GCP #2211

jpraynaud opened this issue Jan 13, 2025 · 0 comments · May be fixed by #2233
Assignees
Labels
feature 🚀 New implemented feature

Comments

@jpraynaud
Copy link
Member

jpraynaud commented Jan 13, 2025

Why

We need to adapt the GcpUploader implementation to support uploading of immutable file archives only if they don't already exist in the bucket. We also need to make sure that these archives are not deleted after 14 days by the GCP bucket life-cycle rules implemented for the bucket

What

Have GcpUploader support an overwriting feature (simple, no hash comparison at first) and adapt the GCP bucket life-cycle rules.

How

  • Add an overwriting option in the GcpUploader which avoids uploading a file that already exists remotely (check on existence only)
  • Add a destination folder option in the GcpUploader which specifies the folder in the GCP bucket where the file is uploaded
  • Modify the dependency injection of the GCP uploaders to support the new options
  • Upload the CardanoImmutableFilesFull snapshot archive to a cardano-immutable-files-full folder
  • Upload the CardanoDatabase ancillary archives to a cardano-database/ancillary folder
  • Upload the CardanoDatabase immutable archives to a cardano-database/immutable folder
  • Upload the CardanoDatabase digests file to a cardano-database/digests folder
  • Update terraform scripts to support prefixes of files deleted after 14 days

Later

  • Extend the overwriting option to verify the hash of the local file vs remote file
  • Manual clean of the non deleted archives located at the root of the bucket 14 days after deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature 🚀 New implemented feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant