[Chunk Data Pack] Refactor Chunk data packs modules to use generic storage interface #6892
Conversation
| @@ -60,18 +69,28 @@ func run(*cobra.Command, []string) { | |||
| metrics := &metrics.NoopCollector{} | |||
| transactionResults := badger.NewTransactionResults(metrics, db, badger.DefaultCacheSize) | |||
| commits := badger.NewCommits(metrics, db) | |||
| chunkDataPacks := badger.NewChunkDataPacks(metrics, db, badger.NewCollections(db, badger.NewTransactions(metrics, db)), badger.DefaultCacheSize) | |||
There was a problem hiding this comment.
The previous implementation was wrong. Since we have moved chunk data packs to pebble, we should use pebble based chunk data packs module for rolling executed height.
| headers := bstorage.NewHeaders(metrics, db) | ||
| txResults := bstorage.NewTransactionResults(metrics, db, bstorage.DefaultCacheSize) | ||
| commits := bstorage.NewCommits(metrics, db) | ||
| chunkDataPacks := store.NewChunkDataPacks(metrics, pebbleimpl.ToDB(pdb), bstorage.NewCollections(db, bstorage.NewTransactions(metrics, db)), bstorage.DefaultCacheSize) |
There was a problem hiding this comment.
This is the actual change.
| require.NoError(t, err) | ||
| require.NoError(t, chunkBatch.Commit()) | ||
| err2 := batch.Flush() |
There was a problem hiding this comment.
This is the actual change.
| require.NoError(t, err) | ||
| require.NoError(t, chunkBatch.Commit()) | ||
| err2 := batch.Flush() | ||
| require.NoError(t, err2) |
There was a problem hiding this comment.
This is the actual change.
| @@ -0,0 +1,24 @@ | |||
| package operation | |||
There was a problem hiding this comment.
Moved from storage/pebble/operation/chunk_data_packs.go
| @@ -0,0 +1,133 @@ | |||
| package store_test | |||
There was a problem hiding this comment.
Moved from storage/pebble/chunk_data_packs_test.go
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6892 +/- ##
==========================================
- Coverage 41.11% 41.09% -0.02%
==========================================
Files 2116 2118 +2
Lines 185737 185840 +103
==========================================
+ Hits 76368 76374 +6
- Misses 102952 103065 +113
+ Partials 6417 6401 -16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| @@ -11,9 +11,6 @@ const ( | |||
| //lint:ignore U1000 Ignore unused variable warning | |||
| // job queue consumers and producers | |||
| codeJobConsumerProcessed = 70 | |||
|
|
|||
| // legacy codes (should be cleaned up) | |||
| codeChunkDataPack = 100 | |||
There was a problem hiding this comment.
should this code be reserved for some period of time?
There was a problem hiding this comment.
If reserving is to reverting the changes, I would revert by reverting the whole feature branch, rather than doing at the code level.
So I think it's OK to just clean up
zhangchiqing
force-pushed
the
leo/storage-refactor-chunk-data-packs
branch
from
5c8a3a5 to
c2cd930
Compare
This PR refactors the chunk data packs modules to use the generic storage interfaces. This allows us to reuse functions that made for the generic storage interface to be used on chunk data packs when implementing chunk data pack pruner.
Since chunk data packs are stored in pebble now, this PR also fixed some test cases and tools that was still dealing with chunk data packs stored in badger.