Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[server][dvc] Open store before bootstrapping from blob transfer. #1290

Merged
merged 3 commits into from
Nov 12, 2024

Conversation

jingy-li
Copy link
Collaborator

@jingy-li jingy-li commented Nov 7, 2024

Summary

Open the store before bootstrapping from blob transfer, otherwise the offset record and StorageVersionState can not update successfully.

How was this PR tested?

Run at 2 dvc hosts.

  1. The first host successfully bootstraps from Kafka, because there is no any available hosts, and successfully generate snapshot.
    Screenshot 2024-11-07 at 11 11 46 AM
  2. The second host successfully bootstraps from blob transfer, and without snapshot, because snapshot will generate when it is ingested from Kafka.
    Screenshot 2024-11-07 at 11 16 41 AM

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to explain your proposed changes and call out the behavior change.

@@ -79,6 +80,7 @@ public void startConsumption(VeniceStoreVersionConfig storeConfig, int partition
if (!storeAndVersion.getFirst().isBlobTransferEnabled() || blobTransferManager == null) {
runnable.run();
} else {
storageService.openStore(storeConfig, svsSupplier);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FelixGV
I saw you modified this interface to pass svsSupplier in the past:
#44

For the usage in this class, it is a little confusing:

Supplier<StoreVersionState> svsSupplier = () -> storageMetadataService.getStoreVersionState(storeVersion);
    syncStoreVersionConfig(storeAndVersion.getFirst(), storeConfig);

    Runnable runnable = () -> {
      AbstractStorageEngine storageEngine =
          storageService.openStoreForNewPartition(storeConfig, partition, svsSupplier);

So it first looks up SVS state from the metadata partition and then pass it to StorageService#openStoreForNewPartition, I wondered whether it is necessary to use this pattern as internally if versionStateCache is null in the get path, it will try to do the lookup internally.

My main question is that whether this param: svsSupplier is necessary or not in openStore and openStoreForNewPartition.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the user can pass a null supplier for such usage, right?

gaojieliu
gaojieliu previously approved these changes Nov 8, 2024
Copy link
Contributor

@gaojieliu gaojieliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code change itself looks good and I left an open-question to @FelixGV .

sixpluszero
sixpluszero previously approved these changes Nov 8, 2024
@jingy-li jingy-li merged commit 5752ad4 into linkedin:main Nov 12, 2024
45 checks passed
@jingy-li jingy-li deleted the fix-bug-for-offset-svs-sync branch November 12, 2024 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants