Design for Velero uploader configuration integration and extensibility #7005

qiuming-best · 2023-10-24T01:29:46Z

Thank you for contributing to Velero!

Please add a summary of your change

Does your change fix a particular issue?

Fixes #(issue)
#6607

Please indicate you've done the following:

Accepted the DCO. Commits without the DCO will delay acceptance.
Created a changelog file or added /kind changelog-not-required as a comment on this pull request.
Updated the corresponding documentation in site/content/docs/main.

codecov · 2023-10-24T01:41:21Z

Codecov Report

Merging #7005 (a86b394) into main (19f38f9) will increase coverage by 0.01%.
Report is 9 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #7005      +/-   ##
==========================================
+ Coverage   61.03%   61.05%   +0.01%     
==========================================
  Files         251      251              
  Lines       26837    26846       +9     
==========================================
+ Hits        16381    16390       +9     
  Misses       9303     9303              
  Partials     1153     1153

see 4 files with indirect coverage changes

Lyndon-Li · 2023-10-24T04:03:15Z

design/velero-uploader-configuration.md

+```
+
+The process is as follows: Users pass the ParallelFilesUpload parameter and its value through the Velero CLI. This parameter and its value are stored as a sub-option within UploaderConfig and then placed into the Backup CR. When users perform file-system level backups, UploaderConfig is passed to the PodVolumeBackup CR. When users use the Data-mover for backups, it is passed to the DataUpload CR. Each respective controller within the CRs calls the uploader, and the ParallelFilesUpload from UploaderConfig in CRs is passed to the uploader. When the uploader subsequently calls the Kopia API, it can use the ParallelFilesUpload to set the MaxParallelFileReads parameter.
+


Let's mention the situation for Restic here --- this option is not supported by Restic uploader.

Lyndon-Li · 2023-10-24T04:04:03Z

design/velero-uploader-configuration.md

+}
+```
+
+The process is as follows: Users pass the ParallelFilesUpload parameter and its value through the Velero CLI. This parameter and its value are stored as a sub-option within UploaderConfig and then placed into the Backup CR. When users perform file-system level backups, UploaderConfig is passed to the PodVolumeBackup CR. When users use the Data-mover for backups, it is passed to the DataUpload CR. Each respective controller within the CRs calls the uploader, and the ParallelFilesUpload from UploaderConfig in CRs is passed to the uploader. When the uploader subsequently calls the Kopia API, it can use the ParallelFilesUpload to set the MaxParallelFileReads parameter.


file-system level backups -> file system backups. file system backup is a particular term of Velero, let's make it consistent.

Lyndon-Li · 2023-10-24T04:04:24Z

design/velero-uploader-configuration.md

+```
+
+### Configuration Propagated to Different CRDs
+The configuration specified in `UploaderConfig` needs to be effective for both file-system level backups and data-mover backups. Therefore, the `UploaderConfig` field value from the backup CRD should be propagated to `PodVolumeBackup` and `DataUpload` CRDs:


file-system level backups -> file system backups. file system backup is a particular term of Velero, let's make it consistent.

Lyndon-Li · 2023-10-24T04:05:31Z

design/velero-uploader-configuration.md

+```
+
+### Configuration Propagated to Different CRDs
+The configuration specified in `UploaderConfig` needs to be effective for both file-system level backups and data-mover backups. Therefore, the `UploaderConfig` field value from the backup CRD should be propagated to `PodVolumeBackup` and `DataUpload` CRDs:


Let's add Restore the UploaderConfig to Restore CRD as well and propagate it to PodVolumeRestore and DataDownload CRDs, considering an option may be for restores

Signed-off-by: Ming <mqiu@vmware.com>

reasonerjt · 2023-10-24T07:59:48Z

design/velero-uploader-configuration.md

+below the sub-option `ParallelFilesUpload` is added into UploaderConfig:
+
+```go
+type UploaderConfig struct {


Is it safe to assume most attributes to be added to this configuration, applies to both backup and restore workflow?

i.e. there won't be an attribute xyz that only makes sense in backup but not restore?

I think here the plan is to have the backup options and restore options mingling in this UploaderConfig struct, so that we don't need to introduce separate struct for backups and restores.
Actually, quite lots of options are for either backups or restores, not for the both.

it does not matter even if we have one attribute 'xyz' both make sense in backup and restore.
As the command created by Velero CLI could not take effect both in backup and restore at the same time. The command may backup or restore, the UploaderConfig will store in Backup or Restore accordingly

reasonerjt · 2023-10-24T08:00:32Z

design/velero-uploader-configuration.md

+
+```go
+type UploaderConfig struct {
+    ParallelFilesUpload int `json:"parallelFilesUpload,omitempty"`


If it is left unset the value will be 0, does 0 has the same meaning as 1?

0 means use the default policy and treaded as not assigned value if the user specifies with 0.
it's different from 1

reasonerjt · 2023-10-24T08:03:03Z

design/velero-uploader-configuration.md

+below are the key steps that should be added to support this new feature.
+
+#### Velero CLI 
+The Velero CLI will support a `--parallel-files-upload` flag, allowing users to set the `ParallelFilesUpload` value when creating backups.


is it only for backup?

We should be careful with the doc to explain this parm only impacts the flow when the files are handled by uploader like kopia and restic

yes, it's only for backup. and in --parallel-files-upload flag usage clearly states that it can only be used for backups using kopia type of uploader

draghuram · 2023-10-24T20:10:46Z

This design looks good to me. I am guessing that we can add a boolean field called "sparseRestore" to implement #6664. If you agree, how do you want to proceed with the work? Do you want to update this proposal itself with the new field or perhaps check this in first and then write a different proposal?

qiuming-best · 2023-10-25T00:19:54Z

@draghuram > This design looks good to me. I am guessing that we can add a boolean field called "sparseRestore" to implement #6664. If you agree, how do you want to proceed with the work? Do you want to update this proposal itself with the new field or perhaps check this in first and then write a different proposal?

sparseRestore is good.

once this proposal is merged, I think you could add a new sub-section in this design about the configuration sparseRestore which would be similar to the sub-section Parallel Files Upload of Sub-options in UploaderConfig

qiuming-best added the kind/changelog-not-required PR does not require a user changelog. Often for docs, website, or build changes label Oct 24, 2023

github-actions bot added the Area/Design Design Documents label Oct 24, 2023

github-actions bot assigned qiuming-best Oct 24, 2023

github-actions bot requested review from blackpiglet and reasonerjt October 24, 2023 01:30

qiuming-best changed the title ~~Velero Uploader Configuration Integration and Extensibility~~ Design for Velero uploader configuration integration and extensibility Oct 24, 2023

qiuming-best force-pushed the kopia-parallelism-design branch 2 times, most recently from c402333 to c03a96c Compare October 24, 2023 03:52

qiuming-best requested a review from Lyndon-Li October 24, 2023 03:56

qiuming-best force-pushed the kopia-parallelism-design branch from c03a96c to 9bca1b8 Compare October 24, 2023 04:00

Lyndon-Li reviewed Oct 24, 2023

View reviewed changes

qiuming-best force-pushed the kopia-parallelism-design branch from 9bca1b8 to b9a37ee Compare October 24, 2023 06:08

Velero Uploader Configuration Integration and Extensibility

a86b394

Signed-off-by: Ming <mqiu@vmware.com>

qiuming-best force-pushed the kopia-parallelism-design branch from b9a37ee to a86b394 Compare October 24, 2023 06:10

Lyndon-Li approved these changes Oct 24, 2023

View reviewed changes

Lyndon-Li mentioned this pull request Oct 24, 2023

Allow "sparse" option for Kopia & Restic restore #6664

Closed

qiuming-best mentioned this pull request Oct 24, 2023

Make Kopia file parallelism configurable #7000

Merged

3 tasks

reasonerjt reviewed Oct 24, 2023

View reviewed changes

blackpiglet approved these changes Oct 26, 2023

View reviewed changes

qiuming-best merged commit 3b22ff3 into vmware-tanzu:main Oct 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Design for Velero uploader configuration integration and extensibility #7005

Design for Velero uploader configuration integration and extensibility #7005

qiuming-best commented Oct 24, 2023

codecov bot commented Oct 24, 2023 •

edited

Loading

Lyndon-Li Oct 24, 2023

qiuming-best Oct 24, 2023

Lyndon-Li Oct 24, 2023 •

edited

Loading

qiuming-best Oct 24, 2023

Lyndon-Li Oct 24, 2023

qiuming-best Oct 24, 2023

Lyndon-Li Oct 24, 2023

qiuming-best Oct 24, 2023 •

edited

Loading

reasonerjt Oct 24, 2023

Lyndon-Li Oct 24, 2023

qiuming-best Oct 24, 2023

reasonerjt Oct 24, 2023

qiuming-best Oct 24, 2023 •

edited

Loading

reasonerjt Oct 24, 2023

qiuming-best Oct 24, 2023

draghuram commented Oct 24, 2023

qiuming-best commented Oct 25, 2023 •

edited

Loading

		```

		The process is as follows: Users pass the ParallelFilesUpload parameter and its value through the Velero CLI. This parameter and its value are stored as a sub-option within UploaderConfig and then placed into the Backup CR. When users perform file-system level backups, UploaderConfig is passed to the PodVolumeBackup CR. When users use the Data-mover for backups, it is passed to the DataUpload CR. Each respective controller within the CRs calls the uploader, and the ParallelFilesUpload from UploaderConfig in CRs is passed to the uploader. When the uploader subsequently calls the Kopia API, it can use the ParallelFilesUpload to set the MaxParallelFileReads parameter.

Design for Velero uploader configuration integration and extensibility #7005

Design for Velero uploader configuration integration and extensibility #7005

Conversation

qiuming-best commented Oct 24, 2023

Please add a summary of your change

Does your change fix a particular issue?

Please indicate you've done the following:

codecov bot commented Oct 24, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lyndon-Li Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiuming-best Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiuming-best Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

draghuram commented Oct 24, 2023

qiuming-best commented Oct 25, 2023 • edited Loading

codecov bot commented Oct 24, 2023 •

edited

Loading

Lyndon-Li Oct 24, 2023 •

edited

Loading

qiuming-best Oct 24, 2023 •

edited

Loading

qiuming-best Oct 24, 2023 •

edited

Loading

qiuming-best commented Oct 25, 2023 •

edited

Loading