Velero Uploader Configuration Integration and Extensibility

Signed-off-by: Ming <mqiu@vmware.com>

design/files-parallelism-upload.md

@@ -0,0 +1,80 @@
+# Velero Uploader Configuration Integration and Extensibility
+
+## Abstract
+This design proposal aims to make Velero Uploader configurable by introducing a structured approach for managing Uploader settings. Initially, we will focus on enabling the configuration of the number of parallel file uploads during backups, and we will define and standardize a data structure to facilitate future additions to Uploader configurations. This enhancement provides a template for extending Uploader-related options.
+
+## Background
+Velero is widely used for backing up and restoring Kubernetes clusters. In certain scenarios, particularly when dealing with large datasets, optimizing the backup process to improve execution speed is crucial. Additionally, as time progresses, there may be a need to introduce more configuration options related to the Uploader. Therefore, a standardized configuration template is required.
+
+## Goals
+1. **Extensible Uploader Configuration**: Provide an extensible approach to manage Uploader configurations, making it easy to add and modify configuration options related to the backup process.
+
+2. **Parallel Files Upload Configuration**: Introduce a new configuration option, `ParallelFilesUpload`, allowing users to specify the number of parallel file uploads during backups.
+
+3. **Flexible Configuration**: Allow users to configure the `ParallelFilesUpload` value when creating backups using the Velero CLI. This value will be stored in the backup Custom Resource Definition (CRD) to ensure its application during the backup process.
+
+## Non Goals
+1. This design does not encompass the configuration settings of other components or modules.
+2. It specifically supports file system backups based on the Kopia Uploader and does not address configurations related to Restic uploader.
+
+## High-Level Design
+To achieve extensibility in Velero Uploader configurations, the following key components and changes are proposed:
+
+### UploaderConfig Structure
+A new data structure, `UploaderConfig`, will be defined to store Uploader configurations. This structure will include the `ParallelFilesUpload` setting and provide room for future configuration options:
+
+```go
+type UploaderConfig struct {
+    ParallelFilesUpload int `json:"parallelFilesUpload,omitempty"`
+    // other options
+}
+```
+
+### Integration with Backup CRD
+The Velero CLI will support a `--parallel-files-upload` flag, allowing users to set the `ParallelFilesUpload` value when creating backups. This value will be stored in the `UploaderConfig` field within the backup CRD:
+
+```go
+type BackupSpec struct {
+    UploaderConfig shared.UploaderConfig `json:"uploaderConfig,omitempty"`
+}
+```
+
+### Configuration Application to Different CRDs
+The configuration specified in `UploaderConfig` needs to be effective for both file-system level backups and data-mover backups. Therefore, the `UploaderConfig` field value from the backup CRD should be propagated to `PodVolumeBackup` and `DataUpload` CRDs:
+
+```go
+type PodVolumeBackupSpec struct {
+    ...
+    UploaderConfig shared.UploaderConfig `json:"uploaderConfig,omitempty"`
+}
+
+type DataUploadSpec struct {
+    ...
+    UploaderConfig shared.UploaderConfig `json:"uploaderConfig,omitempty"`
+}
+```
+
+By doing this, any future additions to Uploader configurations can be accommodated by simply adding new fields to the `UploaderConfig` structure.
+
+### Kopia Parallel Upload Policy
+Velero Uploader has the ability to set upload policies when calling Kopia APIs. In the Kopia codebase, the structure for upload policies is defined as follows:
+
+```go
+// UploadPolicy describes policy to apply when uploading snapshots.
+type UploadPolicy struct {
+    ...
+    MaxParallelFileReads *OptionalInt `json:"maxParallelFileReads,omitempty"`
+}
+```
+
+Velero can set the `MaxParallelFileReads` parameter for Kopia's upload policy as follows:
+
+```golang
+curPolicy := getDefaultPolicy()
+if uploaderCfg.ParallelFilesUpload > 0 {
+    curPolicy.UploadPolicy.MaxParallelFileReads = newOptionalInt(uploaderCfg.ParallelFilesUpload)
+}
+```
+
+## Alternatives Considered
+To enhance extensibility further, the option of storing `UploaderConfig` in a Kubernetes ConfigMap can be explored. This approach would allow the addition and modification of configuration options without the need to modify the backup CRD.