[Feat] Multi region support (Topology Aware Provisioning) #280

komer3 · 2024-10-04T00:27:18Z

Overview

This PR enhances the CSI driver's controller server to provision volumes in regions outside of its deployment region. The improvement allows for more flexible and efficient volume management across different geographical locations.

Key Concepts

Remote Region: A region where worker nodes are deployed, separate from the control plane region.
Example: Control plane in us-ord, worker node in us-central.

Current Deployment Setup

Controller server: Deployed on worker nodes
Node servers: Deployed as a DaemonSet on all nodes

Changes Implemented

1. Dynamic Region Detection

Previously: Used metadata service in the controller service class to determine the region.
Now: Utilizes topology-based dynamic provisioning.

2. Volume Provisioning Process

External-provisioner makes an API call to the api-server to retrieve the Linode region.
This region information is included in the CreateVolumeRequest object during the CreateVolume gRPC call.
Enables volume provisioning in the region where the pod is scheduled.

3. Volume Binding Mode

Changed from 'Immediate' to 'WaitForFirstConsumer'.
Allows the provisioner time to gather necessary region data before volume creation.

Benefits

Improved flexibility in volume provisioning across different regions.
Better alignment of volume location with pod scheduling.

General:

Have you removed all sensitive information, including but not limited to access keys and passwords?
Have you checked to ensure there aren't other open or closed Pull Requests for the same bug/feature/question?

Pull Request Guidelines:

Does your submission pass tests?
Have you added tests?
Are you addressing a single feature in this PR?
Are your commits atomic, addressing one change per commit?
Are you following the conventions of the language?
Have you saved your large formatting changes for a different PR, so we can focus on your work?
Have you explained your rationale for why this feature is needed?
Have you linked your PR to an open issue

- Updated CreateVolume method to include accessibility requirements. - Modified attemptCreateLinodeVolume and createLinodeVolume methods to accept and utilize accessibility requirements for region selection. - Adjusted createAndWaitForVolume to pass accessibility requirements to attemptCreateLinodeVolume. - Ensured volume context includes the region derived from accessibility requirements.

… the volume context, for validation

codecov · 2024-10-04T02:26:32Z

Codecov Report

Attention: Patch coverage is 84.44444% with 7 lines in your changes missing coverage. Please review.

Project coverage is 76.33%. Comparing base (1748a1b) to head (47706e2).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/driver/controllerserver_helper.go	85.71%	3 Missing and 2 partials ⚠️
internal/driver/controllerserver.go	80.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #280      +/-   ##
==========================================
+ Coverage   76.32%   76.33%   +0.01%     
==========================================
  Files          21       21              
  Lines        1622     1644      +22     
==========================================
+ Hits         1238     1255      +17     
- Misses        286      289       +3     
- Partials       98      100       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…fail

…here controller server is deployed

…sioning works

…oyed

…from CreateVolume()

tchinmai7 · 2024-10-09T22:55:46Z

docs/topology-aware-provisioning.md

+
+#### Provisioning Process
+
+1. CO determines required topology based on application needs and cluster layout.


can you explain what CO is?

Updated the doc. Hope its more clear now! Let me know if there is anything else that is unclear :)

tchinmai7 · 2024-10-09T22:57:20Z

internal/driver/controllerserver_helper.go

 		return nil, errRegionMismatch(volumeData.Region, cs.metadata.Region)
 	}

+	// If a topology is specified, the source volume must be in the same region as the specified topology
+	if accessibilityRequirements != nil {
+		if volumeData.Region != accessibilityRequirements.GetPreferred()[0].GetSegments()[VolumeTopologyRegion] {


is this guaranteed to be an array of len > 0 && is the key guaranteed to exist?

Its pretty much guaranteed to exist and the first key is going to be the region we want to use.

for more info check this link out! https://kubernetes-csi.github.io/docs/topology.html?highlight=VOLUME_ACCESSIBILITY_CONSTRAINTS#implementing-topology-in-your-csi-driver

I was curious because getRegionFromTopology operates on the same accessibilityRequirements but includes all those checks, Thanks for the link - will check it out!

could also use that here - to validate

The only time that is not passed (accessibilityRequirements) is when you have a broken multi region cluster (I had this issue while trying to make konnectivity work). In that senario, I'm falling back to using the region from the metadata (region where controller server is running)

Good catch. Let me include those checks here as well

Updated it!

tchinmai7 · 2024-10-09T23:04:58Z

docs/topology-aware-provisioning.md

+```
+
+This example demonstrates how to set up topology-aware provisioning using the Linode Block Storage CSI Driver. The StorageClass defines the provisioner and reclaim policy, while the PersistentVolumeClaim requests storage from this class. The Pod specification shows how to use the PVC and includes a node selector for region-specific deployment.
+


should we also mention that the cluster itself must be started with --feature-gates=CSINodeInfo=true ? https://kubernetes-csi.github.io/docs/topology.html?highlight=VOLUME_ACCESSIBILITY_CONSTRAINTS#kubernetes-cluster-setup

Yeah good point. Let me add that as well.

Also, that feature flag is going to be turn on by default now with future csi releases

docs/topology-aware-provisioning.md

Khaja Omer added 6 commits October 3, 2024 19:05

feat: Update ControllerPublishVolume() to use the region, provided in…

eafca43

… the volume context, for validation

test: update the unit test cases based on the previous changes

eff8321

refactor: reducing the nested if complexity for golangci :)

2c4ad48

fix: update test cases that were failing and minor refactors

e30ba15

fix: linting

92e0a49

komer3 and others added 5 commits October 4, 2024 07:36

Merge branch 'main' into multi-region-support

6f7f536

fix: fixing logic that causing the controller server sanity tests to …

95b617e

…fail

Merge branch 'main' into multi-region-support

0fb44bc

Merge branch 'main' into multi-region-support

576771c

feat: updating the logic to handle volume cloning within remote region

4ffbc3f

komer3 changed the title ~~[Feat] Multi region support - Volume creation (WIP)~~ [Feat] Multi region support (WIP) Oct 8, 2024

Khaja Omer added 6 commits October 8, 2024 19:25

feat: update the provisioner args to enable topology based provisioning

a98c894

Testing to see if it all works with out fallback to metadata region w…

d5d4237

…here controller server is deployed

test: run upstream e2e to make sure changes with topology based provi…

14752ae

…sioning works

revert to having a region fallback to where controller server is depl…

ac526d0

…oyed

no need to double check volume context since we guarantee to pass it …

328086d

…from CreateVolume()

feat: update docs and manifest

2b7d27a

komer3 changed the title ~~[Feat] Multi region support (WIP)~~ [Feat] Multi region support Oct 9, 2024

komer3 marked this pull request as ready for review October 9, 2024 22:53

komer3 requested review from a team as code owners October 9, 2024 22:53

Merge branch 'main' into multi-region-support

200e115

tchinmai7 reviewed Oct 9, 2024

View reviewed changes

updates to doc based on comments

30fd9cb

tchinmai7 reviewed Oct 9, 2024

View reviewed changes

Khaja Omer added 2 commits October 9, 2024 18:11

adding info about sidecare feature flag

e24ea9d

refactor: simplify the region checking logic by reusing code/func

0579192

test: Adding test cases from new getRegionFromTopology() func

1c31cfd

srust reviewed Oct 10, 2024

View reviewed changes

docs/topology-aware-provisioning.md Show resolved Hide resolved

komer3 changed the title ~~[Feat] Multi region support~~ [Feat] Multi region support (Topology Aware Provisioning) Oct 10, 2024

Switch storage class naming

47706e2

eljohnson92 approved these changes Oct 11, 2024

View reviewed changes

komer3 merged commit 0c9e508 into main Oct 11, 2024
9 checks passed

komer3 deleted the multi-region-support branch October 11, 2024 20:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feat] Multi region support (Topology Aware Provisioning) #280

[Feat] Multi region support (Topology Aware Provisioning) #280

komer3 commented Oct 4, 2024 •

edited

Loading

codecov bot commented Oct 4, 2024 •

edited

Loading

tchinmai7 Oct 9, 2024

komer3 Oct 9, 2024

tchinmai7 Oct 9, 2024

komer3 Oct 9, 2024

tchinmai7 Oct 9, 2024

tchinmai7 Oct 9, 2024

komer3 Oct 9, 2024

komer3 Oct 9, 2024

komer3 Oct 10, 2024

tchinmai7 Oct 9, 2024

komer3 Oct 9, 2024


		#### Provisioning Process

		1. CO determines required topology based on application needs and cluster layout.

		```

		This example demonstrates how to set up topology-aware provisioning using the Linode Block Storage CSI Driver. The StorageClass defines the provisioner and reclaim policy, while the PersistentVolumeClaim requests storage from this class. The Pod specification shows how to use the PVC and includes a node selector for region-specific deployment.

[Feat] Multi region support (Topology Aware Provisioning) #280

[Feat] Multi region support (Topology Aware Provisioning) #280

Conversation

komer3 commented Oct 4, 2024 • edited Loading

Overview

Key Concepts

Current Deployment Setup

Changes Implemented

1. Dynamic Region Detection

2. Volume Provisioning Process

3. Volume Binding Mode

Benefits

General:

Pull Request Guidelines:

codecov bot commented Oct 4, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komer3 commented Oct 4, 2024 •

edited

Loading

codecov bot commented Oct 4, 2024 •

edited

Loading