Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: CNS writes CNI conflist on Windows. #2281

Closed

Conversation

nddq
Copy link
Contributor

@nddq nddq commented Oct 9, 2023

Reason for Change:

Allow CNS to write CNI conflist for SWIFT and SWIFT Overlay scenario on Windows.

Issue Fixed:

Requirements:

Notes:

@nddq nddq added cns Related to CNS. cni Related to CNI. swift Related to SWIFT networking. windows labels Oct 9, 2023
@nddq nddq self-assigned this Oct 9, 2023
@nddq nddq requested review from a team as code owners October 9, 2023 16:07
@nddq nddq force-pushed the feat/cns-writes-cni-conflist-on-windows branch 2 times, most recently from eb34786 to d1fa944 Compare October 9, 2023 20:27
@nddq
Copy link
Contributor Author

nddq commented Oct 10, 2023

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@nddq nddq enabled auto-merge (squash) October 10, 2023 21:16
@nddq
Copy link
Contributor Author

nddq commented Oct 10, 2023

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@nddq nddq force-pushed the feat/cns-writes-cni-conflist-on-windows branch 4 times, most recently from 71293ae to ad104ca Compare October 16, 2023 16:29
Comment on lines +49 to +60
"ExceptionList": [
"10.240.0.0/16",
"10.0.0.0/8"
]
}`),
},
{
Name: "EndpointPolicy",
Value: []byte(`{
"Type": "ROUTE",
"DestinationPrefix": "10.0.0.0/8",
"NeedEncap": true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you fix the indentation spaces here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nddq it's a mix of spaces and tabs, you should replace it with all tabs

Comment on lines +104 to +115
"ExceptionList": [
"10.240.0.0/16",
"10.0.0.0/8"
]
}`),
},
{
Name: "EndpointPolicy",
Value: []byte(`{
"Type": "ROUTE",
"DestinationPrefix": "10.0.0.0/8",
"NeedEncap": true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here too

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could come in a later PR, but we should test this feature on our pipeline clusters

@nddq nddq force-pushed the feat/cns-writes-cni-conflist-on-windows branch from ad104ca to ec788d4 Compare October 17, 2023 19:56
@rbtr rbtr requested a review from jaer-tsun October 23, 2023 20:08
@rbtr rbtr force-pushed the feat/cns-writes-cni-conflist-on-windows branch from ec788d4 to 327b97b Compare October 24, 2023 19:31
@nddq nddq force-pushed the feat/cns-writes-cni-conflist-on-windows branch from 327b97b to 10324cc Compare October 27, 2023 00:55
@@ -0,0 +1,52 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that in Windows AKS, the CNI conflist is generated dynamically based on a few things:

  • If the customer specified DisableOutboundNAT, then a certain ExceptionList for an EndpointPolicy will not be present
    • In the conflist I commented down below, this block is what I'm talking about:
{
     "Name":  "EndpointPolicy",
     "Value":  {
                   "Type":  "OutBoundNAT",
                   "ExceptionList":  [
                                         "fddc:be24:d690:87d::/64"
                                     ]
               }
 },

There are other settings that are dynamic, based on a few customer inputs, so we'll need to take all that into account when generating these Windows conflists

Here is where the Conflists are configured (altered) in Windows AKS: AgentBaker - azurecnifunc.ps1

I would also talk to @AbelHu about what all settings in a customer cluster, can change the Conflist (or read the script I linked, he gave it to me originally)

@@ -0,0 +1,52 @@
{
"cniVersion": "0.3.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is an example overlay conflist today in AKS, from a cluster that I just made once:

{
    "cniVersion":  "0.3.0",
    "name":  "azure",
    "adapterName":  "",
    "plugins":  [
                    {
                        "type":  "azure-vnet",
                        "mode":  "bridge",
                        "bridge":  "azure0",
                        "capabilities":  {
                                             "portMappings":  true,
                                             "dns":  true
                                         },
                        "ipam":  {
                                     "type":  "azure-cns",
                                     "mode":  "overlay"
                                 },
                        "dns":  {
                                    "Nameservers":  [
                                                        "10.0.0.10",
                                                        "168.63.129.16"
                                                    ],
                                    "Search":  [
                                                   "svc.cluster.local"
                                               ]
                                },
                        "AdditionalArgs":  [
                                               {
                                                   "Name":  "EndpointPolicy",
                                                   "Value":  {
                                                                 "Type":  "OutBoundNAT",
                                                                 "ExceptionList":  [
                                                                                       "10.244.0.0/16"
                                                                                   ]
                                                             }
                                               },
                                               {
                                                   "Name":  "EndpointPolicy",
                                                   "Value":  {
                                                                 "Type":  "OutBoundNAT",
                                                                 "ExceptionList":  [
                                                                                       "fddc:be24:d690:87d::/64"
                                                                                   ]
                                                             }
                                               },
                                               {
                                                   "Name":  "EndpointPolicy",
                                                   "Value":  {
                                                                 "Type":  "ACL",
                                                                 "Protocols":  "6",
                                                                 "Action":  "Block",
                                                                 "Direction":  "Out",
                                                                 "RemoteAddresses":  "168.63.129.16/32",
                                                                 "RemotePorts":  "80",
                                                                 "Priority":  200,
                                                                 "RuleType":  "Switch"
                                                             }
                                               },
                                               {
                                                   "Name":  "EndpointPolicy",
                                                   "Value":  {
                                                                 "Type":  "ACL",
                                                                 "Action":  "Allow",
                                                                 "Direction":  "In",
                                                                 "Priority":  65500
                                                             }
                                               },
                                               {
                                                   "Name":  "EndpointPolicy",
                                                   "Value":  {
                                                                 "Type":  "ACL",
                                                                 "Action":  "Allow",
                                                                 "Direction":  "Out",
                                                                 "Priority":  65500
                                                             }
                                               }
                                           ],
                        "windowsSettings":  {
                                                "enableLoopbackDSR":  true
                                            }
                    }
                ]
}

@nddq nddq force-pushed the feat/cns-writes-cni-conflist-on-windows branch from 10324cc to 26c1d07 Compare November 2, 2023 20:27
auto-merge was automatically disabled December 19, 2023 23:02

Merge queue setting changed

Copy link

This pull request is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days

@github-actions github-actions bot added the stale Stale due to inactivity. label Dec 20, 2023
Copy link

Pull request closed due to inactivity.

@github-actions github-actions bot closed this Dec 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cni Related to CNI. cns Related to CNS. stale Stale due to inactivity. swift Related to SWIFT networking. windows
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants