feat: Support for Azure CNI without overlay & BYO CNI

[Bryce-Soghigian]

Fixes #367

Description
This PR adds support for azure cni without overlay, as well as introduces some makefile goodness for creating clusters or other cni configurations.

Why Do We Need Secondary IP Configs For AZ CNI Without Overlay?

When a pod is created, the Azure CNI plugin allocates an IP address from the pool of secondary IP addresses configured on the NIC of the node where the pod is scheduled. The Azure CNI plugin manages the allocation and de-allocation of these IP addresses through the IP Address Manager (IPAM), ensuring each pod receives a unique IP address and tracking the usage of these addresses.

In this setup, pods are assigned IP addresses from the node's subnet, allowing for direct IP connectivity. This enables pods within the same virtual network to communicate without the need for Network Address Translation (NAT). The node's NIC routes traffic to the appropriate pod based on the assigned IP.

Flow

1. Node NIC: Primary IP and multiple secondary IP addresses are assigned to the nic on node creation, and the nic is assigned to the node
2. Pod Create: Pod requests an IP address on intialization
3. Azure CNI Plugin: Assigns a secondary IP to the pod from the node’s NIC.
4. Network Interface within Node: We use transparent mode, which doesn't change any properties of the eth0 interface on the nic. Azure CNI creates and adds host-side pod veth pair interfaces that are added to the host network.
5. Pod to Pod Communication: Pods communicate using their assigned IPs directly within the virtual network. Pod to Pod communication is over layer 3 and L3 routing rules route the pod traffic.

Learn more about specifics here

How was this change tested?

• e2e tests see: https://github.com/Azure/karpenter-provider-azure/actions/runs/10950653145/job/30407746822
• unit tests
• manual scale ups and goldpinger

What this PR does not include

• E2E Suite for our Upstream E2Es here testing other network connectivity(ingress + egress)
• Port of our E2Es to run fully each time on azure cni without overlay
  Does this change impact docs?
• Yes, PR includes docs updates
• Yes, issue opened: #
• No

Release Note

support added for static AzureCNI without overlay networking

[Bryce-Soghigian]

/test

[Bryce-Soghigian]

E2E Run: https://github.com/Azure/karpenter-provider-azure/actions/runs/10950653145/job/30407746822

[tangyouzzz]

From what I can tell, in terms of the contract there are two places where NETWORK_PLUGIN is used

https://github.com/Azure/AgentBaker/blob/1c4f87e96b26cd8933a16cc31eaa9d18025932f1/self-contained/bootstrap_install.sh#L62

installNetworkPlugin() {
    if [[ "${NETWORK_PLUGIN}" = "azure" ]]; then
        installAzureCNI
    fi
    installCNI  #reference plugins. Mostly for kubenet but loop back used by contaierd until containerd 2
    rm -rf $CNI_DOWNLOADS_DIR &
}

https://github.com/Azure/AgentBaker/blob/1c4f87e96b26cd8933a16cc31eaa9d18025932f1/self-contained/bootstrap_config.sh#L283

configureCNIIPTables() {
    if [[ "${NETWORK_PLUGIN}" = "azure" ]]; then
        mv $CNI_BIN_DIR/10-azure.conflist $CNI_CONFIG_DIR/
        chmod 600 $CNI_CONFIG_DIR/10-azure.conflist
        if [[ "${NETWORK_POLICY}" == "calico" ]]; then
          sed -i 's#"mode":"bridge"#"mode":"transparent"#g' $CNI_CONFIG_DIR/10-azure.conflist
        elif [[ "${NETWORK_POLICY}" == "" || "${NETWORK_POLICY}" == "none" ]] && [[ "${NETWORK_MODE}" == "transparent" ]]; then
          sed -i 's#"mode":"bridge"#"mode":"transparent"#g' $CNI_CONFIG_DIR/10-azure.conflist
        fi
        /sbin/ebtables -t nat --list
    fi
}

but we need it to support BYO CNI it seems based on these two references in Agentbaker

After adding the NETWORK-POLICY variable, Karpenter should be able to be used in underlay network and Azure Cni AKS

[tallaxes]

Looks good, mosly minor things

[pdeva]

thank you for fixing this folks! hope a new release is cut soon. we cant wait to use this on our AKS cluster.