Kubernetes Virtual Kubelet with ACI

Azure Container Instances (ACI) provides a hosted environment for running containers in Azure. When using ACI, there is no need to manage the underlying compute infrastructure since Azure handles this management overhead. When running containers in ACI, users are charged based on the container lifecycle in seconds.

The ACI provider for the Virtual Kubelet configures ACI service as a virtual node in a Kubernetes cluster. Hence, pods scheduled on the virtual node can be run on ACI instances. This configuration allows users to take advantage of both the capabilities of Kubernetes and the management value and cost benefit of ACI.

Features

Virtual Kubelet's ACI provider relies heavily on the feature set that ACI service provides. Please check the Azure documentations for region availability, pricing and new features. The list here presents a sample reference for the features that ACI provider supports. Note: Users should NOT expect feature parities between Virutal Kubelet and real Kubernetes Kubelet.

Supported

• Volumes: empty dir, github repo, projection, Azure Files, Azure Files CSI drivers
• Secure env variables, config maps
• Virtual network integration (VNet)
• Network security group support
• Exec support for container instances
• Azure Monitor integration ( aka OMS)
• Support for init-containers (use init containers)

Limitations (Not supported)

• Using service principal credentials to pull ACR images (see workaround)
• Liveness and readiness probes
• Limitations with VNet
• VNet peering
• Argument support for exec
• Host aliases support
• Downward APIs (i.e podIP)
• Projected volumes
• Potentially any new features introduced in real Kubelet since 1.24.

Installation

Using Azure Portal for AKS clusters

Please follow this offical document to install virtual node for an AKS cluster using Azure Portal.

Using Azure CLI for AKS clusters

Please follow this offical document to install virtual node for an AKS cluster using Azure CLI.

Deploy Virtual Kubelet manually using Helm

In the following cases, users may need to install Virtual Kubelet manually.

• The ACI provider releases are rolled out to all AKS clusters that are configured with virtual node addon progressively. If the AKS managed version has an issue, or there is an immediate need to use a new release that has not been rolled out yet, you can follow the downgrade document to use a previous version or the upgrade document to use the latest released version. In either case, a new virtual node will be created in the cluster.
• Install a virtual node to support running windows ACI instances (document).

Note: Mannually installed virtual nodes are NOT managed by AKS anymore, they will not be upgraded automatically.

Non-AKS Kubernetes clusters

We do not support non-AKS Kubernetes. If you have specific requests for those clusters, please file an issue to describe the needs.

Demo

To validate that the Virtual Kubelet has been installed, check and find the virtual node in the Kubernetes cluster.

kubectl get nodes
NAME                                        STATUS    ROLES     AGE       VERSION
virtual-kubelet-aci-linux                   Ready     agent     2m        v1.13.1
aks-nodepool1-XXXXXXXX-0                    Ready     agent     22h       v1.12.6
aks-nodepool1-XXXXXXXX-1                    Ready     agent     22h       v1.12.6
aks-nodepool1-XXXXXXXX-2                    Ready     agent     22h       v1.12.6

Create a test Pod virtual-kubelet-test.yaml.

apiVersion: v1
kind: Pod
metadata:
  name: helloworld
spec:
  containers:
  - image: microsoft/aci-helloworld
    imagePullPolicy: Always
    name: helloworld
    resources:
      requests:
        memory: 1G
        cpu: 1
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    - containerPort: 443
      name: https
  dnsPolicy: ClusterFirst
  nodeSelector:
    kubernetes.io/role: agent
    beta.kubernetes.io/os: linux
    type: virtual-kubelet
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
  - key: azure.com/aci
    effect: NoSchedule

Note that virtual nodes are tainted by default to avoid unexpected pods running on them, i.e., kube-proxy. To schedule a pod to them, you need to add the toleration to the pod spec and a node selector:

...
  nodeSelector:
    kubernetes.io/role: agent
    beta.kubernetes.io/os: linux
    type: virtual-kubelet
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
  - key: azure.com/aci
    effect: NoSchedule

If your image is in a private registry, you need to add a kubernetes secret to your cluster and reference it in the pod spec.

...
  spec:
    containers:
    - name: aci-helloworld
      image: <registry name>.azurecr.io/aci-helloworld:v1
      ports:
      - containerPort: 80
    imagePullSecrets:
    - name: <K8 secret name>

Run the application.

kubectl apply -f virtual-kubelet-test.yaml

Note that the helloworld pod is running on the virtual-kubelet node.

NAME                                            READY     STATUS    RESTARTS   AGE       IP             NODE
aci-helloworld-2559879000-XXXXXX                 1/1       Running   0          39s       52.179.XXX.XXX   virtual-kubelet-aci-linux

If the AKS cluster was configured with a virtual network, then the output will look like the following. The container instance will get a private IP rather than a public one.

NAME                            READY     STATUS    RESTARTS   AGE       IP           NODE
aci-helloworld-9b55975f-XXXXX   1/1       Running   0          4m        10.241.XXX.XXX   virtual-kubelet-aci-linux

To validate that the container is running in an Azure Container Instance, use the az container list Azure CLI command.

az container list -o table

Result

Name                             ResourceGroup    ProvisioningState    Image                     IP:ports         CPU/Memory       OsType    Location
-------------------------------  ---------------  -------------------  ------------------------  ---------------  ---------------  --------  ----------
helloworld-2559879000-XXXXXX  myResourceGroup    Succeeded            microsoft/aci-helloworld  52.179.XXX.XXX:80  1.0 core/1.5 gb  Linux     eastus

Uninstallation

For manual installation, you can remove the virtual node by deleting the Helm deployment. Run the following command:

helm uninstall $ChartName

If it is an AKS managed virtual node, please follow the steps here to remove the add-on.