myakscluster

Here are blog articles related to this repository:

• setup a performant aks cluster
• private aks and private acr, safer you are
• my own custom and private azure pipelines agent as a docker container
• flexible kured deployment with its helm chart
• kubernetes network policies, how to secure the communications between your pods

To properly setup and secure your AKS cluster, there is a couple of features and components to enable in order to properly setup your Security Posture, here is the list:

Basic - without cost:

• Managed Identity #62
• Azure Lock #21
• kured #13
• Disable K8S Dashboard #24
• Network Policy with Calico #9
• AAD #10
• Azure Policy for AKS #11
• Azure KeyVault for Azure pipelines #3

Advanced - with cost:

• Uptime SLA #66
• Private AKS cluster + Private ACR + Jumpbox with Azure Private Endpoint #57
  • Alternatively: Use Authorized IP addresses with AKS and Azure Service Endpoint with ACR.
• Limit Egress Traffic with Azure Firewall #16

Complementary Azure services to leverage:

• Azure Advisor for AKS
• Azure Security Center for ACR
• Azure Security Center for AKS

Provisioning

Prerequisites

• Install Azure CLI
• Install Terraform
  • Optional if you would like to leverage Terraform instead of Azure CLI.
• Install Azure ConnectedK8s and KubernetesConfiguration CLI extensions
  • Optional if you would like to use Azure Arc enabled Kubernetes.

Configuration pre-provisioning

az login
#az account list -o table
#az account set -s <subscriptionId>

export LOCATION='canadacentral'
export NODES_COUNT=3
export NODE_SIZE='Standard_DS2_v2'
export ZONES=false

./run-pre-checks.sh

randomSuffix=$(shuf -i 1000-9999 -n 1)
export AKS=FIXME$randomSuffix
export K8S_VERSION=$(az aks get-versions \
  -l $LOCATION \
  --query "orchestrators[?isPreview==null].orchestratorVersion | [-1]" \
  -o tsv)

Provisioning Option 1: Azure CLI

cd cli
./create-aks-cluster.sh

FYI, current issues/workarounds with Azure CLI:

• az aks create|nodepool add --labels - labels ignored if prefix contains '.'
  • Need to manually apply the kubernetes.azure.com/mode=user k8s label on each user node.

Provisioning Option 2: Terraform

cd tf
sudo terraform init
terraform plan \
  -var aks_name=$AKS \
  -var k8s_version=$K8S_VERSION \
  -var location=$LOCATION \
  -var aks_node_count=$NODES_COUNT \
  -var aks_node_size=$NODE_SIZE
terraform apply \
  -auto-approve \
  -var aks_name=$AKS \
  -var k8s_version=$K8S_VERSION \
  -var location=$LOCATION \
  -var aks_node_count=$NODES_COUNT \
  -var aks_node_size=$NODE_SIZE

Configuration post-provisioning

You need to connect to the Jumpbox VM via the Bastion host and run the commands below:

az login \
  --service-principal \
  -u FIXME \
  -p FIXME \
  --tenant-id FIXME
az aks get-credentials \
  -g $RG \
  -n $RG
./configure-aks-cluster.sh

Pricing estimation

• AKS
  • $0.10 per cluster per hour.
• Virtual Machines related to AKS worker nodes
  • Estimation: For 2 Node pools X 3 Standard_DS2_v2 Ubuntu worker nodes, ~ $xx USD/month
  • To decrease the cost: you may want to leverage Azure Reserved VMs Instances to reduce the cost of your AKS worker nodes
• Managed Disks (related to the AKS's Nodes OS Disk)
  • Estimation: ~ $xx USD/month
• Log Analytics Workspace (related to the Azure Monitor for Containers add-on)
  • Estimation: For xx GB/month, ~ $xx USD/month
  • To decrease the cost: you may want to leverage logs filtering, retention by data type or New capacity-based pricing option.
• Standard Load Balancer
  • Estimation: TODO
  • To decrease the cost: you may want to leverage Basic Load Balancer instead which is free but you won't be able to use Availability Zones nor VNET Peering features
• Azure Firewall
  • Estimation: TODO
• Public IP
  • Related to the SLB + FW
  • Estimation: TODO
• Bandwidth (related to Availability Zones)
  • Estimation: TODO
  • Note: you may want to leverage PodAffinity to avoid cross-zones communications with Pods too chatty
• Private Endpoint
  • Estimation: For 2 Private Endpoints (AKS + ACR)
• Bastion - TODO

Other considerations:

• Regulatory compliance
• Security hardening in AKS virtual machine hosts
• Regularly update to the latest version of Kubernetes
• Audit logging in AKS
• Azure AD PIM
• Azure DDOS Protection
• Azure Front Door - TODO

Resources

• Camp Cloud Native, Best Practices in Securing Your Kubernetes Applications
• Attack matrix for Kubernetes
• Detect large-scale cryptocurrency mining attack against Kubernetes clusters
• Trusted Cloud: security, privacy, compliance, resiliency, and IP
• Microsoft Cloud Adoption Framework for Azure
• Azure webinar series - Help Deliver Applications Securely with DevSecOps
• Enterprise security in the era of containers and Kubernetes
• Azure Kubernetes Services: Container Security for a Cloud Native World
• 11 Ways (Not) to Get Hacked
• Tutorial: Bullet-Proof Kubernetes: Learn by Hacking - Luke Bond & Ana-Maria Calin
• Tutorial: Building Security into Kubernetes Deployment Pipelines - Michael Hough & Sam Irvine
• How Spotify Accidentally Deleted All its Kube Clusters with No User Impact
• Introduction to Kubernetes Security with GitOps
• Open Sourcing the Kubernetes Security Audit
• Kubernetes Security book
• https://aka.ms/aks/secure
• Exploiting and protecting containers with a few lines of scripting