This is a modified version of Kodekloud's Kubeadm Cluster setup for VMWare Workstation rather than Oracle Virtualbox for improved performance and usability.
- Install Vagrant
- Install VMWare Workstation
- Install the Vagrant VMWare Desktop Utility
- Install the Vagrant VMWare Workstation Provider
Ref : https://kubernetes.io/docs/setup/production-environment/container-runtimes/
Verify that the br_netfilter module is loaded on each node by running lsmod | grep br_netfilter.
To load it explicitly, run sudo modprobe br_netfilter
In order for a Linux node's iptables to correctly view bridged traffic, verify that net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config. For example:
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# Apply sysctl params without reboot
sudo sysctl --systemRef: https://github.com/containerd/containerd/blob/main/docs/getting-started.md
Choosing to install via option 2 - from apt-get or dnf (https://docs.docker.com/engine/install/ubuntu/)
Remove any pre-existing versions of Docker
sudo apt-get remove docker docker-engine docker.io containerd runcPrereqs to allow for apt to use repos over HTTPS
sudo apt-get update
sudo apt-get install \
ca-certificates \
curl \
gnupg \
lsb-releaseAdd Docker's official GPG Key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpgSet up the repository
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/nullInstall Docker Engine (containerd included!)
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-pluginVerify installation
sudo service docker start
sudo docker run hello-worldAs a prerequisite, Swap MUST be disbled for Kubelet to work; run on all nodes required:
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstabUpdate the apt package index and install packages needed to use the Kubernetes apt repository:
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curlDownload the Google Cloud public signing key:
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpgAdd the Kubernetes apt repository:
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.listUpdate apt package index, install kubelet, kubeadm and kubectl, and pin their version:
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectlRef: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
Run kubeadm init <args>, supplying <args> as required based on the following steps
- If using a HA-setup, specify
--control-plane-endpointto set the shared endpoint for all control plane nodes. Not applicable for what's in this repo at present (1 control-plane, 1 worker) but worth noting. - Specify the Pod Network CIDR depending on the Pod Network Addon chosen. Done via
--pod-network-cidr. - Optional but may be required - Specify the endpoint of the container runtime via
--cri-socket-argumentref - Specify the network interface for the API server via
--apiserver-advertise=flag.
kubeadm init --pod-network-cidr=<cidr> --apiserver-advertise-address=<control plane IP> --cri-socket=unix:////run/containerd/containerd.sockNote: When using
containerd, I had to use a minor workaround to the toml config file to get kubeadm to work. I can't understand why this line would be uncommented by default, but I'm sure there was a valid reason that I'm unaware of! Be sure to runsystemctl daemon-reload && systemctl restart containerdafter the workaround.
Once run successfully, either:
-
As a regular user, run:
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
-
As root user:
export KUBECONFIG=/etc/kubernetes/admin.conf
Then deploy a pod network to the cluster - I personally go with either Calico or Flannel.
- Reference
- Make sure to edit the following custom resource definition file to match up with the pod network cidr set!
Join the worker nodes by running as root:
kubeadm join 192.169.190.2:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash>To determine the Virtual Network Settings in VMWare Workstation, navigate to Edit -> Virtual Network Preferences
During Kubeadm Init on the ControlPlane or Kubeadm Join in the worker nodes, Kubelet may fail to start. In my case, I found this to be an issue with the CGroup Driver used by Kubeadm and Docker conflicting. The following steps can be used to resolve this (reference StackOverFlow Thread):
-
Check the kubeadm environment variables defined in the config file referenced by kubelet.
systemctl status kubelet cat /var/lib/kubelet/kubeadm-flags.env
-
Check Docker's config info
sudo docker info | grep Cgroup -
If the Docker config requires alteration:
cat << EOF > /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } EOF
-
Update the Kubelet CGroup Driver Configuration
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
a. Add the following before ExecStart:
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=<cgroupfs or systemd>"b. Add
$KUBELET_CGROUP_ARGSto ExecStart in the file. -
Reboot the machine (manually or via vagrant) and run the following before running kubeadm init/join again:
kubeadm reset rm -rf ~/.kube/ -
Properly configure the Systemd CGroup Driver as it's weird: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd