I know what your saying “This blog is called Azure for All, so why a Kubernetes article?”, well stay with me on this – all will become clear!
I've recently been conducting more and more technical reviews of the Azure Kubernetes Service (AKS), and one thing that has become clear is that due to AKS being a managed service, some knowledge of how Kubernetes hangs together is not always truly understood. So, I decided to write an article on how to create a K8s (Yep I’m not writing Kubernetes anymore) cluster on Hyper-V for testing and exam prep reasons.
Now, I know you can run Minikube or even Docker with K8s enabled, but let’s be honest here, its not a true production like environment is it? so with this post I am going to guide you through creating your own production like cluster. It may not have all the bells and whistles (we are only going to have one Master node at this point), but its going to give you a better insight into how a cluster should look.
I am also going to give you a few useful links below that I have used before, and still use as reference for AKS and native K8S.
For those of you who have looked at AKS and done your homework, you will know its nodes actually run Ubuntu 16.04 LTS not 126.96.36.199 LTS as we are going to build here, and we are going to install K8s version 1.17.1 again the default version for AKS, 1.13.12. However, remember we are not trying to replicate AKS here, just get a similar environment so we can understand the components.
These are a few of the links I tend to use on a regular basis:
So lets get started on the techie stuff!
In this environment we are going to run 1 K8s Master node and 2 Worker nodes, which will give us a basic cluster.
The nodes will run on a CIDR block of 10.10.10.0/24 and our Pod Network CNI (which we will use Calico for this deployment) will run on 10.244.0.0/16. This will ensure no conflicts between the range I run in my own environment.
Lastly as mentioned above we will be using the following software versions:
For the Hyper-V networking I am using an Internal virtual switch called "K8s-Switch". This is to isolate the cluster as I am running on a laptop.
I also want them to reside on a different CIDR, namely 10.10.10.0/24, but I also need them to have internet access. So for this setup we create a new NAT to allow them to connect to my default connected adaptor using PowerShell.
I'm running this on my Windows 10 device using Hyper-V, but if you have a good understanding of Virtual Machines you can change this for your desired solution (OSX with VMWare Fusion or Virtualbox for example).
The nodes for Hyper-V need to be setup as virtual machines like this:
*(Note that for Ubuntu 18.04.3 LTS to work as a Gen 2 VM, you need to disable Secure Boot in the Virtual Machine Settings)
Virtual Machine #1
Virtual Machine #2
Virtual Machine #3
Operating System Configuration
Run the following Docker and K8s Installs on both Master and Worker nodes
The Docker Install
Ok so now we have our three VMs running, we can start installing the components. Firstly lets update the package list and get Docker installed as our container runtime.
We run the following command to update the package list.
Now we install Docker.
Now we can check the version of Docker that was installed by running.
This will return the details below. Note that for this example we are installing the latest version, but I will show you how to be specific in a later step.
Lastly we enable Docker to start automatically after reboot.
The Kubernetes Install
Ok so now we have our three VMs running, we can start installing K8s. I recommend you connect to each of the servers over SSH using your chosen terminal, for me I tend to stick with Putty so I can save my sessions, but its up to you. You will need to run as root account or use sudo or sudo -i for the commands.
The first thing you need to do once connected is disable the Swap file, K8s will not install if this is running.
Disable the swap file
Now we have disabled the swap file, but we need to ensure it will not be enabled after a restart, so we need to disable this in /etc/fstab by running the following command and adding a # to the line starting with swap.
The next step is to to add both the google repositories and K8s repositories to the machine, to do this we run the following commands.
Now we install kubeadm, kubelet and kubectl.
Lastly we mark the three components on hold to block any auto updating.
Run these commands on the Master node only
Master Node Configuration
Now we need to initialize the control-plane on the Master node using the command below.
Note that in this example we are going to use Calico as our Pod Network CNI Addon, which means we need to pass an argument during the init. You can find out more about the different network addons here kubernetes.io/docs/concepts/cluster-administration/addons/ . Calico is a very simple addon to configure, however I would also recommended looking at Weave Net which has the ability create a mesh overlay between the nodes in the cluster. A great comparison of both of these plus some other CNI providers can be found on Rancher.com https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/ .
Once the control-plane has initiated you should receive a message similar to that below.
You ideally need to copy the kubeadm join command to notepad so that you can use it to join the Worker nodes. However, if you miss this step I will show you how this information can be gathered again in a later step.
Now as the message states, we need to run the following commands.
Once this is completed we need to deploy a pod network. Now, as I mentioned we are using Calico and we have passed the chosen CIDR to the kubeadm init.
By default Calico uses a CIDR 0f 192.168.0.0/16, which in most cases will clash on a home setup, so what we have to do is download the .yaml files and apply them using kubectl.
Firstly we download them.
Then we edit the file to modify the default CIDR.
Scroll through the yaml and find the area relaiting to the default IPV4 pool and make the ammendment to the CIDR block.
Save the file and then apply the calico.yaml manifest using the following command.
Now, in order to check our cluster we can run a couple of commands. The first is the "kubectl get pods" command but with the "-o wide" arguments to allow us to see some more information.
This will return the pods and their current state. Note that they will show "Pending" for a few minutes while the network initiates. Note that the "coredns-xxx" services should have IP addresses in our CIDR.
The second command is the "kubectl get nodes" command, again using the "-o wide" arguments we are able to list the details.
This will return all the nodes and their current state in the cluster
Now if you forgot to save the join details to your notepad, don't panic! As promised here is how you can regnerate them.
Firstly your need to retrieve the token using the following command.
This should show you the token similar to below. Note the token expires after 24 hours, so if this is the case you can recreate it from the information on this link https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
Lastly we need to retrieve the discovery-token-ca-cert-hash using the following command.
This should return the hash which we can combine using the template:
kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>
Run this on each Worker node to join them to the K8s cluster
In order to join the Worker nodes we need to run the kubeadm join command on each one. Remeber to run this with sudo.
Once completed, a successful message will look like this,
Run this on both the nodes, then on the Master run the following command again.
The result should now show all our nodes are joined to the cluster and running as the desired roles.
We can also re-run the following command to see how our services replicated across our nodes in the cluster.
The result will look like this.
So if all has gone according to plan, we should now have a fully functional K8s cluster. For those of you who noticed the jump in IP address allocation from the Master node to the Worker node, this is deliberate. By leaving a gap we can neatly slot in a second Master node to allow for high availability, which I plan to cover in the next part of this blog along with deploying components to the environment and configuring services.