Blog Projects
Kubernetes

Running a Kubernetes Cluster in Vagrant With Kubeadm

Introduction

This post explains how to run a multiple-node Kubernetes cluster using kubeadm running on Vagrant. It assumes that you have a basic understanding of Vagrant and Kubernetes.

The main obstacle to running a Kubernetes cluster on Vagrant is that Vagrant and VirtualBox do not give you sensible IP addresses by default. You must create additional Network Interfaces and instruct Kubernetes to use them.

Here is a repo containing a working example: https://gitlab.com/helm108/kubernetes-vagrant-example

Vagrant

The first step is to ensure that Vagrant assigns each box (master and workers) a unique IP address. I am using Vagrant with VirtualBox, which by default assigns all VMs it runs the same IP address - 10.0.2.15. Understandably this makes it difficult for your Kubernetes nodes to communicate with each other. While you can configure your VMs in VirtualBox directly, it’s better to configure the IP addresses in your Vagrantfile - this way your configuration is in your code and doesn’t need to be set up manually on every machine you launch your Kubernetes cluster on.

Vagrantfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
WORKER_COUNT = 1

Vagrant.configure(2) do |config|
  config.vm.box = "bento/ubuntu-16.04"

  config.vm.provider "virtualbox" do |v|
    v.memory = 1024
  end

  config.vm.define "master", primary:true do |master|
    master.vm.hostname = "master"
    master.vm.network "private_network", ip: "172.17.8.100"

    # Install Ansible.
    master.vm.provision "shell", path: "provision.sh"

    # Provision with Ansible.
    master.vm.provision "ansible" do |ansible|
      ansible.playbook = "playbooks/vagrant.yml"
      ansible.compatibility_mode = "2.0"
    end
  end

  (1..WORKER_COUNT).each do |i|
    config.vm.define "worker#{i}" do |worker|
      worker.vm.hostname = "worker#{i}"
      worker.vm.network "private_network", ip: "172.17.8.10#{i}"

      # Install Ansible.
      worker.vm.provision "shell", path: "provision.sh"

      # Provision with Ansible.
      worker.vm.provision "ansible" do |ansible|
        ansible.playbook = "playbooks/vagrant.yml"
        ansible.compatibility_mode = "2.0"
      end
    end
  end
end

This gives us a Master node with the IP 172.17.8.100, and a worker with the IP address 172.17.8.101. Each subsequent worker will increment that IP address (though if you asked for 10 workers the last one would have the IP address 172.17.8.1010 and that would cause some problems).

Kubernetes

Now that your Master and Worker are running inside VMs that have unique IP addresses, we need to tell Kubernetes about them. Kubernetes will use whichever is the default network interface unless you tell it otherwise, but you can configure the interfaces that it should listen for connections on.

So the two things we have to work with here are:”

  1. Telling the Master node which IP address it should listen for connections on
  2. Telling the Worker nodes the IP address of the Master node

Master Node

On the master, kubeadm init is run like so:

1
kubeadm init --apiserver-advertise-address {{ ansible_eth1.ipv4.address }} --apiserver-cert-extra-sans {{ ansible_eth1.ipv4.address }}

I am using Ansible to provision my Vagrant boxes but the key part is that you pass in the IP address of the network interface you defined manually in your Vagrantfile.

–apiserver-advertise-address

The IP address the API Server will advertise it’s listening on.

This tells the Kubernetes Master to listen for connections from the given IP address.

–apiserver-cert-extra-sans

Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate.

kubeadm defaults to the default network interface when registering the certificate, so you must pass in the one you actually want to use as an extra.

Worker Nodes

All that the worker nodes need to do now is be instructed to join the cluster. The master can generate the join command (kubeadm token create --print-join-command), and it will be generated using the eth1 IP address.

Why not use Minikube?

Minikube is a single-node Kubernetes implementation - your master node is also your worker node. This gives you a really simple way of testing your deployments against a Kubernetes install locally. If you want to learn how to bring up a multi-node Kubernetes cluster, it’s nice to be able to do that locally - it’s faster and cheaper than doing everything against a cloud solution.