Highly Available Kubernetes Setup Without Cloud Provider Lock-in


Anphy Jose

19 Apr 2020

Kubernetes is one of the trending and fastest adopted tools in the DevOps process due to its features like scalability, high availability, network management, and auto-healing. It can group ‘n’ number of containers into one logical unit for managing and deploying them easily. It works brilliantly with all kinds of infrastructure i.e. cloud, hybrid and on-premise.


This blog is intended for users who are willing to create a highly available cluster for stateful applications, on bare metal. This is certainly a challenging case for teams who are trying to avoid cloud provider lock-in.


HashedIn had one such opportunity to set up a HA K8s cluster for stateful application and deploying the microservice-based application on it. It was for one of the telecom giants in India and thus high availability and scale was a very important consideration.


K8s HA Rules


Infrastructure requirement


Since many of the users use managed K8s services like EKS, AKS, PKS, etc., which provide HA by default, it is easy to miss the High Availability aspect of a K8s cluster while setting it up on bare metal. Based on the above constraints and HA aspects on bare metal, here are few infrastructure pieces that we need to have to design HA K8s cluster –



Architecture Design

* IPs are representational. You may use any CIDR.


Installations Required


These steps are relevant to Ubuntu and must be replaced with an equivalent for each OS –


  1. Setup Bastion host
  2. Setup K8s clusters
  3. Setup HAProxy servers
  4. Setup Helm and Tiller

Installation steps for Kubernetes HA cluster :


Kubespray is used to set up the K8s master and cluster. Kubespray uses Ansible internally to connect to all nodes and complete the configurations. To know more on the same please read the following link https://github.com/kubernetes-sigs/kubespray


  1. Disable swap-on all the K8s servers and check the connectivity between all the servers using the following command: swapoff -a
  2. Enable IP forwarding on all servers: echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
  3. Install dependencies from ”requirements.txt”: sudo pip install -r requirements.txt
  4. Copy “inventory/sample” as “inventory/mycluster” : cp -rfp inventory/sample inventory/mycluster
  5. Update Ansible inventory file with inventory builder :
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo touch /etc/apt/sources.list.d/kubernetes.list
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a  
sudo apt-get update
sudo apt-get install -y kubectl

Review and change parameters under “inventory/mycluster/group_vars” :

cat inventory/mycluster/group_vars/all/all.yml 
## We can add the load balancer details over here.
cat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml  
## change the networking and other data as per your requirements.

Deploy the script as the root user:

 ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml

Installation of HAProxy:

HAProxy will be installed as a package.


  1. apt update
  2. apt install haproxy</code



Change the necessary configuration, according to your needs. Test the connection by accessing the URL of haproxy IP with the opened port.


Installing Workstation:


Here are the requirements for setting up a Bastion host/workstation. We will use Ansible for server configurations of all servers –


  1. Install Ansible version >=2.7.2 on a workstation. A workstation should have SSH access to all K8s servers over port 22
  2. Python3 should be installed on all servers including workstation
  3. For further help on Ansible installation please refer – https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible-on-ubuntu
  4. Copy the /etc/kubernetes/admin.conf from one of the master servers to  ~/.kube/config file present in the workstation server.
  5. Install kubectl on the workstation server

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo touch /etc/apt/sources.list.d/kubernetes.list
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a  
sudo apt-get update
sudo apt-get install -y kubectl

Now you can run the kubectl commands from here for the desired requirements.


Installation of Helm and Tiller:


Helm is a package manager and it has two parts to it, the client(CLI) and the server(Tiller). The client lives on your local workstation and the server lives on the Kubernetes cluster to execute what is needed. The idea is that you use the CLI to push the resources you need and the tiller will make sure that state is, in fact, the case by creating/updating/deleting resources from the chart.


The following command downloads and runs the shell script for helm:

curl -L https://git.io/get_helm.sh | bash

Configure a service account for the tiller and install:

kubectl create serviceaccount \
--namespace kube-system \
kubectl create clusterrolebinding tiller-cluster-role \
--clusterrole=cluster-admin \
helm init --service-account tiller
helm version

Storage requirements – Dynamic Provisioning for stateful application:

The application required a dynamic allocation of DB and other storage. We used NFS for shared volume across servers. Here are steps you should use to set up dynamic provisioning of storage if your application is stateful.


Create one storage class for dynamic provisioning of DB on NFS:

kind: StorageClass
apiVersion: storage.k8.io/v1
  name: managed-nfs-storage
provisioner: example.com/nfs
  archiveOnDelete: false


Apply above manifest file and run this command to verify:

kubectl get storageclass/sc


For dynamic provisioning, we can execute the below command, which will install the NFS provisioner with the help of helm chart:

helm install --set nfs.server=x.x.x.x --set nfs.path=/exported/path stable/nfs-client-provisioner


If you’re going for an on-premise setup of Kubernetes, this guide should help you give a high-level picture that often is missed when going for managed K8s services. Few important takeaways and things to explore in further details –

  1. Provisioning dynamic storage with NFS.
  2. Using a bastion host instead of a master node to control your K8s cluster by copying admin.conf file. You may also read about K8s RBAC to tighten up the security while using a bastion host.
  3. No single point of failure is an essential part of any self-managed setup.

Have a question?

Need Technology advice?


+1 669 253 9011


linkedIn youtube