Managing multiple Kubernetes clusters

5 minute read

Working with multiple Kubernetes (K8s) clusters has become commonplace, thanks to the availability of managed clusters by every major cloud provider and advances in Infrastructure-as-Code. Production-ready clusters can now be deployed in minutes and so the clusters themselves are treated more like cattle and less like pets.
(Also, it sounds a lot less risky to have multiple small clusters for different purposes, than one big .. erm .. monolith. Sound familiar?)

At IDBS, we are currently running about 9 K8s clusters, 4 of which are production clusters, serving customers around the world.
This comes with its own set of challenges - In this article we will talk about a specific subset of them - How to keep track of all your clusters when working on the commandline.

You know the old problem

Jenkins just deployed your new code into Kubernetes, but somehow it doesn’t work. You stare at your console in mild anger

Last login: Wed Apr  1 09:21:31 on ttys015
reschenburg:~ $ 

You know the name of your helm deployment and you’re tired of your new code not working, so you just go feral on it

reschenburg:~ $ helm delete --purge my-helm-deployment
release "my-helm-deployment" deleted

Then the sinking feeling sets in.. hang on.. what was the last thing you did before this build? You were checking container logs on a production cluster for that weird customer issue.. oh no.. what kubeconfig are you using?? How do you check again? A quick google later..

reschenburg:~ $ kubectl cluster-info
Kubernetes master is running at https://api.production-cluster.acme.com
Heapster is running at https://api.production-cluster.acme.com/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://api.production-cluster.acme.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

And there you have it. You just deleted your deployment from a production cluster.

How do you know which K8s cluster your console is currently pointing at? (aka what is your active kubeconfig or kube context?)
How do you manage multiple kubeconfigs in the first place?
How do you switch between different clusters (kubeconfigs or contexts) painlessly via CLI?

Kubeconfig 101

Kubeconfigs (default ~/.kube/config) are files that hold information that allows you to gain access to particular K8s clusters. The main building blocks of that information are clusters, users and contexts.

To make kubectl use a particular kubeconfig, you can:

use --kubeconfig=<filename> (no merging, kubectl will only use this file)
specify one or more files in the KUBECONFIG environment variable (files will be merged)
put the file as ~/.kube/config

This is also the order of precedence if you have more than one of those options configured.

Cluster Context

The context concept in the kubeconfig is important. It is a named combination of cluster and user (and namespace):

- context:
    cluster: my-k8s-cluster
    namespace: kube-system
    user: my-k8s-cluster-admin-basic-auth
  name: demo-context
current-context: demo-context

This allows to keep information about several clusters and users in one kubeconfig file. You can then switch context to switch which cluster you want to connect to (or which user to use, to gain or lose permissions. For example:

reschenburg:~ $ kubectl config get-contexts
CURRENT   NAME             CLUSTER          AUTHINFO                         NAMESPACE
          k8s1-context     cluster-1        cluster-1-admin
*         demo-context     my-k8s-cluster   my-k8s-cluster-admin-basic-auth  kube-system

Here you can see that my current active context is the demo-context, with a basic-auth admin user and I’m operating in the kube-system namespace.

How to manage multiple kubeconfig files

yakc - abbreviation - yet another kube config

Chances are, your cluster is provisioned by some automated system and the kubeconfig is the output after a successful cluster deployment. You’ll want to use it yourself right away and you want to share it with colleagues who’ll also want to use yakc.

At this point, you could extract the cluster, user and context information from the new kubeconfig and update your ~/.kube/config with the new information. Alternatively, you can just run every kubectl command with --kubeconfig=new_kubeconfig (maybe use a different k alias for different clusters?).

But you may recall from earlier that the KUBECONFIG environment variable supports multiple files and merges them for us in memory, which sounds like it could be useful!

Thus a .bashrc function was born.

function kubeconfig() {
    local kubeconfigDir="$HOME/Code/kubeconfigs"
    unset KUBECONFIG
    for config in $(ls -d ${kubeconfigDir}/*); do
        export KUBECONFIG=$KUBECONFIG:$config
    done
}
kubeconfig 

put all your individual kubeconfigs into a directory that contains nothing else (e.g. $HOME/Code/kubeconfigs)
put the above function into your ~/.bashrc and source it (source ~/.bashrc)
???
confirm it works by running kubectl config view (aka Profit!)

When you receive a new kubeconfig file, just move it into your directory and run kubeconfig on the commandline.

Now you have access to all those clusters at your fingertips and all you have to do is switch cluster context:

reschenburg:~ $ kubectl config use-context k8s1-context
Switched to context "k8s1-context".
reschenburg:~ $ kubectl config current-context
k8s1-context
reschenburg:~ $

Which brings us to our next problem:

How do I know which context I’m in right now?

Of course there is a tool for that. How does this look for a command prompt?

reschenburg:~ (⎈ |demo-context:kube-system)$

I’m using the demo-context context and the kube-system namespace.

This is a tool called kube-ps1 and can be found here.

To enable the prompt display, you need to update your PS1 environment variable, ideally somewhere like your ~/.bash_profile:

source "/usr/local/opt/kube-ps1/share/kube-ps1.sh"
export PS1='\u:\W $(kube_ps1)\$ '

You then simply run kubeon to enable the prompt or kubeoff to disable it.

If you are working with kubernetes a lot from the CLI, I would recommend to add kubeon to your ~/.bashrc, so it is always enabled and can be turned off if you need to.

Quickly switching context on the CLI

You can use a tool called kubectx that can also be found on github and comes with another tool called kubens which allows for quick and easy switching of the context’s active namespace (thus, no longer requiring the -n my-namespace on each kubectl command).

On Mac, simply brew install kubectx, on windows, consider your bash for windows a linux environment and follow the linux install guide from github.

Both kubectx and kubens are greatly enhanced by fzf, enabling interactive menus (Windows users see here).

If you do, I’d recommend the following export in your ~/.bashrc:

export FZF_DEFAULT_OPTS="--layout=reverse --border --height 40%"

Some other useful shortcuts

When you work with K8s a lot from the CLI, I consider the following aliases essential:

alias k='kubectl'
alias ksn='kubens'
alias ktx='kubectx'

A note on minikube & docker desktop

Minikube and docker desktop will configure your ~/.kube/config file to enable context switching - aka kubectl config use-context minikube or ktx minikube.

Using the above mentioned method of configuring the KUBECONFIG environment variable, you will also have to copy your ~/.kube/config file into the directory with all the other kubeconfigs to get access to the minikube context.

Roland Eschenburg