I will guide you through the setup of a Kubernetes Cluster Mesh using OpenStack as hypervisor, RKE as K8S deployer, MetalLB as Load Balancer provider and Cilium as CNI. In this tutorial we will use a single network, but more complex network setup should work just fine as long as routing is properly in place.

Overall Architecture

Pre-requisites

• OpenStack
  • Image: Ubuntu bionic - 18.04 (Linux kernel >= 4.9 is a requirement from Cilium)
  • Flavor: m1.medium or bigger
  • Network: one or multiple networks; if multiple networks, they should be able to route traffic between each other, at least from the subnets, or IP range, allocated to the MetalLB. Network used for K8S nodes needs internet connectivity.
  • CLI access: to disable anti-spoofing on K8S VM’s port. We need to allow traffic from the LoadBalancer ExternalIP that MetalLB will setup (explanation)
• RKE v1.0.0 (but any version should work fine)
• Kubectl >= 1.9

Deploy the clusters - RKE

We need two three nodes cluster for this demo.

Create VMs

As we deploy in OpenStack, there are multiple ways to create the VMs.

We will use a HEAT template that will:

• create 3 VMs
• setup Docker using RKE script
• mount the BPF filesystem

Make sure to edit the name, the image, the flavor, the network and the key_name.

For the first cluster, we will use name k8s-cluster1-node-%index% and for the second cluster we will use k8s-cluster2-node-%index%

Here is the HEAT template:

Either through the UI, or using OpenStack CLI, create both HEAT stack:

$ openstack stack create -t k8s-cluster-1.yaml k8s-cluster-1
$ openstack stack create -t k8s-cluster-2.yaml k8s-cluster-2

You should get the following in Horizon:

Openstack Instances

Openstack Stacks

Deploy Kubernetes

Now that we have our 6 VMs provisioned with Docker we can deploy Kubernetes.

Attention

• you must not deploy any CNI, as we will deploy Cilium in following step.
• you will need to define distinct cluster_cidr for each cluster

All we need to do is to run RKE as follow; this will prompt you multiple questions to guide you through the cluster setup.

$ rke config

Once done, a cluster.yml file have been generated; you need to edit the file to set network plugin to none, and to set unique cluster_cidr. Then you can deploy the cluster

Make sure to have each cluster.yml file in seperated folder, as the process will write in the folder where the command is executed the file kube_config_cluster.yml which is the KubeConfig to access the cluster.

# Deploy the cluster
$ rke up

Here is the cluster.yml I used:

After a few minutes, you should have the kube_config_cluster.yml present in the folder, and the cluster available.

Let’s validate the nodes are up:

$ export KUBECONFIG=kube_config_cluster.yml
$ kubectl get nodes
NAME             STATUS     ROLES                      AGE     VERSION
10.195.197.168   NotReady   controlplane,etcd,worker   2m29s   v1.16.3
10.195.197.173   NotReady   controlplane,etcd,worker   2m29s   v1.16.3
10.195.197.177   NotReady   controlplane,etcd,worker   2m30s   v1.16.3

So the nodes are up, but marked as NotReady. This is expected, because we didn’t deploy any CNI, hence the DNS isn’t up and running:

$ kubectl get pods -n kube-system
NAME                                      READY   STATUS              RESTARTS   AGE
coredns-5c59fd465f-c95t6                  0/1     ContainerCreating   0          3m30s
coredns-autoscaler-d765c8497-9g47j        0/1     ContainerCreating   0          3m29s
metrics-server-64f6dffb84-np78x           0/1     ContainerCreating   0          3m25s
rke-coredns-addon-deploy-job-rnslc        0/1     Completed           0          3m33s
rke-ingress-controller-deploy-job-pwmz8   0/1     Completed           0          3m23s
rke-metrics-addon-deploy-job-k5shj        0/1     Completed           0          3m28s

$ kubectl describe pod coredns-5c59fd465f-c95t6 -n kube-system
--[cut]--
  Warning  NetworkNotReady  3m12s (x25 over 3m59s)  kubelet, 10.195.197.173  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

So let’s deploy Cilium CNI!

Deploy Cilium CNI

To do so, simply follow the installation documentation provided by Cilium:

Deploy MetalLB

To do. so, simply follow the installation documentation provided by MetalLB.

Configure MetalLB

We will use L2 configuration; you need a subnet or an IP range. See Layer 2 Configuration

$ echo 'apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.195.197.190-10.195.197.199' > metalld.yaml

$ kubectl apply -f metalld.yaml

Deploy the cluster mesh

To do so, follow the instruction from Cilium.

Attention

• At the step Expose the Cilium etcd to other clusters where you need to expose the Cilium etc to other cluster, this is where MetalLB will be of use; it will be the one provisioning the IP for the service.

  cilium-etcd-external.yml

  ``` apiVersion: v1 kind: Service metadata: name: cilium-etcd-external namespace: kube-system spec: ports: - port: 2379 selector: app: etcd etcd_cluster: cilium-etcd io.cilium/app: etcd-operator type: LoadBalancer ```

    $ kubectl apply -f cilium-etcd-external.yml
    $ kubectl get svc cilium-etcd-external
    NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
    cilium-etcd-external   LoadBalancer   10.45.157.32   10.195.197.190   2379:30363/TCP   31s
  

• Before the step Establish connections between clusters, you need to disable anti-spoofing on the VM ports in OpenStack, for the IPs of the cilium-etcd-external services.

  Get the list of port, you’ll need the ID

    $ openstack port list
  

  Then, for each port, add to the allowed-address list the external-ip of each cilium-etc-external service.

    $ openstack port set e9a533a9-e6be-4930-ba74-ba25b9fed059 --allowed-address ip-address=10.195.197.190
    $ openstack port set e9a533a9-e6be-4930-ba74-ba25b9fed059 --allowed-address ip-address=10.195.197.201
  

  You can then proceed by loading the clustermesh.yaml secret file and restart the cilium agent.

Validate cluster mesh

Cilium documentation provides an extensive list of things you can do to troubleshoot and validate the mesh is successful.

Here is the command to see whether the mesh is established

$ kubectl -n kube-system exec -ti cilium-vb6vv cilium node list
Name                      IPv4 Address     Endpoint CIDR   IPv6 Address   Endpoint CIDR
cluster1/10.195.197.162   10.195.197.162   10.42.0.0/24
cluster1/10.195.197.164   10.195.197.164   10.42.2.0/24
cluster1/10.195.197.167   10.195.197.167   10.42.1.0/24
cluster2/10.195.197.163   10.195.197.163   10.44.1.0/24
cluster2/10.195.197.166   10.195.197.166   10.44.0.0/24
cluster2/10.195.197.172   10.195.197.172   10.44.2.0/24

As you can see, Cilium agent from Cluster1 is able to see nodes from Cluster2. The way the cluster mesh works is by having each Cilium agent watching event on the cilium etcd of the other cluster; that way, when there is a change, it is propagate to all the clusters.