Kubernetes Cluster on Raspberry Pi zero using Cluster Hat

Once I've read this blog(3日間クッキング【Kubernetes のラズペリーパイ包み “サイバーエージェント風”】), I wanted to deploy k8s cluster on Raspberry Pi.
However, It's too expensive for me in this blog's way. Need three Raspberry Pi 3 Model B and switching hub ... etc.
So I've created k8s cluster Raspberry Pi Zero, but It was very hard for me than I thought.
I'll introduce the creation of k8s cluster on raspberry pi zero in this time.
And this is totally useless because of using the older version, and not for production. In addition unstable behavior and very slowly.
I made only for my pleasure. If you're interested in, make it.


Component List

Prepare the following components.
We need at least one Raspberry Pi Zero

  • Cluster Hat v2.3 x 1 (23.33£)
  • Raspberry Pi B+ x 1 ($35)
  • Raspberry Pi Zero x 1-4 (3.88£/per)
  • Micro SD Card x 2-5
  • Power Cable x 1
  • Lan Cable or USB Wifi Adaptor

※You can buy Cluster Hat and Raspberry Pi in Pimoroni(https://shop.pimoroni.com/)


Component Role

Hardware Role of ClusterHat Role of K8S
Raspberry Pi B+ Controller Control-Plane
Raspberry Pi Zero x (1 - 4) Node Node
ClusterHat - -

Raspberry Pi Zero( or Zero W)
This is my environment(actually, I used 3 raspberry pi zero)

1. Create Raspberry Pi Zero Image for ClusterHat and k8s

1-1. Create old raspbian image

Create Raspberry Pi Zero image from 2017-11-29-raspbian-stretch-lite because latest raspbian image can unavailable cpuset.
execute the following command in Controller Node of ClusterHat

# wget http://ftp.jaist.ac.jp/pub/raspberrypi/raspbian_lite/images/raspbian_lite-2017-12-01/2017-11-29-raspbian-stretch-lite.zip
# unzip 2017-11-29-raspbian-stretch-lite.zip
# apt install kpartx
# git clone https://github.com/burtyb/clusterhat-image
# cd clusterhat-image/build
# mkdir {img,mnt,dest,mnt2}
# mv 2017-11-29-raspbian-stretch-lite.img img/
# create.sh 2017-11-29

if it is well, the Raspbian image should be in dest directory.


1-2. Write the image to micro sd

Download this Raspbian image to local.
Write CNAT.img to Controller Node(Raspberry Pi B+), others to the node(Raspberry Pi Zero).
Create ssh file at boot directory for connecting ssh each node.
If you're on a Mac, execute the following command after writing.

# touch /Volumes/boot/ssh

1-3. Start Controller Node

Confirm connecting to Controller Node
From local machine.

# ssh pi@[controller ip_addr]

From Controller Node

# ssh pi@[]

Password is raspberry

2. Setting Node of ClusterHat

2-1. Install library to Controller Node

Install python-smbus because lacked library for clusterhat command. Execute the following command.

# apt-get install python-smbus

After install, execute clusterhat command.

# clusterhat status

2-2. Setting Nat

For Node connect to the internet, Setting Nat.
Execute the following command

# apt-get -y install iptables-persistent
# iptables -t nat -A POSTROUTING -s ! -o brint -j MASQUERADE
# iptables -A FORWARD -i brint ! -o brint -j ACCEPT
# iptables -A FORWARD -o brint -m conntrack --ctstate  RELATED,ESTABLISHED -j ACCEPT
# sh -c "iptables-save > /etc/iptables/rules.v4"

3. Setting All Node of ClusterHat

3-1. Hold kernel version

# apt-mark hold raspberrypi-kernel

3-2. upgrade

# apt-get upgrade -s

Check it has not changed kernel version

# uname -a
Linux p1 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux

3-3. Install Docker

Install with fixed version(18.06.1~ce~3-0~raspbian)

# curl -sSL https://get.docker.com | sh
# usermod -aG docker pi
# apt-get install docker-ce=18.06.1~ce~3-0~raspbian

If docker command not well, reinstall docker.

# apt-get remove docker-ce
# apt-get install docker-ce=18.06.1~ce~3-0~raspbian

3-4. Hold Docker version

# apt-mark hold docker-ce

3-5. Install Kubernetes

Install kubernetes with below version.

# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - 
# echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
# apt-get update
# apt-cache madison kubeadm
# apt-get install kubelet=1.5.6-00 kubeadm=1.5.6-00 kubectl=1.5.6-00 kubernetes-cni=0.5.1-00

4. Setting the Control-Plane Node(Controller Node of ClusterHat)

4-1. Initialize a Kubernetes Control-Plane node

Execute the following command on control-plane

# kubeadm init --pod-network-cidr
Probably it should stop at the following log (in 5-10minitus)
<master/apiclient> created API client, waiting for the control plane to become ready
Check the logs,you should see the following logs
# journalctl -eu kubelet
"error: failed to run Kubelet: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt"

※because creating ca(ca.pem) file name and refference ca(ca.crt) file name are different

Solution of this trouble

Change file name as follows

cd /etc/kubernetes/pki
cp ca.pem ca.crt

After change file name, waiting for a while.
if it well, memo the following such a output.

# kubeadm join --token=18b644.0438585f9e20e864

4-2. Install flannel

Create file as follows

apiVersion: v1
kind: ServiceAccount
  name: flannel
  namespace: kube-system
kind: ConfigMap
apiVersion: v1
  name: kube-flannel-cfg
  namespace: kube-system
    tier: node
    app: flannel
  cni-conf.json: |
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
  net-conf.json: |
      "Network": "",
      "Backend": {
        "Type": "vxlan"
apiVersion: extensions/v1beta1
kind: DaemonSet
  name: kube-flannel-ds
  namespace: kube-system
    tier: node
    app: flannel
        tier: node
        app: flannel
      hostNetwork: true
        beta.kubernetes.io/arch: arm
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.7.1-arm
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
          privileged: true
        - name: POD_NAME
              fieldPath: metadata.name
        - name: POD_NAMESPACE
              fieldPath: metadata.namespace
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: quay.io/coreos/flannel:v0.7.1-arm
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
        - name: run
            path: /run
        - name: cni
            path: /etc/cni/net.d
        - name: flannel-cfg
            name: kube-flannel-cfg

Execute the following command.

#kubectl create --validate=false -f kube-flannel.yml

※ it takes a long time(about 30〜40minutes)

5. Setting the Kubernetes Node(Node of ClusterHat)

5-1. Copy ca.crt from Control-Plane

Download ca.crt from Control-Plane, and Upload the file to node.

scp pi@[control-plane ip_addr]:/etc/kubernetes/pki/ca.crt .
scp pi@[node ip_addr]:ca.crt /etc/kubernetes/pki/ca.crt

because when executed join command, not created ca.crt(probably bug)
if you execute join command without this 5-1, you should face the following trouble.

after executing join command, the following output.

Node joins complete:

but node not joined.

Check the log.

# journalctl -eu kubelet
"error: failed to run Kubelet: unable to load client CA file /etc/kubernetes/pki/ca.crt"

5-2. Join to Control-Plane

Execute the following command(at 4-1's command) each node.

# kubeadm join --token=18b644.0438585f9e20e864

6. Check if the installation was successful
If you can see the following output, you got success.

# kubectl get nodes
NAME      STATUS         AGE
cnat      Ready,master   2h
p1        Ready          1m
p2        Ready          9m
p3        Ready          19m

and look this command output

# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
kube-system   dummy-2501624643-8jrkj            1/1       Running   0          2h   cnat
kube-system   etcd-cnat                         1/1       Running   0          2h   cnat
kube-system   kube-apiserver-cnat               1/1       Running   0          2h   cnat
kube-system   kube-controller-manager-cnat      1/1       Running   0          2h   cnat
kube-system   kube-discovery-2202902116-pwwaf   1/1       Running   0          2h   cnat
kube-system   kube-dns-2334855451-aq9pj         3/3       Running   1          2h      cnat
kube-system   kube-flannel-ds-5tz4s             2/2       Running   2          22m    p3
kube-system   kube-flannel-ds-k5vsc             2/2       Running   0          3m    p1
kube-system   kube-flannel-ds-kasde             2/2       Running   0          1h   cnat
kube-system   kube-flannel-ds-uumv1             2/2       Running   2          11m    p2
kube-system   kube-proxy-2ku07                  1/1       Running   0          2h   cnat
kube-system   kube-proxy-d9xxo                  1/1       Running   0          22m    p3
kube-system   kube-proxy-dz146                  1/1       Running   0          11m    p2
kube-system   kube-proxy-gvcs1                  1/1       Running   0          3m    p1
kube-system   kube-scheduler-cnat               1/1       Running   0          2h   cnat

What kind of trouble occurred

1. Unavailable cpuset on latest raspbian image (1-1)

created old raspbian image with being able to use cpuset.

2. Can't initialize kubernetes on kubeadm(4-1)

kubeadm created ca(ca.pem) file, also kubeadm reference ca(ca.crt) file, but both file name is not same.
So Renamed the ca.pem file to the ca.crt.

3. Can't deploy the latest flannel(4-2)

The latest flannel can unavailable in this version of kubeadm.
I used an old flannel version.

4. Node can't join to Control-Plane(5-1)

When execute join command, node can't join to Control-Node.
The cause of this problem is the node doesn't have ca.crt. Probably should be created automatically when joining.
This can be solved by copying the ca.crt from Control-Plane to Node