Raspberry Pi: From 0 to k3s cluster in 5 min with k3sup and Ansible

Topic created ยท 1 Posts ยท 4 Views
  • This project is based on these two projects:


    Their code and explanations are better than mines ๐Ÿ˜‰
    You can find my code here.

    What is k3s?

    K3s is a lightweight implementation of Kubernetes' API. Small binary easy to install and deploy. That makes it ideal for using it on a Raspberry pi.

    Having fun and learning about Ansible, Kubernetes and K3s.



    4 x Raspberry Pi 4 Model B with 4GB RAM
    4 x Sandisk SDSQUAR-032G-GN6MA Ultra 32GB Micro SD
    1 x 4-Layer Acrylic Cluster Case With Cooling Fan For Raspberry Pi 4B/3B
    NETGEAR ProSafe 5-Port Gigabit Unmanaged Switch
    RAVPower 60W 12A 6-Port USB Charger Desktop Charging Station
    Fasgear USB Type C Cable (1ft 5 Pcs) Short USB C to USB A Cable Nylon Braided
    4 x Ethernet Cable Cat 6 30cm

    Configuration k3s cluster:

    1 master node
    3 worker nodes

    The first node is by far the most critical. We use the first node's wireless connection to provide internet access to the rest of the nodes (a local image cache would be useful) and use it as jump host in Ansible. Cluster creation is done with an untility called k3sup (said 'ketchup'). K3sup is a light-weight utility that allows you to create a k3s cluster easily and many other features.

    Download Raspbian Buster Lite from repo

    You have to repeat the following process with all SD cards.

    Insert SD card in your PC and umount it in case it was mounted automatically. (NOTE: mount points can be different in your system)

    umount device
    umount /dev/mmcblk0p1

    Write the image into the SD card:

    sudo dd bs=4M if=2019-09-26-raspbian-buster-lite.img of=/dev/mmcblk0 status=progress  conv=fsync
    536+0 records in
    536+0 records out
    2248146944 bytes (2.2 GB, 2.1 GiB) copied, 103.182 s, 21.8 MB/s

    Check the partitions:

    sudo fdisk -l /dev/mmcblk0
    Disk /dev/mmcblk0: 29.74 GiB, 31914983424 bytes, 62333952 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x6c586e13
    Device         Boot  Start     End Sectors  Size Id Type
    /dev/mmcblk0p1        8192  532479  524288  256M  c W95 FAT32 (LBA)
    /dev/mmcblk0p2      532480 4390911 3858432  1.9G 83 Linux

    We need to mount the SD card as we are provisioning some initial configuration:

    udisksctl mount -b /dev/mmcblk0p1
    Mounted /dev/mmcblk0p1 at /media/kaderno/boot.
    udisksctl mount -b /dev/mmcblk0p2
    Mounted /dev/mmcblk0p2 at /media/kaderno/rootfs.

    Enabling SSH, setting up WiFi connection and adding my ssh keys:

    touch /media/kaderno/boot/ssh
    sudo  mkdir -m 700 /media/kaderno/rootfs/home/pi/.ssh
    sudo cp ~/.ssh/id_rsa.pub /media/kaderno/rootfs/home/pi/.ssh/authorized_keys
    sudo chown -R 1000:1000 /media/kaderno/rootfs/home/pi/.ssh/

    Enable container features:

    We need to enable container features in the kernel, edit /boot/cmdline.txt and add the following to the end of the line:

    cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

    Configuring video memory on /boot/config.txt

    # Set the GPU memory split to 16mb

    On the master node you also need to set up the Wi-Fi connection:

    vi /media/kaderno/boot/wpa_supplicant.conf
    ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev

    Now umount both partitions and repeat until you have all the SDs ready.

    umount /media/kaderno/boot
    umount /media/kaderno/rootfs

    Before applying the playbooks, you have to install the change-hostname playbook by running the following:

    ansible-galaxy install mrlesmithjr.change-hostname

    Once you have all SD cards you can proceed to set up the cluster. Insert the SDs on the Raspberries and power on only the master node. From your local router you should be able to find out the ip address of the master node. You need to update the file inventory/hosts.inv and set the juphost ip address to the ip of your master node. From there, you can apply the ansible playbooks to create the cluster under 5 mins.

    Set up master node:

    ansible-playbook playbooks/master.yml

    When it finishes, start all the other nodes. Once they are up, we are going to update their host-names and and reboot all the nodes:

    ansible-playbook playbooks/change_hostname.yml

    Install and configure the k3s cluster:

    ansible-playbook playbooks/cluster.yml

    Once it finishes, from the master node, you should be able to run:

    pi@master:~ $ sudo kubectl get nodes
    worker-2   Ready    <none>   5m14s   v1.17.2+k3s1
    master     Ready    master   5m41s   v1.17.2+k3s1
    worker-3   Ready    <none>   4m33s   v1.17.2+k3s1
    worker-4   Ready    <none>   3m57s   v1.17.2+k3s1

    Now you need to configure your kubectl to access this cluster. The configuration is available at /etc/rancher/k3s/k3s.yaml in the master node. Add it to your ~/.kube/config file and don't forget to update the cluster ip address (same as the jumphost) or alternatively, get from this file the username and password and run the following commands:

    kubectl config set-credentials admin/k3s.local --username=your_username_most_likely_is_admin --password=the_password_from_k3s.yaml_file_on_master
    User "admin/k3s.local" set.
    kubectl config set-cluster k3s.local --insecure-skip-tls-verify=true --server=https://your_jumphost_ip_address
    Cluster "k3s.local" set.
    kubectl config set-context default/k3s.local/admin --user=admin/k3s.local --namespace=default --cluster=k3s.local
    Context "default/k3s.local/admin" created.
    kubectl config use-context default/k3s.local/admin
    Switched to context "default/k3s.local/admin".
    kubectl config set-cluster k3s.local --insecure-skip-tls-verify=true --server=https://your_jumphost_ip_address:6443
    Cluster "k3s.local" set.

    Now you can start using your fresh installed cluster from your local computer:

    kubectl get nodes
    worker-2   Ready    <none>   46h   v1.17.2+k3s1
    master     Ready    master   46h   v1.17.2+k3s1
    worker-3   Ready    <none>   46h   v1.17.2+k3s1
    worker-4   Ready    <none>   46h   v1.17.2+k3s1
    kubectl run --generator=run-pod/v1 nginx --image=nginx-slim-arm
    pod/nginx created
    kubectl get pods
    NAME                     READY   STATUS         RESTARTS   AGE
    nginx-85ff79dd56-bqlbr   1/1     Running        1          43h


    References: https://www.raspberrypi.org/documentation/installation/installing-images/linux.md https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ https://github.com/mrlesmithjr/ansible-rpi-k8s-cluster https://www.raspberrypi.org/documentation/configuration/wireless/headless.md

    Ansible: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html https://www.edureka.co/blog/ansible-tutorial/ https://raspberrypi.stackexchange.com/questions/13137/how-can-i-mount-a-raspberry-pi-linux-distro-image

Log in to reply