Group Details Private


  • Let's say we have a Kubernetes cluster in Google Cloud and we've performed the deployment of a Mariadb cluster with 1 primary instances and 2 read replicas.

    After some time of activity, we notice that the Mariadb main pod keeps crashing and restarting and you wonder, what could it be?

    Review the logs

    The very first step of troubleshooting our pods it's to check the logs.
    To do so, identify first the pod name in our namespace:

    kubectl  get pods -n mariadb-1
     kubectl  get pods -n mariadb-1
    NAME                                  READY   STATUS      RESTARTS   AGE
    mariadb-1-deployer-1225393596-czs5w   0/1     Completed   0          13d
    mariadb-1-mariadb-0                   2/2     Running     0          96m
    mariadb-1-mariadb-secondary-0         1/1     Running     1          6d4h
    mariadb-1-mariadb-secondary-1         1/1     Running     1          13d

    Then, get the logs by using:

    kubectl logs mariadb-1-mariadb-0  -n mariadb-1
    2020-03-04  1:36:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2626073630
    2020-03-04  1:36:18 0 [Note] InnoDB: Last binlog file './mysql-bin.000057', position 718622
    2020-03-04  1:36:19 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
    2020-03-04  1:36:19 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
    2020-03-04  1:36:19 0 [Note] InnoDB: Creating shared tablespace for temporary tables
    2020-03-04  1:36:19 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
    2020-03-04  1:36:19 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
    2020-03-04  1:36:19 0 [Note] InnoDB: 10.3.22 started; log sequence number 2626073639; transaction id 73045993

    We can see clearly on the first line that our database is recovering from a crash. Nevertheless, this log won't give us more details.

    The next step would be to connect to our pod and try to get more information. To so:

    kubectl exec -it mariadb-1-mariadb-0  -n mariadb-1 /bin/bash

    Once inside the pod, we can perform some basic linux troubleshooting. We can try by using dmesg

    [1178727.643114] Memory cgroup out of memory: Kill process 1338764 (k8s_metadata) score 119 or sacrifice child
    [1178727.653058] Killed process 1338764 (k8s_metadata) total-vm:525388kB, anon-rss:23952kB, file-rss:2796kB, shmem-rss:0kB

    Interesting fact, we can see a process kill due to lack of memory

    Additionally we can check the namespace events and describe the pods to gather extra information

    kubectl describe pod/mariadb-1-mariadb-0 -n mariadb-1
    kubectl get events -n mariadb-1

    Review the metrics

    If we go to our cloud provider monitoring our defined tool and we look for our pod in the period of time when it crashed
    we can observe the Used Memory and the Requested

    alt text

    As per the official documentantion: Requests and limits are the mechanisms Kubernetes uses to control resources such as CPU and memory. Requests are what the container is guaranteed to get. If a container requests a resource, Kubernetes will only schedule it on a node that can give it that resource.

    Taking this in consideration, we need to guarantee that our Pod will have at least our currant usage plus any possible spike.

    If we check the official mariadb documentation or use the mysql calculator with the default parameters:, the minimum to run is about 576mb

    Update the requests

    Now that we have identified the problem, it's time to fix our Deployment or Statefulset.
    To do so, we will identify first the Statefulset or Deployment name:

    kubectl get statefulset -n mariadb-1
    kubectl get statefulset -n mariadb-1
    NAME                          READY   AGE
    mariadb-1-mariadb             1/1     13d
    mariadb-1-mariadb-secondary   2/2     13d

    Then, we can edit by typing:

    kubectl edit statefulset mariadb-1-mariadb -n mariadb-1

    Once in the edit mode, we need to locate the requests parameters:

                cpu: 100m
                memory: 100Mi

    We just need to adjust the values for the memory parameter:

                cpu: 100m
                memory: 800Mi

    Once we save the changes, please be aware that the current pods will be terminated and new pods will be deployed

    posted in Kubernetes
  • This is for test / non prod environments


    Even for a test environment, considering that both Elasticsearch and Kibana will consume memory anc cpu, we need at least 1 VM with 16g of ram and 4vCPU


    We need to install the following packages in our system:

    • openjdk-11-jre-headless
    • apt-transport-https
    • elasticsearch
    • kibana

    To do so we will execute the following commands as root:

    apt install openjdk-11-jre-headless
    apt-get install apt-transport-https

    Add the elasticsearch GPG key and repo

    wget -qO - | sudo apt-key add -
    add-apt-repository "deb stable main"

    Then, proceed to install Kibana and elasticsearch

    apt-get update
    apt-get install kibana
    apt-get install elasticsearch


    Prior to start the services, we will need to perform a initial configuration

    Edit the file /etc/elasticsearch/elasticsearch.yml and add the following parameters:

    Replace the cluster name, node name, publish host and initial mater nodes values by your host IP address mycluster elkserver
    node.master: true true
    action.auto_create_index: true
    http.port: 9200
    index.number_of_replicas: 0

    Then, edit the kibana config file /etc/kibana/kibana.yml and add the following parameters:
    (replace the elastic host IP for your IP)

    server.port: 5601 "" "elkserver"
    elasticsearch.hosts: [""]
    kibana.index: ".kibana"

    Start Services

    Then, start the elastic service by executing

    systemctl enable elasticsearch.service
    systemctl start elasticsearch.service

    Start also the kibana service:

    systemctl enable kibana.service
    systemctl start kibana.service


    From any other node, you can verify that Elasticsearch is accessible by querying its status by running:

    curl -X GET ""


    curl -X GET ""
      "name" : "elkserver",
      "cluster_name" : "mycluster",
      "cluster_uuid" : "q3aOcBEsQWOGfuT0_5bWqg",
      "version" : {
        "number" : "7.6.0",
        "build_flavor" : "default",
        "build_type" : "deb",
        "build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
        "build_date" : "2020-02-06T00:09:00.449973Z",
        "build_snapshot" : false,
        "lucene_version" : "8.4.0",
        "minimum_wire_compatibility_version" : "6.8.0",
        "minimum_index_compatibility_version" : "6.0.0-beta1"
      "tagline" : "You Know, for Search"

    You can also verify the node:

    curl -XGET ''
    curl -XGET ''
    ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name           45          37   1    0.00    0.01     0.04 dilm      *      elkserver

    Then, you can create your first index

    curl -XPUT ""

    And verify its creation

    curl -XGET ''


    curl -XGET ''
    health status index                    uuid                   pri rep docs.count docs.deleted store.size
    green  open   .kibana_task_manager_1   jHpC_e8mRzq5A8JdwHVu_g   1   0          2            0     43.4kb         43.4kb
    green  open   .apm-agent-configuration yoVyzzG5S56Q05Mqxghaog   1   0          0            0       283b           283b
    green  open   .kibana_1                2oE1URIITm6KNwqYgEd9ng   1   0         16            0     36.7kb         36.7kb
    green  open   myfirstindex             0bNB5LNMT6m31rHeTy_usQ   1   1          0            0       230b           230b

    Finally, access the Kibana dashboard by typing in your browser your host url on port 5601

    alt text

    posted in Linux General
  • This project is based on these two projects:

    Their code and explanations are better than mines 😉
    You can find my code here.

    What is k3s?

    K3s is a lightweight implementation of Kubernetes' API. Small binary easy to install and deploy. That makes it ideal for using it on a Raspberry pi.

    Having fun and learning about Ansible, Kubernetes and K3s.



    4 x Raspberry Pi 4 Model B with 4GB RAM
    4 x Sandisk SDSQUAR-032G-GN6MA Ultra 32GB Micro SD
    1 x 4-Layer Acrylic Cluster Case With Cooling Fan For Raspberry Pi 4B/3B
    NETGEAR ProSafe 5-Port Gigabit Unmanaged Switch
    RAVPower 60W 12A 6-Port USB Charger Desktop Charging Station
    Fasgear USB Type C Cable (1ft 5 Pcs) Short USB C to USB A Cable Nylon Braided
    4 x Ethernet Cable Cat 6 30cm

    Configuration k3s cluster:

    1 master node
    3 worker nodes

    The first node is by far the most critical. We use the first node's wireless connection to provide internet access to the rest of the nodes (a local image cache would be useful) and use it as jump host in Ansible. Cluster creation is done with an untility called k3sup (said 'ketchup'). K3sup is a light-weight utility that allows you to create a k3s cluster easily and many other features.

    Download Raspbian Buster Lite from repo

    You have to repeat the following process with all SD cards.

    Insert SD card in your PC and umount it in case it was mounted automatically. (NOTE: mount points can be different in your system)

    umount device
    umount /dev/mmcblk0p1

    Write the image into the SD card:

    sudo dd bs=4M if=2019-09-26-raspbian-buster-lite.img of=/dev/mmcblk0 status=progress  conv=fsync
    536+0 records in
    536+0 records out
    2248146944 bytes (2.2 GB, 2.1 GiB) copied, 103.182 s, 21.8 MB/s

    Check the partitions:

    sudo fdisk -l /dev/mmcblk0
    Disk /dev/mmcblk0: 29.74 GiB, 31914983424 bytes, 62333952 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x6c586e13
    Device         Boot  Start     End Sectors  Size Id Type
    /dev/mmcblk0p1        8192  532479  524288  256M  c W95 FAT32 (LBA)
    /dev/mmcblk0p2      532480 4390911 3858432  1.9G 83 Linux

    We need to mount the SD card as we are provisioning some initial configuration:

    udisksctl mount -b /dev/mmcblk0p1
    Mounted /dev/mmcblk0p1 at /media/kaderno/boot.
    udisksctl mount -b /dev/mmcblk0p2
    Mounted /dev/mmcblk0p2 at /media/kaderno/rootfs.

    Enabling SSH, setting up WiFi connection and adding my ssh keys:

    touch /media/kaderno/boot/ssh
    sudo  mkdir -m 700 /media/kaderno/rootfs/home/pi/.ssh
    sudo cp ~/.ssh/ /media/kaderno/rootfs/home/pi/.ssh/authorized_keys
    sudo chown -R 1000:1000 /media/kaderno/rootfs/home/pi/.ssh/

    Enable container features:

    We need to enable container features in the kernel, edit /boot/cmdline.txt and add the following to the end of the line:

    cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

    Configuring video memory on /boot/config.txt

    # Set the GPU memory split to 16mb

    On the master node you also need to set up the Wi-Fi connection:

    vi /media/kaderno/boot/wpa_supplicant.conf
    ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev

    Now umount both partitions and repeat until you have all the SDs ready.

    umount /media/kaderno/boot
    umount /media/kaderno/rootfs

    Before applying the playbooks, you have to install the change-hostname playbook by running the following:

    ansible-galaxy install mrlesmithjr.change-hostname

    Once you have all SD cards you can proceed to set up the cluster. Insert the SDs on the Raspberries and power on only the master node. From your local router you should be able to find out the ip address of the master node. You need to update the file inventory/hosts.inv and set the juphost ip address to the ip of your master node. From there, you can apply the ansible playbooks to create the cluster under 5 mins.

    Set up master node:

    ansible-playbook playbooks/master.yml

    When it finishes, start all the other nodes. Once they are up, we are going to update their host-names and and reboot all the nodes:

    ansible-playbook playbooks/change_hostname.yml

    Install and configure the k3s cluster:

    ansible-playbook playbooks/cluster.yml

    Once it finishes, from the master node, you should be able to run:

    pi@master:~ $ sudo kubectl get nodes
    worker-2   Ready    <none>   5m14s   v1.17.2+k3s1
    master     Ready    master   5m41s   v1.17.2+k3s1
    worker-3   Ready    <none>   4m33s   v1.17.2+k3s1
    worker-4   Ready    <none>   3m57s   v1.17.2+k3s1

    Now you need to configure your kubectl to access this cluster. The configuration is available at /etc/rancher/k3s/k3s.yaml in the master node. Add it to your ~/.kube/config file and don't forget to update the cluster ip address (same as the jumphost) or alternatively, get from this file the username and password and run the following commands:

    kubectl config set-credentials admin/k3s.local --username=your_username_most_likely_is_admin --password=the_password_from_k3s.yaml_file_on_master
    User "admin/k3s.local" set.
    kubectl config set-cluster k3s.local --insecure-skip-tls-verify=true --server=https://your_jumphost_ip_address
    Cluster "k3s.local" set.
    kubectl config set-context default/k3s.local/admin --user=admin/k3s.local --namespace=default --cluster=k3s.local
    Context "default/k3s.local/admin" created.
    kubectl config use-context default/k3s.local/admin
    Switched to context "default/k3s.local/admin".
    kubectl config set-cluster k3s.local --insecure-skip-tls-verify=true --server=https://your_jumphost_ip_address:6443
    Cluster "k3s.local" set.

    Now you can start using your fresh installed cluster from your local computer:

    kubectl get nodes
    worker-2   Ready    <none>   46h   v1.17.2+k3s1
    master     Ready    master   46h   v1.17.2+k3s1
    worker-3   Ready    <none>   46h   v1.17.2+k3s1
    worker-4   Ready    <none>   46h   v1.17.2+k3s1
    kubectl run --generator=run-pod/v1 nginx --image=nginx-slim-arm
    pod/nginx created
    kubectl get pods
    NAME                     READY   STATUS         RESTARTS   AGE
    nginx-85ff79dd56-bqlbr   1/1     Running        1          43h




    posted in Kubernetes
  • The following documentation has been tested on a lab / Dev Environment. For a PROD usage, please check the official documentation

    What is tekton

    Tekton is Kubernetes-native pipeline resource
    The Tekton Pipelines project provides Kubernetes-style resources for declaring CI/CD-style pipelines.

    The main objective of Tekton Pipelines is to run your Task individually or as a part of a Pipeline. Every Task runs as a Pod on your Kubernetes cluster with each step as its own container.

    Source of information

    We've performed our deployment based on the following documentation:

    Deploying tekton

    Assuming we a have a local openshift 4.2 or kubernetes cluster, first we will need to create a PV and a PVC for the storage. You can use the following yaml file as an example.

    Create 2 files named pv.yaml and pvc.taml with the following content:

    kind: PersistentVolume
    apiVersion: v1
      name: config-logging
        type: local
        storage: 40Gi
        - ReadWriteMany
        path: "/mnt/glusterfs/config-logging"
    kind: PersistentVolumeClaim
    apiVersion: v1
      name: config-logging
      namespace: tekton-pipelines
        - ReadWriteMany
          storage: 40Gi
      volumeName: config-logging
      volumeMode: Filesystem

    Then, create the volumes by typing

    oc create namespace tekton-pipelines
    oc create -f pv.yaml
    oc create -f pvc.yaml

    As per the document, download the official release yaml file

    Before applying the file, we need to modify the following 2 iterations in order to use our local storage

          - configMap:
              name: config-logging
            name: config-logging


          - name: config-logging
              claimName: config-logging

    then, perform the deployment by executing:

    oc apply -f release.yaml

    And then verify that the pods are up and running by using:

     oc get pods --namespace tekton-pipelines
    NAME                                          READY   STATUS    RESTARTS   AGE
    tekton-pipelines-controller-8b78bdf57-qmrlx   1/1     Running   0          22m
    tekton-pipelines-webhook-66987c8b96-d9vsq     1/1     Running   0          22m

    Once up and running, we can also download the tekton dashboard from its git repo

    There's no need to edit the yaml file here, so we can apply it straight forward:

    oc apply -f dashboard_latest_release.yaml

    Verify again your pods, and create an ingress (in kubernetes) or a route in openshift and access the dashboard

    alt text

    Create service account

    As a final step, to ensure that the pipelines that we create have the appropriate permissions to store images and perform the deployments, we need to create a Service account:

    oc create serviceaccount pipeline
    oc adm policy add-scc-to-user privileged -z pipeline
     oc adm policy add-role-to-user edit -z pipeline
    posted in Pipelines
  • This document is for dev/test environments. Do not use on a prod environment

    What is glusterfs

    Gluster is a scalable network filesystem. Using common off-the-shelf hardware, you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks. Gluster is free.

    Why user glusterfs?

    Usually, when you run your infrastructure on a cloud provider, the storage is automatically provisioned. Nevertheless, when running your own installation in a VM or bare metal, you won't have any kind of storage.

    By using glusterfs, you have a reliable and fast system to create your volumes in a easy manner


    The process it's quite simple. As per the official documentation, which states:

    Please use the Gluster Community PPAs for the latest version of GlusterFS:

    Let's say we have 3 kubernetes nodes. Our master node, and node1 and node2.
    The following commands should be executed in all the nodes

    The steps that we need to perform are:

    sudo add-apt-repository ppa:gluster/glusterfs-7
    sudo apt-get update
    apt install glusterfs-server
    sudo systemctl start glusterd
    sudo systemctl enable glusterd

    Adding nodes

    Ensure that our 3 hosts have the corresponding entry on the /etc/hosts

    From the master node, we need to perform the following actions:

    gluster peer probe node1
    gluster peer probe node2

    Then, we need to verify that our nodes have been added by running:

    gluster peer status

    which will return:

    Number of Peers: 2
    Hostname: node1
    Uuid: 76a57408-5f50-420f-927a-4cfc6843f1d4
    State: Peer in Cluster (Connected)
    Hostname: node2
    Uuid: 9e76c3a8-9a48-4a07-a6de-d3a9c3659a43
    State: Peer in Cluster (Connected)
    root@master:/home/istio/istio-1.4.3# gluster pool list
    UUID                                    Hostname        State
    76a57408-5f50-420f-927a-4cfc6843f1d4    node1           Connected
    9e76c3a8-9a48-4a07-a6de-d3a9c3659a43    node2           Connected
    df901df3-f164-45c7-afaf-92efc5964f10    localhost       Connected

    Setup Distributed GlusterFS Volume

    Now that we have the nodes, it's time to add the actual shared storage.
    We need to add an additional disk to our instances (50 / 100 gb)

    Then, scan for new disks with fdisk -l and create the vg / lv as per below on each node:

    vgcreate glustervg /dev/sdb
    lvcreate -L +51G -n glusterlv /dev/glustervg
    mkfs.xfs /dev/glustervg/glusterlv
    mkdir /gluster

    Then, get the blkid for the new lv and add it to /etc/fstab

    blkid /dev/glustervg/glusterlv
    vi /etc/fstab
    UUID=99f57d21-8844-44b9-b2e3-85fc80ce5886 /gluster xfs defaults 0 0
    mount /gluster

    From the master node, we will create the replica by executing:

    gluster volume create vol01 replica 3 transport tcp master:/glusterfs node1:/gluster node2:/gluster force
    gluster volume start vol01

    And validate by checking the volume information:

    gluster volume info vol01
    Volume Name: vol01
    Type: Replicate
    Volume ID: 63f13452-27f6-44a9-a24f-66ec0a4df33e
    Status: Started
    Snapshot Count: 0
    Number of Bricks: 1 x 3 = 3
    Transport-type: tcp
    Brick1: master:/glusterfs
    Brick2: node1:/gluster
    Brick3: node2:/gluster
    Options Reconfigured:
    transport.address-family: inet
    storage.fips-mode-rchecksum: on
    nfs.disable: on
    performance.client-io-threads: off

    Mount our gluster volume

    On each node we will create a new mount point

    mkdir -p /mnt/glusterfs

    For redundancy, we will be mounting the different servers on each node. As an example, on the master node, we mount node2, on node1 we mount master and in node2 we mount node1

    Edit /etc/fstab and add:


    node2:/vol01 /mnt/glusterfs glusterfs defaults,_netdev 0 0


    master:/vol01 /mnt/glusterfs glusterfs defaults,_netdev 0 0


    node1:/vol01 /mnt/glusterfs glusterfs defaults,_netdev 0 0

    Then, on each node, mount the volume by doing a mount -a

    Create a kuberntes PV

    Now that we have our shared storage, we can create a Persistent Volume.
    To do so, in our master node we will run the following command


    Then, create a yaml file named storage.yaml with the following content

    kind: PersistentVolume
    apiVersion: v1
      name: test
        type: local
        storage: 5Gi
        - ReadWriteMany
        path: "/mnt/glusterfs/test"

    Once done, create the pv with:

    kubectl apply -f storage.yml

    And validate with:

    kubectl get pv
    test   5Gi        RWX            Retain           Available                                   4s

    Create the PVC

    One we have our volume create, we need a Persistent volume claim (or pvc) in order to allow our pods to use it.
    To do so, we will create a file named pvc.yaml with the following content:

    apiVersion: v1
    kind: PersistentVolumeClaim
      name: test
      namespace: default
      - ReadWriteMany
          storage: 5Gi
      volumeName: test

    Then, create it by using:

    kubectl create -f pvc.yaml

    Finally, validate its creation:

    kubectl get pvc
    test   Bound    test     5Gi        RWX                           115s
    posted in Kubernetes
  • This is a exercise for testing purposes


    In order to create our Kubernetes test cluster, we will need 3 nodes (1 master and 2 slaves) with specs as the following ones.
    You can go down to 4gb of RAM for this POC, but we recommend having at least 8gb

    • 2 vCPUs
    • 8 GB RAM
    • Ubuntu 18.04 on each node

    You can find really cheap cloud providers to run this kind of test. For example offers a similar instance for 20$ per month per instance

    Requirements pre-installation

    First, we will need to generate an ssh key-pair on each node

    root@node1:~# ssh-keygen
    root@node2:~# ssh-keygen
    root@node3:~# ssh-keygen

    You will need to set a password for the root user, as by default it doesn't have one (run on each node)

    sudo passwd
    [sudo] password for xxxx: 
    Enter new UNIX password: 
    Retype new UNIX password: 
    passwd: password updated successfully

    Once done and before moving forward, we need to disable the following entries from our /etc/hosts file on each node

    #::1    ip6-localhost   ip6-loopback
    #       virtualserver01

    get your node ips and add them to /etc/hosts on each node node1 node2 node3

    Then copy the keys to each node (this needs to be done from each node)

    ssh-copy-id -i /root/.ssh/ root@node1
    ssh-copy-id -i /root/.ssh/ root@node2
    ssh-copy-id -i /root/.ssh/ root@node3

    Update the packages on each node

    apt-get update
    sudo apt-get install \
        apt-transport-https \
        ca-certificates \
        curl \
        software-properties-common -y

    Install docker

    We need to install Docker on each node

    curl -fsSL | sudo apt-key add 
        sudo add-apt-repository \
       "deb [arch=amd64] \
       $(lsb_release -cs) \
     sudo apt-get update
     apt-get install docker-ce -y

    Install Kubernetes

    We need to add the Kubernetes signing key on each node
    To do so. run the following command:

    curl -s | sudo apt-key add

    Also, we need to add the Xenial Kubernetes Repository on each node
    Run the following command to do so and then update the package list:

    sudo apt-add-repository "deb kubernetes-xenial main"
    apt-get update

    The next step in the installation process is to install Kubeadm on all the nodes through the following command:

    sudo apt install kubeadm -y

    Once the installation finishes, you can check the version by running

    kubeadm version

    Before moving on, we need to disable the SWAP memory in all the nodes
    You need to disable swap memory on both the nodes as Kubernetes does not perform properly on a system that is using swap memory.

    Run the following command to do so:

    sudo swapoff -a

    Then, only in our master node (node1), initialize kubelet by typing the following command

    kubeadm init --pod-network-cidr=

    Once the process finishes, take note of the output as it's really important.
    We will be using later this information to join the worker nodes to our cluster.

    Don't execute it yet

    Your Kubernetes control-plane has initialized successfully!
    To start using your cluster, you need to run the following as a regular user:
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
    Then you can join any number of worker nodes by running the following on each as root:
    kubeadm join --token ahlqlw.bk3iqojsp519rf7d \
        --discovery-token-ca-cert-hash sha256:ebf7c6b895b52a059872d198a44b942267bef89d8d1d6802bbd3cc8082ebe600

    Run the following commands on the master node

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

    Then, on the master node download the calico network manifest using the following command:

    curl -O

    edit the file calico.yaml and modify as per below (defining the CIDR block that we configured during the kube installation)

            - name: CALICO_IPV4POOL_CIDR
              value: ""


            - name: CALICO_IPV4POOL_CIDR
              value: ""

    The section will look like:

    alt text

    then apply the manifest by running

    kubectl apply -f calico.yaml

    After a few minutes, you can query the status of the pods by using:

    kubectl get pods --all-namespaces

    alt text

    And also verify that your master node is now ready by using:

    kubectl get nodes

    alt text

    Joining worker nodes to the cluster

    Now you can go to node2 and node 3 and run the command generated during the installtion to join the worker nodes to our cluster

    kubeadm join --token ahlqlw.bk3iqojsp519rf7d \
        --discovery-token-ca-cert-hash sha256:ebf7c6b895b52a059872d198a44b942267bef89d8d1d6802bbd3cc8082ebe600

    after a couple of minutes you can check the nodes status:

    kubectl get nodes

    alt text

    And get additional node info

    kubectl get nodes -o wide

    alt text

    Access the cluster from your workstation

    In order to access from our laptop or a remote machine, we need to create a cluster role and cluster role binding.
    In the master node create a yaml file witht the content:

    We are creating a role with full admin permissions. In a PROD environment we should follow the best practices and limit the permissions.

    kind: ClusterRole
      annotations: "true"
      name: remote
    - apiGroups:
      - '*'
      - '*'
      - '*'
    - nonResourceURLs:
      - '*'
      - '*'

    then create the cluster role binding by executing the command from the master node

    kubectl create clusterrolebinding remote   --clusterrole=remote  --user=remote   --group=system:serviceaccounts

    get the cluster token by running

    kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | awk '/^deployment-controller-token-/{print $1}') | awk '$1=="token:"{print $2}'

    you will get a long token string such as the following output.
    take note of it


    Then, from your workstation, open a shell or powershell and perform the following commands

    kubectl config set-cluster kubernetes --server=https://your_node1_public_ip:6443 --insecure-skip-tls-verify=true
     kubectl config set-context kubernetes --cluster=kubernetes
     kubectl config set-credentials remote --token=your_token
     kubectl config set-context kubernetes --user=remote
     kubectl config use-context kubernetes

    you can then run the usual commands to check your cluster info

    kubectl get nodes
    kuectl get cluster-info

    Deploy a stateless app

    In order to test our cluster, we can deploy a stateless example app based on the official kubernetes doc, create a yaml file with the following content in your workstation:

    apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
    kind: Deployment
      name: nginx-deployment
          app: nginx
      replicas: 2 # tells deployment to run 2 pods matching the template
            app: nginx
          - name: nginx
            image: nginx:1.7.9
            - containerPort: 80

    Then, perform

    kubectl apply -f my-file.yaml

    And then verify if your deployment is running and where

    kubectl.exe get pods --all-namespaces -o wide
    NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
    default       nginx-deployment-54f57cf6bf-n7nfc          1/1     Running   0          2m11s     node2   <none>           <none>
    default       nginx-deployment-54f57cf6bf-wrtsd          1/1     Running   0          2m11s     node3   <none>           <none>

    Access your app

    Now, you can access your nginx server by doing a kubernetes port-forward.
    Replace the pod name by the value returned by the previous command

    kubectl port-forward nginx-deployment-54f57cf6bf-n7nfc 8080:80
    Forwarding from -> 80
    Forwarding from [::1]:8080 -> 80
    Handling connection for 8080
    Handling connection for 8080

    Then, open a web browser to http://localhost:8080

    Using octant

    As you know, you can deploy and use the kubernetes dashboard.
    Nevertheless, we recommend you to take a look to octant

    Octant is a tool for developers to understand how applications run on a Kubernetes cluster. It aims to be part of the developer's toolkit for gaining insight and approaching complexity found in Kubernetes.

    The benefit is that it´s installed in your workstation and the access is based in your permissions, meaning that no additional deployments are needed in the cluster.
    You can download the last release here

    Once installed, just execute it by running the following command, and a tab in your web browser will launch the console.
    You can check the deployment and pods that we just triggered before.


    alt text

    posted in Kubernetes
  • Register to our blog

    We've just started with this blog, but if you would like to comment, collaborate or whatsoever, please feel free to register and let us know what you think.

    Thank you!

    posted in Announcements
  • Purpose

    It's possible that due to some requirement in our environment, we might need to run a Docker image which has Docker running inside of it.

    An example would be the need of creating a custom Jenkins slave image with specific libraries or packages for our builds in Kubernetes.

    If there are no needs for specific libraries or packages, we recommend to use Kaniko

    In this build we are including:

    • Python 3.6
    • wget
    • unzip
    • curl

    Build Process

    We need an environment with Docker installed (could be a linux server, our workstation or laptop).
    Then, we need to create a Dockerfile with the following content:

    The following dockerfile is an example with several dependencies.

    FROM centos:7
    RUN yum -y install curl systemd yum-utils
    ENV container docker
    RUN rm -f /lib/systemd/system/* && rm -f /etc/systemd/system/*.wants/* && rm -f /lib/systemd/system/* && \
        rm -f /lib/systemd/system/*udev* && rm -f /lib/systemd/system/*initctl* && rm -f /lib/systemd/system/* && \
        rm -f /lib/systemd/system/*
    RUN yum-config-manager --add-repo && \
        yum clean all && rm -Rf /var/cache/yum/
    RUN yum install -y
    RUN yum install docker-ce initscripts epel-release openssh openssh-server openssh-clients openssl-libs wget unzip git python36 -y && \
        yum clean all; systemctl enable docker.service && systemctl enable sshd.service
    RUN sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config && rm -f /etc/nologin /run/nologin && mkdir /jenkins && chmod -R 777 /jenkins && \
        mkdir -p /etc/docker/ && chmod -R 777 /etc/docker/
    WORKDIR /jenkins
    COPY ./ /jenkins/
    COPY ./  /jenkins/
    COPY ./daemon.json /etc/docker/daemon.json
    EXPOSE 80
    EXPOSE 8080
    EXPOSE 50000
    CMD ["/jenkins/"]

    We need to create to additional files. The first file would be

    for (( ; ; ))
       echo "Pres CTRL+C to stop..."
       sleep 1

    And the second file would be

    setsid /jenkins/ > output.txt &
    exec /usr/sbin/init && service docker start 

    Don't forget to give execution permissions to our scripts:

    chmod a+x

    Finally, create a file named daemon.json
    As we will be running docker inside a container, we will be using the vfs storage driver

     "storage-driver": "vfs",
     "data-root": "/tmp"

    Then we can build our docker image with the command:

    docker build . -t mycustomimage

    Testing the image

    Now if you run the following command, you'll see your new build

    docker images
    REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
    mycustomimage       latest              f2a59057bc67        23 seconds ago      1.16GB

    Try and run it by typing:

    docker run -d --privileged=true -it mycustomimage

    Then check that it's running:

    docker ps
    CONTAINER ID        IMAGE               COMMAND               CREATED             STATUS              PORTS                         NAMES
    f8f27c1cb50e        mycustomimage       "/jenkins/"   13 seconds ago      Up 11 seconds       80/tcp, 8080/tcp, 50000/tcp   sad_goodall

    Then, connect to our running container (remember to change for your container id) by using:

     docker exec -it f8f27c1cb50e /bin/bash

    Once inside the container, try to run

    docker ps
    docker pull nginx

    Tweaking the image

    As you might have noticed, the image grew up more than 500mb. This is because we are installing several packages.
    You can remove the packages that you're not going to use and customize your docker builds inside the container properly.

    Build cleanup

    Once you've finished building your images and you've pushed them to your container registry, you can clean up all the images and cached information.


    If you run the following command it will delete all your images in the system.
    Execute it wisely

    docker system prune -a
    posted in Docker
  • Install let's encrypt certbot


    Let's Encrypt provides free SSL certificates which last 3 months since the time of its generation.

    In our example we will be using Centos 7

    Firts, we need to install the epel-release repo

    yum -y install epel-release 

    Then, install the following packages:

    yum -y install yum-utils certbot 

    Certbot will be the binary that we will be using for the certificate generation.

    There are a few ways to go on from now. The most common way is to install certbot on the server itself and let it automatically update our config files for apache or nginx. Nevertheless, even that it might take a few more minutes, we strongly recommend to generate the SSL certificate by creating a TXT DNS record.

    The reason for this is quite simple. We can generate our certificate anywhere and then copy the files to our corresponding server and ensure that our configuration is not automatically updated.

    Generate the certifciate

    certbot  -d **** --manual --preferred-challenges dns certonly

    This will generate the following output

    [root@myserver]#  certbot  -d --manual --preferred-challenges dns certonly
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator manual, Installer None
    Starting new HTTPS connection (1):
    Obtaining a new certificate
    Performing the following challenges:
    dns-01 challenge for
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    NOTE: The IP of this machine will be publicly logged as having requested this
    certificate. If you're running certbot in manual mode on a machine that is not
    your server, please ensure you're okay with that.
    Are you OK with your IP being logged?
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    (Y)es/(N)o: y
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Please deploy a DNS TXT record under the name with the following value:
    Before continuing, verify the record is deployed.
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Press Enter to Continue

    Now, what we need to to is generate the requested TXT record with the corresponding value. We recommend set the TTL of 1 min for our new record.
    Once done, wait 1/2 minutes and press enter

    This will generate the necessary files under /etc/letsencrypt/archive/

    [root@myserver]# ls -la /etc/letsencrypt/archive/
    total 20
    drwxr-xr-x.  2 root root   79 Jan 13 21:40 .
    drwx------. 11 root root 4096 Jan 13 21:40 ..
    -rw-r--r--.  1 root root 1915 Jan 13 21:40 cert1.pem
    -rw-r--r--.  1 root root 1647 Jan 13 21:40 chain1.pem
    -rw-r--r--.  1 root root 3562 Jan 13 21:40 fullchain1.pem
    -rw-------.  1 root root 1704 Jan 13 21:40 privkey1.pem

    You can copy then the fullchain1.pem and privkey1.pem to your corresponding web server
    In the example of an nginx server, the config file will look similar to:

        listen 443 ssl;
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
        ssl_certificate /etc/nginx/certs/fullchain1.pem;
        ssl_certificate_key /etc/nginx/certs/privkey1.pem;
        ssl_verify_client off;
        ssl_session_timeout 5m;
        ssl_protocols TLSv1.2;
        ssl_prefer_server_ciphers on;

    Finally, in your DNS provider, to not forget to create a CAA record.
    A Certification Authority Authorization (CAA) record is used to specify which certificate authorities (CAs) are allowed to issue certificates for a domain. The purpose of the CAA record is to allow domain owners to declare which certificate authorities are allowed to issue a certificate for a domain

    It should contain a record as in:

    128 issue ""
    posted in Web