Group Details Private

administrators

  • Let's say we have a Kubernetes cluster in Google Cloud and we've performed the deployment of a Mariadb cluster with 1 primary instances and 2 read replicas.

    After some time of activity, we notice that the Mariadb main pod keeps crashing and restarting and you wonder, what could it be?

    Review the logs

    The very first step of troubleshooting our pods it's to check the logs.
    To do so, identify first the pod name in our namespace:

    kubectl  get pods -n mariadb-1
    
     kubectl  get pods -n mariadb-1
    NAME                                  READY   STATUS      RESTARTS   AGE
    mariadb-1-deployer-1225393596-czs5w   0/1     Completed   0          13d
    mariadb-1-mariadb-0                   2/2     Running     0          96m
    mariadb-1-mariadb-secondary-0         1/1     Running     1          6d4h
    mariadb-1-mariadb-secondary-1         1/1     Running     1          13d
    

    Then, get the logs by using:

    kubectl logs mariadb-1-mariadb-0  -n mariadb-1
    
    2020-03-04  1:36:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2626073630
    2020-03-04  1:36:18 0 [Note] InnoDB: Last binlog file './mysql-bin.000057', position 718622
    2020-03-04  1:36:19 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
    2020-03-04  1:36:19 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
    2020-03-04  1:36:19 0 [Note] InnoDB: Creating shared tablespace for temporary tables
    2020-03-04  1:36:19 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
    2020-03-04  1:36:19 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
    2020-03-04  1:36:19 0 [Note] InnoDB: 10.3.22 started; log sequence number 2626073639; transaction id 73045993
    

    We can see clearly on the first line that our database is recovering from a crash. Nevertheless, this log won't give us more details.

    The next step would be to connect to our pod and try to get more information. To so:

    kubectl exec -it mariadb-1-mariadb-0  -n mariadb-1 /bin/bash
    

    Once inside the pod, we can perform some basic linux troubleshooting. We can try by using dmesg

    [1178727.643114] Memory cgroup out of memory: Kill process 1338764 (k8s_metadata) score 119 or sacrifice child
    [1178727.653058] Killed process 1338764 (k8s_metadata) total-vm:525388kB, anon-rss:23952kB, file-rss:2796kB, shmem-rss:0kB
    

    Interesting fact, we can see a process kill due to lack of memory

    Additionally we can check the namespace events and describe the pods to gather extra information

    kubectl describe pod/mariadb-1-mariadb-0 -n mariadb-1
    
    kubectl get events -n mariadb-1
    

    Review the metrics

    If we go to our cloud provider monitoring our defined tool and we look for our pod in the period of time when it crashed
    we can observe the Used Memory and the Requested

    alt text

    As per the official documentantion: Requests and limits are the mechanisms Kubernetes uses to control resources such as CPU and memory. Requests are what the container is guaranteed to get. If a container requests a resource, Kubernetes will only schedule it on a node that can give it that resource.

    Taking this in consideration, we need to guarantee that our Pod will have at least our currant usage plus any possible spike.

    If we check the official mariadb documentation or use the mysql calculator with the default parameters: https://www.mysqlcalculator.com/, the minimum to run is about 576mb

    Update the requests

    Now that we have identified the problem, it's time to fix our Deployment or Statefulset.
    To do so, we will identify first the Statefulset or Deployment name:

    kubectl get statefulset -n mariadb-1
    
    kubectl get statefulset -n mariadb-1
    NAME                          READY   AGE
    mariadb-1-mariadb             1/1     13d
    mariadb-1-mariadb-secondary   2/2     13d
    

    Then, we can edit by typing:

    kubectl edit statefulset mariadb-1-mariadb -n mariadb-1
    

    Once in the edit mode, we need to locate the requests parameters:

            resources:
              requests:
                cpu: 100m
                memory: 100Mi
    

    We just need to adjust the values for the memory parameter:

            resources:
              requests:
                cpu: 100m
                memory: 800Mi
    

    Once we save the changes, please be aware that the current pods will be terminated and new pods will be deployed

    posted in Kubernetes
  • This is for test / non prod environments

    Requirements

    Even for a test environment, considering that both Elasticsearch and Kibana will consume memory anc cpu, we need at least 1 VM with 16g of ram and 4vCPU

    Installation

    We need to install the following packages in our system:

    • openjdk-11-jre-headless
    • apt-transport-https
    • elasticsearch
    • kibana

    To do so we will execute the following commands as root:

    apt install openjdk-11-jre-headless
    apt-get install apt-transport-https
    
    

    Add the elasticsearch GPG key and repo

    wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
    add-apt-repository "deb https://artifacts.elastic.co/packages/7.x/apt stable main"
    

    Then, proceed to install Kibana and elasticsearch

    apt-get update
    apt-get install kibana
    apt-get install elasticsearch
    

    Configuration

    Prior to start the services, we will need to perform a initial configuration

    Edit the file /etc/elasticsearch/elasticsearch.yml and add the following parameters:

    Replace the cluster name, node name, publish host and initial mater nodes values by your host IP address

    cluster.name: mycluster
    node.name: elkserver
    network.host: 192.168.1.73
    network.publish_host: 192.168.1.73
    node.master: true
    node.data: true
    action.auto_create_index: true
    http.port: 9200
    cluster.initial_master_nodes: 192.168.1.73
    index.number_of_replicas: 0
    

    Then, edit the kibana config file /etc/kibana/kibana.yml and add the following parameters:
    (replace the elastic host IP for your IP)

    server.port: 5601
    server.host: "0.0.0.0"
    server.name: "elkserver"
    elasticsearch.hosts: ["http://192.168.1.73:9200"]
    kibana.index: ".kibana"
    

    Start Services

    Then, start the elastic service by executing

    systemctl enable elasticsearch.service
    systemctl start elasticsearch.service
    

    Start also the kibana service:

    systemctl enable kibana.service
    systemctl start kibana.service
    

    Validate

    From any other node, you can verify that Elasticsearch is accessible by querying its status by running:

    curl -X GET "http://192.168.1.73:9200/?pretty"
    

    Result:

    curl -X GET "http://192.168.1.73:9200/?pretty"
    {
      "name" : "elkserver",
      "cluster_name" : "mycluster",
      "cluster_uuid" : "q3aOcBEsQWOGfuT0_5bWqg",
      "version" : {
        "number" : "7.6.0",
        "build_flavor" : "default",
        "build_type" : "deb",
        "build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
        "build_date" : "2020-02-06T00:09:00.449973Z",
        "build_snapshot" : false,
        "lucene_version" : "8.4.0",
        "minimum_wire_compatibility_version" : "6.8.0",
        "minimum_index_compatibility_version" : "6.0.0-beta1"
      },
      "tagline" : "You Know, for Search"
    }
    

    You can also verify the node:

    curl -XGET 'http://192.168.1.73:9200/_cat/nodes?v&pretty'
    
    curl -XGET 'http://192.168.1.73:9200/_cat/nodes?v&pretty'
    ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
    192.168.1.73           45          37   1    0.00    0.01     0.04 dilm      *      elkserver
    

    Then, you can create your first index

    curl -XPUT "http://192.168.1.73:9200/myfirstindex"
    

    And verify its creation

    curl -XGET 'http://192.168.1.73:9200/_cat/indices?v&pretty'
    

    Result:

    curl -XGET 'http://192.168.1.73:9200/_cat/indices?v&pretty'
    health status index                    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    green  open   .kibana_task_manager_1   jHpC_e8mRzq5A8JdwHVu_g   1   0          2            0     43.4kb         43.4kb
    green  open   .apm-agent-configuration yoVyzzG5S56Q05Mqxghaog   1   0          0            0       283b           283b
    green  open   .kibana_1                2oE1URIITm6KNwqYgEd9ng   1   0         16            0     36.7kb         36.7kb
    green  open   myfirstindex             0bNB5LNMT6m31rHeTy_usQ   1   1          0            0       230b           230b
    

    Finally, access the Kibana dashboard by typing in your browser your host url on port 5601

    alt text

    posted in Linux General
  • This project is based on these two projects:

    https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/
    https://github.com/mrlesmithjr/ansible-rpi-k8s-cluster
    

    Their code and explanations are better than mines 😉
    You can find my code here.

    What is k3s?

    K3s is a lightweight implementation of Kubernetes' API. Small binary easy to install and deploy. That makes it ideal for using it on a Raspberry pi.
    Motivation:

    Having fun and learning about Ansible, Kubernetes and K3s.

    Hardware:

    cluster

    4 x Raspberry Pi 4 Model B with 4GB RAM
    4 x Sandisk SDSQUAR-032G-GN6MA Ultra 32GB Micro SD
    1 x 4-Layer Acrylic Cluster Case With Cooling Fan For Raspberry Pi 4B/3B
    NETGEAR ProSafe 5-Port Gigabit Unmanaged Switch
    RAVPower 60W 12A 6-Port USB Charger Desktop Charging Station
    Fasgear USB Type C Cable (1ft 5 Pcs) Short USB C to USB A Cable Nylon Braided
    4 x Ethernet Cable Cat 6 30cm
    

    Setup:
    Configuration k3s cluster:

    1 master node
    3 worker nodes
    

    The first node is by far the most critical. We use the first node's wireless connection to provide internet access to the rest of the nodes (a local image cache would be useful) and use it as jump host in Ansible. Cluster creation is done with an untility called k3sup (said 'ketchup'). K3sup is a light-weight utility that allows you to create a k3s cluster easily and many other features.

    Steps:
    Download Raspbian Buster Lite from repo

    You have to repeat the following process with all SD cards.

    Insert SD card in your PC and umount it in case it was mounted automatically. (NOTE: mount points can be different in your system)

    umount device
    umount /dev/mmcblk0p1
    

    Write the image into the SD card:

    sudo dd bs=4M if=2019-09-26-raspbian-buster-lite.img of=/dev/mmcblk0 status=progress  conv=fsync
    536+0 records in
    536+0 records out
    2248146944 bytes (2.2 GB, 2.1 GiB) copied, 103.182 s, 21.8 MB/s
    

    Check the partitions:

    sudo fdisk -l /dev/mmcblk0
    Disk /dev/mmcblk0: 29.74 GiB, 31914983424 bytes, 62333952 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x6c586e13
    
    Device         Boot  Start     End Sectors  Size Id Type
    /dev/mmcblk0p1        8192  532479  524288  256M  c W95 FAT32 (LBA)
    /dev/mmcblk0p2      532480 4390911 3858432  1.9G 83 Linux
    

    We need to mount the SD card as we are provisioning some initial configuration:

    udisksctl mount -b /dev/mmcblk0p1
    Mounted /dev/mmcblk0p1 at /media/kaderno/boot.
    udisksctl mount -b /dev/mmcblk0p2
    Mounted /dev/mmcblk0p2 at /media/kaderno/rootfs.
    

    Enabling SSH, setting up WiFi connection and adding my ssh keys:

    touch /media/kaderno/boot/ssh
    sudo  mkdir -m 700 /media/kaderno/rootfs/home/pi/.ssh
    sudo cp ~/.ssh/id_rsa.pub /media/kaderno/rootfs/home/pi/.ssh/authorized_keys
    sudo chown -R 1000:1000 /media/kaderno/rootfs/home/pi/.ssh/
    

    Enable container features:

    We need to enable container features in the kernel, edit /boot/cmdline.txt and add the following to the end of the line:

    cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
    

    Configuring video memory on /boot/config.txt

    # Set the GPU memory split to 16mb
    gpu_mem=16
    

    On the master node you also need to set up the Wi-Fi connection:

    vi /media/kaderno/boot/wpa_supplicant.conf
    
    wpa_supplicant.conf:
    
    country=US
    ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
    update_config=1
    
    network={
        ssid="your_real_wifi_ssid"
        scan_ssid=1
        psk="your_real_password"
        key_mgmt=WPA-PSK
    }
    

    Now umount both partitions and repeat until you have all the SDs ready.

    umount /media/kaderno/boot
    umount /media/kaderno/rootfs
    

    Before applying the playbooks, you have to install the change-hostname playbook by running the following:

    ansible-galaxy install mrlesmithjr.change-hostname
    

    Once you have all SD cards you can proceed to set up the cluster. Insert the SDs on the Raspberries and power on only the master node. From your local router you should be able to find out the ip address of the master node. You need to update the file inventory/hosts.inv and set the juphost ip address to the ip of your master node. From there, you can apply the ansible playbooks to create the cluster under 5 mins.

    Set up master node:

    ansible-playbook playbooks/master.yml
    

    When it finishes, start all the other nodes. Once they are up, we are going to update their host-names and and reboot all the nodes:

    ansible-playbook playbooks/change_hostname.yml
    

    Install and configure the k3s cluster:

    ansible-playbook playbooks/cluster.yml
    

    Once it finishes, from the master node, you should be able to run:

    pi@master:~ $ sudo kubectl get nodes
    NAME       STATUS   ROLES    AGE     VERSION
    worker-2   Ready    <none>   5m14s   v1.17.2+k3s1
    master     Ready    master   5m41s   v1.17.2+k3s1
    worker-3   Ready    <none>   4m33s   v1.17.2+k3s1
    worker-4   Ready    <none>   3m57s   v1.17.2+k3s1
    

    Now you need to configure your kubectl to access this cluster. The configuration is available at /etc/rancher/k3s/k3s.yaml in the master node. Add it to your ~/.kube/config file and don't forget to update the cluster ip address (same as the jumphost) or alternatively, get from this file the username and password and run the following commands:

    kubectl config set-credentials admin/k3s.local --username=your_username_most_likely_is_admin --password=the_password_from_k3s.yaml_file_on_master
    User "admin/k3s.local" set.
    kubectl config set-cluster k3s.local --insecure-skip-tls-verify=true --server=https://your_jumphost_ip_address
    Cluster "k3s.local" set.
    kubectl config set-context default/k3s.local/admin --user=admin/k3s.local --namespace=default --cluster=k3s.local
    Context "default/k3s.local/admin" created.
    kubectl config use-context default/k3s.local/admin
    Switched to context "default/k3s.local/admin".
    kubectl config set-cluster k3s.local --insecure-skip-tls-verify=true --server=https://your_jumphost_ip_address:6443
    Cluster "k3s.local" set.
    

    Now you can start using your fresh installed cluster from your local computer:

    kubectl get nodes
    NAME       STATUS   ROLES    AGE   VERSION
    worker-2   Ready    <none>   46h   v1.17.2+k3s1
    master     Ready    master   46h   v1.17.2+k3s1
    worker-3   Ready    <none>   46h   v1.17.2+k3s1
    worker-4   Ready    <none>   46h   v1.17.2+k3s1
    
    kubectl run --generator=run-pod/v1 nginx --image=nginx-slim-arm
    pod/nginx created
    kubectl get pods
    NAME                     READY   STATUS         RESTARTS   AGE
    nginx-85ff79dd56-bqlbr   1/1     Running        1          43h
    

    TO BE CONTINUED...

    References: https://www.raspberrypi.org/documentation/installation/installing-images/linux.md https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ https://github.com/mrlesmithjr/ansible-rpi-k8s-cluster https://www.raspberrypi.org/documentation/configuration/wireless/headless.md

    Ansible: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html https://www.edureka.co/blog/ansible-tutorial/ https://raspberrypi.stackexchange.com/questions/13137/how-can-i-mount-a-raspberry-pi-linux-distro-image

    posted in Kubernetes
  • The following documentation has been tested on a lab / Dev Environment. For a PROD usage, please check the official documentation

    What is tekton

    Tekton is Kubernetes-native pipeline resource
    The Tekton Pipelines project provides Kubernetes-style resources for declaring CI/CD-style pipelines.

    The main objective of Tekton Pipelines is to run your Task individually or as a part of a Pipeline. Every Task runs as a Pod on your Kubernetes cluster with each step as its own container.

    Source of information

    We've performed our deployment based on the following documentation:

    Deploying tekton

    Assuming we a have a local openshift 4.2 or kubernetes cluster, first we will need to create a PV and a PVC for the storage. You can use the following yaml file as an example.

    Create 2 files named pv.yaml and pvc.taml with the following content:

    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: config-logging
      labels:
        type: local
    spec:
      capacity:
        storage: 40Gi
      accessModes:
        - ReadWriteMany
      hostPath:
        path: "/mnt/glusterfs/config-logging"
    
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: config-logging
      namespace: tekton-pipelines
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 40Gi
      volumeName: config-logging
      volumeMode: Filesystem
    

    Then, create the volumes by typing

    oc create namespace tekton-pipelines
    oc create -f pv.yaml
    oc create -f pvc.yaml
    

    As per the document, download the official release yaml file

    Before applying the file, we need to modify the following 2 iterations in order to use our local storage

          volumes:
          - configMap:
              name: config-logging
            name: config-logging
    

    For:

          volumes:
          - name: config-logging
            persistentVolumeClaim:
              claimName: config-logging
    

    then, perform the deployment by executing:

    oc apply -f release.yaml
    

    And then verify that the pods are up and running by using:

     oc get pods --namespace tekton-pipelines
    NAME                                          READY   STATUS    RESTARTS   AGE
    tekton-pipelines-controller-8b78bdf57-qmrlx   1/1     Running   0          22m
    tekton-pipelines-webhook-66987c8b96-d9vsq     1/1     Running   0          22m
    

    Once up and running, we can also download the tekton dashboard from its git repo

    There's no need to edit the yaml file here, so we can apply it straight forward:

    oc apply -f dashboard_latest_release.yaml
    

    Verify again your pods, and create an ingress (in kubernetes) or a route in openshift and access the dashboard

    alt text

    Create service account

    As a final step, to ensure that the pipelines that we create have the appropriate permissions to store images and perform the deployments, we need to create a Service account:

    oc create serviceaccount pipeline
    oc adm policy add-scc-to-user privileged -z pipeline
     oc adm policy add-role-to-user edit -z pipeline
    
    posted in Pipelines
  • This document is for dev/test environments. Do not use on a prod environment

    What is glusterfs

    Gluster is a scalable network filesystem. Using common off-the-shelf hardware, you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks. Gluster is free.

    Why user glusterfs?

    Usually, when you run your infrastructure on a cloud provider, the storage is automatically provisioned. Nevertheless, when running your own installation in a VM or bare metal, you won't have any kind of storage.

    By using glusterfs, you have a reliable and fast system to create your volumes in a easy manner

    Installation

    The process it's quite simple. As per the official documentation, which states:

    Please use the Gluster Community PPAs for the latest version of GlusterFS:
    https://launchpad.net/~gluster
    

    Let's say we have 3 kubernetes nodes. Our master node, and node1 and node2.
    The following commands should be executed in all the nodes

    The steps that we need to perform are:

    sudo add-apt-repository ppa:gluster/glusterfs-7
    sudo apt-get update
    apt install glusterfs-server
    sudo systemctl start glusterd
    sudo systemctl enable glusterd
    

    Adding nodes

    Ensure that our 3 hosts have the corresponding entry on the /etc/hosts

    From the master node, we need to perform the following actions:

    gluster peer probe node1
    gluster peer probe node2
    

    Then, we need to verify that our nodes have been added by running:

    gluster peer status
    

    which will return:

    Number of Peers: 2
    
    Hostname: node1
    Uuid: 76a57408-5f50-420f-927a-4cfc6843f1d4
    State: Peer in Cluster (Connected)
    
    Hostname: node2
    Uuid: 9e76c3a8-9a48-4a07-a6de-d3a9c3659a43
    State: Peer in Cluster (Connected)
    root@master:/home/istio/istio-1.4.3# gluster pool list
    UUID                                    Hostname        State
    76a57408-5f50-420f-927a-4cfc6843f1d4    node1           Connected
    9e76c3a8-9a48-4a07-a6de-d3a9c3659a43    node2           Connected
    df901df3-f164-45c7-afaf-92efc5964f10    localhost       Connected
    

    Setup Distributed GlusterFS Volume

    Now that we have the nodes, it's time to add the actual shared storage.
    We need to add an additional disk to our instances (50 / 100 gb)

    Then, scan for new disks with fdisk -l and create the vg / lv as per below on each node:

    vgcreate glustervg /dev/sdb
    lvcreate -L +51G -n glusterlv /dev/glustervg
    mkfs.xfs /dev/glustervg/glusterlv
    mkdir /gluster
    

    Then, get the blkid for the new lv and add it to /etc/fstab

    blkid /dev/glustervg/glusterlv
    
    vi /etc/fstab
    add:
    UUID=99f57d21-8844-44b9-b2e3-85fc80ce5886 /gluster xfs defaults 0 0
    
    mount /gluster
    

    From the master node, we will create the replica by executing:

    gluster volume create vol01 replica 3 transport tcp master:/glusterfs node1:/gluster node2:/gluster force
    gluster volume start vol01
    

    And validate by checking the volume information:

    gluster volume info vol01
    
    Volume Name: vol01
    Type: Replicate
    Volume ID: 63f13452-27f6-44a9-a24f-66ec0a4df33e
    Status: Started
    Snapshot Count: 0
    Number of Bricks: 1 x 3 = 3
    Transport-type: tcp
    Bricks:
    Brick1: master:/glusterfs
    Brick2: node1:/gluster
    Brick3: node2:/gluster
    Options Reconfigured:
    transport.address-family: inet
    storage.fips-mode-rchecksum: on
    nfs.disable: on
    performance.client-io-threads: off
    

    Mount our gluster volume

    On each node we will create a new mount point

    mkdir -p /mnt/glusterfs
    

    For redundancy, we will be mounting the different servers on each node. As an example, on the master node, we mount node2, on node1 we mount master and in node2 we mount node1

    Edit /etc/fstab and add:

    Master:

    node2:/vol01 /mnt/glusterfs glusterfs defaults,_netdev 0 0
    

    Node1:

    master:/vol01 /mnt/glusterfs glusterfs defaults,_netdev 0 0
    

    Node2:

    node1:/vol01 /mnt/glusterfs glusterfs defaults,_netdev 0 0
    

    Then, on each node, mount the volume by doing a mount -a

    Create a kuberntes PV

    Now that we have our shared storage, we can create a Persistent Volume.
    To do so, in our master node we will run the following command

    /mnt/glusterfs/test
    

    Then, create a yaml file named storage.yaml with the following content

    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: test
      labels:
        type: local
    spec:
      capacity:
        storage: 5Gi
      accessModes:
        - ReadWriteMany
      hostPath:
        path: "/mnt/glusterfs/test"
    

    Once done, create the pv with:

    kubectl apply -f storage.yml
    

    And validate with:

    kubectl get pv
    NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
    test   5Gi        RWX            Retain           Available                                   4s
    

    Create the PVC

    One we have our volume create, we need a Persistent volume claim (or pvc) in order to allow our pods to use it.
    To do so, we will create a file named pvc.yaml with the following content:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      annotations:
      name: test
      namespace: default
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 5Gi
      volumeName: test
    

    Then, create it by using:

    kubectl create -f pvc.yaml
    

    Finally, validate its creation:

    kubectl get pvc
    NAME   STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    test   Bound    test     5Gi        RWX                           115s
    
    posted in Kubernetes
  • This is a exercise for testing purposes

    Specs

    In order to create our Kubernetes test cluster, we will need 3 nodes (1 master and 2 slaves) with specs as the following ones.
    You can go down to 4gb of RAM for this POC, but we recommend having at least 8gb

    • 2 vCPUs
    • 8 GB RAM
    • Ubuntu 18.04 on each node

    You can find really cheap cloud providers to run this kind of test. For example https://www.vultr.com/products/cloud-compute/ offers a similar instance for 20$ per month per instance

    Requirements pre-installation

    First, we will need to generate an ssh key-pair on each node

    root@node1:~# ssh-keygen
    root@node2:~# ssh-keygen
    root@node3:~# ssh-keygen
    

    You will need to set a password for the root user, as by default it doesn't have one (run on each node)

    sudo passwd
    [sudo] password for xxxx: 
    Enter new UNIX password: 
    Retype new UNIX password: 
    passwd: password updated successfully
    

    Once done and before moving forward, we need to disable the following entries from our /etc/hosts file on each node

    #::1    ip6-localhost   ip6-loopback
    #127.0.1.1      virtualserver01.xxx.xx       virtualserver01
    

    get your node ips and add them to /etc/hosts on each node

    172.16.0.15 node1
    172.16.0.16 node2
    172.16.0.17 node3
    

    Then copy the keys to each node (this needs to be done from each node)

    ssh-copy-id -i /root/.ssh/id_rsa.pub root@node1
    ssh-copy-id -i /root/.ssh/id_rsa.pub root@node2
    ssh-copy-id -i /root/.ssh/id_rsa.pub root@node3
    

    Update the packages on each node

    apt-get update
    sudo apt-get install \
        apt-transport-https \
        ca-certificates \
        curl \
        software-properties-common -y
    

    Install docker

    We need to install Docker on each node

    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add 
    
        sudo add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
       $(lsb_release -cs) \
       stable"
    
     sudo apt-get update
    
     apt-get install docker-ce -y
    
    

    Install Kubernetes

    We need to add the Kubernetes signing key on each node
    To do so. run the following command:

    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
    

    Also, we need to add the Xenial Kubernetes Repository on each node
    Run the following command to do so and then update the package list:

    sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
    apt-get update
    

    The next step in the installation process is to install Kubeadm on all the nodes through the following command:

    sudo apt install kubeadm -y
    

    Once the installation finishes, you can check the version by running

    kubeadm version
    

    Before moving on, we need to disable the SWAP memory in all the nodes
    You need to disable swap memory on both the nodes as Kubernetes does not perform properly on a system that is using swap memory.

    Run the following command to do so:

    sudo swapoff -a
    

    Then, only in our master node (node1), initialize kubelet by typing the following command

    kubeadm init --pod-network-cidr=10.244.0.0/16
    

    Once the process finishes, take note of the output as it's really important.
    We will be using later this information to join the worker nodes to our cluster.

    Don't execute it yet

    Your Kubernetes control-plane has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 172.16.0.15:6443 --token ahlqlw.bk3iqojsp519rf7d \
        --discovery-token-ca-cert-hash sha256:ebf7c6b895b52a059872d198a44b942267bef89d8d1d6802bbd3cc8082ebe600
    

    Run the following commands on the master node

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    

    Then, on the master node download the calico network manifest using the following command:

    curl https://docs.projectcalico.org/v3.11/manifests/calico.yaml -O
    

    edit the file calico.yaml and modify as per below (defining the CIDR block that we configured during the kube installation)

            - name: CALICO_IPV4POOL_CIDR
              value: "192.168.0.0/16"
    

    for

            - name: CALICO_IPV4POOL_CIDR
              value: "10.244.0.0/16"
    

    The section will look like:

    alt text

    then apply the manifest by running

    kubectl apply -f calico.yaml
    

    After a few minutes, you can query the status of the pods by using:

    kubectl get pods --all-namespaces
    

    alt text

    And also verify that your master node is now ready by using:

    kubectl get nodes
    

    alt text

    Joining worker nodes to the cluster

    Now you can go to node2 and node 3 and run the command generated during the installtion to join the worker nodes to our cluster

    kubeadm join 172.16.0.15:6443 --token ahlqlw.bk3iqojsp519rf7d \
        --discovery-token-ca-cert-hash sha256:ebf7c6b895b52a059872d198a44b942267bef89d8d1d6802bbd3cc8082ebe600
    

    after a couple of minutes you can check the nodes status:

    kubectl get nodes
    

    alt text

    And get additional node info

    kubectl get nodes -o wide
    

    alt text

    Access the cluster from your workstation

    In order to access from our laptop or a remote machine, we need to create a cluster role and cluster role binding.
    In the master node create a yaml file witht the content:

    We are creating a role with full admin permissions. In a PROD environment we should follow the best practices and limit the permissions.

    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      annotations:
        rbac.authorization.kubernetes.io/autoupdate: "true"
      labels:
      name: remote
    rules:
    - apiGroups:
      - '*'
      resources:
      - '*'
      verbs:
      - '*'
    - nonResourceURLs:
      - '*'
      verbs:
      - '*'
    

    then create the cluster role binding by executing the command from the master node

    kubectl create clusterrolebinding remote   --clusterrole=remote  --user=remote   --group=system:serviceaccounts
    

    get the cluster token by running

    kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | awk '/^deployment-controller-token-/{print $1}') | awk '$1=="token:"{print $2}'
    

    you will get a long token string such as the following output.
    take note of it

    eyJhbGciOiJSUzI1NiIsImtpZCI6ImxnRlJxeFpOZUF2bWFKSktXSzNLVGdvYjRkQk5rU2tyT0JNWVdMbWtvYzAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZXBsb3ltZW50LWNvbnRyb2xsZXItdG9rZW4tZ2pxenEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVwbG95bWVudC1jb250cm9sbGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2NkMWI0ZTItM2NlOC00MzhjLWIzNmItMjMxYzI1ZTU2ZGU4Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRlcGxveW1lbnQtY29udHJvbGxlciJ9.EV06Ljf3jZH3iH7Fi4YHFjKoSFPK8tLQIoR68GEzt1CGNTwKWKsDSkp6VhvZ3LJO9k4H60bvm6GIfj8aU8qXfE1JRw70hp83_oVsyj-0D-r1THlBiXazeAcw8bP02bWp0_005cNsZZ88UEVfWq1f1cFWUS4aqESfo8w5LFmudEnrA1j2tIikR61u__Mr0WJJrzvc7HhIt79RC2_A1004tjaUWJRhXJM1U52WC6o_B4Iyv1k9xs6rNeFwW3C6vSsfyB381ZN3ItiHGBDPPIZnWhsQb7m1HTRX3kCWi949ngdFtbo0_fJttgob8YIkQaTm77gZcdAERNMwZsfr6NtDiw
    

    Then, from your workstation, open a shell or powershell and perform the following commands

    kubectl config set-cluster kubernetes --server=https://your_node1_public_ip:6443 --insecure-skip-tls-verify=true
    
     kubectl config set-context kubernetes --cluster=kubernetes
    
     kubectl config set-credentials remote --token=your_token
    
     kubectl config set-context kubernetes --user=remote
    
     kubectl config use-context kubernetes
    

    you can then run the usual commands to check your cluster info

    kubectl get nodes
    kuectl get cluster-info
    

    Deploy a stateless app

    In order to test our cluster, we can deploy a stateless example app based on the official kubernetes doc, create a yaml file with the following content in your workstation:

    apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
    kind: Deployment
    metadata:
      name: nginx-deployment
    spec:
      selector:
        matchLabels:
          app: nginx
      replicas: 2 # tells deployment to run 2 pods matching the template
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.7.9
            ports:
            - containerPort: 80
    

    Then, perform

    kubectl apply -f my-file.yaml
    

    And then verify if your deployment is running and where

    kubectl.exe get pods --all-namespaces -o wide
    NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
    default       nginx-deployment-54f57cf6bf-n7nfc          1/1     Running   0          2m11s   10.244.104.1     node2   <none>           <none>
    default       nginx-deployment-54f57cf6bf-wrtsd          1/1     Running   0          2m11s   10.244.135.1     node3   <none>           <none>
    

    Access your app

    Now, you can access your nginx server by doing a kubernetes port-forward.
    Replace the pod name by the value returned by the previous command

    kubectl port-forward nginx-deployment-54f57cf6bf-n7nfc 8080:80
    
    Forwarding from 127.0.0.1:8080 -> 80
    Forwarding from [::1]:8080 -> 80
    Handling connection for 8080
    Handling connection for 8080
    

    Then, open a web browser to http://localhost:8080

    Using octant

    As you know, you can deploy and use the kubernetes dashboard.
    Nevertheless, we recommend you to take a look to octant

    Octant is a tool for developers to understand how applications run on a Kubernetes cluster. It aims to be part of the developer's toolkit for gaining insight and approaching complexity found in Kubernetes.

    The benefit is that it´s installed in your workstation and the access is based in your permissions, meaning that no additional deployments are needed in the cluster.
    You can download the last release here

    Once installed, just execute it by running the following command, and a tab in your web browser will launch the console.
    You can check the deployment and pods that we just triggered before.

    octant
    

    alt text

    posted in Kubernetes
  • Register to our blog

    We've just started with this blog, but if you would like to comment, collaborate or whatsoever, please feel free to register and let us know what you think.

    https://blog.cloudgsx.es/register

    Thank you!

    posted in Announcements
  • Purpose

    It's possible that due to some requirement in our environment, we might need to run a Docker image which has Docker running inside of it.

    An example would be the need of creating a custom Jenkins slave image with specific libraries or packages for our builds in Kubernetes.

    If there are no needs for specific libraries or packages, we recommend to use Kaniko

    In this build we are including:

    • Python 3.6
    • wget
    • unzip
    • curl

    Build Process

    We need an environment with Docker installed (could be a linux server, our workstation or laptop).
    Then, we need to create a Dockerfile with the following content:

    The following dockerfile is an example with several dependencies.

    FROM centos:7
    RUN yum -y install curl systemd yum-utils
    ENV container docker
    RUN rm -f /lib/systemd/system/multi-user.target.wants/* && rm -f /etc/systemd/system/*.wants/* && rm -f /lib/systemd/system/local-fs.target.wants/* && \
        rm -f /lib/systemd/system/sockets.target.wants/*udev* && rm -f /lib/systemd/system/sockets.target.wants/*initctl* && rm -f /lib/systemd/system/basic.target.wants/* && \
        rm -f /lib/systemd/system/anaconda.target.wants/*
    RUN yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && \
        yum clean all && rm -Rf /var/cache/yum/
    RUN yum install https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm -y
    RUN yum install docker-ce initscripts epel-release openssh openssh-server openssh-clients openssl-libs wget unzip git python36 -y && \
        yum clean all; systemctl enable docker.service && systemctl enable sshd.service
    RUN sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config && rm -f /etc/nologin /run/nologin && mkdir /jenkins && chmod -R 777 /jenkins && \
        mkdir -p /etc/docker/ && chmod -R 777 /etc/docker/
    WORKDIR /jenkins
    COPY ./start.sh /jenkins/start.sh
    COPY ./slave.sh  /jenkins/slave.sh
    COPY ./daemon.json /etc/docker/daemon.json
    
    EXPOSE 80
    EXPOSE 8080
    EXPOSE 50000
    
    CMD ["/jenkins/start.sh"]
    

    We need to create to additional files. The first file would be slave.sh:

    #!/bin/bash
    for (( ; ; ))
    do
       echo "Pres CTRL+C to stop..."
       sleep 1
    done
    

    And the second file would be start.sh

    #!/bin/bash
    setsid /jenkins/slave.sh > output.txt &
    exec /usr/sbin/init && service docker start 
    

    Don't forget to give execution permissions to our scripts:

    chmod a+x slave.sh start.sh
    

    Finally, create a file named daemon.json
    As we will be running docker inside a container, we will be using the vfs storage driver

    {
     "storage-driver": "vfs",
     "data-root": "/tmp"
    }
    

    Then we can build our docker image with the command:

    docker build . -t mycustomimage
    

    Testing the image

    Now if you run the following command, you'll see your new build

    docker images
    REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
    mycustomimage       latest              f2a59057bc67        23 seconds ago      1.16GB
    

    Try and run it by typing:

    docker run -d --privileged=true -it mycustomimage
    

    Then check that it's running:

    docker ps
    CONTAINER ID        IMAGE               COMMAND               CREATED             STATUS              PORTS                         NAMES
    f8f27c1cb50e        mycustomimage       "/jenkins/start.sh"   13 seconds ago      Up 11 seconds       80/tcp, 8080/tcp, 50000/tcp   sad_goodall
    

    Then, connect to our running container (remember to change for your container id) by using:

     docker exec -it f8f27c1cb50e /bin/bash
    

    Once inside the container, try to run

    docker ps
    docker pull nginx
    

    Tweaking the image

    As you might have noticed, the image grew up more than 500mb. This is because we are installing several packages.
    You can remove the packages that you're not going to use and customize your docker builds inside the container properly.

    Build cleanup

    Once you've finished building your images and you've pushed them to your container registry, you can clean up all the images and cached information.

    Warning

    If you run the following command it will delete all your images in the system.
    Execute it wisely

    docker system prune -a
    
    posted in Docker
  • Install let's encrypt certbot

    info

    Let's Encrypt provides free SSL certificates which last 3 months since the time of its generation.

    In our example we will be using Centos 7

    Firts, we need to install the epel-release repo

    yum -y install epel-release 
    

    Then, install the following packages:

    yum -y install yum-utils certbot 
    

    Certbot will be the binary that we will be using for the certificate generation.

    There are a few ways to go on from now. The most common way is to install certbot on the server itself and let it automatically update our config files for apache or nginx. Nevertheless, even that it might take a few more minutes, we strongly recommend to generate the SSL certificate by creating a TXT DNS record.

    The reason for this is quite simple. We can generate our certificate anywhere and then copy the files to our corresponding server and ensure that our configuration is not automatically updated.

    Generate the certifciate

    certbot  -d **mydomain.com** --manual --preferred-challenges dns certonly
    

    This will generate the following output

    [root@myserver]#  certbot  -d  mydomain.com --manual --preferred-challenges dns certonly
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator manual, Installer None
    Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
    Obtaining a new certificate
    Performing the following challenges:
    dns-01 challenge for  mydomain.com
    
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    NOTE: The IP of this machine will be publicly logged as having requested this
    certificate. If you're running certbot in manual mode on a machine that is not
    your server, please ensure you're okay with that.
    
    Are you OK with your IP being logged?
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    (Y)es/(N)o: y
    
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Please deploy a DNS TXT record under the name
    _acme-challenge.mydomain.com with the following value:
    
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    
    Before continuing, verify the record is deployed.
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Press Enter to Continue
    

    Now, what we need to to is generate the requested TXT record with the corresponding value. We recommend set the TTL of 1 min for our new record.
    Once done, wait 1/2 minutes and press enter

    This will generate the necessary files under /etc/letsencrypt/archive/mydomain.com

    [root@myserver]# ls -la /etc/letsencrypt/archive/mydomain.com/
    total 20
    drwxr-xr-x.  2 root root   79 Jan 13 21:40 .
    drwx------. 11 root root 4096 Jan 13 21:40 ..
    -rw-r--r--.  1 root root 1915 Jan 13 21:40 cert1.pem
    -rw-r--r--.  1 root root 1647 Jan 13 21:40 chain1.pem
    -rw-r--r--.  1 root root 3562 Jan 13 21:40 fullchain1.pem
    -rw-------.  1 root root 1704 Jan 13 21:40 privkey1.pem
    

    You can copy then the fullchain1.pem and privkey1.pem to your corresponding web server
    In the example of an nginx server, the config file will look similar to:

        listen 443 ssl;
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
        ssl_certificate /etc/nginx/certs/fullchain1.pem;
        ssl_certificate_key /etc/nginx/certs/privkey1.pem;
        ssl_verify_client off;
        ssl_session_timeout 5m;
        ssl_protocols TLSv1.2;
        ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;;
        ssl_prefer_server_ciphers on;
    

    Finally, in your DNS provider, to not forget to create a CAA record.
    A Certification Authority Authorization (CAA) record is used to specify which certificate authorities (CAs) are allowed to issue certificates for a domain. The purpose of the CAA record is to allow domain owners to declare which certificate authorities are allowed to issue a certificate for a domain

    It should contain a record as in:

    128 issue "mydomain.com"
    
    posted in Web