Skip to main content

1. Table of Content

2. Intro

This document contains my notes on preparation course of Certified Kubernetes Administrators (CKA) Certification by Mumshad Mannambeth.

3. Certification

4. CKA Lecture and Lab Notes

5. Core Concepts

5.1. Kubernetes Architecture​

  • Worker Nodes: hosts application as containers
    • Worker Nodes Components:
      • kubelet: is an agent which runs on Worker Nodes
      • kube-proxy: is a service responsible for networking on Worker Nodes
  • Master Node: manage, plan schedule, monitor etc. nodes using Control Plane Components
    • Control Plane Components:
      • kube-apiserver: is responsible for orchestration of all operations within k8s cluster Control Plane Components
      • ETCD Cluster: is a database which stores information in a key-value format
      • kube-scheduler: identifies right node based on containers resource requirements
      • Controller-Manager
        • Node-Controller
        • Replication-Controller

5.2. Open Container Initiative (OCI)​

  • imagespec: specifications how container image must be built
  • runtimespec: specifications how conainer runtime should be developed

5.3. Container Runtime Interface (CRI)​

  • Kubernetes introduced Container Runtime Interface (CRI) which allows any vendor to work as container runtime as long as it adheres to OCI specification and not only Docker which used initially dockershim.
  • Starting from version v1.24 Kubernetes deleted dockershim and Docker support

5.4. Docker vs ContainerD​

5.4.1. Docker​

  • Docker is not only container runtime, but also consist of multiple tools
    • CLI
    • API
    • BUILD
    • Volumes
    • AUTH
    • SECURITY
    • ContainerD
  • Docker uses container runtime ContainerD

5.4.2. ContainerD​

  • ContainerD is part of Cloud Native Computing Foundation (CNCF)
  • ContainerD is a higher-level tool that manages the entire lifecycle of containers.
  • ContainerD is a deamon which manages runc and compatable with Container Runtime Interface (CRI)
  • runc is a container runtime
  • ContainerD works as container runtime on its own separate from Docker
  • ContainerD comes with crt CLI tool mainly made for debugging container. nerdctl (similar to docker CLI) provides stable and humand-friendly user experience
  • crictl is another CLI tool from k8s community compatable with CRI which works accross different container runtime. Mainly used for debugging
  • Read more

5.5. ETCD​

  • ETCD is a distributed reliable key-value store that is simple, secure and fast
  • Key-Value Store stores information if form of documents, pages (i.e. files) which can be in any format and structure

5.5.1. Get started quickly with ETCD​

  • Download the relevant binary for your operating system from github releases page (https://github.com/etcd-io/etcd/releases)

    For Example: To download ETCD v3.5.6, run the below curl command

    $ curl -LO https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz

  • Extract it.

    $ tar xvzf etcd-v3.5.6-linux-amd64.tar.gz

  • Run the ETCD Service

    $ ./etcd

  • When you start ETCD it will by default listen on port 2379

  • The default client that comes with ETCD is the etcdctl client. You can use it to store and retrieve key-value pairs.

    Syntax: To Store a Key-Value pair
    $ ./etcdctl put key1 value1
    Syntax: To retrieve the stored data
    $ ./etcdctl get key1
    Syntax: To view more commands. Run etcdctl without any arguments
    $ ./etcdctl

    # etcdctl command in API 3 version
    etcdctl snapshot save 
    etcdctl endpoint health
    etcdctl get
    etcdctl put

    export ETCDCTL_API=3           # set the environment variable ETCDCTL_API for desired API version

    # Certs for ETCDCTL to authenticate to the ETCD API Server
    --cacert /etc/kubernetes/pki/etcd/ca.crt     
    --cert /etc/kubernetes/pki/etcd/server.crt     
    --key /etc/kubernetes/pki/etcd/server.key

    # Example: 
    kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl get / --prefix --keys-only --limit=10 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt  --key /etc/kubernetes/pki/etcd/server.key"

5.5.2. ETCD in Kubernetes​

  • The ETCD Datastore stores following cluster information:

    • Nodes
    • PODS
    • Configs
    • Secrets
    • Accounts
    • Roles
    • Bindings
    • Others

  • kubectl get command retrieves all information from the ETCD Server

5.5.3. ETCD Setup - Manual vs kubeadm​

5.6. Kube-API Server​

  • Kube-API Server is a primary management component in K8s
  • kube-api server performs following actions when kubectl command is executed:
    • Authenticate User
    • Validate Request
    • Retrieve dataf
  • Read more

5.7. Kube Controller Manager​

  • Kube Controller Manager manages various controllers in kubernetes
  • Controller is a process that continuously monitors the state of the components within the system and ensures the whole system to have the desired functioning state
  • Controllers:
    • Node Controller - monitors health of Nodes every 5s
      • Node Monitor Period = 5s
      • Node Monitor Grace Period = 40s (i.e. waits 40s before marking it as unreachable)
      • POD Eviction Timeout = 5m (i.e. waits 5m to come back up if unreachable before removing pods and provisioning resources to the healthy nodes if it is part of replica set)
    • Replication Controller - monitors state of replica sets
    • Deployment Controller
    • Namespace Controller
    • Endpoint-Controller
    • CronJob
    • Job Controller
    • PV Protection Controller
    • Service Account Controller
    • Stateful Set
    • Replica Set
    • PV Binder Controller

5.7.1. Installation of Kube-Controller-Manager​

5.8. Kube Scheduler​

  • kube-scheduler is responsible for scheduling pods on nodes. The kube-scheduler is only responsible for deciding which pod goes on which node depending on certain criteria (e.g. Filter Nodes, Rank Nodes). It doesn't actually place the pod on the nodes, that's the job of the kubelet.
  • Read more

5.9. Kubelet​

  • kubelet is a node agent which is responsible for creating the pods on the nodes and the sole point of contact for the kubernetes cluster
  • kubelet also collects reports and sends regularly to master node

5.10. Kube Proxy​

  • POD Network is an internal virtual network that spans across all nodes in the cluster to which all pods are connected. Within k8s cluster every pod can reach every other pod which is accomplished by deploying networking solution, i.e. POD Network.
  • kube-proxy is a network component in Kubernetes that helps manage network communication to and from the Pods in a k8s cluster. Key function of kube-proxy:
    • Traffic Routing
    • Load Balancing
    • Service Discovery
    • Protocol Support

5.11. Pods​

  • Pod is a logical host for one or more containers that are deployed together on the same host
# Example of Pod definition with YAML

apiVersion: v1
kind: Pod               
metadata:                       # dictionary
    name: myapp-pod
    namespace: dev
    labels:                     # under labels custom key:value allowed
        app: myapp
        type: front-end

spec:                           # dictionary
    containers:                 # list/array
        - name: nginx-controller
          image: nginx

5.12. ReplicaSets​

  • ReplicaSet and ReplicationController (deprecated in favor of ReplicaSet) are both Kubernetes resources for managing pod replication to ensure a specified number of pod replicas running at any given time
# Example of ReplicationController definition with YAML

apiVersion: v1
kind: ReplicationController
metadata:                       # dictionary
    name: myapp-rc
    namespace: dev
    labels:                     # under labels custom key:value allowed
        app: myapp
        type: front-end

spec:                           # dictionary
    template:                   # Pod template for replication
        metadata:                       
            name: myapp-pod
            labels:                     
                app: myapp
                type: front-end

        spec:                           
            containers:                 
                - name: nginx-controller
                image: nginx
    
    replicas: 3
# Example of ReplicaSet definition with YAML

apiVersion: apps/v1
kind: ReplicaSet
metadata:                       # dictionary
    name: myapp-rs
    namespace: dev
    labels:                     # under labels custom key:value allowed
        app: myapp
        type: front-end

spec:                           # dictionary
    template:                   # Pod template for replication
        metadata:                       
            name: myapp-pod
            labels:                     
                app: myapp
                type: front-end

        spec:                           
            containers:                 
                - name: nginx-controller
                image: nginx
    
    replicas: 3
    selector:                   # Major difference from ReplicationController
        matchLabels:
            type: front-end     # To select all existing Pods matching defined label

5.13. Labels and Selectors​

  • Labels are key/value pairs that are attached to objects such as Pods for categorizing resources.
  • Selectors are queries to filter resources based on labels.

5.14. Annotations​

  • Annotations are key-value pairs that can be attached to various Kubernetes objects (e.g. pods, services, deployments etc.) to store arbitrary metadata that is not used for identification or selection.
  • Usage of Annotations:
    • Storing build or version information
    • Providing links to documentation or related resources
    • Specifying configuration options for tools that interact with the resource (e.g., deployment tools, monitoring systems)

5.15. Deployments​

  • Deployment is a higher-level abstraction that manages a ReplicaSet, which in turn manages the Pods.
# Example of Deployment definition with YAML

apiVersion: apps/v1
kind: Deployment
metadata:                       # dictionary
    name: myapp-rs
    namespace: dev
    labels:                     # under labels custom key:value allowed
        app: myapp
        type: front-end

spec:                           # dictionary
    template:                   # Pod template for replication
        metadata:                       
            name: myapp-pod
            labels:                     
                app: myapp
                type: front-end

        spec:                           
            containers:                 
                - name: nginx-controller
                image: nginx
    
    replicas: 3
    selector:                   
        matchLabels:
            type: front-end     # To select all existing Pods matching defined label

5.16. Services​

  • Service is used to expose a set of Pods and enable communication between them
  • Services enable communication between different components of an application, allowing Pods to discover and interact with each other reliably
  • There are different types of Services, each serving a specific purpose
  • Service acts as built-in load balancer to distribute load across different Pods (Algorithm: Random, SessionAffinity: Yes)

5.16.1. ClusterIP​

  • This is the default service type
  • It exposes the service on a cluster-internal IP
  • Only accessible from within the cluste
  • Useful for internal communication between services
# Example of Service of type ClusterIP definition with YAML

apiVersion: v1
kind: Service               
metadata:                       # dictionary
    name: back-end
    namespace: dev

spec:
    type: ClusterIP
    ports:
      - port: 80                # The port that the service will expose
        targetPort: 8080        # The port on the pod that the service should forward to
    selector:
        app: myapp
        type: back-end

5.16.2. NodePort​

  • Service of type NodePort listens on the static port of each node's IP address in cluster and forward request to the Pods
  • Node's port range 30 000 to 32 767
# Example of Service of type NodePort definition with YAML

apiVersion: v1
kind: Service               
metadata:                       # dictionary
    name: myapp-service
    namespace: dev

spec:
    type: NodePort
    ports:
      - port: 80                # The port that the service will expose
        targetPort: 8080        # The port on the pod that the service should forward to
        nodePort: 30008         # The port on each node that will forward to the service
    selector:
        app: myapp
        type: front-end

5.16.3. LoadBalancer​

  • Creates an external load balancer (if supported by the cloud provider)
  • Automatically assigns a public IP address to the service
  • Distributes incoming traffic across multiple nodes
  • Ideal for production environments where you need to handle large amounts of traffic
# Example of Service of type LoadBalancer definition with YAML

apiVersion: v1
kind: Service               
metadata:                       # dictionary
    name: myapp-service
    namespace: dev

spec:
    type: LoadBalancer
    ports:
      - port: 80                # The port that the service will expose
        targetPort: 8080        # The port on the pod that the service should forward to
        nodePort: 30008         # The port on each node that will forward to the service

5.16.4. ExternalName​

5.17. Namespaces​

  • Namespace is a way to organize and manage resources within a cluster which provides a mechanism for isolating groups of resources
  • Key Points of Namespaces:
    • Isolation
    • Resource Management
    • Access Control
    • Resource Quotas
    • Network Policies
    • Scoped Resources

Info: A pattern to access a service in a different namespace using DNS:

<service-name>.<namespace>.svc.cluster.local

  • <service-name>: name of the service to access
  • <namespace>: namespace where the service is located
  • svc: indicates that a service being accessed
  • cluster.local: default domain name for services in the k8s cluster (can be customized)
# Example of Namespace definition with YAML

apiVersion: v1
kind: Namespace
metadata:
    name: dev

5.17.1. Resource Quota​

# Example of ResourceQuota definition with YAML

apiVersion: v1
kind: ResourceQuota
metadata:
    name: compute-quota
    namespace: dev

spec: 
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: "5Gi"
    limits.cpu: "10"
    limits.memory: "10Gi"

5.18. Imperative vs. Declarative​

In the context of Infrastructure as Code (IaC), imperative and declarative are two different approaches to defining and managing infrastructure.

  • In an imperative approach, you specify how to achieve a desired state by providing a sequence of commands or instructions.
  • In a declarative approach, you specify what the desired state of the infrastructure should be, without detailing the steps to achieve that state.
# IMPERAIVE APPROACH

# create objects
kubectl run --image=nginx nginx 
kubectl create deployment --image=nginx nginx 
kubectl expose deployment nginx --port 80

# update objects
kubectl edit deployment nginx
kubectl scale deployment nginx --replicas=5
kubectl set image deployment nginx nginx=nginx:1.18

kubectl create -f nginx.yaml
kubectl replace -f nginx.yaml
kubectl delete -f nginx.yaml


# DECLARATIVE APPROACH

kubectl apply -f object-definition-file.yaml

6. Scheduling

  • Scheduling is a process of assigning pods (the smallest deployable units in Kubernetes) to nodes (the machines or virtual machines in the cluster) based on resource availability and other constraints.

6.1. Manual Scheduling​

apiVersion: v1
kind: Pod
metadata:
 name: nginx
 labels:
  name: nginx
spec:
 containers:
 - name: nginx
   image: nginx
   ports:
   - containerPort: 8080
 nodeName: node02                            # Specifies the name of Node, by default is not set

6.1.1. Binding Object​

apiVersion: v1
kind: Binding
metadata:
  name: my-pod
  namespace: default
target:
  apiVersion: v1
  kind: Node
  name: my-node

6.2. Taints and Tolerations​

  • Taints are applied to nodes to prevent certain pods from being scheduled on them
  • Tolerations are applied to pods to allow them to be scheduled on nodes with specific taints
kubectl taint nodes <NODE-NAME> <KEY>=<VALUE>:<TAINT-EFFECT>

# TAINT-EFFECT defines what happens to Pods that do not TOLERATE this taint.
# TAINT-EFFECTs: NoSchedule, PreferNoSchedule, NoExecute



# TAINTING NODE EXAMPLE
kubectl taint nodes node1 app=blue:NoSchedule

# TOLERATION DEFINITION OF POD EXAMPLE
apiVersion: v1
kind: Pod
metadata:
 name: nginx
 labels:
  name: nginx
spec:
 containers:
 - name: nginx
   image: nginx
   ports:
   - containerPort: 8080
 tolerations:
 - key: "app"
   operator: "Equal" 
   value: "blue"
   effect: "NoSchedule"

6.3. Node Selectors​

  • Label a node:
kubectl label nodes node01 size=Large
  • Assign a Pod to the Node
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
spec:
  containers:
  - name: nginx
    image: nginx
  nodeSelector:
    size: Large
  • Node Selector's Limitations:
    • Advanced expressions, e.g. Large OR Medium, NOT Small etc.

6.4. Node Affinity​

Node affinity is conceptually similar to nodeSelector, allowing you to constrain which nodes your Pod can be scheduled on based on node labels.

apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
spec:
  containers:
  - name: nginx
    image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: size
            operator: In                                        # operators: In, NotIn, Exists, DoesNotExist
            values:
            - Large
            - Medium

6.4.1. Node Affinity Types​

  • Available:
    • requiredDuringSchedulingIgnoredDuringExecution
    • preferredDuringSchedulingIgnoredDuringExecution
  • Planned:
    • requiredDuringSchedulingRequiredDuringExecution

Two states in the lifecycle of the Pod considering Node Affinity:

  • DuringScheduling is a state when Pod does not exists and is created for the first time. If reqired then a node with specified lable must exist in order the Pod to be scheduled.
  • DuringExecution is a state where the Pod is running and the change made in environment which affects Node Affinity, e.g. change in the label of the Node. If ignored, the Pod will continue running if node label is changed or removed.

Read more.

6.5. Taints and Tolerations vs Node Affinity​

Taints and Tolerations: Taints are applied to nodes to repel certain pods unless those pods have matching tolerations. This mechanism is useful for reserving nodes for specific workloads. Node Affinity: Node affinity allows you to specify rules based on node labels that determine which nodes a pod can be scheduled on. It helps ensure that pods are placed on nodes with specific characteristics.

  • Taints and Tolerations do not guarantee that the pods will only prefer these nodes.
  • As such, a combination of taints and tolerations and node affinity rules can be used together to completely dedicate nodes for specific parts.

6.6. Resource Requirements and Limits​

Note: Requests and Limits for resources are set per container in the pod.

6.6.1. Default Behaviour​

6.6.2. Resource Requests​

apiVersion: v1
kind: Pod
metadata:
 name: nginx
 labels:
  name: nginx
spec:
 containers:
 - name: nginx
   image: nginx
   ports:
   - containerPort: 8080
   resources:
     requests:
       memory: "4Gi"  
       cpu: 2                           # 1 CPU Count: 1 AWS vCPU, 1 GCP Core, 1 Azure Core, 1 Hyperthread

6.6.3. Resource Limits​

  • Limit = Requests if Request is not defined
apiVersion: v1
kind: Pod
metadata:
 name: nginx
 labels:
  name: nginx
spec:
 containers:
 - name: nginx
   image: nginx
   ports:
   - containerPort: 8080
   resources:
     requests:
       memory: "4Gi"  
       cpu: 1                           # 1 CPU Count: 1 AWS vCPU, 1 GCP Core, 1 Azure Core, 1 Hyperthread
    limits:
       memory: "8Gi"  
       cpu: 2

6.6.4. LimitRange​

  • Applies in Namespace level (Namespace level object)
# LimitRange CPU
apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-resource-constraint
spec:
  limits:
  - default: # this section defines default limits
      cpu: 500m
    defaultRequest: # this section defines default requests
      cpu: 500m
    max: # max and min define the limit range
      cpu: "1"
    min:
      cpu: 100m
    type: Container


# LimitRange Memory
apiVersion: v1
kind: LimitRange
metadata:
  name: memory-resource-constraint
spec:
  limits:
  - default:
      memory: 1Gi
    defaultRequest:
      memory: 1Gi
    max: 
      memory: 1Gi
    min:
      memory: 500Mi
    type: Container

6.6.5. ResourceQuota​

apiVersion: v1
kind: ResourceQuota
metadata:
  name: my-resource-quota
spec:
  hard:
    requests.cpu: 4
    requests.memory: 4Gi
    limit.cpu: 10
    limit.memory: 10Gi

6.7. DeamonSets​

  • DeamonSet ensures that one copy of pod is present in all nodes of cluster
  • Similar to ReplicaSet but deployes a pod on each nodes
  • A pod is deployed on new node and removed when node is removed
  • Use cases
    • Monitoring solution
    • Logs Viewer
    • Examples:
      • kube-proxy
      • weave-net (networking solution)
# deamon-set-definition.yaml

apiVersion: apps/v1
kind: DeamonSet
metadata:
  name: monitoring-deamon
spec:
  selector:
    matchLabels:
      app: monitoring-agent
  template:
    metadata:
      labels:
        app: monitoring-agent
    spec:
      containers:
      - name: monitoring-agent
        image: monitoring-agent

6.8. Static Pods​

6.8.1. What is Static Pods​

  • Static Pods are pods with kubelet without kube-apiserver or other k8s control plain components
  • kubelet can manage pod independently from kube-apiserver
  • Pod definition files must be provided to kubelet manually by putting them into /etc/kubernetes/manifests directory designated to store information about pods
  • kubelet periodically checks /etc/kubernetes/manifests
  • kubelet creates a pod and makes sure it stays alive and removes pods if definition file is removed from manifests directory
  • On Static Pods kubelet works in pod level and it is not possible to create ReplicaSets, Deployments or Services by copying definition files into manifest directory as it requires other control plane components, such as replication, deployment-controller etc.
  • --pod-manifest-path=/etc/kubernetes/manifests flag or --config=kubeconfig.yaml with content: staticPodPath: /etc/kubernetes/manifests is used to set a designated directory to be scanned by kubelet
  • Check running containers with: (i: no kubectrl util as kube-apiserver not available)
    • docker ps for Docker
    • crictl ps for cri-o
    • nerdctl ps for containerd

6.8.2. Static Pods in Cluster​

  • kubelet takes requests to create pod from different inputs:
    • from definition file from static pod folder
    • from http api endpoint (i.e. from kube-apiserver)
  • kubectl get pods will also list pods created statically but they can NOT be edited, deleted etc. from kube-apiserver
  • Static pods can be recognized by their name where node name is appended to it

6.8.3. Static Pods Use Case​

  • To deploy Control Plane components as Static Pods. E.g. kubeadm tool uses a mechanist of Static Pods to create a k8s cluster. E.g. by just putting controller-manager.yaml, apiserver.yaml, etcd.yaml into static pod folder and the k8s control plane components will be started as pods

6.8.4. Static Pods vs. DaemonSets​

Static PodsDaemonSets
Created by kubeletCreated kube-apiserver (DaemonSet Controller)
Deploy Control Plane components as Static PodsDeploy Monitoring Agents, Logging Agents on nodes
Ignored by kube-schedulerIgnored by kube-scheduler

6.9. Multiple Schedulers​

6.10. Configuring Scheduler Profiles​

6.11. Admission Controllers​

6.12. Validating and Mutating Admission Controllers​

6.12.1. Validating Admission Controllers​

  • Validates the request and allow/deny it

6.12.2. Mutating Admission Controller​

  • Mutates/Changes object before it is created (e.g. DefaultStorageClass)

6.12.3. Custom Validating/Mutating Admission Controllers​

  • MutatingAdmissionWebhook

  • ValidatingAdmissionWebhook

  • Webhooks above will reference Admission Webhook Server with custom logic

7. Logging and Monitoring

7.1. Monitor Cluster Components​

7.1.1. Tools​

  • Prometheus
  • DataDog
  • Elastic Stack
  • Dynatrace
  • Mentics Server: in-memory solution without storing data option.
    • kubelet contains sub-component called cAdvisor (i.e. container advisor) responsible for retrieving performance metrics and exposing them through kubelet API for Metrics Server.
    • Enable Metrics Service:
      • git clone https://github.com/kubernetes-incubator/metrics-server.git & kubectl create -f deploy/1.8+
    • Cluster Performance view:
      • kubectl top node
      • kubectl top pod

7.2. Managing Application Logs​

8. Application Lifecycle Management

8.1. Rolling Updates and Rollbacks​

8.1.1. Deployment Strategy​

  • Recreate: destroy all and create all
  • Rolling Update: destroy and create one-by-one (DEFAULT Deployment Strategy)

8.2. Configure Applications​

  • Configuring Command and Arguments on applications
  • Configure Environment Variables
  • Configure Secrets

8.2.1. Commands and Arguments in Docker and K8s​

  • Read more
  • Containers (unlike VMs) are not meant to host OS, but run specific process.
FROM nginx

ENTRYPOINT ["sleep"]
CMD ["5"]
apiVersion: v1
kind: Pod
metadata:
 name: nginx-sleeper-pod
spec:
 containers:
 - name: nginx
   image: nginx
   command: ["sleep"]                    # Overrides executables ENTRYPOINT in Dockerfile
   args: ["10"]                          # Overrides CMD defined after ENTRYPOINT as arguments in Dockerfile

8.2.2. Configuring Environment Variables in Applications​

apiVersion: v1
kind: Pod
metadata:
  name: envar-demo
spec:
  containers:
  - name: envar-demo-container
    image: nginx
    env:
    - name: MY_VAR
      value: "My Value"
# ENV Value Types

# 1. Plain Key Value
...
    env:
    - name: MY_VAR
      value: "My Value"
...

# 2. ConfigMap
...
    env:
    - name: MY_VAR
      valueFrom: 
        configMapKeyRef:
...

# 3. Secrets
...
    env:
    - name: MY_VAR
      valueFrom: 
        secretKeyRef:
...

8.2.3. Configuring ConfigMaps in Applications​

apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-demo
data:
  # property-like keys; each key maps to a simple value
  player_initial_lives: "3"
  ui_properties_file_name: "user-interface.properties"

  # file-like keys
  game.properties: |
    enemy.types=aliens,monsters
    player.maximum-lives=5    
  user-interface.properties: |
    color.good=purple
    color.bad=yellow
    allow.textmode=true  
apiVersion: v1
kind: Pod
metadata:
  name: env-configmap
spec:
  containers:
    - name: app
      command: ["/bin/sh", "-c", "printenv"]
      image: busybox:latest
      envFrom:
        - configMapRef:
            name: configmap-demo
# ConfigMaps in Pods

# 1. ENV
envFrom:
  - configMapRef:
      name: app-config

# 2. SINGLE ENV
env:
  - name: APP_COLOR
    valueFrom:
      configMapKeyRef:
        name: app-config
        key: APP_COLOR

# 3. VOLUME
volumes:
- name: app-config-value
  configMap:
    name: app-config

8.2.4. Configure Secrets an Applications​

8.2.4.1. Create Secrets​

  • Imperative approach - see Commands
  • Declarative approach
# kubectl create -f secret-data.yaml

# Secrets are only encoded, NOT Encryped
# echo -n 'password' | base64
# echo -n 'cGFzc3dvcmQ=' | base64 --decode

# secret-data.yaml

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
data:
  DB_User: root
  DB_Password: dmFsdWUtMg0KDQo=
---
apiVersion: v1
kind: Pod
metadata:
  name: sapmple-secret-pod
spec:
  containers:
    - name: test-container
      image: registry.k8s.io/busybox
      envFrom:
        - secretRef:
            name: app-secret                           # ref to secret name
# Secrets in Pods

# 1. ENV
envFrom:
  - secretRef:
      name: app-config

# 2. SINGLE ENV
env:
  - name: DB_password
    valueFrom:
      secretKeyRef:
        name: app-secret
        key: DB_password

# 3. VOLUME
volumes:
- name: app-secret-volume
  secret:
    secretName: app-secret

8.2.4.2. Encrypting Secret Data at Rest​

Secrets are not encrypted, so it is not safer in that sense. However, some best practices around using secrets make it safer. As in best practices like:

  • Not checking-in secret object definition files to source code repositories.
  • Enabling Encryption at Rest for Secrets so they are stored encrypted in ETCD.

8.2.4.3. Secret Store Providers​

  • Third-party secrets store providers: AWS Provider, Azure Provider, GCP Provider, Vault Provider (HashiCorp Vault)
  • Helm Secrets plugin

8.3. Scale Applications​

8.4. Multi Container Pods​

# pod-definition.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80
  - name: nginx-2
    image: nginx:1.14.2
    ports:
    - containerPort: 81

8.4.1. Multi-container Pods Design Patterns​

  • Sidecar (e.g. service and logging containers)
  • Adapter
  • Ambassador

8.5. InitContainers​

# pod-definition.yaml
# initContainer must run to a completion before the real container hosting the application starts. 

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox
    command: ['sh', '-c', 'git clone <some-repository-that-will-be-used-by-application> ; done;']

8.6. Self Healing Applications​

Kubernetes supports self-healing applications through ReplicaSets and Replication Controllers. The replication controller helps in ensuring that a POD is re-created automatically when the application within the POD crashes.

8.7. Intro to Autoscaling​

8.7.1. Scaling Cluster Infra vs Scaling Workloads​

  • Scaling Cluster Infra by increasing number of nodes horizontally or increasing size of resources (CPU, RAM) on existing node
  • Scaling Workloads by increasing the number of pods horizontally or increase resources allocated to existing pods

8.7.2. Manual vs Automated Scaling​

8.7.3. Horizontal Pod Autoscaler (HPA)​

  • Increate the running instances of application horizontally
  • HPA relies on
    • Metrics Server or Custom Metrics Adapter for internal sources
    • External Adapter for e.g. DataDog, Dynatrace etc.

8.7.3.1. Scaling a workload manually​

# Manually check the load and increase replicas

kubectl top pod my-app-pod
kubectl scale deployment my-app --replicas=3

8.7.3.2. HPA​

HPA:

  • Observes metrics
  • Adds pods
  • Balances thresholds
  • Tracks multiple metrics
# IMPERATIVE: Create HPA for specific deployment

kubectl autoscale deployment my-app \
          --cpu-percent=50 --min=1 --max=10                       # CPU thresholds of 50% and min 1 pod and max 10 pods
# DECLARATIVE: Create HPA for specific deployment

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

8.7.4. Vertical Pod Autoscaling (VPA)​

  • Manual:
    • kubectl edit deployment my-app and manually change resources
  • VPA:
    • Observe metrics
    • Adjust pod resources: e.g. increase the size of resources vertically, such as CPU, RAM
    • Balances threshold
apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
  name: stress-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: high-cpu-utilization-deployment
  updatePolicy:
    updateMode: Auto                # Modes: Off, Initial, Recreate and Auto         
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 50Mi
        maxAllowed:
          cpu: 200m                 # maximum vpa will be allocating this many cpus even if demand is higher.
          memory: 500Mi
        controlledResources: ["cpu", "memory"]

8.7.4.1. Deployment of VPA Components​

  • VPA does NOT come built-in with K8s, but must be deployed
    • kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vertical-pod-autoscaler.yaml
  • VPA includes following pods:
    • vpa recommender - continuously monitoring resource usage from Metrics Server and provides recommendation for optimal cpu, ram values
    • vpa updater - get an information from recommender and evicts pods needed to be updated with recommended resources
    • vpa admission controller - controller then creates new pod with new resources values recommended

8.7.5. In-place resize Pods​

  • As of kubernetes version 1.32, if the resource definition parameter is changed, the default behaviour is delete running instance of pod and spin up a new pod with changes
  • InPlacePodVerticalScaling: Resize CPU and Memory Resources assigned to Containers without restart
    • FEATURE STATE: K8s v1.27 [alpha][enabled by default:false]
    • Enable feature: FEaTURE_GATES:InPlacePodVerticalScaling=true
apiVersion: v1
kind: Pod
metadata:
  name: resize-demo
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.8
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired                   # Default, but explicit here
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      limits:
        memory: "200Mi"
        cpu: "700m"
      requests:
        memory: "200Mi"
        cpu: "700m"

8.7.5.1. Limitations​

  • Only CPU and Memory resources can be changed
  • Pod QoS Class cannot change
  • InitContainers and Ephemeral COntainers cannot be resized
  • Resource requests and limits cannot be removed once set
  • A container's memor limit may not be reduced below its usage. If a request pts a container in this state, the resize status will remain in InProgress until the resired memory limit becomes feasible.

8.7.6. VPA vs. HPA​

FeatureVPA (Vertical Scaling)HPA (Horizontal Scaling)
Scaling MethodIncrease CPU & memory of existing PodsAdds/Removes Pods based on load
Pods BehaviorRestarts Pods to apply new resource valuesKeeps existing Pods running
Handles Traffic Spikes?NO, because scaling requires a Pod restartYES, instantly adds more Pods
Optimizes Costs?Prevents over-provisioning of CPU/RAMAvoids unnecessary idle Pods
Best forStateful workloads, CPU/RAM-heavy appsStateless services, web apps, microservices
Use casesDBs, JVM-baseds apps, AI/ML workloadsWeb servers (Nginx, API services), message queues, microservices

9. Cluster Maintenance

9.1. OS Upgrade​

  • Drain and cordon nodes (see Command section)
  • Uncordon nodes (see Command section)

9.2. Kubernetes Software Versions​

Kubernetes version 1.27

  • kube-apiserver: v1.27.0
  • kube-scheduler: v1.27.0
  • kube-controller-manager: v1.27.0
  • kubelet: v1.27.0
  • kube-proxy: v1.27.0
  • etcd: v3.5.7 (or similar, depending on the Kubernetes version)
  • coredns: v1.10.1 (or similar, depending on the Kubernetes version)
  • container runtime: Versions can vary (e.g., containerd v1.6.0 or Docker v20.10.0)

9.3. Cluster Upgrade Process​

  • Kubenrnetes Supports only 3 last versions

  • It is recommended to upgrade one MINOR version at a time.

  • Upgrade possibilies of different versions.

    • kube-apiserver: vX
    • kube-controller-manager: vX-1
    • kube-scheduler: vX-1
    • kubelet: vX-2
    • kube-proxy: vX-2
    • kubectl: vX+1, vX, vX-1

  • Upgrade using kubeadm

    • kubeadm upgrade plan
    • kubeadm upgrade apply

  • Upgrade Sequence:

    • First Master Node
    • Then Worker Nodes

  • Upgrade Strategies of Worker Nodes

    • Upgrade ALL of Worker Nodes at once
    • Upgrade ONE node as a time
    • Add NEW node with new k8s version. I.e. provision new nodes and decomission old one.

9.3.1. kubeadm upgrade​

Upgrade Master Node:

  • kubeadm upgrade plan # kubeadm does not upgrade kubelet
  • apt-get upgrade -y kubeadm=1.12.0-00
  • kubeadm upgrade apply v1.12.0
  • apt-get upgrade -y kubelet=1.12.0-00
  • systemctl restart kubelet

Upgrade Worker Nodes:

  • kubectl drain node-1
  • apt-get upgrade -y kubeadm=1.12.0-00
  • apt-get upgrade -y kubelet=1.12.0-00
  • kubeadm upgrade node config --kubelet-version 1.12.0
  • systemctl restart kubelet
  • kubectl get nodes
  • kubectl uncordon node-1

9.3.2. Cluster Upgrade - Demo​

9.3.3. Backup and Restore Methods​

9.3.3.1. Backup Candidates​

  • Resource Configurations (namespace, configmap, secrets etc.)
    • kubectl get all --all-namespaces -o yaml > all-resources.yaml
  • ETCD Cluster (Stores information about the state of k8s cluster)
    • ETCD Cluster is hosted on Master Nodes
    • ETCD Cluster data dir to backup --data-dir=/var/lib/etcd
    • Create a Snapshot of ETCD with
      ETCDCTL_API=3 etcdctl snapshot save etcd_snapshot.db
    • Restore ETCD from Backup 
      service kube-apiserver stop
      ETCDCTL_API=3 etcdctl snapshot restore etcd_snapshot.db --data-dir /var/lib/etcd-from-backup
      systemctl daemon-reload
      service etcd restart
      service kube-apiserver start
    • Read more about backup tools
  • Persistence Volumes

10. Security

10.1. Kubernetes Security Primitives​

10.1.1. Cluster Hosts Security​

  • All access to cluster host must be secured
  • root access disabled
  • Password based authentication disabled
  • SSH Key based authentication enabled

10.1.2. Kubernetes Security​

  • Securing the access to kube-apiserver is the first line of diffence
    • Authentication, i.e. who can access it?
      • Files - Username and Passwords
      • Files - Username and Tokens
      • Certificates
      • External Authentication providers - LDAP
      • Service Accounts
    • Authorization, i.e. what can they do?
      • RBAC Authorization
      • ABAC Authorization
      • Node Authorization
      • Webhook Mode

10.1.3. TLS Certificates​

  • All communication between kube-apiserver and
  • ETCD Cluster
  • Kubelet
  • Kube Proxy
  • Kube Scheduler
  • Kube Controller Manager is secured TLS Encryption.

10.1.4. Application Communication within Cluster​

  • By default all Pods can access other Pods within the Cluster
  • The Pods access by another Pods can be restricted by Network Policies

10.2. Authentication​

  • Users who may access K8s Cluster
    • Users
      • Admins
      • Developers
    • Service Accounts
      • Bots (CI/CD etc.)
    • Application End Users (Usually managed by Application Authentication Management)

10.2.1. Users​

All user access are managed by kube-apiserver whether access with kubectl or curl https://kube-server-ip:6443/ by

  • First Authenticate Users
  • Then Process Requests

kube-apiserver Authentication Mechanisms:

  • Static Password File (not recommended)
  • Static Token File (not recommended)
  • Certificates
  • Identity Services (e.g. third Party Auth protocol LDAP, Cerberus etc.)

10.2.1.1. Basic Authentication Mechanism - Static Password File​

# basic-auth.csv
password123,user1,u001,group1
password123,user2,u002,group2
password123,user3,u003,group3
kube-apiserver --basic-auth-file=/path/to/basic-auth.csv
systemctl restart kube-apiserver

Then kube-apiserver can be access with

curl -v -k https://master-node-ip:6443/api/v1/pods -u "user1:password123"

10.2.1.2. Basic Authentication Mechanism - Static Token File​

# basic-token-auth.csv
a3f5c8e1d2b4a6e7,user1,u001,group1
9b1e2c3d4f5a6b7c,user2,u002,group2
7e8f9a0b1c2d3e4f,user3,u003,group3
kube-apiserver --token-auth-file=/path/to/basic-token-auth.csv
systemctl restart kube-apiserver

Then kube-apiserver can be access with

curl -v -k https://master-node-ip:6443/api/v1/pods --header "Authorization: Bearer a3f5c8e1d2b4a6e7"

10.2.2. Service Accounts​

  • kubectl create serviceaccount sa1

10.3. TLS Certificates in K8s​

10.3.1. TLS Basics​

Transport Layer Security (TLS) is primarily associated with securing communications over the internet and it is used in various applications and protocols, like: SSH, HTTPS, Email Protocols(SMTP, IMAP), FPN, FTPS, APIs(RESTful APIs), WebSocket Secure (WSS), DB Connections etc.

  • TLS certificates are digital certificates that authenticate the identity of a website or server and enable encrypted communication between clients and servers over the internet, ensuring data privacy and integrity during transmission.
  • Key Pair Generation: When you create a key pair, a cryptographic algorithm (like RSA or ECC) generates two keys: a private key and a public key. These keys are mathematically linked.
  • Private Key: is used to decrypt information or sign data and is kept secret.
  • Public Key: is used to encrypt information or verify signatures and can be shared with anyone.
  • Certificate Signing Request (CSR): includes the public key along with information about the entity requesting a certificate (e.g. organization’s name, domain). When you generate a CSR, you create it using your public key.
  • Certificate (CRT or Digital Certificate): contains the public key along with information about the certificate holder (e.g. their identity, domain). It is issued by a Certificate Authority (CA) after verifying the information in the CSR. The CRT allows others to trust that the public key belongs to the entity it claims to represent.
  • Digital Signature: When you sign data with your private key, others can use your public key to verify that the signature is valid. This ensures that the data hasn’t been tampered with and confirms the identity of the sender.
  • Certificate Authority (CA): An entity that issues digital certificates, verifying the identity of the certificate holder and ensuring the integrity of the public key.

10.3.1.1. Private vs Public Key​

  • Key Pair
  • Data can be encrytped with ANY of them and ONLY decrypted with other
  • Data can NOT be encrypted with one and descrypted with the same key
  • If data is encrypted with Private Key, anyone who has Public Key can decrypt data!
  • Signing is done with a private key and Verification is done with a public key
  • Encryption is done with a public key and Decryption is done with a private key

10.3.1.2. Symmetric Encryption​

Symmetric encryption is a cryptographic method where the same key is used for both encrypting and decrypting data, ensuring that only parties with the shared key can access the original information.

10.3.1.3. Asymmetric Encryption​

Asymmetric encryption is a cryptographic method that uses a pair of keys—a public key for encryption and a private key for decryption—allowing secure communication where the public key can be shared openly while the private key remains confidential.

10.3.1.4. Asymmetric Encryption - SSH​

Asymmetric encryption in SSH (Secure Shell) uses a pair of keys—a public key and a private key—to secure communications between a client and a server. The public key is shared with the server, while the private key remains confidential on the client.

10.3.1.5. Asymmetric Encryption - HTTPS​

In HTTPS (Hypertext Transfer Protocol Secure), both symmetric and asymmetric encryption are used to secure communications between a web browser and a server:

  • Asymmetric Encryption: Initially used during the SSL/TLS handshake process to establish a secure connection. The server presents its public key to the client, allowing the client to encrypt a shared secret (session key) that only the server can decrypt with its private key. This ensures that the client is communicating with the legitimate server.
  • Symmetric Encryption: Once the secure connection is established and the session key is shared, symmetric encryption is used for the actual data transmission. This is because symmetric encryption is faster and more efficient for encrypting large amounts of data compared to asymmetric encryption.

10.3.1.6. Certificate Authority (CA)​

  • Public CAs have also Private and Public Keys

    • Private CA Service can be hosted within ones infrastructure

  • CA uses Private Key to sign Certificates

  • Public Keys of CAs are available in e.g. Browsers (Trusted Root Certification Authorities)

  • Browser uses Public Key of CA to verify that a specific Certificate was signed with Private Key of Official CA

  • Symantec

  • Comodo

  • GlobalSign

  • Digicert

10.3.1.7. Naming Convention of Public and Private Keys​

  • Certificate (Public Key)
    • *.crt, *.pem:
      • server.crt
      • server.pem
      • client.crt
      • client.pem
  • Private Key
    • *.key, *-key.pem:
      • server.key
      • server-key.pem
      • client.key
      • client-key.pem

10.3.1.8. TLS Use Cases​

  • Admins generate Key Pairs to secure SSH
  • Web Server has Key Pairs to secure Website with HTTPS
  • CA Generates its own Key Pairs to sign/verify Certificates
  • End Users/Browser only generates single Symetric Key

10.3.2. TLS in Kubernetes​

10.3.3. TLS in Kubernetes - Certificate Creation​

openssl s_client -connect <ip-of-host>

11. Storage

12. Networking

13. Design and Install a Kubernetes Cluster

14. Install "K8s the kubeadm way"

15. Helm Basics

16. Kustomize Basics

17. End to End tests on a K8s Cluster

18. Troubleshooting

19. Other Topics

20. Ligtning Labs

21. Mock Exams

22. Course Conclusion

23. Commands

# GET

kubectl get all

kubectl get pods --watch
kubectl get pods -n kube-system
kubectl get pods -o wide

kubectl get pods -l environment=production,tier=frontend
kubectl get pods -l 'environment in (production),tier in (frontend)'
kubectl get pods -l 'environment,environment notin (frontend)'

kubectl get pods --selector app=App1
kubectl get pods --selector app=App1 --no-headers | wc -l

kubectl get pod <pod-name> --all-namespaces

kubectl get replicationcontroller
kubectl get replicaset
kubectl get replicaset myapp-rs -o yaml

kubectl get deployments

kubectl get services

kubectl get configmaps

kubectl get secrets
kubectl get secret app-secret -o yaml

kubectl get hpa

kubectl get all --all-namespaces -o yaml > all-resources.yaml                 # Backup all resource configurations

kubectl get serviceaccount





# DESCRIBE

kubectl describe pod myapp-pod
kubectl describe node kubemaster






# APPLY

kubectl apply -f object-definition-file.yaml                                   # Create or Update (DEFAULT Rolling Update) k8s object 
kubectl apply -f /path/to/object-definition-files





# CREATE

kubectl create deployment --image=nginx nginx --namespace=dev
kubectl create deployment my-deployment --image=nginx nginx --replicas=3 --dry-run=client -o yaml

kubectl create service nodeport myapp-service --tcp=8080:8080 --dry-run=client -o yaml

kubectl create namespace dev

kubectl create configmap <config-name> --from-literal=APP_COLOR=blue
kubectl create configmap <config-name> --from-file=app_config.properties

kubectl create secret generic <secret-name> --from-literal=<key>=<value>
kubectl create secret generic <secret-name> --from-file=<path-to-file>

kubectl create serviceaccount sa1



# EXEC

kubectl exec etcd-master -n kube-system etcdctl get / --prefix -keys-only





# RUN

kubectl run nginx --image=nginx --namespace=default --dry-run=client -o yaml
kubectl run nginx --image=nginx -- <arg1> <arg2> ... <argN>
kubectl run nginx --image=nginx --command -- <cmd> <arg1> <arg2> ... <argN>






# EDIT

kubectl edit replicaset






# LABEL

kubectl label nodes <node-name> <label-key>=<label-value>





# REPLACE

kubectl replace --force -f replicaset-definition.yaml





# SCALE

kubectl scale --replicas=6 -f replicaset-definition.yaml
kubectl scale --replicas=6 -f replicaset myapp-rs





# AUTOSCALE 

kubectl autoscale deployment foo --cpu-percent=50 --min=1 --max=10                # Create HPA for a deployment "foo" with CPU thresholds of 50% and min 1 pod and max 10 pods




# DELETE

kubectl delete replicaset myapp-replicaset               # deletes all underlying Pods 





# CONFIG

kubectl config set-context $(kubectl config current-context) --namespace=dev
kubectl config use-context <cluster-name>





# TAINT

kubectl taint nodes <NODE-NAME> <KEY>=<VALUE>:<TAINT-EFFECT>                    # TAINT-EFFECTs: NoSchedule, PreferNoSchedule, NoExecute
kubectl taint nodes <NODE-NAME> <KEY>=<VALUE>:<TAINT-EFFECT>-                   # UNTAINT




# LOGS

kubectl logs -f <pod-name>
kubectl logs -f <pod-name> [container-name]




# AUTH

kubectl auth can-i --list                                                      # check current user permissions
kubectl auth can-i <verb> <resource> -n <namespace>                            # check if the current user can perform a specific action on a resource
kubectl auth can-i create pods -n <namespace>                                  




# ROLLOUT

kubectl rollout status deployment/<depoyment-name>                                 # Check the status of a rollout
kubectl rollout history deployment/<depoyment-name>                                # Show the history of rollouts
kubectl rollout undo deployment/<depoyment-name> --to-revision=<revision_number>   # Undo a rollout to the specific revision




# DRAIN

kubectl drain <node-name>                                   # drain the node (will be marked as unschedulable) for maintenance purpose to gracefully terminate pods running on it and to reschedule them. 




# CORDON

kubectl cordon <node-name>                                  # cordon node (mark as unschedulable) so that no new pods are scheduled on it




# UNCORDON

kubectl uncordon <node-name>                                # uncordon the node so that pods can be scheduled on it after drain




# KUBE-CONTROLLER-MANAGER

kube-controller-manager --pod-eviction-timeout=5m0s                          # set time to wait pods to be evicted. I.e. control plane waits 5 minutes before considering the pod as dead and redeploy it




# ETCDCTL

ETCDCTL_API=3 etcdctl member list
ETCDCTL_API=3 etcdctl snapshot save etcd_snapshot.db \
                      --endpoints=https://127.0.0.1:2379 \
                      --cacert=/etc/etcd/ca.crt \
                      --cert=/etc/etcd/etcd-server.crt \
                      --key=/etc/etcd/etcd-server.key
ETCDCTL_API=3 etcdctl snapshot status etcd_snapshot.db
ETCDCTL_API=3 etcdctl snapshot restore etcd_snapshot.db --data-dir /var/lib/etcd-from-backup

24. References