1. Table of Content
- 1. Table of Content
- 2. Intro
- 3. Certification
- 4. CKA Lecture and Lab Notes
- 5. Core Concepts
- 5.1. Kubernetes Architecture
- 5.2. Open Container Initiative (OCI)
- 5.3. Container Runtime Interface (CRI)
- 5.4. Docker vs ContainerD
- 5.5. ETCD
- 5.6. Kube-API Server
- 5.7. Kube Controller Manager
- 5.8. Kube Scheduler
- 5.9. Kubelet
- 5.10. Kube Proxy
- 5.11. Pods
- 5.12. ReplicaSets
- 5.13. Labels and Selectors
- 5.14. Annotations
- 5.15. Deployments
- 5.16. Services
- 5.17. Namespaces
- 5.18. Imperative vs. Declarative
- 6. Scheduling
- 6.1. Manual Scheduling
- 6.2. Taints and Tolerations
- 6.3. Node Selectors
- 6.4. Node Affinity
- 6.5. Taints and Tolerations vs Node Affinity
- 6.6. Resource Requirements and Limits
- 6.7. DeamonSets
- 6.8. Static Pods
- 6.9. Multiple Schedulers
- 6.10. Configuring Scheduler Profiles
- 6.11. Admission Controllers
- 6.12. Validating and Mutating Admission Controllers
- 7. Logging and Monitoring
- 8. Application Lifecycle Management
- 8.1. Rolling Updates and Rollbacks
- 8.2. Configure Applications
- 8.3. Scale Applications
- 8.4. Multi Container Pods
- 8.5. InitContainers
- 8.6. Self Healing Applications
- 8.7. Intro to Autoscaling
- 9. Cluster Maintenance
- 10. Security
- 10.1. Kubernetes Security Primitives
- 10.2. Authentication
- 10.3. TLS Certificates in K8s
- 10.4. Certificate API
- 10.5. KubeConfig
- 10.6. Persistent Key/Value Store
- 10.7. API Group
- 10.8. Authorization
- 10.9. Service Accounts
- 10.10. Image Security
- 10.11. Docker Security
- 10.12. Security Context
- 10.13. Network Policies
- 10.14. Kubectx and Kubens
- 10.15. Custorm Resource Definition (CRD)
- 10.16. Custom Controllers
- 10.17. Operator Framework
- 11. Storage
- 12. Networking
- 13. Design and Install a Kubernetes Cluster
- 14. Install "K8s the kubeadm way"
- 15. Helm Basics
- 16. Kustomize Basics
- 17. End to End tests on a K8s Cluster
- 18. Troubleshooting
- 19. Other Topics
- 20. Ligtning Labs
- 21. Mock Exams
- 22. Course Conclusion
- 23. Commands
- 24. References
2. Intro
This document contains my notes on preparation course of Certified Kubernetes Administrators (CKA) Certification by Mumshad Mannambeth.
3. Certification
4. CKA Lecture and Lab Notes
5. Core Concepts
5.1. Kubernetes Architecture
- Worker Nodes: hosts application as containers
- Worker Nodes Components:
- kubelet: is an agent which runs on Worker Nodes
- kube-proxy: is a service responsible for networking on Worker Nodes
- Worker Nodes Components:
- Master Node: manage, plan schedule, monitor etc. nodes using Control Plane Components
- Control Plane Components:
- kube-apiserver: is responsible for orchestration of all operations within k8s cluster Control Plane Components
- ETCD Cluster: is a database which stores information in a key-value format
- kube-scheduler: identifies right node based on containers resource requirements
- Controller-Manager
- Node-Controller
- Replication-Controller
- Control Plane Components:
5.2. Open Container Initiative (OCI)
- imagespec: specifications how container image must be built
- runtimespec: specifications how conainer runtime should be developed
5.3. Container Runtime Interface (CRI)
- Kubernetes introduced Container Runtime Interface (CRI) which allows any vendor to work as container runtime as long as it adheres to OCI specification and not only Docker which used initially dockershim.
- Starting from version v1.24 Kubernetes deleted dockershim and Docker support
5.4. Docker vs ContainerD
5.4.1. Docker
- Docker is not only container runtime, but also consist of multiple tools
- CLI
- API
- BUILD
- Volumes
- AUTH
- SECURITY
- ContainerD
- Docker uses container runtime ContainerD
5.4.2. ContainerD
- ContainerD is part of Cloud Native Computing Foundation (CNCF)
- ContainerD is a higher-level tool that manages the entire lifecycle of containers.
- ContainerD is a deamon which manages runc and compatable with Container Runtime Interface (CRI)
- runc is a container runtime
- ContainerD works as container runtime on its own separate from Docker
- ContainerD comes with
crt
CLI tool mainly made for debugging container.nerdctl
(similar to docker CLI) provides stable and humand-friendly user experience crictl
is another CLI tool from k8s community compatable with CRI which works accross different container runtime. Mainly used for debugging- Read more
5.5. ETCD
- ETCD is a distributed reliable key-value store that is simple, secure and fast
- Key-Value Store stores information if form of documents, pages (i.e. files) which can be in any format and structure
5.5.1. Get started quickly with ETCD
-
Download the relevant binary for your operating system from github releases page (https://github.com/etcd-io/etcd/releases)
For Example: To download ETCD v3.5.6, run the below curl command
$ curl -LO https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz
-
Extract it.
$ tar xvzf etcd-v3.5.6-linux-amd64.tar.gz
-
Run the ETCD Service
$ ./etcd
-
When you start
ETCD
it will by default listen on port2379
-
The default client that comes with
ETCD
is theetcdctl
client. You can use it to store and retrieve key-value pairs.Syntax: To Store a Key-Value pair
$ ./etcdctl put key1 value1Syntax: To retrieve the stored data
$ ./etcdctl get key1Syntax: To view more commands. Run etcdctl without any arguments
$ ./etcdctl
# etcdctl command in API 3 version
etcdctl snapshot save
etcdctl endpoint health
etcdctl get
etcdctl put
export ETCDCTL_API=3 # set the environment variable ETCDCTL_API for desired API version
# Certs for ETCDCTL to authenticate to the ETCD API Server
--cacert /etc/kubernetes/pki/etcd/ca.crt
--cert /etc/kubernetes/pki/etcd/server.crt
--key /etc/kubernetes/pki/etcd/server.key
# Example:
kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl get / --prefix --keys-only --limit=10 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key"
5.5.2. ETCD in Kubernetes
-
The ETCD Datastore stores following cluster information:
- Nodes
- PODS
- Configs
- Secrets
- Accounts
- Roles
- Bindings
- Others
-
kubectl get
command retrieves all information from the ETCD Server
5.5.3. ETCD Setup - Manual vs kubeadm
- Read more
5.6. Kube-API Server
- Kube-API Server is a primary management component in K8s
- kube-api server performs following actions when
kubectl
command is executed:- Authenticate User
- Validate Request
- Retrieve dataf
- Read more
5.7. Kube Controller Manager
- Kube Controller Manager manages various controllers in kubernetes
- Controller is a process that continuously monitors the state of the components within the system and ensures the whole system to have the desired functioning state
- Controllers:
- Node Controller - monitors health of Nodes every
5s
Node Monitor Period = 5s
Node Monitor Grace Period = 40s
(i.e. waits 40s before marking it as unreachable)POD Eviction Timeout = 5m
(i.e. waits 5m to come back up if unreachable before removing pods and provisioning resources to the healthy nodes if it is part of replica set)
- Replication Controller - monitors state of replica sets
- Deployment Controller
- Namespace Controller
- Endpoint-Controller
- CronJob
- Job Controller
- PV Protection Controller
- Service Account Controller
- Stateful Set
- Replica Set
- PV Binder Controller
- Node Controller - monitors health of Nodes every
5.7.1. Installation of Kube-Controller-Manager
- Read more
5.8. Kube Scheduler
- kube-scheduler is responsible for scheduling pods on nodes. The kube-scheduler is only responsible for deciding which pod goes on which node depending on certain criteria (e.g. Filter Nodes, Rank Nodes). It doesn't actually place the pod on the nodes, that's the job of the
kubelet
. - Read more
5.9. Kubelet
- kubelet is a node agent which is responsible for creating the pods on the nodes and the sole point of contact for the kubernetes cluster
- kubelet also collects reports and sends regularly to master node
5.10. Kube Proxy
- POD Network is an internal virtual network that spans across all nodes in the cluster to which all pods are connected. Within k8s cluster every pod can reach every other pod which is accomplished by deploying networking solution, i.e. POD Network.
- kube-proxy is a network component in Kubernetes that helps manage network communication to and from the Pods in a k8s cluster. Key function of kube-proxy:
- Traffic Routing
- Load Balancing
- Service Discovery
- Protocol Support
5.11. Pods
- Pod is a logical host for one or more containers that are deployed together on the same host
# Example of Pod definition with YAML
apiVersion: v1
kind: Pod
metadata: # dictionary
name: myapp-pod
namespace: dev
labels: # under labels custom key:value allowed
app: myapp
type: front-end
spec: # dictionary
containers: # list/array
- name: nginx-controller
image: nginx
5.12. ReplicaSets
- ReplicaSet and ReplicationController (deprecated in favor of ReplicaSet) are both Kubernetes resources for managing pod replication to ensure a specified number of pod replicas running at any given time
# Example of ReplicationController definition with YAML
apiVersion: v1
kind: ReplicationController
metadata: # dictionary
name: myapp-rc
namespace: dev
labels: # under labels custom key:value allowed
app: myapp
type: front-end
spec: # dictionary
template: # Pod template for replication
metadata:
name: myapp-pod
labels:
app: myapp
type: front-end
spec:
containers:
- name: nginx-controller
image: nginx
replicas: 3
# Example of ReplicaSet definition with YAML
apiVersion: apps/v1
kind: ReplicaSet
metadata: # dictionary
name: myapp-rs
namespace: dev
labels: # under labels custom key:value allowed
app: myapp
type: front-end
spec: # dictionary
template: # Pod template for replication
metadata:
name: myapp-pod
labels:
app: myapp
type: front-end
spec:
containers:
- name: nginx-controller
image: nginx
replicas: 3
selector: # Major difference from ReplicationController
matchLabels:
type: front-end # To select all existing Pods matching defined label
5.13. Labels and Selectors
- Labels are key/value pairs that are attached to objects such as Pods for categorizing resources.
- Selectors are queries to filter resources based on labels.
5.14. Annotations
- Annotations are key-value pairs that can be attached to various Kubernetes objects (e.g. pods, services, deployments etc.) to store arbitrary metadata that is not used for identification or selection.
- Usage of Annotations:
- Storing build or version information
- Providing links to documentation or related resources
- Specifying configuration options for tools that interact with the resource (e.g., deployment tools, monitoring systems)
5.15. Deployments
- Deployment is a higher-level abstraction that manages a ReplicaSet, which in turn manages the Pods.
# Example of Deployment definition with YAML
apiVersion: apps/v1
kind: Deployment
metadata: # dictionary
name: myapp-rs
namespace: dev
labels: # under labels custom key:value allowed
app: myapp
type: front-end
spec: # dictionary
template: # Pod template for replication
metadata:
name: myapp-pod
labels:
app: myapp
type: front-end
spec:
containers:
- name: nginx-controller
image: nginx
replicas: 3
selector:
matchLabels:
type: front-end # To select all existing Pods matching defined label
5.16. Services
- Service is used to expose a set of Pods and enable communication between them
- Services enable communication between different components of an application, allowing Pods to discover and interact with each other reliably
- There are different types of Services, each serving a specific purpose
- Service acts as built-in load balancer to distribute load across different Pods (
Algorithm: Random, SessionAffinity: Yes
)
5.16.1. ClusterIP
- This is the default service type
- It exposes the service on a cluster-internal IP
- Only accessible from within the cluste
- Useful for internal communication between services
# Example of Service of type ClusterIP definition with YAML
apiVersion: v1
kind: Service
metadata: # dictionary
name: back-end
namespace: dev
spec:
type: ClusterIP
ports:
- port: 80 # The port that the service will expose
targetPort: 8080 # The port on the pod that the service should forward to
selector:
app: myapp
type: back-end
5.16.2. NodePort
- Service of type NodePort listens on the static port of each node's IP address in cluster and forward request to the Pods
- Node's port range
30 000 to 32 767
# Example of Service of type NodePort definition with YAML
apiVersion: v1
kind: Service
metadata: # dictionary
name: myapp-service
namespace: dev
spec:
type: NodePort
ports:
- port: 80 # The port that the service will expose
targetPort: 8080 # The port on the pod that the service should forward to
nodePort: 30008 # The port on each node that will forward to the service
selector:
app: myapp
type: front-end
5.16.3. LoadBalancer
- Creates an external load balancer (if supported by the cloud provider)
- Automatically assigns a public IP address to the service
- Distributes incoming traffic across multiple nodes
- Ideal for production environments where you need to handle large amounts of traffic
# Example of Service of type LoadBalancer definition with YAML
apiVersion: v1
kind: Service
metadata: # dictionary
name: myapp-service
namespace: dev
spec:
type: LoadBalancer
ports:
- port: 80 # The port that the service will expose
targetPort: 8080 # The port on the pod that the service should forward to
nodePort: 30008 # The port on each node that will forward to the service
5.16.4. ExternalName
5.17. Namespaces
- Namespace is a way to organize and manage resources within a cluster which provides a mechanism for isolating groups of resources
- Key Points of Namespaces:
- Isolation
- Resource Management
- Access Control
- Resource Quotas
- Network Policies
- Scoped Resources
Info: A pattern to access a service in a different namespace using DNS:
<service-name>.<namespace>.svc.cluster.local
<service-name>
: name of the service to access<namespace>
: namespace where the service is locatedsvc
: indicates that a service being accessedcluster.local
: default domain name for services in the k8s cluster (can be customized)
# Example of Namespace definition with YAML
apiVersion: v1
kind: Namespace
metadata:
name: dev
5.17.1. Resource Quota
# Example of ResourceQuota definition with YAML
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
namespace: dev
spec:
hard:
pods: "10"
requests.cpu: "4"
requests.memory: "5Gi"
limits.cpu: "10"
limits.memory: "10Gi"
5.18. Imperative vs. Declarative
In the context of Infrastructure as Code (IaC), imperative and declarative are two different approaches to defining and managing infrastructure.
- In an imperative approach, you specify how to achieve a desired state by providing a sequence of commands or instructions.
- In a declarative approach, you specify what the desired state of the infrastructure should be, without detailing the steps to achieve that state.
# IMPERAIVE APPROACH
# create objects
kubectl run --image=nginx nginx
kubectl create deployment --image=nginx nginx
kubectl expose deployment nginx --port 80
# update objects
kubectl edit deployment nginx
kubectl scale deployment nginx --replicas=5
kubectl set image deployment nginx nginx=nginx:1.18
kubectl create -f nginx.yaml
kubectl replace -f nginx.yaml
kubectl delete -f nginx.yaml
# DECLARATIVE APPROACH
kubectl apply -f object-definition-file.yaml
6. Scheduling
- Scheduling is a process of assigning pods (the smallest deployable units in Kubernetes) to nodes (the machines or virtual machines in the cluster) based on resource availability and other constraints.
6.1. Manual Scheduling
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 8080
nodeName: node02 # Specifies the name of Node, by default is not set
6.1.1. Binding Object
apiVersion: v1
kind: Binding
metadata:
name: my-pod
namespace: default
target:
apiVersion: v1
kind: Node
name: my-node
6.2. Taints and Tolerations
- Taints are applied to nodes to prevent certain pods from being scheduled on them
- Tolerations are applied to pods to allow them to be scheduled on nodes with specific taints
kubectl taint nodes <NODE-NAME> <KEY>=<VALUE>:<TAINT-EFFECT>
# TAINT-EFFECT defines what happens to Pods that do not TOLERATE this taint.
# TAINT-EFFECTs: NoSchedule, PreferNoSchedule, NoExecute
# TAINTING NODE EXAMPLE
kubectl taint nodes node1 app=blue:NoSchedule
# TOLERATION DEFINITION OF POD EXAMPLE
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 8080
tolerations:
- key: "app"
operator: "Equal"
value: "blue"
effect: "NoSchedule"
6.3. Node Selectors
- Label a node:
kubectl label nodes node01 size=Large
- Assign a Pod to the Node
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
spec:
containers:
- name: nginx
image: nginx
nodeSelector:
size: Large
- Node Selector's Limitations:
- Advanced expressions, e.g. Large OR Medium, NOT Small etc.
6.4. Node Affinity
Node affinity is conceptually similar to nodeSelector
, allowing you to constrain which nodes your Pod can be scheduled on based on node labels.
apiVersion: v1
kind: Pod
metadata:
name: with-node-affinity
spec:
containers:
- name: nginx
image: nginx
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: size
operator: In # operators: In, NotIn, Exists, DoesNotExist
values:
- Large
- Medium
6.4.1. Node Affinity Types
- Available:
requiredDuringSchedulingIgnoredDuringExecution
preferredDuringSchedulingIgnoredDuringExecution
- Planned:
requiredDuringSchedulingRequiredDuringExecution
Two states in the lifecycle of the Pod considering Node Affinity:
- DuringScheduling is a state when Pod does not exists and is created for the first time. If reqired then a node with specified lable must exist in order the Pod to be scheduled.
- DuringExecution is a state where the Pod is running and the change made in environment which affects Node Affinity, e.g. change in the label of the Node. If ignored, the Pod will continue running if node label is changed or removed.
Read more.
6.5. Taints and Tolerations vs Node Affinity
Taints and Tolerations: Taints are applied to nodes to repel certain pods unless those pods have matching tolerations. This mechanism is useful for reserving nodes for specific workloads. Node Affinity: Node affinity allows you to specify rules based on node labels that determine which nodes a pod can be scheduled on. It helps ensure that pods are placed on nodes with specific characteristics.
- Taints and Tolerations do not guarantee that the pods will only prefer these nodes.
- As such, a combination of taints and tolerations and node affinity rules can be used together to completely dedicate nodes for specific parts.
6.6. Resource Requirements and Limits
Note: Requests and Limits for resources are set per container in the pod.
6.6.1. Default Behaviour
6.6.2. Resource Requests
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 8080
resources:
requests:
memory: "4Gi"
cpu: 2 # 1 CPU Count: 1 AWS vCPU, 1 GCP Core, 1 Azure Core, 1 Hyperthread
6.6.3. Resource Limits
- Limit = Requests if
Request
is not defined
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 8080
resources:
requests:
memory: "4Gi"
cpu: 1 # 1 CPU Count: 1 AWS vCPU, 1 GCP Core, 1 Azure Core, 1 Hyperthread
limits:
memory: "8Gi"
cpu: 2
6.6.4. LimitRange
- Applies in Namespace level (Namespace level object)
# LimitRange CPU
apiVersion: v1
kind: LimitRange
metadata:
name: cpu-resource-constraint
spec:
limits:
- default: # this section defines default limits
cpu: 500m
defaultRequest: # this section defines default requests
cpu: 500m
max: # max and min define the limit range
cpu: "1"
min:
cpu: 100m
type: Container
# LimitRange Memory
apiVersion: v1
kind: LimitRange
metadata:
name: memory-resource-constraint
spec:
limits:
- default:
memory: 1Gi
defaultRequest:
memory: 1Gi
max:
memory: 1Gi
min:
memory: 500Mi
type: Container
6.6.5. ResourceQuota
apiVersion: v1
kind: ResourceQuota
metadata:
name: my-resource-quota
spec:
hard:
requests.cpu: 4
requests.memory: 4Gi
limit.cpu: 10
limit.memory: 10Gi
6.7. DeamonSets
- DeamonSet ensures that one copy of pod is present in all nodes of cluster
- Similar to
ReplicaSet
but deployes a pod on each nodes - A pod is deployed on new node and removed when node is removed
- Use cases
- Monitoring solution
- Logs Viewer
- Examples:
- kube-proxy
- weave-net (networking solution)
# deamon-set-definition.yaml
apiVersion: apps/v1
kind: DeamonSet
metadata:
name: monitoring-deamon
spec:
selector:
matchLabels:
app: monitoring-agent
template:
metadata:
labels:
app: monitoring-agent
spec:
containers:
- name: monitoring-agent
image: monitoring-agent
6.8. Static Pods
6.8.1. What is Static Pods
Static Pods
are pods withkubelet
withoutkube-apiserver
or other k8s control plain componentskubelet
can manage pod independently fromkube-apiserver
- Pod definition files must be provided to
kubelet
manually by putting them into/etc/kubernetes/manifests
directory designated to store information about pods kubelet
periodically checks/etc/kubernetes/manifests
kubelet
creates a pod and makes sure it stays alive and removes pods if definition file is removed from manifests directory- On
Static Pods
kubelet
works in pod level and it is not possible to createReplicaSets
,Deployments
orServices
by copying definition files into manifest directory as it requires other control plane components, such asreplication
,deployment-controller
etc. --pod-manifest-path=/etc/kubernetes/manifests
flag or--config=kubeconfig.yaml with content: staticPodPath: /etc/kubernetes/manifests
is used to set a designated directory to be scanned bykubelet
- Check running containers with: (i: no
kubectrl
util askube-apiserver
not available)docker ps
forDocker
crictl ps
forcri-o
nerdctl ps
forcontainerd
6.8.2. Static Pods in Cluster
kubelet
takes requests to create pod from different inputs:- from definition file from static pod folder
- from http api endpoint (i.e. from
kube-apiserver
)
kubectl get pods
will also list pods created statically but they can NOT be edited, deleted etc. fromkube-apiserver
- Static pods can be recognized by their name where node name is appended to it
6.8.3. Static Pods Use Case
- To deploy Control Plane components as Static Pods. E.g.
kubeadm
tool uses a mechanist of Static Pods to create a k8s cluster. E.g. by just puttingcontroller-manager.yaml
,apiserver.yaml
,etcd.yaml
into static pod folder and the k8s control plane components will be started as pods
6.8.4. Static Pods vs. DaemonSets
Static Pods | DaemonSets |
---|---|
Created by kubelet | Created kube-apiserver (DaemonSet Controller) |
Deploy Control Plane components as Static Pods | Deploy Monitoring Agents, Logging Agents on nodes |
Ignored by kube-scheduler | Ignored by kube-scheduler |
6.9. Multiple Schedulers
6.10. Configuring Scheduler Profiles
6.11. Admission Controllers
6.12. Validating and Mutating Admission Controllers
6.12.1. Validating Admission Controllers
- Validates the request and allow/deny it
6.12.2. Mutating Admission Controller
- Mutates/Changes object before it is created (e.g.
DefaultStorageClass
)
6.12.3. Custom Validating/Mutating Admission Controllers
-
MutatingAdmissionWebhook
-
ValidatingAdmissionWebhook
-
Webhooks above will reference
Admission Webhook Server
with custom logic
7. Logging and Monitoring
7.1. Monitor Cluster Components
7.1.1. Tools
- Prometheus
- DataDog
- Elastic Stack
- Dynatrace
- Mentics Server: in-memory solution without storing data option.
kubelet
contains sub-component calledcAdvisor
(i.e. container advisor) responsible for retrieving performance metrics and exposing them throughkubelet
API for Metrics Server.- Enable Metrics Service:
git clone https://github.com/kubernetes-incubator/metrics-server.git & kubectl create -f deploy/1.8+
- Cluster Performance view:
kubectl top node
kubectl top pod
7.2. Managing Application Logs
8. Application Lifecycle Management
8.1. Rolling Updates and Rollbacks
8.1.1. Deployment Strategy
- Recreate: destroy all and create all
- Rolling Update: destroy and create one-by-one (DEFAULT Deployment Strategy)
8.2. Configure Applications
- Configuring Command and Arguments on applications
- Configure Environment Variables
- Configure Secrets
8.2.1. Commands and Arguments in Docker and K8s
- Read more
- Containers (unlike VMs) are not meant to host OS, but run specific process.
FROM nginx
ENTRYPOINT ["sleep"]
CMD ["5"]
apiVersion: v1
kind: Pod
metadata:
name: nginx-sleeper-pod
spec:
containers:
- name: nginx
image: nginx
command: ["sleep"] # Overrides executables ENTRYPOINT in Dockerfile
args: ["10"] # Overrides CMD defined after ENTRYPOINT as arguments in Dockerfile
8.2.2. Configuring Environment Variables in Applications
apiVersion: v1
kind: Pod
metadata:
name: envar-demo
spec:
containers:
- name: envar-demo-container
image: nginx
env:
- name: MY_VAR
value: "My Value"
# ENV Value Types
# 1. Plain Key Value
...
env:
- name: MY_VAR
value: "My Value"
...
# 2. ConfigMap
...
env:
- name: MY_VAR
valueFrom:
configMapKeyRef:
...
# 3. Secrets
...
env:
- name: MY_VAR
valueFrom:
secretKeyRef:
...
8.2.3. Configuring ConfigMaps in Applications
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-demo
data:
# property-like keys; each key maps to a simple value
player_initial_lives: "3"
ui_properties_file_name: "user-interface.properties"
# file-like keys
game.properties: |
enemy.types=aliens,monsters
player.maximum-lives=5
user-interface.properties: |
color.good=purple
color.bad=yellow
allow.textmode=true
apiVersion: v1
kind: Pod
metadata:
name: env-configmap
spec:
containers:
- name: app
command: ["/bin/sh", "-c", "printenv"]
image: busybox:latest
envFrom:
- configMapRef:
name: configmap-demo
# ConfigMaps in Pods
# 1. ENV
envFrom:
- configMapRef:
name: app-config
# 2. SINGLE ENV
env:
- name: APP_COLOR
valueFrom:
configMapKeyRef:
name: app-config
key: APP_COLOR
# 3. VOLUME
volumes:
- name: app-config-value
configMap:
name: app-config
8.2.4. Configure Secrets an Applications
8.2.4.1. Create Secrets
- Imperative approach - see Commands
- Declarative approach
# kubectl create -f secret-data.yaml
# Secrets are only encoded, NOT Encryped
# echo -n 'password' | base64
# echo -n 'cGFzc3dvcmQ=' | base64 --decode
# secret-data.yaml
apiVersion: v1
kind: Secret
metadata:
name: app-secret
data:
DB_User: root
DB_Password: dmFsdWUtMg0KDQo=
---
apiVersion: v1
kind: Pod
metadata:
name: sapmple-secret-pod
spec:
containers:
- name: test-container
image: registry.k8s.io/busybox
envFrom:
- secretRef:
name: app-secret # ref to secret name
# Secrets in Pods
# 1. ENV
envFrom:
- secretRef:
name: app-config
# 2. SINGLE ENV
env:
- name: DB_password
valueFrom:
secretKeyRef:
name: app-secret
key: DB_password
# 3. VOLUME
volumes:
- name: app-secret-volume
secret:
secretName: app-secret
8.2.4.2. Encrypting Secret Data at Rest
Secrets are not encrypted, so it is not safer in that sense. However, some best practices around using secrets make it safer. As in best practices like:
- Not checking-in secret object definition files to source code repositories.
- Enabling Encryption at Rest for Secrets so they are stored encrypted in ETCD.
- Read more about Encrypting Confidential Data at Rest
- Or here
8.2.4.3. Secret Store Providers
- Third-party secrets store providers: AWS Provider, Azure Provider, GCP Provider, Vault Provider (HashiCorp Vault)
- Helm Secrets plugin
8.3. Scale Applications
8.4. Multi Container Pods
# pod-definition.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
- name: nginx-2
image: nginx:1.14.2
ports:
- containerPort: 81
8.4.1. Multi-container Pods Design Patterns
- Sidecar (e.g. service and logging containers)
- Adapter
- Ambassador
8.5. InitContainers
# pod-definition.yaml
# initContainer must run to a completion before the real container hosting the application starts.
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox:1.28
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
initContainers:
- name: init-myservice
image: busybox
command: ['sh', '-c', 'git clone <some-repository-that-will-be-used-by-application> ; done;']
8.6. Self Healing Applications
Kubernetes supports self-healing applications through ReplicaSets and Replication Controllers. The replication controller helps in ensuring that a POD is re-created automatically when the application within the POD crashes.
8.7. Intro to Autoscaling
8.7.1. Scaling Cluster Infra vs Scaling Workloads
- Scaling Cluster Infra by increasing number of nodes horizontally or increasing size of resources (CPU, RAM) on existing node
- Scaling Workloads by increasing the number of pods horizontally or increase resources allocated to existing pods
8.7.2. Manual vs Automated Scaling
- Manual Scaling of
- Cluster Infra:
- Horizontally:
kubeadm join ...
- Vertically: rare
- Horizontally:
- Workloads:
- Horizontally:
kubectl scale ...
- Vertically:
kubectl edit ...
- Horizontally:
- Cluster Infra:
- Automatied Scaling of
- Cluster Infra:
- Cluster Autoscaler
- Workloads:
- Cluster Infra:
8.7.3. Horizontal Pod Autoscaler (HPA)
- Increate the running instances of application horizontally
- HPA relies on
- Metrics Server or Custom Metrics Adapter for internal sources
- External Adapter for e.g. DataDog, Dynatrace etc.
8.7.3.1. Scaling a workload manually
# Manually check the load and increase replicas
kubectl top pod my-app-pod
kubectl scale deployment my-app --replicas=3
8.7.3.2. HPA
HPA:
- Observes metrics
- Adds pods
- Balances thresholds
- Tracks multiple metrics
# IMPERATIVE: Create HPA for specific deployment
kubectl autoscale deployment my-app \
--cpu-percent=50 --min=1 --max=10 # CPU thresholds of 50% and min 1 pod and max 10 pods
# DECLARATIVE: Create HPA for specific deployment
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
8.7.4. Vertical Pod Autoscaling (VPA)
- Manual:
kubectl edit deployment my-app
and manually change resources
- VPA:
- Observe metrics
- Adjust pod resources: e.g. increase the size of resources vertically, such as CPU, RAM
- Balances threshold
apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
name: stress-vpa
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: high-cpu-utilization-deployment
updatePolicy:
updateMode: Auto # Modes: Off, Initial, Recreate and Auto
resourcePolicy:
containerPolicies:
- containerName: '*'
minAllowed:
cpu: 100m
memory: 50Mi
maxAllowed:
cpu: 200m # maximum vpa will be allocating this many cpus even if demand is higher.
memory: 500Mi
controlledResources: ["cpu", "memory"]
8.7.4.1. Deployment of VPA Components
- VPA does NOT come built-in with K8s, but must be deployed
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vertical-pod-autoscaler.yaml
- VPA includes following pods:
vpa recommender
- continuously monitoring resource usage from Metrics Server and provides recommendation for optimal cpu, ram valuesvpa updater
- get an information from recommender and evicts pods needed to be updated with recommended resourcesvpa admission controller
- controller then creates new pod with new resources values recommended
8.7.5. In-place resize Pods
- As of kubernetes version 1.32, if the resource definition parameter is changed, the default behaviour is delete running instance of pod and spin up a new pod with changes
- InPlacePodVerticalScaling: Resize CPU and Memory Resources assigned to Containers without restart
- FEATURE STATE: K8s v1.27 [alpha][enabled by default:false]
- Enable feature:
FEaTURE_GATES:InPlacePodVerticalScaling=true
apiVersion: v1
kind: Pod
metadata:
name: resize-demo
spec:
containers:
- name: pause
image: registry.k8s.io/pause:3.8
resizePolicy:
- resourceName: cpu
restartPolicy: NotRequired # Default, but explicit here
- resourceName: memory
restartPolicy: RestartContainer
resources:
limits:
memory: "200Mi"
cpu: "700m"
requests:
memory: "200Mi"
cpu: "700m"
8.7.5.1. Limitations
- Only CPU and Memory resources can be changed
- Pod QoS Class cannot change
- InitContainers and Ephemeral COntainers cannot be resized
- Resource requests and limits cannot be removed once set
- A container's memor limit may not be reduced below its usage. If a request pts a container in this state, the resize status will remain in InProgress until the resired memory limit becomes feasible.
8.7.6. VPA vs. HPA
Feature | VPA (Vertical Scaling) | HPA (Horizontal Scaling) |
---|---|---|
Scaling Method | Increase CPU & memory of existing Pods | Adds/Removes Pods based on load |
Pods Behavior | Restarts Pods to apply new resource values | Keeps existing Pods running |
Handles Traffic Spikes? | NO, because scaling requires a Pod restart | YES, instantly adds more Pods |
Optimizes Costs? | Prevents over-provisioning of CPU/RAM | Avoids unnecessary idle Pods |
Best for | Stateful workloads, CPU/RAM-heavy apps | Stateless services, web apps, microservices |
Use cases | DBs, JVM-baseds apps, AI/ML workloads | Web servers (Nginx, API services), message queues, microservices |
9. Cluster Maintenance
9.1. OS Upgrade
- Drain and cordon nodes (see Command section)
- Uncordon nodes (see Command section)
9.2. Kubernetes Software Versions
Kubernetes version 1.27
- kube-apiserver: v1.27.0
- kube-scheduler: v1.27.0
- kube-controller-manager: v1.27.0
- kubelet: v1.27.0
- kube-proxy: v1.27.0
- etcd: v3.5.7 (or similar, depending on the Kubernetes version)
- coredns: v1.10.1 (or similar, depending on the Kubernetes version)
- container runtime: Versions can vary (e.g., containerd v1.6.0 or Docker v20.10.0)
9.3. Cluster Upgrade Process
-
Kubenrnetes Supports only 3 last versions
-
It is recommended to upgrade one MINOR version at a time.
-
Upgrade possibilies of different versions.
- kube-apiserver: vX
- kube-controller-manager: vX-1
- kube-scheduler: vX-1
- kubelet: vX-2
- kube-proxy: vX-2
- kubectl: vX+1, vX, vX-1
-
Upgrade using
kubeadm
kubeadm upgrade plan
kubeadm upgrade apply
-
Upgrade Sequence:
- First Master Node
- Then Worker Nodes
-
Upgrade Strategies of Worker Nodes
- Upgrade ALL of Worker Nodes at once
- Upgrade ONE node as a time
- Add NEW node with new k8s version. I.e. provision new nodes and decomission old one.
9.3.1. kubeadm upgrade
Upgrade Master Node:
kubeadm upgrade plan
#kubeadm
does not upgradekubelet
apt-get upgrade -y kubeadm=1.12.0-00
kubeadm upgrade apply v1.12.0
apt-get upgrade -y kubelet=1.12.0-00
systemctl restart kubelet
Upgrade Worker Nodes:
kubectl drain node-1
apt-get upgrade -y kubeadm=1.12.0-00
apt-get upgrade -y kubelet=1.12.0-00
kubeadm upgrade node config --kubelet-version 1.12.0
systemctl restart kubelet
kubectl get nodes
kubectl uncordon node-1
9.3.2. Cluster Upgrade - Demo
- Read more
9.3.3. Backup and Restore Methods
9.3.3.1. Backup Candidates
- Resource Configurations (namespace, configmap, secrets etc.)
kubectl get all --all-namespaces -o yaml > all-resources.yaml
- ETCD Cluster (Stores information about the state of k8s cluster)
- ETCD Cluster is hosted on Master Nodes
- ETCD Cluster data dir to backup
--data-dir=/var/lib/etcd
- Create a Snapshot of ETCD with
ETCDCTL_API=3 etcdctl snapshot save etcd_snapshot.db
- Restore ETCD from Backup
service kube-apiserver stop
ETCDCTL_API=3 etcdctl snapshot restore etcd_snapshot.db --data-dir /var/lib/etcd-from-backup
systemctl daemon-reload
service etcd restart
service kube-apiserver start - Read more about backup tools
- Persistence Volumes
10. Security
10.1. Kubernetes Security Primitives
10.1.1. Cluster Hosts Security
- All access to cluster host must be secured
root
access disabled- Password based authentication disabled
- SSH Key based authentication enabled
10.1.2. Kubernetes Security
- Securing the access to
kube-apiserver
is the first line of diffence- Authentication, i.e. who can access it?
- Files - Username and Passwords
- Files - Username and Tokens
- Certificates
- External Authentication providers - LDAP
- Service Accounts
- Authorization, i.e. what can they do?
- RBAC Authorization
- ABAC Authorization
- Node Authorization
- Webhook Mode
- Authentication, i.e. who can access it?
10.1.3. TLS Certificates
- All communication between
kube-apiserver
and - ETCD Cluster
- Kubelet
- Kube Proxy
- Kube Scheduler
- Kube Controller Manager is secured TLS Encryption.
10.1.4. Application Communication within Cluster
- By default all Pods can access other Pods within the Cluster
- The Pods access by another Pods can be restricted by Network Policies
10.2. Authentication
- Users who may access K8s Cluster
- Users
- Admins
- Developers
- Service Accounts
- Bots (CI/CD etc.)
- Application End Users (Usually managed by Application Authentication Management)
- Users
10.2.1. Users
All user access are managed by kube-apiserver
whether access with kubectl
or curl https://kube-server-ip:6443/
by
- First Authenticate Users
- Then Process Requests
kube-apiserver
Authentication Mechanisms:
- Static Password File (not recommended)
- Static Token File (not recommended)
- Certificates
- Identity Services (e.g. third Party Auth protocol LDAP, Cerberus etc.)
10.2.1.1. Basic Authentication Mechanism - Static Password File
# basic-auth.csv
password123,user1,u001,group1
password123,user2,u002,group2
password123,user3,u003,group3
kube-apiserver --basic-auth-file=/path/to/basic-auth.csv
systemctl restart kube-apiserver
Then kube-apiserver
can be access with
curl -v -k https://master-node-ip:6443/api/v1/pods -u "user1:password123"
10.2.1.2. Basic Authentication Mechanism - Static Token File
# basic-token-auth.csv
a3f5c8e1d2b4a6e7,user1,u001,group1
9b1e2c3d4f5a6b7c,user2,u002,group2
7e8f9a0b1c2d3e4f,user3,u003,group3
kube-apiserver --token-auth-file=/path/to/basic-token-auth.csv
systemctl restart kube-apiserver
Then kube-apiserver
can be access with
curl -v -k https://master-node-ip:6443/api/v1/pods --header "Authorization: Bearer a3f5c8e1d2b4a6e7"
10.2.2. Service Accounts
kubectl create serviceaccount sa1
10.3. TLS Certificates in K8s
10.3.1. TLS Basics
Transport Layer Security (TLS) is primarily associated with securing communications over the internet and it is used in various applications and protocols, like: SSH, HTTPS, Email Protocols(SMTP, IMAP), FPN, FTPS, APIs(RESTful APIs), WebSocket Secure (WSS), DB Connections etc.
- TLS certificates are digital certificates that authenticate the identity of a website or server and enable encrypted communication between clients and servers over the internet, ensuring data privacy and integrity during transmission.
- Key Pair Generation: When you create a key pair, a cryptographic algorithm (like RSA or ECC) generates two keys: a private key and a public key. These keys are mathematically linked.
- Private Key: is used to decrypt information or sign data and is kept secret.
- Public Key: is used to encrypt information or verify signatures and can be shared with anyone.
- Certificate Signing Request (CSR): includes the public key along with information about the entity requesting a certificate (e.g. organization’s name, domain). When you generate a CSR, you create it using your public key.
- Certificate (CRT or Digital Certificate): contains the public key along with information about the certificate holder (e.g. their identity, domain). It is issued by a Certificate Authority (CA) after verifying the information in the CSR. The CRT allows others to trust that the public key belongs to the entity it claims to represent. I.e. Certificate is so called Validated User ID and Key is Password.
- Digital Signature: When you sign data with your private key, others can use your public key to verify that the signature is valid. This ensures that the data hasn’t been tampered with and confirms the identity of the sender.
- Root Certificate is a special type of digital certificate that belongs to a trusted organization called a Certificate Authority (CA).
- Certificate Authority (CA): An entity that issues digital certificates, verifying the identity of the certificate holder and ensuring the integrity of the public key.
10.3.1.1. Private vs Public Key
- Key Pair
- Data can be encrytped with ANY of them and ONLY decrypted with other
- Data can NOT be encrypted with one and descrypted with the same key
- If data is encrypted with Private Key, anyone who has Public Key can decrypt data!
- Signing is done with a private key and Verification is done with a public key
- Encryption is done with a public key and Decryption is done with a private key
10.3.1.2. Symmetric Encryption
Symmetric encryption is a cryptographic method where the same key is used for both encrypting and decrypting data, ensuring that only parties with the shared key can access the original information.
10.3.1.3. Asymmetric Encryption
Asymmetric encryption is a cryptographic method that uses a pair of keys—a public key for encryption and a private key for decryption-allowing secure communication where the public key can be shared openly while the private key remains confidential.
10.3.1.4. Asymmetric Encryption - SSH
Asymmetric encryption in SSH (Secure Shell) uses a pair of keys—a public key and a private key—to secure communications between a client and a server. The public key is shared with the server, while the private key remains confidential on the client.
10.3.1.5. Asymmetric Encryption - HTTPS
In HTTPS (Hypertext Transfer Protocol Secure), both symmetric and asymmetric encryption are used to secure communications between a web browser and a server:
- Asymmetric Encryption: Initially used during the SSL/TLS handshake process to establish a secure connection. The server presents its public key to the client, allowing the client to encrypt a shared secret (session key) that only the server can decrypt with its private key. This ensures that the client is communicating with the legitimate server.
- Symmetric Encryption: Once the secure connection is established and the session key is shared, symmetric encryption is used for the actual data transmission. This is because symmetric encryption is faster and more efficient for encrypting large amounts of data compared to asymmetric encryption.
10.3.1.6. Certificate Authority (CA)
- Public CAs have also Private and Public Keys
- Private CA Service can be hosted within ones infrastructure
- CA uses Private Key to sign Certificates
- Public Keys of CAs are available in e.g. Browsers (Trusted Root Certification Authorities)
- Browser uses Public Key of CA to verify that a specific Certificate was signed with Private Key of Official CA
Public CAs:
- Symantec
- Comodo
- GlobalSign
- Digicert
10.3.1.7. Naming Convention of Public and Private Keys
- Certificate (Public Key)
- *.crt, *.pem:
- server.crt
- server.pem
- client.crt
- client.pem
- *.crt, *.pem:
- Private Key
- *.key, *-key.pem:
- server.key
- server-key.pem
- client.key
- client-key.pem
- *.key, *-key.pem:
10.3.1.8. TLS Use Cases
- Admins generate Key Pairs to secure SSH
- Web Server has Key Pairs to secure Website with HTTPS
- CA Generates its own Key Pairs to sign/verify Certificates
- End Users/Browser only generates single Symetric Key
10.3.2. TLS in Kubernetes
- Root Certificates (on CA Servers)
- Server Certificates signed by CA (on Servers)
- Client Certificates signed by CA (on Clients)
Certificate Authority
- CA
- ca.crt, ca.key
Server Certificates for Servers
- kube-apiserver (server)
- kube-apiserver.crt, kube-apiserver.key
- etcd (server)
- etcd.crt, etcd.key
- kubelet (server)
- kubelet.crt, kubelet.key
Client Certificates for Clients
- kube-apiserver (client)
- etcd (server)
- kube-apiserver-etcd.crt, kube-apiserver-etcd.key
- kubelet (server)
- kube-apiserver-kubelet.crt, kube-apiserver-kubelet.key
- etcd (server)
- kubelet (client)
- kube-apiserver (server)
- kubelet-kube-apiserver.crt, kubelet-kube-apiserver.key
- kube-apiserver (server)
- admin (client)
- kube-apiserver (server)
- admin-kube-apiserver.crt, admin-kube-apiserver.key
- kube-apiserver (server)
- kube-scheduler (client)
- kube-apiserver (server)
- kube-scheduler-kube-apiserver.crt, kube-scheduler-kube-apiserver.key
- kube-apiserver (server)
- kube-controller-manager (client)
- kube-apiserve (server)
- kube-controller-manager-kube-apiserver.crt, kube-controller-manager-kube-apiserver.key
- kube-apiserve (server)
- kube-proxy (client)
- kube-apiserve (server)
- kube-proxy-kube-apiserver.crt, kube-proxy-kube-apiserver.key
- kube-apiserve (server)
10.3.3. TLS in Kubernetes - Certificate Creation
Tools to create certificates
- EASYRSA
- OPENSSL
- CFSSL
# CA --------------------------------------------------------------------------
# Generate Private Key ca.key for CA
openssl genrsa -out ca.key 2048
# Generate Certificate Signing Request (CSR) ca.crs using ca.key for CA. CSR is a certificate with all your details but without signature
openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr
# Sign (ROOT) Certificate ca.crt using ca.csr and ca.key for CA (self-signed). CA ca.crt must be in ALL k8s hosts (servers/clients)
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt
# CLIENT Certificates ---------------------------------------------------------
# Generate Private Key admin.key for Admin
openssl genrsa -out admin.key 2048
# Generate Certificate Signing Request (CSR) admin.crs using admin.key for Admin. CSR is a certificate with all your details but without signature. Certificate is so called Validated User ID and Key is Password
# "/CN=kube-admin" is admin name which e.g. appears in audit logs
# "/O=system:masters" is a Group Detail added to Certificate to differentiat admin user from simple user.
# Groups named "system:masters", "system:kube-scheduler", "system:kube-controller-manager", "system:kube-proxy", "system:nodes" exists in k8s with admin/relevant priviledges
openssl req -new -key admin.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr
# Sign Admin admin.crt using ca.csr and ca.key. CA Key Pair is used to sign all server and client certificates
openssl x509 -req -in ca.csr -signkey ca.key -out admin.crt
# SERVER Certificates ---------------------------------------------------------
# Generate Private Key kube-apiserver.key for kube-apiserver
openssl genrsa -out kube-apiserver.key 2048
# Generate Certificate Signing Request (CSR) kube-apiserver.crs using kube-apiserver.key for kube-apiserver. CSR is a certificate with all your details but without signature
openssl req -new -key kube-apiserver.key -subj "/CN=kube-apiserver" -out kube-apiserver.csr -config openssl.cnf
# openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation,
subjectAltName = @alt_names
[alt_names] # Only clients refering to kube-apiserver by defined names can establish valid connection
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.deafult.svc.cluster.local
IP.1 = 10.96.0.1
IP.2 = 172.17.0.87
# Sign kube-apiserver kube-apiserver.crt using ca.csr and ca.key. CA Key Pair is used to sign all server and client certificates
openssl x509 -req -in kube-apiserver.csr -signkey ca.key -out kube-apiserver.crt
10.3.4. View Certificate Details
# View Certificate Details and Perform a healthcheck of Certificates ----------
# Check Details:
# - Common Name (e.g. kube-apiserver)
# - Subject Alternative Name (kubernetes, kubernetes.default etc.)
# - Validity (e.g. expirity date)
# - Issuer (e.g. CA)
# - See https://kubernetes.io/docs/setup/best-practices/certificates/ or https://github.com/mmumshad/kubernetes-the-hard-way/tree/master/tools
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
# Show certificate of server --------------------------------------------------
openssl s_client -connect <ip-of-host>
# Inspect Service Logs --------------------------------------------------------
journalctl -u etcd.service -l
kubectl logs etcd-master
10.4. Certificate API
- In a Kubernetes cluster, a Certificate Authority (CA) server is responsible for managing and issuing digital certificates
- If a new developer needs to access k8s cluster, he/she must create Private Key and provide CSR which then will be signed with CA private key on CA server by k8s admins
- The built-in Certificate API in Kubernetes is a set of resources and functionalities that facilitate the management of TLS certificates within a Kubernetes cluster (e.g. sign and rotate certificates etc.)
- Admin creates
CertificateSigningRequest
object (e.g. Instead of signing to CA server and signing certificate manually) - Request then can be reviewed
- And approved using
kubectl get csr && kubectl certificate approve <name-of-csr>
commands - Certificates then can be extracted and shared with users with
kubectl get csr <name-of-csr> -o yaml && echo "certificate-encoded-in-base64" | base64 --decode
- Kube Controller Manager is responsible for operations with certificates (components: CSR-APPROVING, CSR-SIGNING)
- Admin creates
10.4.1. Steps to generate certificate for a new User
# USER creates Private Key and CSR
openssh genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr
# ADMIN encodes in with base64, creates CertificateSigningRequest object, approve and share certificate with user
cat jane.csr | base64 -w 0
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: jane
spec:
request: <base64-encoded-csr>
expirationSeconds: 600
usages:
- digital signature
- key encipherment
- server auth
kubect get csr
kubect certificate approve jane
kubectl get csr <name-of-csr> -o yaml
echo "certificate-encoded-in-base64" | base64 --decode
10.5. KubeConfig
# View KubeConfig
kubectl config view --kubeconfig=/custom/path/to/config
kubectl config use-context prod-user@production
kubectl config -h
# Authenticate using Certificates with curl,kubectl or define all values in $HOME/.kube/config
curl https://kube-apiserver:6443/api/v1/pods --key admin.key --cert admin.crt --cacert ca.crt
kubect get pods --server kube-apiserver:6443 --client-key admin.key --client-certificate admin.crt --certificate-authority ca.crt
# $HOME/.kube/config
apiVersion: v1
kind: Config
current-context: kubernetes-admin@kubernetes-cluster
clusters: # Clusters (e.g. dev, prod, google)
- name: kubernetes-cluster
cluster:
certificate-authority: ca.crt # alternatively use "certificate-authority-data: <base64-encoded-certificate>"
server: https://kube-apiserver:6443
kind: Config
users: # Users (e.g. admin, devuser, produser)
- name: kubernetes-admin
user:
client-certificate: admin.crt
client-key: admin.key
contexts: # Contexts (<user>@<cluster> e.g. admin@dev, devuser@prod)
- name: kubernetes-admin@kubernetes-cluster
context:
cluster: kubernetes-cluster
user: kubernetes-admin
namespace: kubernetes-namespace
10.6. Persistent Key/Value Store
10.7. API Group
API groups make it easier to extend the Kubernetes API. The API group is specified in a REST path and in the apiVersion field of a serialized object.
- /metrics
- /healthz
- /version
- /logs
- /api (responsible for cluster functionality)
- /apis (responsible for cluster functionality)
API groups in Kubernetes:
- The core (also called legacy) group is found at REST path
/api/v1
. The core group is not specified as part of the apiVersion field, for example, apiVersion: v1. - The named groups are at REST path
/apis/$GROUP_NAME/$VERSION
, more organized and use apiVersion: VERSION (for example, apiVersion: batch/v1).
Core groups:
- /api
- /v1
- namespaces
- pods
- rc
- events
- endpoints
- nodes
- bindings
- PV
- PVC
- configmaps
- secrets
- services
- /v1
Named groups:
- /apis
- /apps (API Groups)
- /v1 (aka apiVersion)
- /deployments (aka RESOURCES)
- list, get, create, delete, update, watch (aka VERBS)
- /replicasets
- /statefulsets
- /deployments (aka RESOURCES)
- /v1 (aka apiVersion)
- /extensions
- /networking.k8s.io
- /v1
- /networkpolicies
- /v1
- /storage.k8s.io
- /authentication.k8s.io
- /certificates.k8s.io
- /v1
- /certificatesigningrequests
- /v1
- /apps (API Groups)
# Start Kubectl Proxy (i.e. http proxy service) locally which uses credentials from KubeConfig and provides access to Kube ApiServer
kubectl proxy
# List available API Groups in K8s Cluster/Kube ApiServer
curl https://localhost:6443 -k
10.8. Authorization
- Admins
- Developers
- Bots
10.8.1. Authorization Modes
Authorization modes refer to the specific configurations or settings that determine which authorization mechanisms are active:
- AlwaysAllow
- Node
- ABAC
- RBAC
- Webhook
- AlwaysDeny
10.8.2. Authorization Mechanisms
- Node based
- used to access within the cluster, e.g. node access to apiserver
- Attribute Based Authorization (ABAC)
- difficult to manage as it is managed by policy file defined in Kube ApiServer
- external access to apiserver
- Role Based Authorization (RBAC)
- Webhook
# Flag used to set authorization mode on Kube ApiServer. Authorization check in defined ORDER of authorization mode
---authorization-mode = Node,RBAC,Webhook \\
10.8.2.1. Node Authorizer
e.g. is managed by certificates
- Kube ApiServer (server)
- Kubelet (client, certificate group part of "system:node:node1")
- Read:
- Services, Endpoints, Nodes, Pods
- Write
- Node status, Pod status, events
- Read:
- Kubelet (client, certificate group part of "system:node:node1")
10.8.2.2. ABAC
e.g. is managed by policy file defined in Kube ApiServer (see below)
- Kube ApiServer (server)
- User (client, e.g. dev-user-1)
- Can View, Create, Delete Pods
- User (client, e.g. dev-user-1)
// ABAC policy file
{"kind": "Policy", "spec": {"user":"dev-user-1", "namespace": "*", "resource": "pods", "apiGroup": "*" }}
10.8.2.3. RBAC
e.g. is managed by policy file defined in Kube ApiServer
- Kube ApiServer (server)
- Developers
- User (client, e.g. dev-user-1)
- Can View Pods
- User (client, e.g. dev-user-1)
- Admins
- User (client, e.g. admin-user-1)
- Can View, Create, Delete Pods
- User (client, e.g. admin-user-1)
- Developers
10.8.2.4. Webhook
- Authorization managed by external tool
- Open Policy Agent
10.8.3. RBAC
Roles and RoleBindings are namespaces, i.e. created within namespace. They created in default
namespace if the namespace is not defined in description file.
# Check RBAC
kubectl roles
kubectl rolebindings
kubectl auth can-i --list
-
Create a Role
# Role definition file developer-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: developer-role
namespace: dev-namespace
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods", "services"]
verbs: ["get", "list", "create", "update", "delete"]
resourceName: ["blue", "orange"] # Specify Pod Names
- apiGroups: ["apps"] # For resources in the apps API group
resources: ["deployments"]
verbs: ["get", "list", "create", "update", "delete"] -
Link a Role to User
# Bind a Role to User in developer-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: developer-rolebinding
namespace: dev-namespace
subjects:
- kind: User
name: dev-user
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: developer-role
apiGroup: rbac.authorization.k8s.io
10.8.4. Cluster Roles and Role Bindings
Resources:
- namespaced
- pods, replicasets, jobs, deployments, services, secrets, roles, rolebindings, configmaps, PVC
- cluster-scoped
- nodes
- nodes are cluster wide/scoped resources and can not be assosiated to any particular namespace
- clusterroles
- clusterrolebindings
- PV
- certificatesigningrequests
- namespaces
- nodes
# List namespaced and non-namespaces api-resources
kubectl api-resources --namespaced=true/false
E.g. of cluster roles:
- Cluster Admin
- Can view, create or delete Nodes
- Storage Admin
- Can view, create or delete PVs/PVCs
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-administrator
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["nodes"]
verbs: ["list", "get", "create", "delete"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin-clusterrolebinding
subjects:
- kind: User
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-administrator
apiGroup: rbac.authorization.k8s.io
10.9. Service Accounts
Types of user accounts:
- User account
- used by humans
- Service account
- used by machines, bots, application (e.g. Premetheus, Jenkins)
- has an automatically (must be manually created from version v1.24) created token, stored as secret
- every namespace has default service account, which associated to pod on creation
kubectl get serviceaccount
kubectl create serviceaccount <service-account-name>
10.10. Image Security
# Create a secret of type docker-registry
kubectl create secret docker-registry <secret-name> \
--docker-server=string \
--docker-username=user \
--docker-password=password \
--docker-email=email \
[--from-file=[key=]source] [--dry-run=server|client|none]
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
spec:
containers:
- name: nginx-container
image: nginx:latest
imagePullSecrets:
- name: <docker-registry-secret-name>
10.11. Docker Security
# Capability of root user list under
cat /usr/include/linux/capability.h
# E.g.: CHOWN, DAC, KILL, SETFCAP, SETPCAP, SETGID, SETUID, NET_BIND, NET_RAW, MAC_ADMIN, BROADCAST, NET_ADMIN, SYS_ADMIN, SYS_CHROOT, AUDIT_WRITE, MANY MORE
# Root user in linux and docker container does not have the same capabilities
# Add or Drop additional capabilities to docker container with
docker run --cap-add MAC_ADMIN ubuntu
docker run --cap-drop KILL ubuntu
# Run docker container with all capabilities
docker run --privilided ubuntu
10.12. Security Context
apiVersion: v1
kind: Pod
metadata:
name: ubuntu-sleep-pod
labels:
app: ubuntu-sleep
spec:
securityContext: # Pod Level Security Context
runAsUser: 1000 # Run the container as user ID 1000
runAsGroup: 3000 # Run the container as group ID 3000
fsGroup: 2000 # Set the group ID for mounted volumes
containers:
- name: ubuntu-container
image: ubuntu:latest
command: ["sleep"]
args: ["3600"]
securityContext: # Container Level Security Context
runAsUser: 1000 # Run the container as user ID 1000
runAsGroup: 3000 # Run the container as group ID 3000
capabilities: # Capabilities are supported only in container level
add: ["MAC_ADMIN"]
10.13. Network Policies
A NetworkPolicy in Kubernetes is a resource that defines rules for controlling the ingress and egress traffic to and from Pods, allowing administrators to specify which Pods can communicate with each other and with external endpoints.
# Egress poicity type is not defined, hence egress response traffic not isolated. But egress originiating from db still requires egress policy definition
# Below db only accepts traffic from api-pod on port 3306
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-policy
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
ingress:
- from:
- podSelector: # all Pods will be allowed to reach db within defined namespace if podSelector not defined (AND operation)
matchLabels:
name: api-pod
namespaceSelector: # to restrict Network Policy to specific namespace traffic flow (if - in front of it, then separate rule)
matchLabels:
name: prod
- ipBlock:
cidr: 192.168.5.10/32 # Allow traffic from specific ip (OR operation with 1st rule "podSelector")
ports:
- protocol: TCP
port: 3306
# Below db only accepts traffic from api-pod on port 3306 AND can push backups to external server with defined ip
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-policy
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector: # all Pods will be allowed to reach db within defined namespace if podSelector not defined (AND operation)
matchLabels:
name: api-pod
ports:
- protocol: TCP
port: 3306
egress:
- to:
- ipBlock:
cidr: 192.168.5.10/32
ports:
- protocol: TCP
port: 80
10.14. Kubectx and Kubens
kubectx is a tool to switch between contexts (clusters) kubens is a tool to switch between Kubernetes namespaces
# kubens and kubectx installation
sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx
sudo ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx
sudo ln -s /opt/kubectx/kubens /usr/local/bin/kubens
10.15. Custorm Resource Definition (CRD)
- Resource - objects that represent the state of the cluster
- ReplicaSet
- Deployments
- Job
- CronJob
- Statefulset
- Namespace
- Controller - control loops that manage the lifecycle of resources to maintain the desired state
- ReplicaSet
- Deployments
- Job
- CronJob
- Statefulset
- Namespace
Custom resource
# flight-ticket.yaml
apiVersion: flights.com/v1
kind: FlightTicket
metadata:
name: my-flight-ticket
spec:
from: Berlin
to: London
number: 2
Custom Resource Definition (CRD)
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: flighttickets.flights.com
spec:
group: flights.com # value of "apiVersion" of definition file
scope: Namespaced
names:
kind: FlightTicket # value of "kind" of definition file
singular: flightticket
plural: flighttickets
shortnames:
- ft
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
from:
type: string
to:
type: string
number:
type: integer
minumum: 1
maximum: 10
kubectl create -f flight-ticket.yaml
kubectl get flightticket
kubectl delete flightticket
10.16. Custom Controllers
To create Custom Controller
- Install Go
- Clone sample of custom controller
- Customize controller.go
- Build it with
go build -o sample-controller .
- Run it with
./sample-controller -kubeconfig=$HOME/.kube/config
- Custom Controller can be containerized and run as k8s pod
// flightticket_controller.go
package flightticket
var controllerKind =
apps.SchemeGroupVersion.WithKind("Flightticket")
// Run begins watching and syncing
func (dc *FligjtTicketController) Run (worker int, stopCh <-chan struct{})
// Cal BookFlightAPIReplicaSet
func (dc *FlightTicketController) callBookFlightAPI(obj interface{})
10.17. Operator Framework
- Operator Framework can be used to package Custom Resource Definition and Custom Controller together as simple entity
- etcd operator is famous example to deploy and manage an etcd cluster within kubernetes
- CRS
- EtcdCluster
- EtcdBackup
- EtcdRestore
- CC
- ETCD Controller
- Backup Operator
- Restore Operator
- CRS
- Check OperatorHub.io for available operators
11. Storage
11.1. Docker Storage
- Storage Drivers
- Volume Drivers
11.1.1. Docker Storage Drivers and File Systems
Storage Drivers in Docker enables Layered Architecture and helps to manage storage on images and containers. Common Storage Drivers:
- AUFS (default SD on Ubuntu)
- ZFS
- BTRFS
- Device Mapper
- Overlay
- Overlay2
By default Docker stores data in
- /var/lib/docker
- /aufs
- /containers (layered architecture, transient, container layers are read/write)
- /image (layered architecture, image layers are read-only)
- /volumes
- /data_volume
# Volume created with
docker volume create data_volume
# and use either with VOLUME MOUNT:
docker run -v data_volume:/var/lib/mysql mysql
# BIND MOUNT:
docker run -v /data/mysql:/var/lib/mysql mysql
docker run --mount type=bind, source=/data/mysql, target=/var/lib/mysql mysql
- /data_volume
11.1.2. Volume Driver Plugins in Docker
Volumes in Docker are handled by Volume Driver Plugins and not by Storage Drivers.
Volume Driver Plugins:
- Local (default)
- Azure File Storage
- Convoy
- DigitalOcean Block Storage
- Flocker
- gce-docker
- GlusterFS
- NetApp
- RexRay
- Portworx
- VMware vSphere Storage
# AWS Cloud Storage, Amazon EBS
docker run -it --name mysql --volume-driver rexray/ebs --mount src=ebs-vol, target=/var/lib/mysql mysql
11.2. Container Interfaces
Kubernetes:
- Container Runtime Interface (CRI)
- rkt
- docker
- cri-o
- Container Network Interface (CNI)
- waveworks
- flannel
- cilium
- Container Storage Interface (CSI)
- portworx
- Amazon EBS
- DELL EMC
- GlusterFS
11.2.1. Container Storage Interface (CSI)
Container Storage Interface (CSI) is not kubernetes specific but universal standard which enables any orchestration tool to work with any storage vendor. Currently Kubernetes, Cloud Foundry and Mesos are with CSI. CSI definces RPCs as:
- CreateVolume
- DeleteVolume
- ControllerPublishVolume
11.3. Volumes
11.3.1. Volumes and Mounts
apiVersion: v1
kind: Pod
metadata:
name: random-number-generator
spec:
containers:
- image: alpine
name: alpine
command: ["/bin/sh","-c"]
args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"]
volumeMounts:
- mountPath: /opt
name: data-volume
volumes:
- name: data-volume
hostPath: # hostPath (bounded to node)
path: /data
type: Directory
11.3.2. Volume Types
Volume Types:
- NFS
- GlusterFS
- Flocker
- Ceph
- Scaleio
- AWS
volumes:
- name: data-volume
hostPath: # hostPath (bounded to node)
path: /data
type: Directory
volumes:
- name: data-volume
awsElasticBlockStore: # awsElasticBlockStore (not bounded to node)
volumeID: <volume-id>
fsType: ext4
11.4. Persistent Volumes (PV)
Persistent Volumes (PV) is a cluster-wide pool of storage volumes configured by administrator. Resources can select storage from PV pool using Persistent Volume Claims (PVC).
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-vol1
spec:
accessModes: # accessMode - how a volume should be mounted on the host
- ReadWriteOnce # ReadOnlyMany, ReadWriteOnce, ReadWriteMany
persistentVolumeReclaimPolicy: Retain # Defines what happens to the volume when it is released. Recycle, Delete, Retain (will not be deleted)
capacity:
storage: 1Gi
# 1. Option
hostPath: # Not for PROD
path: /tmp/data
# 2. Option
awsElasticBlockStore:
volumeID: <volume-id>
fsType: ext4
11.5. Persistent Volume Claims (PVC)
Resources will be boind to a single Persistent Volume (PV) by Persistent Volume Claims (PVC). During the bind following criterias are considered:
- Sufficient Capacity
- Access Modes
- Volume Modes
- Storage Class
- Selector (is used to bind PVC to specific PV)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-persistent-volume-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
---
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/var/www/html"
name: mypd
volumes:
- name: mypd
persistentVolumeClaim:
claimName: my-persistent-volume-claim
11.6. Storage Class
11.6.1. Static Provisioning of Volume
- Before PV is created a disk on e.g. google-cloud must be created manually
gcloud beta compute disks create --size 1GB --region us-east1 pd-disk
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-persistent-volume-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
11.6.2. Dynamic Provisioning of Volume
- PV Definition is not needed for DPV but created automatically by StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: google-storage
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: none
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-persistent-volume-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: google-storage # Definition of storageClassName to dynamically define PV and create disk
resources:
requests:
storage: 500Mi
12. Networking
12.1. CoreDNS
Read more about CoreDNS here:
https://github.com/kubernetes/dns/blob/master/docs/specification.md https://coredns.io/plugins/kubernetes/
13. Design and Install a Kubernetes Cluster
14. Install "K8s the kubeadm way"
15. Helm Basics
15.1. What is Helm
Helm is a package manager for Kubernetes.
16. Kustomize Basics
17. End to End tests on a K8s Cluster
18. Troubleshooting
19. Other Topics
20. Ligtning Labs
21. Mock Exams
22. Course Conclusion
23. Commands
23.1. Kubernetes
# GET
kubectl get all
kubectl get pods --watch
kubectl get pods -n kube-system
kubectl get pods -o wide
kubectl get pods -l environment=production,tier=frontend
kubectl get pods -l 'environment in (production),tier in (frontend)'
kubectl get pods -l 'environment,environment notin (frontend)'
kubectl get pods --selector app=App1
kubectl get pods --selector app=App1 --no-headers | wc -l
kubectl get pod <pod-name> --all-namespaces
kubectl get replicationcontroller
kubectl get replicaset
kubectl get replicaset myapp-rs -o yaml
kubectl get deployments
kubectl get services
kubectl get configmaps
kubectl get secrets
kubectl get secret app-secret -o yaml
kubectl get hpa
kubectl get all --all-namespaces -o yaml > all-resources.yaml # Backup all resource configurations
kubectl get serviceaccount
kubectl get roles
kubectl get rolebindings
# DESCRIBE
kubectl describe pod myapp-pod
kubectl describe node kubemaster
# APPLY
kubectl apply -f object-definition-file.yaml # Create or Update (DEFAULT Rolling Update) k8s object
kubectl apply -f /path/to/object-definition-files
# CREATE
kubectl create deployment --image=nginx nginx --namespace=dev
kubectl create deployment my-deployment --image=nginx nginx --replicas=3 --dry-run=client -o yaml
kubectl create service nodeport myapp-service --tcp=8080:8080 --dry-run=client -o yaml
kubectl create namespace dev
kubectl create configmap <config-name> --from-literal=APP_COLOR=blue
kubectl create configmap <config-name> --from-file=app_config.properties
kubectl create secret generic <secret-name> --from-literal=<key>=<value>
kubectl create secret generic <secret-name> --from-file=<path-to-file>
kubectl create serviceaccount <service-account-name>
kubectl create token <service-account-name>
kubectl create secret docker-registry <secret-name> \
--docker-server=string \
--docker-username=user \
--docker-password=password \
--docker-email=email [--from-file=[key=]source] [--dry-run=server|client|none]
# EXEC
kubectl exec etcd-master -n kube-system etcdctl get / --prefix -keys-only
# RUN
kubectl run nginx --image=nginx --namespace=default --dry-run=client -o yaml
kubectl run nginx --image=nginx -- <arg1> <arg2> ... <argN>
kubectl run nginx --image=nginx --command -- <cmd> <arg1> <arg2> ... <argN>
# EDIT
kubectl edit replicaset
# LABEL
kubectl label nodes <node-name> <label-key>=<label-value>
# REPLACE
kubectl replace --force -f replicaset-definition.yaml
# SCALE
kubectl scale --replicas=6 -f replicaset-definition.yaml
kubectl scale --replicas=6 -f replicaset myapp-rs
# AUTOSCALE
kubectl autoscale deployment foo --cpu-percent=50 --min=1 --max=10 # Create HPA for a deployment "foo" with CPU thresholds of 50% and min 1 pod and max 10 pods
# DELETE
kubectl delete replicaset myapp-replicaset # deletes all underlying Pods
# CONFIG
kubectl config -h
kubectl config view --kubeconfig=/custom/path/to/config
kubectl config use-context <user-name>@<cluster-name>
kubectl config set-context $(kubectl config current-context) --namespace=dev
# TAINT
kubectl taint nodes <NODE-NAME> <KEY>=<VALUE>:<TAINT-EFFECT> # TAINT-EFFECTs: NoSchedule, PreferNoSchedule, NoExecute
kubectl taint nodes <NODE-NAME> <KEY>=<VALUE>:<TAINT-EFFECT>- # UNTAINT
# LOGS
kubectl logs -f <pod-name>
kubectl logs -f <pod-name> [container-name]
# AUTH
kubectl auth can-i --list # check current user permissions
kubectl auth can-i <verb> <resource> -n <namespace> --as <user-name> # check if the current user can perform a specific action on a resource
kubectl auth can-i create pods -n <namespace> --as dev-user
# ROLLOUT
kubectl rollout status deployment/<depoyment-name> # Check the status of a rollout
kubectl rollout history deployment/<depoyment-name> # Show the history of rollouts
kubectl rollout undo deployment/<depoyment-name> --to-revision=<revision_number> # Undo a rollout to the specific revision
# DRAIN
kubectl drain <node-name> # drain the node (will be marked as unschedulable) for maintenance purpose to gracefully terminate pods running on it and to reschedule them.
# CORDON
kubectl cordon <node-name> # cordon node (mark as unschedulable) so that no new pods are scheduled on it
# UNCORDON
kubectl uncordon <node-name> # uncordon the node so that pods can be scheduled on it after drain
# KUBE-CONTROLLER-MANAGER
kube-controller-manager --pod-eviction-timeout=5m0s # set time to wait pods to be evicted. I.e. control plane waits 5 minutes before considering the pod as dead and redeploy it
# ETCDCTL
ETCDCTL_API=3 etcdctl member list
ETCDCTL_API=3 etcdctl snapshot save etcd_snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/etcd/ca.crt \
--cert=/etc/etcd/etcd-server.crt \
--key=/etc/etcd/etcd-server.key
ETCDCTL_API=3 etcdctl snapshot status etcd_snapshot.db
ETCDCTL_API=3 etcdctl snapshot restore etcd_snapshot.db --data-dir /var/lib/etcd-from-backup
23.2. Helm
# INSTALL
helm install <package-name>
# UNINSTALL
helm uninstall <package-name>
# UPGRARE
helm upgrade <package-name>
# ROLLBACK
helm rollback <package-name>