Skip to content

Kubernetes Aide-mémoire

Container orchestration platform for automating deployment and scaling.

01

Getting Started

kubectl Basics

kubectl is the CLI for Kubernetes. get lists resources, create makes new ones, expose creates services, scale changes replica count. describe gives detailed info about a resource.

kubernetes
# cluster info
kubectl cluster-info

# list nodes
kubectl get nodes

# list pods (all namespaces)
kubectl get pods --all-namespaces

# create a deployment
kubectl create deployment nginx --image=nginx

# expose deployment as service
kubectl expose deployment nginx --port=80 --type=LoadBalancer

# scale deployment
kubectl scale deployment nginx --replicas=3

# get detailed info
kubectl describe pod <pod-name>

# view logs
kubectl logs <pod-name>

Output Formatting

The -o flag controls output. yaml/json show full resource spec. jsonpath and custom-columns extract specific fields for scripting. -w watches for live changes. Use --no-headers in scripts to drop column names.

kubernetes
# wide format (more columns)
kubectl get pods -o wide

# YAML output
kubectl get pod nginx -o yaml

# JSON output
kubectl get pod nginx -o json

# jsonpath (extract specific fields)
kubectl get pods -o jsonpath='{.items[*].metadata.name}'

# custom columns
kubectl get pods -o custom-columns=NAME:.metadata.name,STATUS:.status.phase

# show labels
kubectl get pods --show-labels

# watch for changes
kubectl get pods -w

Context & Configuration

A context bundles cluster address, user credentials, and namespace. kubeconfig files (default ~/.kube/config) store these. Multiple files can be combined via KUBECONFIG env var for managing many clusters.

kubernetes
# view current config
kubectl config view

# list contexts
kubectl config get-contexts

# switch context
kubectl config use-context my-cluster

# set default namespace for current context
kubectl config set-context --current --namespace=dev

# show current context
kubectl config current-context

# KUBECONFIG with multiple files
export KUBECONFIG=~/.kube/config:~/.kube/prod-config

Explain & API Resources

api-resources lists what the cluster supports. explain shows the OpenAPI schema for any resource — invaluable for writing YAML without leaving the terminal. Use --recursive to dump the full nested structure.

kubernetes
# list all resource types
kubectl api-resources

# list namespaced vs cluster-scoped
kubectl api-resources --namespaced=true
kubectl api-resources --namespaced=false

# explain a resource's fields
kubectl explain pod
kubectl explain pod.spec.containers
kubectl explain deployment.spec.strategy

# explain with recursion (all nested fields)
kubectl explain pod --recursive

# include description
kubectl explain pod.spec.containers.resources --api-version=apps/v1

Dry Run & Generate YAML

--dry-run=client -o yaml is the standard way to scaffold manifests from imperative commands. --dry-run=server validates against the real apiserver (admission webhooks, schema) without persisting. Great for bootstrapping declarative YAML.

kubernetes
# generate YAML without applying
kubectl create deployment nginx --image=nginx --dry-run=client -o yaml

# pipe generated YAML straight to a file
kubectl create deployment nginx --image=nginx --dry-run=client -o yaml > nginx.yaml

# apply the generated manifest
kubectl apply -f nginx.yaml

# dry-run against the server (validates with apiserver)
kubectl apply -f nginx.yaml --dry-run=server

# generate a service from a deployment
kubectl create service clusterip my-svc --tcp=80:80 --dry-run=client -o yaml

Help & Autocomplete

Every kubectl command supports --help with examples. Shell completion saves enormous typing and exposes subcommands. The alias k=kubectl is near-universal among practitioners — combine with completion for the best experience.

kubernetes
# top-level help
kubectl --help

# subcommand help
kubectl create --help
kubectl explain --help

# generate bash completion
kubectl completion bash > /etc/bash_completion.d/kubectl

# zsh completion
source <(kubectl completion zsh)

# alias k=kubectl with completion
alias k=kubectl
complete -o default -F __start_kubectl k

# shorthand: kubectl -> k
echo 'alias k=kubectl' >> ~/.bashrc
02

Pod

Pod Manifest

A Pod is the smallest deployable unit, holding one or more containers that share network and storage. Containers in a pod are always co-scheduled on the same node. You rarely create pods directly — use a Deployment or Job instead.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
    - name: nginx
      image: nginx:1.25
      ports:
        - containerPort: 80
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 200m
          memory: 256Mi
  restartPolicy: Always

Multi-Container Pod (Sidecar)

Multi-container pods share network (same IP, port space) and volumes. Common patterns: sidecar (helper), adapter (transforms output), ambassador (proxy). The shared emptyDir volume lets the sidecar read logs the main container writes.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: web-with-logs
spec:
  containers:
    - name: app
      image: nginx
      volumeMounts:
        - name: shared-logs
          mountPath: /var/log/nginx
    - name: log-shipper
      image: busybox
      command: ["sh", "-c", "tail -f /var/log/nginx/access.log"]
      volumeMounts:
        - name: shared-logs
          mountPath: /var/log/nginx
          readOnly: true
  volumes:
    - name: shared-logs
      emptyDir: {}

Pod Lifecycle & Phases

A pod's phase is high-level (Pending, Running, Succeeded, Failed). Container states (Waiting/Running/Terminated) give the real detail — e.g. CrashLoopBackOff means the container keeps exiting. Check restart count and events to diagnose.

kubernetes
# pod phases: Pending | Running | Succeeded | Failed | Unknown
kubectl get pod nginx

# show status including container states
kubectl get pod nginx -o jsonpath='{.status}'

# container states: Waiting | Running | Terminated
kubectl describe pod nginx

# common Waiting reasons: ImagePullBackOff, CrashLoopBackOff, CreateContainerConfigError
kubectl get pods --field-selector=status.phase=Pending

# restart count reveals crash loops
kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'

Pod Status & Events

Events are the cluster's audit log for a resource — scheduling, pulling, pulling errors, liveness failures. They expire (~1 hour by default), so capture them while debugging. describe bundles events with the resource spec for quick triage.

kubernetes
# full describe (includes Events at the bottom)
kubectl describe pod nginx

# list cluster events sorted by time
kubectl get events --sort-by=.lastTimestamp

# events for a specific namespace
kubectl get events -n default

# watch events live
kubectl get events -w

# filter events by type
kubectl get events --field-selector type=Warning

# events for a specific pod
kubectl get events --field-selector involvedObject.name=nginx

Exec & Port-Forward

exec runs commands inside a container; -it allocates a TTY for shells. port-forward tunnels a local port to a pod/service without exposing it publicly — essential for debugging databases or web UIs from your laptop.

kubernetes
# open an interactive shell
kubectl exec -it nginx -- sh

# run a one-off command
kubectl exec nginx -- ls /etc/nginx

# specify container in a multi-container pod
kubectl exec -it web-with-logs -c log-shipper -- sh

# forward a local port to a pod port
kubectl port-forward pod/nginx 8080:80

# forward to a service / deployment
kubectl port-forward svc/nginx 8080:80
kubectl port-forward deployment/nginx 8080:80

# forward to a random local port
kubectl port-forward pod/nginx :80

Delete & Force

Deleting a pod controlled by a Deployment/StatefulSet causes an immediate replacement. --force --grace-period=0 skips the TERM signal grace period and can leave stale resources — use sparingly. Stuck terminating pods often need a finalizer removal.

kubernetes
# delete a pod (controller will recreate it)
kubectl delete pod nginx

# force delete immediately (bypasses graceful shutdown)
kubectl delete pod nginx --grace-period=0 --force

# delete by label
kubectl delete pods -l app=nginx

# delete all pods in a namespace
kubectl delete pods --all -n default

# delete a pod and wait for it to be gone
kubectl delete pod nginx --wait

# delete with a timeout
kubectl delete pod nginx --timeout=30s
03

Deployment

Deployment Manifest

A Deployment manages ReplicaSets and provides declarative rolling updates and rollbacks. spec.selector must match spec.template.metadata.labels. The template defines the pod spec that gets rolled out.

kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          ports:
            - containerPort: 80

Rolling Update Strategy

RollingUpdate (default) gradually replaces pods for zero downtime. maxSurge/maxUnavailable control rollout speed and availability. Recreate kills all pods before creating new ones — used when the app cannot run two versions at once (e.g. single-volume writers).

kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%        # extra pods above replicas during update
      maxUnavailable: 25%  # pods allowed down during update
  # alternative: recreate (kills all, then starts new)
  # strategy:
  #   type: Recreate
  selector:
    matchLabels:
      app: nginx
  template:

Rollback

Each change to the pod template creates a ReplicaSet revision. undo flips traffic back to a previous ReplicaSet. pause lets you do multiple fixes between canary increments. Use --record (deprecated) or annotations to track change causes.

kubernetes
# view rollout history
kubectl rollout history deployment/nginx

# see details of a specific revision
kubectl rollout history deployment/nginx --revision=2

# undo to the previous revision
kubectl rollout undo deployment/nginx

# undo to a specific revision
kubectl rollout undo deployment/nginx --to-revision=2

# pause/resume a rollout (for canary pauses)
kubectl rollout pause deployment/nginx
kubectl rollout resume deployment/nginx

# check rollout status
kubectl rollout status deployment/nginx

Scale & HPA

HPA scales replicas based on CPU/memory or custom metrics. Requires metrics-server for CPU/mem. minReplicas and maxReplicas bound scaling. The v2 API supports multiple metrics and behaviors (scale-down stabilization).

kubernetes
# scale manually
kubectl scale deployment/nginx --replicas=5

# horizontal pod autoscaler (CPU target)
kubectl autoscale deployment nginx --min=2 --max=10 --cpu-percent=80

# list HPA
kubectl get hpa

# HPA manifest (v2 with custom metrics)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

Update Image & Restart

set image triggers a new ReplicaSet and rolling update. rollout restart is the cleanest way to pull the latest image or refresh config — it creates a new ReplicaSet with the same spec. edit opens your $EDITOR for ad-hoc fixes.

kubernetes
# update container image
kubectl set image deployment/nginx nginx=nginx:1.26

# update from a specific container
kubectl set image deployment/nginx container=nginx:1.26 --record

# trigger a rollout restart (same image, recreate pods)
kubectl rollout restart deployment/nginx

# edit a deployment in editor
kubectl edit deployment/nginx

# patch a single field
kubectl patch deployment/nginx -p '{"spec":{"replicas":5}}'

# apply a manifest change
kubectl apply -f nginx-deployment.yaml

Deployment Status

A Deployment owns ReplicaSets; only one is active (replicas>0). Old ReplicaSets scale to zero for rollback. pod-template-hash labels each pod with its revision. Check status.conditions for Available, Progressing, and ReplicaFailure signals.

kubernetes
# deployment status
kubectl get deployment nginx

# watch rollout
kubectl rollout status deployment/nginx

# show ReplicaSets owned by the deployment
kubectl get rs -l app=nginx

# show pods across all revisions
kubectl get pods -l app=nginx --show-labels

# pod-template-hash label distinguishes revisions
# kubectl get pods -l app=nginx,pod-template-hash=xxxx

# conditions: Available, Progressing, ReplicaFailure
kubectl get deployment nginx -o jsonpath='{.status.conditions}'
04

Service

Service Types

A Service provides a stable IP/DNS that load-balances across a set of pods. ClusterIP is the default (in-cluster only). NodePort and LoadBalancer expose externally. ExternalName is a DNS alias, not a proxy.

kubernetes
# four service types:
# - ClusterIP (default): reachable inside the cluster
# - NodePort: exposed on each node's IP at a static port
# - LoadBalancer: cloud-provisioned external LB
# - ExternalName: DNS CNAME to an external host

# quick create
kubectl expose deployment nginx --port=80 --type=ClusterIP
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl expose deployment nginx --port=80 --type=LoadBalancer

# external name (no proxying, pure DNS)
kubectl create service externalname my-svc --external-name=db.example.com

ClusterIP Service

ClusterIP gives a stable virtual IP and DNS name within the cluster. The selector matches pod labels; kube-proxy programs iptables/IPVS to load-balance across endpoints. Resolve via <svc>.<ns>.svc.cluster.local or just <svc>.

kubernetes
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - name: http
      port: 80          # service port
      targetPort: 80    # container port
      protocol: TCP
# DNS: nginx.default.svc.cluster.local
# short names also work: nginx, nginx.default
# kube-dns / CoreDNS resolves the service VIP

NodePort Service

NodePort opens the same static port (30000-32767) on every node. Traffic to any node:nodePort reaches a pod (kube-proxy may redirect to a pod on another node). Useful when you can't provision a cloud LB; pair with an external LB for HA.

kubernetes
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080   # optional, 30000-32767 range
# reachable at <any-node-ip>:30080
# also still reachable via clusterIP:80

# get assigned nodePort
kubectl get svc nginx -o jsonpath='{.spec.ports[*].nodePort}'

LoadBalancer Service

LoadBalancer asks the cloud provider to provision an external load balancer pointing at the nodes. externalTrafficPolicy: Local preserves the client source IP (no SNAT) but skips cross-node load balancing. STATUS shows the assigned external IP once ready.

kubernetes
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: LoadBalancer
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
  # cloud-specific annotations
  # annotations:
  #   service.beta.kubernetes.io/aws-load-balancer-type: nlb
  #   service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  # externalTrafficPolicy: Local preserves client IP
  externalTrafficPolicy: Local
# status.ingress shows the external IP/hostname

Headless Service

A headless service (clusterIP: None) has no VIP — DNS returns the pod IPs directly. Required by StatefulSet so each pod gets a stable DNS name (pod-0.svc, pod-1.svc). Also used by clients that want to discover all backends themselves.

kubernetes
apiVersion: v1
kind: Service
metadata:
  name: nginx-headless
spec:
  clusterIP: None       # makes it headless
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
# DNS returns pod IPs directly (not a VIP)
# A records: nginx-headless.default.svc.cluster.local
#   -> 10.1.0.5, 10.1.0.6, 10.1.0.7
# used by StatefulSet for stable pod DNS:
#   <pod-name>.<svc-name>.default.svc.cluster.local

Endpoints & EndpointSlices

Endpoints/EndpointSlices list the pod IPs that actually receive traffic — only Ready pods appear. If a service has no endpoints, the pods aren't matching the selector or aren't Ready. EndpointSlices (v1) scale better for huge services and are the default.

kubernetes
# view endpoints backing a service
kubectl get endpoints nginx

# endpoint slices (newer, more scalable)
kubectl get endpointslices -l kubernetes.io/service-name=nginx

# endpoints track pods that match the selector
# AND pass readiness checks
kubectl describe endpoints nginx

# a service with no ready pods has no endpoints
# - check pod readiness if traffic is refused
# - EndpointSlices are grouped by address type (IPv4/IPv6)

# manually managed endpoints (selector-less service)
05

ConfigMap

Create ConfigMap

ConfigMaps store non-sensitive configuration as key-value pairs. --from-literal for inline values, --from-file to embed whole files (the filename becomes the key). Useful for decoupling config from images without rebuilding.

kubernetes
# from literal values
kubectl create configmap app-config \
  --from-literal=LOG_LEVEL=debug \
  --from-literal=ENABLE_FEATURE=true

# from a file (key = filename, value = contents)
kubectl create configmap app-config --from-file=app.properties

# from a directory (one key per file)
kubectl create configmap app-config --from-file=configs/

# from an env file
kubectl create configmap app-config --from-env-file=app.env

# from a file with a custom key
kubectl create configmap app-config --from-file=custom-key=app.properties

ConfigMap Manifest

data holds string values (each key becomes either an env var or a file when mounted). Multi-line strings use YAML block scalars (|). binaryData accepts base64 for binary blobs. Total size limit is 1 MiB per ConfigMap.

kubernetes
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "debug"
  ENABLE_FEATURE: "true"
  application.yml: |
    server:
      port: 8080
    logging:
      level: debug
binaryData:
  # base64-encoded binary values
  logo.png: iVBORw0KGgoAAAANSUhEUg...

Consume as Environment Variables

envFrom injects every ConfigMap key as an env var in one shot. env with configMapKeyRef picks individual keys. optional: true lets the pod start even if the ConfigMap doesn't exist. Updates to the ConfigMap do NOT refresh env vars in running pods.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:1.0
      env:
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: LOG_LEVEL
      envFrom:
        - configMapRef:
            name: app-config      # injects all keys as env vars
              optional: true      # pod starts even if CM missing

Consume as Volume

Mounting a ConfigMap as a volume projects each key as a file. Updates to the ConfigMap are eventually reflected in mounted files (configured by kubelet sync period) — but apps must re-read them. subPath mounts pin a single key and do NOT update.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:1.0
      volumeMounts:
        - name: config
          mountPath: /etc/config
          readOnly: true
  volumes:
    - name: config
      configMap:
        name: app-config
        items:                       # optional: pick subset
          - key: application.yml
            path: app.yml            # rename file
        defaultMode: 0644
# each ConfigMap key becomes a file under /etc/config

Update & Reload

Mounted ConfigMap files refresh automatically (after kubelet sync), but env vars are immutable for the pod's life — restart pods to pick up env changes. rollout restart is the cleanest way to propagate config updates to a Deployment.

kubernetes
# edit the configmap
kubectl edit configmap app-config

# patch a single key
kubectl patch configmap app-config \
  --type merge -p '{"data":{"LOG_LEVEL":"info"}}'

# mounted files update within ~kubelet sync period (default 60s)
# but env vars do NOT update in running pods

# force pods to pick up new config by rolling them
kubectl rollout restart deployment/app

# verify the new value on a pod
kubectl exec app -- printenv LOG_LEVEL

Immutable ConfigMap

Setting immutable: true freezes the ConfigMap — any change requires delete + recreate. This drastically reduces kubelet-apiserver watch traffic for large, stable configs (a major scalability win at scale). Common in production for app configs that are versioned with the image.

kubernetes
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
immutable: true
data:
  LOG_LEVEL: "debug"
# once immutable, the only change allowed is delete + recreate
# benefits: kubelet does not watch it, lower apiserver load
# useful for large, stable configs in production

# make it mutable again
kubectl patch configmap app-config -p '{"immutable":false}'
# (this fails if currently immutable — must delete instead)
06

Secret

Create Secret

Secrets store sensitive data base64-encoded (NOT encrypted at rest by default — enable encryption at rest for production). Use generic/Opaque for arbitrary data, docker-registry for image pulls, tls for Ingress TLS. Never commit Secret YAML to git.

kubernetes
# generic (opaque) from literals
kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password='S3cur3!'

# from a file (e.g. SSH key, certificate)
kubectl create secret generic ssh-key --from-file=ssh-privatekey=~/.ssh/id_rsa

# docker registry secret (for private image registries)
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=pass \
  [email protected]

# TLS secret from cert/key files
kubectl create secret tls my-tls \
  --cert=fullchain.pem \
  --key=privkey.pem

Secret Manifest

data values must be base64-encoded (echo -n 'value' | base64). stringData is a convenience that accepts plaintext and is merged into data on apply. type Opaque is the default; specialized types (kubernetes.io/tls, kubernetes.io/dockerconfigjson) validate the keys.

kubernetes
apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  username: YWRtaW4=         # base64 of "admin"
  password: UzNjdXIzIQ==     # base64 of "S3cur3!"
# stringData accepts plaintext (auto-encoded on apply)
stringData:
  api-key: "live-key-xyz"
# encode on the command line:
# echo -n 'admin' | base64
# decode:
# echo 'YWRtaW4=' | base64 -d

Use Secret as Environment Variable

Consuming secrets as env vars is identical to ConfigMaps but uses secretKeyRef/secretRef. Note that env vars are visible via /proc/1/environ and to any process in the container — prefer volume mounts for highly sensitive data.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:1.0
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-creds
              key: password
      envFrom:
        - secretRef:
            name: db-creds      # injects all keys
              optional: true

Use Secret as Volume

Mounted secrets are projected as files into a tmpfs (memory) volume — never written to node disk. defaultMode 0400 restricts access. Mounted secrets DO update when the Secret changes (after kubelet sync), unlike env-var secrets.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:1.0
      volumeMounts:
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: db-creds
        defaultMode: 0400       # restrict permissions
        items:
          - key: password
            path: db-password   # written to /etc/secrets/db-password
# memory-backed tmpfs, never written to disk on the node

ImagePullSecret

imagePullSecrets authenticate to private registries when pulling images. Attaching to a ServiceAccount applies it to every pod using that SA — the preferred pattern. The docker-registry secret type produces the right .dockerconfigjson payload automatically.

kubernetes
# attach to a pod
apiVersion: v1
kind: Pod
metadata:
  name: private-app
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: registry.example.com/myapp:1.0

# attach to a service account (auto-applied to all pods using it)
kubectl patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

# verify
kubectl get sa default -o jsonpath='{.imagePullSecrets}'

Secret Types

Specialized secret types enable validation and integration: tls for Ingress, dockerconfigjson for image pulls, service-account-token auto-mounted into pods. Using the right type ensures the keys are validated and consumed by the right controllers.

kubernetes
# Opaque (default) — arbitrary key-value
type: Opaque

# kubernetes.io/service-account-token — auto for SAs
type: kubernetes.io/service-account-token

# kubernetes.io/dockerconfigjson — registry auth
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64>

# kubernetes.io/tls — for Ingress TLS
type: kubernetes.io/tls
data:
  tls.crt: <base64>
  tls.key: <base64>

# kubernetes.io/basic-auth
type: kubernetes.io/basic-auth
stringData:
  username: admin
  password: pass
07

Namespace

Create Namespace

Namespaces partition a cluster for multiple teams or environments (dev/stage/prod). Names of resources must be unique within a namespace. Some resources (Nodes, PVs, StorageClasses) are cluster-scoped and not namespaced.

kubernetes
# imperative
kubectl create namespace dev

# declarative
apiVersion: v1
kind: Namespace
metadata:
  name: dev
  labels:
    environment: development
    team: platform

# apply
kubectl apply -f namespace.yaml

# list namespaces
kubectl get namespaces

# set default namespace for current context
kubectl config set-context --current --namespace=dev

ResourceQuota

ResourceQuota caps aggregate resource usage in a namespace — total CPU/memory requests/limits, object counts (pods, PVCs, services, deployments). Essential for multi-tenant clusters to prevent one team from starving others. Pods without explicit requests are rejected once quota is set.

kubernetes
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: dev
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"
    services.loadbalancers: "2"
    persistentvolumeclaims: "10"
    count/deployments.apps: "5"
# limits total aggregate resource use in the namespace

LimitRange

LimitRange sets per-resource defaults and bounds: default requests/limits for pods that don't specify them, plus min/max to constrain outrageous values. Pairs with ResourceQuota — without default requests, quota would reject every pod that forgets to set them.

kubernetes
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: dev
spec:
  limits:
    - type: Container
      default:               # default limits (if unset)
        cpu: 500m
        memory: 512Mi
      defaultRequest:        # default requests (if unset)
        cpu: 100m
        memory: 128Mi
      max:                   # max allowed limits
        cpu: "2"
        memory: 2Gi
      min:                   # min allowed requests
        cpu: 50m
        memory: 64Mi
    - type: PersistentVolumeClaim
      max:
        storage: 100Gi
      min:
        storage: 1Gi

Delete Namespace & Finalizers

Deleting a namespace cascades all its resources. A namespace stuck in Terminating usually has a finalizer that can't complete — an unavailable APIService (e.g. metrics-server) or a CRD whose controller is gone. Removing the finalizer forces completion but can leak external resources.

kubernetes
# delete a namespace (cascades all resources in it)
kubectl delete namespace dev

# if stuck in Terminating, inspect finalizers
kubectl get namespace dev -o yaml

# remove finalizer to force completion (DANGEROUS)
kubectl patch namespace dev -p '{"metadata":{"finalizers":null}}'

# resources that block deletion:
# - APIService unavailable (metrics-server removed)
# - custom resources whose CRD is gone
# - PVCs with attached PVs

# check what's still in the namespace
kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get -n dev --ignore-not-found

Default Namespaces

kube-system runs the control plane and add-ons (DNS, kube-proxy, CNI). kube-public holds bootstrap data readable by unauthenticated users. kube-node-lease stores node heartbeats as Lease objects for performance. default is where your workloads land if you don't specify a namespace.

kubernetes
# built-in namespaces
kubectl get namespaces
# NAME              STATUS   AGE
# default           Active   30d   <- user workloads land here
# kube-system       Active   30d   <- control plane / addons
# kube-public       Active   30d   <- publicly readable (bootstrap)
# kube-node-lease   Active   30d   <- node heartbeats (leases)

# kube-system holds:
# - kube-dns / CoreDNS
# - kube-proxy
# - metrics-server
# - CNI, CSI, cloud provider controllers

# kube-public is readable by all (used for cluster info discovery)

Namespace-scoped vs Cluster-scoped

Most workload resources are namespaced; infrastructure (nodes, PVs, StorageClasses, ClusterRoles) is cluster-scoped. A Service can only target pods in its own namespace. -A is the shorthand for --all-namespaces — essential for cluster-wide visibility.

kubernetes
# which resources are namespaced?
kubectl api-resources --namespaced=true
kubectl api-resources --namespaced=false

# namespaced: pods, services, deployments, configmaps, secrets
# cluster-scoped: nodes, namespaces, PVs, StorageClasses, ClusterRoles

# access across namespaces:
# - a Service selector only matches pods in the SAME namespace
# - cross-namespace refs require special controllers (e.g. external-dns)

# list resources across all namespaces
kubectl get pods --all-namespaces
kubectl get pods -A   # shorthand
08

Labels & Selectors

Label Syntax

Labels are key-value pairs used to identify and group resources. Keys can have an optional prefix (a DNS subdomain, e.g. app.kubernetes.io/). Labels are meant for querying/filtering, unlike annotations which are for arbitrary non-identifying metadata.

kubernetes
# labels are key=value pairs
# key: prefix (optional DNS subdomain) + name
# value: string, up to 63 chars

metadata:
  labels:
    app: nginx
    tier: frontend
    env: prod
    app.kubernetes.io/name: nginx
    app.kubernetes.io/version: "1.25"

# view labels
kubectl get pods --show-labels

# add/overwrite a label
kubectl label pod nginx env=prod
kubectl label pod nginx env=staging --overwrite

# remove a label (trailing dash)
kubectl label pod nginx env-

Equality & Set Selectors

Two selector flavors: equality-based (=, ==, !=) and set-based (in, notin, exists, !exists). Multiple comma-separated conditions are ANDed. Services and Deployments use equality-based selectors in YAML; kubectl -l supports both.

kubernetes
# equality-based: = == !=
kubectl get pods -l app=nginx
kubectl get pods -l env!=dev
kubectl get pods -l 'app=nginx,env=prod'   # AND of conditions

# set-based: in, notin, exists
kubectl get pods -l 'env in (prod,staging)'
kubectl get pods -l 'env notin (dev)'
kubectl get pods -l 'tier'                 # key exists
kubectl get pods -l '!canary'              # key does not exist

# combine equality and set-based
kubectl get pods -l 'app=nginx,env in (prod,staging)'

Labeling Resources

label mutates identifying metadata — use it for anything selectors should match on. annotate is for non-identifying metadata (owner, ticket, description, tooling hints). Nodes are commonly labeled with topology or hardware hints to drive scheduling.

kubernetes
# label a pod
kubectl label pod nginx app=nginx tier=web

# label all pods matching a selector
kubectl label pods -l app=oldapp app=newapp --overwrite

# label a node (for scheduling hints)
kubectl label node node-1 disktype=ssd

# remove a node label
kubectl label node node-1 disktype-

# list nodes by label
kubectl get nodes -l disktype=ssd

# annotate (non-identifying metadata)
kubectl annotate pod nginx owner=alice
kubectl annotate pod nginx description="web server"

Selectors in Manifests

In YAML, selectors use matchLabels (equality) and matchExpressions (set-based with In/NotIn/Exists/DoesNotExist). For Deployments the selector is immutable after creation — and the pod template labels must match it. Services only support matchLabels.

kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:            # equality-based
      app: nginx
    matchExpressions:       # set-based (optional)
      - key: env
        operator: In
        values: [prod, staging]
      - key: tier
        operator: Exists
  template:
    metadata:
      labels:
        app: nginx          # MUST match selector
        env: prod
        tier: web

Annotations

Annotations hold non-identifying metadata used by tools — kubectl apply stores the last-applied config, cert-manager reads issuer hints, Prometheus reads scrape config, controllers store rollout state. Unlike labels, annotations values can be large JSON blobs.

kubernetes
metadata:
  annotations:
    kubernetes.io/change-cause: "bump nginx to 1.26"
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1",...}
    cert-manager.io/cluster-issuer: "letsencrypt-prod"

# view annotations
kubectl get pod nginx -o jsonpath='{.metadata.annotations}'

# annotate
kubectl annotate pod nginx owner=team-platform

# remove an annotation (trailing dash)
kubectl annotate pod nginx owner-

Recommended Labels

The app.kubernetes.io/* labels are a shared vocabulary so tools (Helm, operators, dashboards) can group all resources of an application. name+instance uniquely identify a release; component/part-of describe hierarchy; managed-by/created-by track ownership.

kubernetes
# shared recommended labels (app.kubernetes.io/*)
metadata:
  labels:
    app.kubernetes.io/name: nginx
    app.kubernetes.io/instance: nginx-prod
    app.kubernetes.io/version: "1.25"
    app.kubernetes.io/component: web
    app.kubernetes.io/part-of: storefront
    app.kubernetes.io/managed-by: helm
    app.kubernetes.io/created-by: controller-manager

# Helm applies these automatically; kubectl doesn't.
# They unify how tooling identifies an application's resources
# across Deployments, Services, Ingresses, ConfigMaps, etc.
09

ReplicaSet

ReplicaSet Manifest

A ReplicaSet maintains a stable set of replica pods matching a selector — it recreates pods when they're deleted or fail. You almost always use a Deployment instead, which wraps a ReplicaSet and adds rolling updates and rollbacks.

kubernetes
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
# kubectl get rs
# kubectl describe rs nginx

How ReplicaSet Works

The ReplicaSet controller reconciles desired vs actual pod count using the selector. Each pod gets an ownerReference pointing at the RS — that's why kubectl get pod shows 'Owned by ReplicaSet/nginx'. Scaling just changes the desired count.

kubernetes
# the controller loop:
# 1. count pods matching selector
# 2. if fewer than replicas -> create from template
# 3. if more than replicas -> delete excess (arbitrary order)

# owned-by reference on each pod
kubectl get pod -l app=nginx -o jsonpath='{.items[*].metadata.ownerReferences}'

# if you delete a pod, the RS creates a new one immediately
kubectl delete pod <pod-name>
kubectl get pods -l app=nginx

# scale by editing replicas
kubectl scale rs nginx --replicas=5

ReplicaSet vs ReplicationController

ReplicationController (the original v1 primitive) only supports equality selectors and is deprecated. ReplicaSet (apps/v1) adds set-based selectors. In practice, you should use Deployment — it manages ReplicaSets for you and provides rolling updates/rollbacks.

kubernetes
# ReplicationController (v1, legacy): equality-only selectors
# ReplicaSet (apps/v1, modern): set-based selectors (in, notin, exists)

# ReplicationController is deprecated; do not use.
apiVersion: v1
kind: ReplicationController      # legacy
# ...
# ReplicaSet is the successor:
apiVersion: apps/v1
kind: ReplicaSet                 # modern
# ...
# Deployment wraps ReplicaSet and is the recommended primitive:
apiVersion: apps/v1
kind: Deployment                 # use this in practice

Manual Scaling

kubectl scale changes the replicas field. If an HPA targets the same resource, it will override manual scaling on the next reconcile. Prefer kubectl scale for one-offs and HPA for ongoing elastic workloads.

kubernetes
# scale a replicaset directly
kubectl scale rs nginx --replicas=5

# scale via patch
kubectl patch rs nginx -p '{"spec":{"replicas":5}}'

# scale multiple resources at once
kubectl scale deployment nginx deployment web --replicas=3

# scale by referencing a file
kubectl scale --replicas=5 -f nginx.yaml

# the autoscaler (HPA) will override manual scaling
# if it targets the same resource

Template Changes & Adoption

Unlike a Deployment, a bare ReplicaSet does not roll out changes to the pod template — only newly created pods (after a delete/scale-up) get the new spec. A ReplicaSet will adopt any orphaned pod matching its selector, which can cause surprising behavior.

kubernetes
# changing the pod template does NOT update existing pods
# (unlike a Deployment, a bare ReplicaSet won't roll out)
kubectl edit rs nginx   # change image -> new pods use it, old ones don't

# delete old pods to force them onto the new template
kubectl delete pods -l app=nginx

# a ReplicaSet will ADOPT existing pods that match its selector
# even pods created independently — useful (and dangerous)

# orphaned pods (no owner) matching the selector are taken over
kubectl get pods -l app=nginx --show-kind

Removing Pods from a ReplicaSet

To remove a pod without the ReplicaSet recreating it, either scale the RS to 0 first, or strip the ownerReferences from the pod so the RS no longer owns it. Useful for debugging a specific instance or extracting logs before cleanup.

kubernetes
# detaching a pod from its ReplicaSet (so it isn't replaced)
kubectl patch pod <pod-name> \
  -p '{"metadata":{"ownerReferences":[]}}'

# now deleting this pod won't trigger a replacement,
# because the RS no longer owns it

# or scale the RS to 0 first, then delete
kubectl scale rs nginx --replicas=0
kubectl delete pod <pod-name>

# useful for debugging a specific failing pod instance
# without the RS immediately recreating it
10

StatefulSet

StatefulSet Manifest

StatefulSet gives pods a stable identity (name, hostname, storage) and ordered deploy/scale. Requires a headless serviceName. volumeClaimTemplates creates a dedicated PVC per pod — so even if pod-0 is rescheduled, it reattaches the same volume.

kubernetes
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql          # required: headless service
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8
          ports:
            - containerPort: 3306
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:       # one PVC per pod
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

Stable Identity

Each pod gets a stable ordinal name (mysql-0, mysql-1, …) and a stable DNS name via the headless service. When a node fails, pod-0 is rescheduled elsewhere but keeps its name and reattaches its volume — critical for clustered databases that depend on stable membership.

kubernetes
# pods are named <statefulset-name>-<ordinal>
# mysql-0, mysql-1, mysql-2
kubectl get pods -l app=mysql

# each pod has stable DNS (via the headless service):
#   mysql-0.mysql.default.svc.cluster.local
#   mysql-1.mysql.default.svc.cluster.local

# stable hostname inside the pod:
kubectl exec mysql-0 -- hostname
# -> mysql-0

# ordinal index also exposed:
kubectl exec mysql-0 -- sh -c 'echo $HOSTNAME'
# -> mysql-0

# pod identity persists across reschedules

Stable Storage

volumeClaimTemplates generates one PVC per pod with a predictable name (data-mysql-0, …). PVCs survive pod and StatefulSet deletion — this prevents data loss but means you must explicitly delete PVCs to reclaim storage. The same volume always reattaches to the same ordinal pod.

kubernetes
# one PVC per pod, named <volumeClaimTemplate-name>-<pod-name>
kubectl get pvc
# NAME        STATUS   VOLUME       CAPACITY   ...
# data-mysql-0   Bound   pvc-aaa      10Gi
# data-mysql-1   Bound   pvc-bbb      10Gi
# data-mysql-2   Bound   pvc-ccc      10Gi

# when pod-0 is rescheduled, it re-binds data-mysql-0
# deleting the StatefulSet does NOT delete PVCs by default
# (protects from accidental data loss)

# to actually delete the data:
kubectl delete pvc data-mysql-0 data-mysql-1 data-mysql-2

Ordered Deployment & Pod Management

OrderedReady (default) creates pods strictly in order 0→1→2 and deletes in reverse — required by systems that need a primary to bootstrap before replicas. Parallel creates/deletes all pods simultaneously — faster for symmetric sharded stores (Cassandra, Elasticsearch).

kubernetes
spec:
  podManagementPolicy: OrderedReady   # default: strict order
  # podManagementPolicy: Parallel      # create/delete all at once

  replicas: 3
  # OrderedReady: create pod-0, then pod-1, then pod-2 (each waits for Ready)
  # deletion is reverse: pod-2, pod-1, pod-0

  # Parallel: all pods created/deleted simultaneously
  # identity and storage remain stable either way

# check policy
kubectl get statefulset mysql -o jsonpath='{.spec.podManagementPolicy}'

# parallel is great for sharded stores that don't need bootstrap order

Rolling Updates

StatefulSet rolling updates go in reverse ordinal order (N-1 down to 0). partition lets you canary the highest-ordinal pods first. OnDelete pauses updates until you manually delete a pod — useful for controlled migrations. Rollouts are slower than Deployments due to ordering.

kubernetes
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2          # only pods with ordinal >= 2 update
      maxUnavailable: 1     # (default 1) allowed down
  # type: OnDelete  -> only update when pod is manually deleted

# partition enables canaries: set partition=N to update
# only the top N pods; lower to 0 to roll out to all.

kubectl rollout status statefulset/mysql
# waits for all pods to be updated

# StatefulSets also support rollout history/undo,
# but only when the change is in the pod template

Headless Service Association

The headless service (clusterIP: None) referenced by spec.serviceName is what makes stable per-pod DNS work. Without it the StatefulSet won't create pods. Clients resolve the service name for round-robin, or pod-0.svc to pin a primary. Don't forget to create the Service.

kubernetes
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  clusterIP: None         # headless
  selector:
    app: mysql
  ports:
    - port: 3306
---
# StatefulSet references this service via spec.serviceName
# this gives each pod a stable DNS:
#   mysql-0.mysql.default.svc.cluster.local

# clients can:
# - resolve mysql -> any pod IP (load-balanced by DNS round-robin)
# - resolve mysql-0.mysql -> pod-0's IP (pin to a specific replica)

# required: serviceName MUST match the headless service name
11

DaemonSet

DaemonSet Manifest

A DaemonSet runs exactly one pod per node (matching its scheduling constraints). Perfect for node-level agents: log collectors, monitoring, storage, CNI, ingress controllers. New nodes automatically get a pod; deleted nodes have theirs garbage-collected.

kubernetes
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
  namespace: kube-system
  labels:
    app: log-agent
spec:
  selector:
    matchLabels:
      app: log-agent
  template:
    metadata:
      labels:
        app: log-agent
    spec:
      containers:
        - name: agent
          image: fluent-bit:2.2
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
      # one pod per (matching) node automatically

Node Scheduling

nodeSelector is the simplest constraint (all key=value must match). nodeAffinity adds In/NotIn/Exists operators and preferred/required rules. DaemonSets respect these like any pod — combined with tolerations you can run agents on tainted control-plane nodes too.

kubernetes
spec:
  template:
    spec:
      nodeSelector:                # simplest: must match all
        disktype: ssd

      # more flexible affinity
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values: [linux]

      # only run on nodes labeled role=logging
      nodeSelector:
        role: logging

      tolerations:                 # tolerate tainted nodes
        - key: node-role.kubernetes.io/control-plane
          operator: Exists

Update Strategy

RollingUpdate (default) updates pods node by node, bounded by maxUnavailable (can be a number or %). OnDelete pauses updates until you manually delete each pod — useful when node-by-node control is required. maxSurge defaults to 0 (DaemonSets rarely want extra pods).

kubernetes
spec:
  updateStrategy:
    type: RollingUpdate           # default
    rollingUpdate:
      maxUnavailable: 1           # how many nodes update at once
      maxSurge: 0                 # (default 0) no extra pods
  # type: OnDelete               # only update when old pod is deleted

# trigger an update (e.g. new image)
kubectl set image daemonset/log-agent agent=fluent-bit:2.3

# watch rollout
kubectl rollout status daemonset/log-agent -n kube-system

# rollout restart (recreate pods on every node)
kubectl rollout restart daemonset/log-agent -n kube-system

Taints & Tolerations

Taints repel pods; tolerations let specific pods ignore a taint. Control-plane nodes are tainted NoSchedule so user workloads don't land there — node-level agents tolerate it to run everywhere. NoExecute also evicts non-tolerating pods already running.

kubernetes
# inspect a node's taints
kubectl describe node node-1 | grep Taints
# Taints: node-role.kubernetes.io/control-plane:NoSchedule

# tolerate control-plane so the agent runs there too
spec:
  template:
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node.kubernetes.io/disk-pressure
          operator: Exists
          effect: NoExecute

# taint effects: NoSchedule | NoExecute | PreferNoSchedule

DaemonSet Use Cases

DaemonSets are the standard deployment shape for node-level infrastructure. They frequently need privileged access (hostNetwork, hostPID, hostPath) to do their job — grant these deliberately and only to trusted images. Never use a DaemonSet for app workloads that should scale horizontally.

kubernetes
# classic DaemonSet workloads (node-level agents):
# - log collection:        fluent-bit, filebeat, promtail
# - monitoring:            node-exporter, datadog-agent
# - networking:            kube-proxy, CNI plugins, ingress controllers
# - storage:               CSI node plugins, Rook/Ceph
# - security:              Falco, Tetragon, runtime security
# - GPU/FPGA drivers:      nvidia-device-plugin

# storage / device plugins often need:
spec:
  template:
    spec:
      hostNetwork: true        # use node network namespace
      hostPID: true            # see node processes
      hostPath:                # mount node paths
        - path: /var/log
        - path: /var/lib/docker/containers

DaemonSet Status

DESIRED equals the number of nodes matching the scheduling constraints; CURRENT/READY show how many pods exist and are passing readiness. -o wide is the easiest way to see which node each pod landed on. DaemonSets support rollout history/undo just like Deployments.

kubernetes
# desired = number of matching nodes
kubectl get daemonset log-agent -n kube-system
# NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
# log-agent    5         5         5       5            5

# show which nodes a DaemonSet pod landed on
kubectl get pods -n kube-system -l app=log-agent -o wide

# rollout history
kubectl rollout history daemonset/log-agent -n kube-system

# rollback
kubectl rollout undo daemonset/log-agent -n kube-system

# check a per-node pod's logs
kubectl logs -n kube-system log-agent-abcde
12

Job & CronJob

Job Manifest

A Job runs pods until a specified number of them complete successfully (completions). restartPolicy must be Never or OnFailure (not Always — that's for long-running workloads). backoffLimit caps retries; activeDeadlineSeconds caps total runtime.

kubernetes
apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  completions: 1              # how many successful pods needed
  parallelism: 1              # how many run concurrently
  backoffLimit: 6             # retries before marking Failed
  activeDeadlineSeconds: 600  # hard timeout
  template:
    spec:
      restartPolicy: Never    # OnFailure | Never
      containers:
        - name: pi
          image: perl
          command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
# kubectl get jobs
# kubectl logs job/pi

Job Completions & Parallelism

completions = how many successful pods satisfy the Job; parallelism = how many run concurrently. Indexed Jobs assign each pod an ordinal (env var JOB_COMPLETION_INDEX) — great for sharded batch processing where each pod owns one slice. NonIndexed (default) just needs N successes from anywhere.

kubernetes
spec:
  completions: 5         # 5 successful pods total
  parallelism: 2         # 2 pods run at a time
  backoffLimit: 4
  # default completionMode: NonIndexed (any 5 successes)
  completionMode: Indexed
  # Indexed: pods get 0..N-1 in JOB_COMPLETION_INDEX env,
  # useful for sharded work (each pod handles one shard)

  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: worker
          image: worker:1.0

# set via kubectl
kubectl create job pi --image=perl -- perl -Mbignum=bpi -wle "print bpi(2000)"

Job Backoff & Timeouts

backoffLimit retries failed pods (exponential backoff). activeDeadlineSeconds is the hard ceiling — if hit, the Job is marked Failed even with retries left. ttlSecondsAfterFinished auto-cleans completed Jobs. podFailurePolicy (1.31+) lets you fail fast on certain exit codes or ignore voluntary evictions.

kubernetes
spec:
  backoffLimit: 6            # default 6, retries on failure
  activeDeadlineSeconds: 3600  # hard wall, even if retries remain
  ttlSecondsAfterFinished: 86400  # auto-delete Job 1 day after done
  startingDeadlineSeconds: 300  # (CronJob only) miss window

  # pod failure handling (v1.31+):
  podFailurePolicy:
    rules:
      - action: FailJob
        onExitCodes:
          operator: In
          values: [42]      # exit 42 -> don't retry, fail fast
      - action: Count
        onPodConditions:
          - type: DisruptionTarget   # don't count voluntary evictions

  template:

CronJob Manifest

A CronJob creates Jobs on a cron schedule (5 fields: minute hour day month weekday). concurrencyPolicy controls overlapping runs: Forbid skips a new run if the previous is still going; Replace kills the old and starts new. timeZone (1.27+) removes the need for UTC arithmetic.

kubernetes
apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup
spec:
  schedule: "0 2 * * *"          # 5-field cron: 02:00 daily
  timeZone: "Asia/Shanghai"
  concurrencyPolicy: Forbid      # Allow | Forbid | Replace
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  startingDeadlineSeconds: 200
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: backup:1.0
# kubectl get cronjob
# kubectl get jobs

CronJob Suspend & History

suspend: true pauses a CronJob without deleting it — handy during incidents. History limits keep the Job list manageable (default 3 success / 1 failure). kubectl create job --from=cronjob/X runs the template immediately, perfect for manual triggers or backfilling missed runs.

kubernetes
spec:
  schedule: "*/5 * * * *"
  suspend: true                # pause without deleting the CronJob
  successfulJobsHistoryLimit: 3   # default 3
  failedJobsHistoryLimit: 1       # default 1

# toggle suspend
kubectl patch cronjob backup -p '{"spec":{"suspend":true}}'
kubectl patch cronjob backup -p '{"spec":{"suspend":false}}'

# trigger a one-off run immediately
kubectl create job --from=cronjob/backup manual-backup

# clean up old jobs manually
kubectl delete jobs -l job-name=backup

CronJob Schedule & Edge Cases

Standard 5-field cron. When both day-of-month and day-of-week are restricted, cron fires on EITHER match (OR semantics) — a classic gotcha. If a CronJob's scheduled time is missed beyond startingDeadlineSeconds, that run is skipped rather than backfilled.

kubernetes
# cron syntax (5 fields):
# ┌───────────── minute (0-59)
# │ ┌───────────── hour (0-23)
# │ │ ┌───────────── day of month (1-31)
# │ │ │ ┌───────────── month (1-12)
# │ │ │ │ ┌───────────── day of week (0-6, Sun=0)
# "*/5 * * * *"      every 5 minutes
# "0 2 * * *"        daily at 02:00
# "0 0 * * 0"        weekly Sunday midnight
# "0 0 1 * *"        monthly on the 1st
# "30 3-5 * * *"     03:30, 04:30, 05:30

# gotchas:
# - day-of-month AND day-of-week both set => OR (not AND)
# - if startingDeadlineSeconds is missed, the run is skipped
# - control-plane restart can skip a tick if the schedule is missed
# - cron interprets literal fields; prefer timeZone over UTC math
13

Ingress

Ingress Manifest

Ingress exposes HTTP/HTTPS routes from outside the cluster to Services, managed by an Ingress controller (nginx, traefik, ALB, …). ingressClassName selects which controller handles it. Each rule maps a host + path to a service+port backend.

kubernetes
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 8080

Ingress with TLS

tls entries bind a TLS secret (type kubernetes.io/tls, holding tls.crt + tls.key) to one or more hosts. The Ingress controller terminates TLS and proxies plain HTTP to the backend. cert-manager can auto-issue and rotate Let's Encrypt certs based on the cluster-issuer annotation.

kubernetes
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-tls
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - app.example.com
        - www.example.com
      secretName: web-tls-secret   # kubernetes.io/tls secret
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

# auto-issue certs with cert-manager:
# annotations:
#   cert-manager.io/cluster-issuer: letsencrypt-prod

Ingress Rules & Default Backend

defaultBackend handles traffic that matches no rule (no host or unmatched path). Multiple rules let one Ingress route many virtual hosts. Each rule can have multiple path backends — a single Ingress can front a whole application's routing topology.

kubernetes
spec:
  ingressClassName: nginx
  defaultBackend:                # catch-all for unmatched traffic
    service:
      name: default-svc
      port:
        number: 80
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service: { name: api, port: { number: 8080 } }
    - host: blog.example.com     # different host -> different svc
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: blog, port: { number: 80 } }

IngressClass

An IngressClass declares a controller (e.g. k8s.io/ingress-nginx). Ingresses reference it via spec.ingressClassName so multiple controllers can coexist. Exactly one class can be marked default with the is-default-class annotation — new Ingresses without an explicit class use it.

kubernetes
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  controller: k8s.io/ingress-nginx

# an Ingress picks a class via spec.ingressClassName
# (replaces the deprecated kubernetes.io/ingress.class annotation)

# multiple IngressClasses in one cluster:
# - nginx    (community ingress-nginx)
# - traefik  (Traefik controller)
# - alb      (AWS ALB)
# only one IngressClass can be marked default

Path Types

Exact matches the path verbatim; Prefix matches whole path segments (/api matches /api and /api/foo but not /apix). ImplementationSpecific defers to the controller — e.g. nginx regex. Always prefer Prefix or Exact for portable behavior; ImplementationSpecific ties you to one controller.

kubernetes
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api/v1
            pathType: Exact        # only /api/v1 (exact match)
            backend: ...
          - path: /api
            pathType: Prefix       # /api, /api/, /api/x (prefix match)
            backend: ...
          - path: /old
            pathType: ImplementationSpecific  # controller decides
            backend: ...

# Prefix matches path segments, not raw string prefix:
# /api   matches /api, /api/, /api/x  but NOT /apix
# Exact  matches the path verbatim
# ImplementationSpecific defers to the controller

Annotations (Nginx Controller)

Controller-specific annotations configure behavior beyond the standard Ingress spec. ingress-nginx has dozens: rewrite-target, ssl-redirect, body-size, timeouts, CORS, basic auth, configuration snippets. They're controller-specific — porting to another controller means rewriting them.

kubernetes
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    kubernetes.io/ingress.class: nginx          # legacy form
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Frame-Options: DENY";
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /v1(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service: { name: api, port: { number: 8080 } }
14

Volume (PV/PVC)

PersistentVolume Manifest

A PersistentVolume (PV) is a cluster-scoped piece of storage provisioned by an admin or dynamically by a StorageClass. accessModes describe how it can be mounted. reclaimPolicy controls what happens to the storage when the PVC is released.

kubernetes
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-1
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce        # RWO: one node R/W
  persistentVolumeReclaimPolicy: Retain   # Retain | Delete | Recycle
  storageClassName: manual
  hostPath:                # for dev only (single node)
    path: /mnt/data
  # nfs:
  #   server: 10.0.0.1
  #   path: /export/data
  # csi:
  #   driver: ebs.csi.aws.com
  #   volumeHandle: vol-xxxx

PersistentVolumeClaim Manifest

A PersistentVolumeClaim (PVC) is a namespaced request for storage. The controller binds it to a matching PV (accessModes, sufficient capacity, same StorageClass). With a default StorageClass, leaving storageClassName empty triggers dynamic provisioning.

kubernetes
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem     # Filesystem | Block
  selector:                  # optional: filter PVs by labels
    matchLabels:
      disktype: ssd
# kubectl get pvc
# STATUS: Pending -> Bound once a matching PV is found

Access Modes

AccessModes describe mount semantics — but actual support depends on the storage backend. EBS/GCE PD/Azure Disk are RWO only. RWX (multi-node read/write) needs NFS, CephFS, or a distributed filesystem. RWOP (1.27+) restricts to one pod even on the same node.

kubernetes
# four access modes (support is driver-specific):
# - ReadWriteOnce  (RWO):  mounted R/W by ONE node
# - ReadOnlyMany   (ROX):  mounted R/O by MANY nodes
# - ReadWriteMany  (RWX):  mounted R/W by MANY nodes
# - ReadWriteOncePod (RWOP): mounted R/W by ONE pod (1.27+)

# examples by backend:
# AWS EBS, GCE PD, Azure Disk:  RWO only
# NFS:                          RWO, ROX, RWX
# CephFS, Portworx:             RWO, RWX
# Block volumes:                RWO

# RWX is required for pods on different nodes
# to share the same volume (e.g. read-only assets)

Reclaim Policy

Retain keeps the data for safety — you must manually wipe and rebind the PV. Delete (default for dynamic provisioning) wipes both PV and underlying storage when the PVC goes. Recycle is deprecated. A Released PV with Retain needs its claimRef cleared before rebinding.

kubernetes
# what happens when a PVC is deleted:
# - Retain:   PV stays, data preserved; PV must be manually
#             reclaimed (delete data, then delete PV) before reuse
# - Delete:   PV and the underlying storage are deleted (dynamic only)
# - Recycle:  deprecated; ran scrub on the volume

# change policy on an existing PV
kubectl patch pv pv-1 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

# a released PV (PVC deleted, policy Retain) is stuck:
#   phase=Released, claimRef still set
# to reuse: clear claimRef
kubectl patch pv pv-1 --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'

PVC in a Pod

Mount a PVC like any volume via persistentVolumeClaim.claimName. subPath mounts a subdirectory — handy for sharing one volume across pods/paths. RWO volumes can be mounted by multiple pods but only on the same node; cross-node mounting causes multi-attach errors.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: db
spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-claim
  containers:
    - name: db
      image: postgres:16
      volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
          subPath: pgdata       # mount a subdirectory of the volume

# a PVC is namespaced and bound to ONE PV
# RWO PVCs can be mounted by multiple pods on the SAME node
# (e.g. one writer, several readers on that node)

Static vs Dynamic Provisioning

Static provisioning requires admins to pre-create PVs — fine for NFS but painful at scale. Dynamic provisioning lets a PVC trigger on-demand volume creation via a StorageClass's provisioner. Modern clusters almost always use dynamic provisioning; static is reserved for shared/legacy storage.

kubernetes
# STATIC: admin pre-creates PVs; PVCs bind to them.
#   Good for: pre-existing NFS exports, manual capacity planning.

# DYNAMIC: PVC requests storage; StorageClass provisions a PV
#   automatically (AWS EBS, GCE PD, Ceph, local-path, etc.)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-claim
spec:
  storageClassName: ebs-sc      # triggers dynamic provisioning
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi

# no PV exists -> provisioner creates one -> binds to PVC
# kubectl get pvc dynamic-claim -> Bound
# kubectl get pv -> shows the dynamically-created PV
15

StorageClass

StorageClass Manifest

A StorageClass describes a 'flavor' of dynamically provisioned storage: the provisioner (CSI driver), parameters (disk type, encryption, fsType), and policies. PVCs reference it via spec.storageClassName. One cluster typically has several classes (fast SSD, bulk HDD, NFS, …).

kubernetes
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
  encrypted: "true"
reclaimPolicy: Delete            # Retain | Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
mountOptions:
  - noatime
# kubectl get storageclass

Default StorageClass

Exactly one StorageClass can be the default (annotation is-default-class: true). PVCs that omit storageClassName automatically use it — convenient for users who don't care which backend they get. Use kubectl get sc to see which class is marked (default).

kubernetes
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

# a PVC with no storageClassName uses the default
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: auto
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 5Gi
# (no storageClassName set -> uses 'standard')

Provisioners

kubernetes.io/no-provisioner means static binding only (no dynamic creation). All real dynamic provisioning uses CSI drivers — cloud providers ship their own (ebs.csi.aws.com, etc.), and self-hosted options (Longhorn, Rook/Ceph, local-path) cover bare metal. Parameters are entirely provisioner-specific.

kubernetes
# built-in (no external driver): only static binding
provisioner: kubernetes.io/no-provisioner

# cloud CSI drivers (dynamic):
provisioner: ebs.csi.aws.com          # AWS EBS
provisioner: pd.csi.storage.gke.io    # GCE PD
provisioner: disk.csi.azure.com       # Azure Disk
provisioner: nfs.csi.k8s.io           # NFS

# popular self-hosted:
provisioner: driver.longhorn.io       # Longhorn
provisioner: rook-ceph.ceph.com       # Rook/Ceph
provisioner: rancher.io/local-path    # local-path (dev)

# parameters are provisioner-specific
# (e.g. AWS: type=gp3|iops=3000|encrypted=true)

Volume Binding Mode

Immediate binds the PVC the moment it's created — risky for zonal disks because the pod might land in a different zone. WaitForFirstConsumer defers binding until a pod actually consumes the PVC, so the volume lands in the right zone. Always use WaitForFirstConsumer for zonal cloud disks.

kubernetes
spec:
  volumeBindingMode: WaitForFirstConsumer
  # ^ default for many CSI drivers
  # alternatives:
  # volumeBindingMode: Immediate

# Immediate: PVC binds/provisions as soon as created
#   risk: volume created in zone A, pod scheduled in zone B
#   -> multi-attach / cross-zone latency issues

# WaitForFirstConsumer: delays binding until a pod using the PVC
#   is scheduled. The PV is then created in the pod's zone.
#   REQUIRED for topology-constrained backends (EBS, etc.)

# topology constraints come from the CSI driver via
# AllowedTopologies

Reclaim Policy per StorageClass

StorageClass.reclaimPolicy sets the policy of every PV it dynamically provisions. Delete (default) destroys the volume when the PVC is deleted — efficient but dangerous for production data. To save data, switch the PV to Retain before deleting the PVC, or use a Retain-class from the start.

kubernetes
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-temp
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete            # default for dynamic volumes
# reclaimPolicy: Retain          # keep data after PVC delete

# the StorageClass policy becomes the PV's policy
# when a PV is dynamically provisioned

# to preserve data, either:
# 1. use a Retain StorageClass
# 2. patch the dynamically-created PV to Retain before deleting PVC:
kubectl patch pv pvc-xxxx -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

# then delete PVC safely; data survives

Volume Expansion

Setting allowVolumeExpansion: true lets users grow a PVC by editing spec.resources.requests.storage. The CSI driver expands the underlying volume, then kubelet expands the filesystem on next mount. Shrinking is never supported. Most cloud CSI drivers support online expansion (no unmount needed).

kubernetes
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true       # must be true to resize

# then resize by editing the PVC's requested size
kubectl patch pvc data-claim -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

# status.conditions shows Resizing / FileSystemResizePending
kubectl describe pvc data-claim

# requirements:
# - StorageClass.allowVolumeExpansion = true
# - driver supports online expansion (most CSI drivers do)
# - shrinking is NOT supported (only grow)
# - FilesystemResize happens after the PV is expanded
16

Helm

Helm Install & Release

Helm packages Kubernetes manifests as a 'chart' and manages them as a 'release'. install creates a release; upgrade applies changes; upgrade --install is idempotent. Releases are tracked by a Secret/ConfigMap in the namespace — that's how rollback works.

kubernetes
# add and update a chart repo
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# search for charts
helm search repo nginx
helm search hub wordpress   # searches Artifact Hub

# install a chart (creates a release)
helm install my-nginx bitnami/nginx --namespace web --create-namespace

# upgrade an existing release
helm upgrade my-nginx bitnami/nginx --set replicaCount=3

# upgrade + install in one idempotent command
helm upgrade --install my-nginx bitnami/nginx -f values.yaml

# uninstall
helm uninstall my-nginx -n web

# list releases
helm list -n web
helm list --all-namespaces

Chart Structure

A chart is a directory of templated YAML plus defaults. Chart.yaml holds metadata; values.yaml holds defaults; templates/ holds Go templates rendered with those values. helm create scaffolds a working chart. helm lint catches schema/template errors before publishing.

kubernetes
my-chart/
├── Chart.yaml          # chart metadata (name, version, apiVersion)
├── values.yaml         # default values (overridable)
├── charts/             # chart dependencies (subcharts)
├── templates/          # Go-templated YAML
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── _helpers.tpl    # named templates (partials)
│   └── NOTES.txt       # post-install notes
├── templates.yaml      # (optional) CRDs to install before templates
└── README.md

# create a new chart skeleton
helm create my-chart

# lint a chart
helm lint my-chart

# package into a .tgz
helm package my-chart

Values & Overrides

values.yaml holds chart defaults. Override with --set (for scalars) or -f (for whole files). helm show values lists every key a chart accepts. helm template renders manifests without touching the cluster — essential for diffing what a release will actually apply.

kubernetes
# values.yaml (chart defaults)
image:
  repository: nginx
  tag: "1.25"
replicaCount: 2
service:
  type: ClusterIP
  port: 80

# override on the command line
helm upgrade --install web my-chart \
  --set replicaCount=5 \
  --set image.tag=1.26 \
  --set service.type=LoadBalancer

# override from a file
helm upgrade --install web my-chart -f prod-values.yaml

# see all values a chart supports
helm show values bitnami/nginx

# rendered output (debug, no install)
helm template web my-chart -f prod-values.yaml

Templates & Built-in Objects

Templates use Go template syntax with Sprig functions. .Release.Name makes every install unique; .Values exposes chart values; .Chart pulls from Chart.yaml. The default function supplies fallbacks. Render-safe naming via .Release.Name lets one chart run many releases in the same namespace.

kubernetes
# built-in template objects:
#   .Values         chart values (after overrides)
#   .Release        release metadata (Name, Namespace, Service, Revision, IsInstall, IsUpgrade)
#   .Chart          Chart.yaml contents
#   .Files          files in the chart
#   .Capabilities   cluster capabilities (KubeVersion, APIVersions)
#   .Template       current template info

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-app
  labels:
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount | default 1 }}
  selector:
    matchLabels:
      app: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ .Release.Name }}

Helm Repository

Helm 3 supports both classic HTTP chart repos and OCI registries (helm push/pull oci://…). OCI is the modern, signing-friendly path. helm repo update refreshes indexes; always run it before upgrade to get the latest chart versions. Artifact Hub (helm search hub) indexes public charts.

kubernetes
# add a chart repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add jetstack https://charts.jetstack.io

# list configured repos
helm repo list

# update repo indexes
helm repo update

# remove a repo
helm repo remove bitnami

# push to OCI registry (Helm 3+)
helm registry login registry.example.com
helm package my-chart
helm push my-chart-0.1.0.tgz oci://registry.example.com/charts

# pull an OCI chart
helm pull oci://registry.example.com/charts/my-chart --version 0.1.0

Rollback & History

Every install/upgrade creates a numbered revision stored as a Secret. helm rollback flips to a previous revision instantly — Helm's killer feature vs plain kubectl apply. helm get values shows what was actually deployed, which is invaluable for debugging drifted state.

kubernetes
# view release history
helm history my-nginx -n web

# rollback to a previous revision
helm rollback my-nginx 1 -n web

# get rendered manifests of a deployed release
helm get manifest my-nginx -n web

# get the values used at install/upgrade
helm get values my-nginx -n web
helm get values my-nginx -n web --all   # incl. chart defaults

# get all info (hooks, manifest, values, notes)
helm get all my-nginx -n web

# wait for resources to be ready on install/upgrade
helm upgrade --install my-nginx bitnami/nginx --wait --timeout 5m
17

NetworkPolicy

NetworkPolicy Manifest

A NetworkPolicy is a firewall rule for pods. An empty podSelector selects all pods in the namespace; an empty rules list denies all traffic of that type. NetworkPolicies are additive — a pod is allowed if ANY policy permits it. They require a supporting CNI; plain flannel ignores them.

kubernetes
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: prod
spec:
  podSelector: {}             # selects ALL pods in the namespace
  policyTypes:
    - Ingress
    - Egress
  # no ingress/egress rules => deny all traffic of that type
# effect: pods in 'prod' can't send or receive ANY traffic
# until another policy explicitly allows it

# REQUIRES a CNI that supports NetworkPolicy
# (Calico, Cilium, Weave, kube-router; NOT flannel alone)

Ingress Rules

Ingress rules allow traffic INTO selected pods. from entries are OR'd; ports are OR'd. podSelector matches pods in the same namespace; namespaceSelector matches whole namespaces (use the auto-added kubernetes.io/metadata.name label). Combining both in one item ANDs them.

kubernetes
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:               # pods in same namespace
            matchLabels:
              app: gateway
        - namespaceSelector:         # whole other namespace
            matchLabels:
              kubernetes.io/metadata.name: monitoring
        - ipBlock:                   # external CIDR
            cidr: 10.0.0.0/8
      ports:
        - protocol: TCP
          port: 8080

Egress Rules

Egress rules control what selected pods can talk TO. A common gotcha: a default-deny egress breaks DNS resolution and pods can't reach kube-dns. Always allow UDP/TCP 53 to the kube-system namespace before denying everything else. ipBlock.except carves out exceptions.

kubernetes
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-dns
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector: {}     # any namespace (kube-dns)
      ports:
        - protocol: UDP
          port: 53
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8          # allow internet, deny internal
      ports:
        - protocol: TCP
          port: 443

# WARNING: egress deny-all blocks DNS and breaks pods.
# Always allow kube-dns first.

Selectors & Namespace Selection

A bare podSelector matches pods in the policy's own namespace. A namespaceSelector matches whole namespaces. Specifying BOTH podSelector AND namespaceSelector in one item ANDs them (pods with label X in namespace Y) — placing them as separate items ORs them. ipBlock matches external IPs.

kubernetes
spec:
  podSelector:                       # selects pods in THIS namespace
    matchLabels:
      app: web
  ingress:
    - from:
        # SAME namespace, matching pods
        - podSelector:
            matchLabels:
              app: gateway

        # DIFFERENT namespace (whole namespace)
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring

        # BOTH: pods in another namespace with specific labels
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: backend
          podSelector:
            matchLabels:
              app: api

        # external IP range
        - ipBlock:
            cidr: 192.168.1.0/24

Default Policies (Deny/Allow All)

Default-deny policies (empty rules) lockdown a namespace so only explicit allow-lists open holes. Apply deny-all ingress + deny-all egress as a baseline, then add targeted allow policies. An empty ingress: [{}] entry allows all ingress — the escape hatch for opt-out.

kubernetes
# --- DEFAULT DENY INGRESS (in a namespace) ---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: prod
spec:
  podSelector: {}
  policyTypes: [Ingress]
---
# --- DEFAULT DENY EGRESS ---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: prod
spec:
  podSelector: {}
  policyTypes: [Egress]
---
# --- ALLOW ALL INGRESS (whitelist opt-out) ---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress
  namespace: prod
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    - {}      # allow from anywhere

Policy Types & Default Behavior

NetworkPolicies are additive — once any policy selects a pod, that traffic type becomes default-deny for the pod unless an explicit rule allows it. With no policies at all, everything is open. List policies matching a pod's labels to see what governs its traffic.

kubernetes
spec:
  policyTypes:
    - Ingress
    - Egress
  # if omitted, defaults are inferred:
  #   - Ingress is always present (a policy always affects ingress)
  #   - Egress is implied if any egress rule exists

# key behaviors:
# - policies are ADDITIVE: a pod is allowed if ANY matching
#   policy permits the traffic
# - if NO policy selects a pod => all traffic allowed (default open)
# - if at least one policy selects a pod => default deny for that type
#   unless an explicit rule allows it

# inspect effective policies for a pod
kubectl get networkpolicy -n prod -l app=web
18

RBAC

Role (Namespaced)

A Role grants permissions on namespaced resources within a single namespace. rules list apiGroups, resources, and verbs. pods/log is a subresource; pods/exec requires the create verb (because exec creates a subresource). Use '*' to wildcard any value.

kubernetes
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: dev
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
# a Role grants permissions WITHIN one namespace
# verbs: get, list, watch, create, update, patch, delete, deletecollection, "*"
# apiGroups: "" (core), "apps", "batch", "rbac.authorization.k8s.io", ...

RoleBinding

A RoleBinding grants a Role (or a ClusterRole, scoped to one namespace) to subjects: Users, Groups, or ServiceAccounts. The binding lives in the namespace where the permissions apply. roleRef is immutable — to change the referenced Role, delete and recreate the binding.

kubernetes
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: dev
subjects:
  - kind: User
    name: alice
    apiGroup: rbac.authorization.k8s.io
  - kind: ServiceAccount
    name: my-sa
    namespace: dev
  - kind: Group
    name: dev-team
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
# binds the Role to users, groups, or service accounts
# RoleBinding can also reference a ClusterRole (scoped to its namespace)

ClusterRole & ClusterRoleBinding

A ClusterRole grants permissions on cluster-scoped resources (nodes, namespaces, PVs) — Roles can't touch these. ClusterRoles can also be bound in any namespace via a RoleBinding for reuse (define once, bind many). ClusterRoleBinding grants a ClusterRole cluster-wide.

kubernetes
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-nodes
subjects:
  - kind: ServiceAccount
    name: monitor
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io
# ClusterRole grants cluster-scoped resources (nodes, PVs, namespaces)
# or can be reused across namespaces via a RoleBinding

Verbs, Resources & API Groups

apiGroups group resources — core is '' (empty string); apps has Deployments/StatefulSets; batch has Jobs/CronJobs. Subresources (pods/log, pods/exec, pods/portforward) are separate permissions. resourceNames restricts a rule to specific instances. nonResourceURLs covers /healthz, /metrics, etc.

kubernetes
rules:
  - apiGroups: [""]                        # core API group
    resources: ["pods", "services", "configmaps", "secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets"]
    verbs: ["*"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["create", "delete", "get"]
  - apiGroups: [""]
    resources: ["pods/log", "pods/exec", "pods/portforward"]
    verbs: ["get", "create"]
  - nonResourceURLs: ["/healthz", "/healthz/*"]   # non-resource URLs
    verbs: ["get"]

# resourceNames: restrict to specific instances
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["my-config"]
    verbs: ["get"]

ServiceAccount

A ServiceAccount is an identity for pods. Assign it via serviceAccountName; the token auto-mounts at /var/run/secrets/kubernetes.io/serviceaccount. Since 1.24, SAs don't have permanent token Secrets — use kubectl create token for short-lived, time-bound tokens. automountServiceAccountToken: false opts out.

kubernetes
# create a service account
kubectl create serviceaccount my-sa -n dev

# assign a RoleBinding (see RoleBinding example)
# mount the SA in a pod
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  serviceAccountName: my-sa
  automountServiceAccountToken: false   # opt out of token mount
  containers:
    - name: app
      image: myapp:1.0

# short-lived tokens (1.24+) - no permanent secret
kubectl create token my-sa -n dev --duration=1h

# list SAs
kubectl get sa -n dev

kubectl auth can-i

kubectl auth can-i checks RBAC permissions for the current (or impersonated) user — essential for debugging 'forbidden' errors. --list shows every allowed verb/resource. --as impersonates a user/SA for testing. Built-in roles (view, edit, admin, cluster-admin) cover common needs.

kubernetes
# check if you can do something
kubectl auth can-i create pods -n dev
kubectl auth can-i delete deployments -n prod

# check on behalf of another user/SA
kubectl auth can-i list secrets --as=system:serviceaccount:dev:my-sa -n dev

# check all allowed verbs on a resource
kubectl auth can-i --list --as=alice -n dev

# whoami (1.25+)
kubectl auth whoami

# reconcile: detect over-privileged bindings
kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide

# common built-in cluster roles:
#   view, edit, admin, cluster-admin
19

Probes (Liveness/Readiness/Startup)

Liveness Probe

A liveness probe tells kubelet when to restart a container. If it fails failureThreshold consecutive times, kubelet kills the container and the restart policy applies. Use it to recover from deadlocks or unrecoverable states — not to check dependencies (that's readiness).

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:1.0
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 15
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3
        successThreshold: 1
# if the probe fails failureThreshold times in a row,
# kubelet kills and restarts the container (per restartPolicy)

Readiness Probe

A readiness probe controls whether the pod receives traffic from Services. A failing readiness probe removes the pod's IP from endpoints — but does NOT restart it. Use it for warm-up (don't route until ready) and to drain traffic during transient failures or graceful shutdown.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:1.0
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5
        failureThreshold: 3
# a failing readiness probe REMOVES the pod from service endpoints,
# but does NOT restart the container.
# use to drain traffic during deploys, warm-up, or transient outages.

Startup Probe

A startup probe (1.18+) is for slow-to-start apps (Java, legacy servers). While it runs, liveness and readiness are disabled — so a slow boot doesn't get killed by liveness. Once startup succeeds, liveness/readiness activate. Use a long failureThreshold × periodSeconds window for the boot.

kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: slow-app
spec:
  containers:
    - name: app
      image: myapp:1.0
      startupProbe:
        httpGet:
          path: /startup
          port: 8080
        failureThreshold: 30
        periodSeconds: 10      # up to 300s to start
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
# while the startup probe runs, liveness/readiness are disabled.
# once startup succeeds, liveness/readiness take over.

Probe Handlers

Three handler types: httpGet (success = HTTP 2xx/3xx), tcpSocket (success = TCP connect), exec (success = exit 0). gRPC probe (1.27+ GA) is the clean option for gRPC services. Prefer httpGet — it lets the app signal 'healthy but busy' (503) without restarting. exec is fragile (needs shell/binaries in image).

kubernetes
# three handler types:

# 1. HTTP GET (most common) - success = 200-399
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
    httpHeaders:
      - name: X-Custom
        value: probe
    scheme: HTTP

# 2. TCP socket - success = port open
livenessProbe:
  tcpSocket:
    port: 3306

# 3. Exec - success = exit code 0
livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - "pg_isready -h localhost"

# gRPC (1.24+, alpha -> GA): use grpc probe
livenessProbe:
  grpc:
    port: 9090
    service: my.package.Health

Probe Parameters

initialDelaySeconds delays the first probe — but a startup probe is a cleaner way to handle slow boots. periodSeconds is the cadence; failureThreshold × periodSeconds is how long a pod can be unhealthy before action. Tighten these for fast failover, loosen for tolerance. terminationGracePeriodSeconds caps shutdown.

kubernetes
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15   # wait before first probe (after start)
  periodSeconds: 10         # probe every 10s
  timeoutSeconds: 2         # max time per probe
  successThreshold: 1       # consecutive successes to mark OK
  failureThreshold: 3       # consecutive failures to mark failed
  terminationGracePeriodSeconds: 30  # (liveness) grace before SIGKILL

# tuning tips:
# - set initialDelaySeconds > typical startup time
# - OR use a startup probe instead (cleaner)
# - shorter period = faster reaction, more load
# - higher failureThreshold = more tolerant of blips

Probe Failure Behavior

Liveness failure kills+restarts the container in place (not rescheduled). Readiness failure just removes the pod from Services — no restart. Startup failure behaves like liveness. Flapping liveness often comes from probes that depend on slow downstream services — keep health checks local and cheap.

kubernetes
# --- LIVENESS FAIL ---
#   container killed -> restarted (per restartPolicy: Always/OnFailure)
#   pod stays on the same node (NOT rescheduled)
#   CrashLoopBackOff if it keeps failing

# --- READINESS FAIL ---
#   pod stays running, but is REMOVED from Service endpoints
#   no traffic until it succeeds again
#   NOT restarted by kubelet

# --- STARTUP FAIL ---
#   container killed and restarted (like liveness)
#   liveness/readiness never activate
#   after failureThreshold × periodSeconds, pod is stuck restarting

# check probe results:
kubectl describe pod app | grep -A5 "Last State"
kubectl get pod app -o jsonpath='{.status.containerStatuses[*].state}'

# common cause of flapping: liveness probe hitting a slow dependency
20

Resources & Advanced kubectl

Requests & Limits

requests are what the scheduler reserves (a pod is scheduled only if a node has spare requests) — set them to your steady-state needs. limits are hard ceilings: CPU is throttled, memory is OOMKilled. cpu is in cores (500m = 0.5), memory in bytes (Mi/Gi). Always set both for production.

kubernetes
spec:
  containers:
    - name: app
      image: myapp:1.0
      resources:
        requests:           # what the scheduler guarantees (reservation)
          cpu: 250m          # 250 millicores = 0.25 core
          memory: 256Mi      # 256 mebibytes
          ephemeral-storage: 1Gi
        limits:             # hard ceiling (enforced)
          cpu: "1"           # 1 core
          memory: 512Mi
          ephemeral-storage: 2Gi

# cpu limit: throttled (cfs quota)
# memory limit: OOMKilled if exceeded
# ephemeral-storage limit: evicted if exceeded
# request only (no limit): best-effort burst

LimitRange Defaults

LimitRange applies default requests/limits to pods that don't specify them — critical when a ResourceQuota requires requests. max/min bound how large/small a single container can go. maxLimitRequestRatio caps how much a container can burst above its request, preventing noisy neighbors.

kubernetes
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: prod
spec:
  limits:
    - type: Container
      default:               # applied as LIMITS when not set
        cpu: 500m
        memory: 512Mi
      defaultRequest:        # applied as REQUESTS when not set
        cpu: 100m
        memory: 128Mi
      max:                   # max allowed LIMITS
        cpu: "4"
        memory: 8Gi
      min:                   # min allowed REQUESTS
        cpu: 50m
        memory: 64Mi
      maxLimitRequestRatio:  # limit/request ratio cap (burst cap)
        cpu: "4"

# a pod with no resources gets default/defaultRequest applied
# a pod with limit=4cpu, request=50m would breach maxLimitRequestRatio

Horizontal Pod Autoscaler

HPA scales replicas based on metrics (CPU/memory via metrics-server, or custom/external via Prometheus Adapter). behavior (v2) tunes aggressiveness: stabilizationWindowSeconds prevents flapping, policies cap rate per minute. Scale up is usually fast/aggressive; scale down is intentionally slow.

kubernetes
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization          # avg across pods
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:                          # (v2) scale up/down tuning
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 50                  # max 50% down per minute
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60

# requires metrics-server for CPU/memory
kubectl get hpa web -w

Pod Disruption Budget

A PodDisruptionBudget limits how many pods of a set can be voluntarily evicted at once — protecting availability during node drains and cluster-autoscaler actions. Use minAvailable or maxUnavailable (not both). It does NOT protect against involuntary disruptions (hardware failure) — only replicas do.

kubernetes
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: prod
spec:
  minAvailable: 2            # at least 2 pods always available
  # OR
  # maxUnavailable: 1        # at most 1 down at a time
  selector:
    matchLabels:
      app: web
# PDB protects against VOLUNTARY disruptions:
#   - kubectl drain (node maintenance)
#   - cluster autoscaler evictions
#   - kubectl delete pod (controller-managed)
# It does NOT prevent INVOLUNTARY disruptions (node failure, kernel panic).

# check PDB status
kubectl get pdb -n prod
kubectl describe pdb web-pdb -n prod

Taints, Tolerations & Affinity

Taints repel pods; tolerations let specific pods ignore them — use to dedicate nodes (GPU, ingress, control-plane). nodeAffinity prefers/requires certain nodes. podAntiAffinity spreads replicas across topology domains (hosts, zones) for HA. Remove a taint with a trailing dash on the same key+effect.

kubernetes
# taint a node
kubectl taint nodes node-1 dedicated=gpu:NoSchedule

# tolerate the taint so a pod can schedule there
spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule

# prefer specific nodes
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: disktype
                operator: In
                values: [ssd]
    podAntiAffinity:               # spread replicas across nodes
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app: web

# remove a taint
kubectl taint nodes node-1 dedicated=gpu:NoSchedule-

Advanced kubectl (patch/wait/top/debug)

patch supports JSON, strategic-merge, and merge types. wait blocks until a condition is met — perfect for scripts. top shows live CPU/memory (needs metrics-server). kubectl debug attaches an ephemeral container to a running pod (great for distroless images) or opens a shell on a node.

kubernetes
# JSON patch (precise field ops)
kubectl patch deployment web --type json -p \
  '[{"op":"replace","path":"/spec/replicas","value":3}]'

# strategic merge patch (default for k8s objects)
kubectl patch svc web -p '{"spec":{"type":"NodePort"}}'

# wait for a condition
kubectl wait --for=condition=Ready pod/web --timeout=60s
kubectl wait --for=delete pod/web --timeout=30s

# resource usage (needs metrics-server)
kubectl top nodes
kubectl top pods -n prod
kubectl top pod web --containers

# ephemeral debug container (1.25+)
kubectl debug -it web --image=busybox --target=web
# debug a node
kubectl debug node/node-1 -it --image=ubuntu

# run a one-off pod
kubectl run tmp --rm -it --image=alpine -- sh

# explain a field
kubectl explain pod.spec.containers.resources

Was this helpful?

Learning path

Learn from scratch

Learn this language from the ground up with structured lessons.