Getting Started
kubectl Basics
kubectl is the CLI for Kubernetes. get lists resources, create makes new ones, expose creates services, scale changes replica count. describe gives detailed info about a resource.
# cluster info
kubectl cluster-info
# list nodes
kubectl get nodes
# list pods (all namespaces)
kubectl get pods --all-namespaces
# create a deployment
kubectl create deployment nginx --image=nginx
# expose deployment as service
kubectl expose deployment nginx --port=80 --type=LoadBalancer
# scale deployment
kubectl scale deployment nginx --replicas=3
# get detailed info
kubectl describe pod <pod-name>
# view logs
kubectl logs <pod-name>Output Formatting
The -o flag controls output. yaml/json show full resource spec. jsonpath and custom-columns extract specific fields for scripting. -w watches for live changes. Use --no-headers in scripts to drop column names.
# wide format (more columns)
kubectl get pods -o wide
# YAML output
kubectl get pod nginx -o yaml
# JSON output
kubectl get pod nginx -o json
# jsonpath (extract specific fields)
kubectl get pods -o jsonpath='{.items[*].metadata.name}'
# custom columns
kubectl get pods -o custom-columns=NAME:.metadata.name,STATUS:.status.phase
# show labels
kubectl get pods --show-labels
# watch for changes
kubectl get pods -wContext & Configuration
A context bundles cluster address, user credentials, and namespace. kubeconfig files (default ~/.kube/config) store these. Multiple files can be combined via KUBECONFIG env var for managing many clusters.
# view current config
kubectl config view
# list contexts
kubectl config get-contexts
# switch context
kubectl config use-context my-cluster
# set default namespace for current context
kubectl config set-context --current --namespace=dev
# show current context
kubectl config current-context
# KUBECONFIG with multiple files
export KUBECONFIG=~/.kube/config:~/.kube/prod-configExplain & API Resources
api-resources lists what the cluster supports. explain shows the OpenAPI schema for any resource — invaluable for writing YAML without leaving the terminal. Use --recursive to dump the full nested structure.
# list all resource types
kubectl api-resources
# list namespaced vs cluster-scoped
kubectl api-resources --namespaced=true
kubectl api-resources --namespaced=false
# explain a resource's fields
kubectl explain pod
kubectl explain pod.spec.containers
kubectl explain deployment.spec.strategy
# explain with recursion (all nested fields)
kubectl explain pod --recursive
# include description
kubectl explain pod.spec.containers.resources --api-version=apps/v1Dry Run & Generate YAML
--dry-run=client -o yaml is the standard way to scaffold manifests from imperative commands. --dry-run=server validates against the real apiserver (admission webhooks, schema) without persisting. Great for bootstrapping declarative YAML.
# generate YAML without applying
kubectl create deployment nginx --image=nginx --dry-run=client -o yaml
# pipe generated YAML straight to a file
kubectl create deployment nginx --image=nginx --dry-run=client -o yaml > nginx.yaml
# apply the generated manifest
kubectl apply -f nginx.yaml
# dry-run against the server (validates with apiserver)
kubectl apply -f nginx.yaml --dry-run=server
# generate a service from a deployment
kubectl create service clusterip my-svc --tcp=80:80 --dry-run=client -o yamlHelp & Autocomplete
Every kubectl command supports --help with examples. Shell completion saves enormous typing and exposes subcommands. The alias k=kubectl is near-universal among practitioners — combine with completion for the best experience.
# top-level help
kubectl --help
# subcommand help
kubectl create --help
kubectl explain --help
# generate bash completion
kubectl completion bash > /etc/bash_completion.d/kubectl
# zsh completion
source <(kubectl completion zsh)
# alias k=kubectl with completion
alias k=kubectl
complete -o default -F __start_kubectl k
# shorthand: kubectl -> k
echo 'alias k=kubectl' >> ~/.bashrcPod
Pod Manifest
A Pod is the smallest deployable unit, holding one or more containers that share network and storage. Containers in a pod are always co-scheduled on the same node. You rarely create pods directly — use a Deployment or Job instead.
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.25
ports:
- containerPort: 80
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
restartPolicy: AlwaysMulti-Container Pod (Sidecar)
Multi-container pods share network (same IP, port space) and volumes. Common patterns: sidecar (helper), adapter (transforms output), ambassador (proxy). The shared emptyDir volume lets the sidecar read logs the main container writes.
apiVersion: v1
kind: Pod
metadata:
name: web-with-logs
spec:
containers:
- name: app
image: nginx
volumeMounts:
- name: shared-logs
mountPath: /var/log/nginx
- name: log-shipper
image: busybox
command: ["sh", "-c", "tail -f /var/log/nginx/access.log"]
volumeMounts:
- name: shared-logs
mountPath: /var/log/nginx
readOnly: true
volumes:
- name: shared-logs
emptyDir: {}Pod Lifecycle & Phases
A pod's phase is high-level (Pending, Running, Succeeded, Failed). Container states (Waiting/Running/Terminated) give the real detail — e.g. CrashLoopBackOff means the container keeps exiting. Check restart count and events to diagnose.
# pod phases: Pending | Running | Succeeded | Failed | Unknown
kubectl get pod nginx
# show status including container states
kubectl get pod nginx -o jsonpath='{.status}'
# container states: Waiting | Running | Terminated
kubectl describe pod nginx
# common Waiting reasons: ImagePullBackOff, CrashLoopBackOff, CreateContainerConfigError
kubectl get pods --field-selector=status.phase=Pending
# restart count reveals crash loops
kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'Pod Status & Events
Events are the cluster's audit log for a resource — scheduling, pulling, pulling errors, liveness failures. They expire (~1 hour by default), so capture them while debugging. describe bundles events with the resource spec for quick triage.
# full describe (includes Events at the bottom)
kubectl describe pod nginx
# list cluster events sorted by time
kubectl get events --sort-by=.lastTimestamp
# events for a specific namespace
kubectl get events -n default
# watch events live
kubectl get events -w
# filter events by type
kubectl get events --field-selector type=Warning
# events for a specific pod
kubectl get events --field-selector involvedObject.name=nginxExec & Port-Forward
exec runs commands inside a container; -it allocates a TTY for shells. port-forward tunnels a local port to a pod/service without exposing it publicly — essential for debugging databases or web UIs from your laptop.
# open an interactive shell
kubectl exec -it nginx -- sh
# run a one-off command
kubectl exec nginx -- ls /etc/nginx
# specify container in a multi-container pod
kubectl exec -it web-with-logs -c log-shipper -- sh
# forward a local port to a pod port
kubectl port-forward pod/nginx 8080:80
# forward to a service / deployment
kubectl port-forward svc/nginx 8080:80
kubectl port-forward deployment/nginx 8080:80
# forward to a random local port
kubectl port-forward pod/nginx :80Delete & Force
Deleting a pod controlled by a Deployment/StatefulSet causes an immediate replacement. --force --grace-period=0 skips the TERM signal grace period and can leave stale resources — use sparingly. Stuck terminating pods often need a finalizer removal.
# delete a pod (controller will recreate it)
kubectl delete pod nginx
# force delete immediately (bypasses graceful shutdown)
kubectl delete pod nginx --grace-period=0 --force
# delete by label
kubectl delete pods -l app=nginx
# delete all pods in a namespace
kubectl delete pods --all -n default
# delete a pod and wait for it to be gone
kubectl delete pod nginx --wait
# delete with a timeout
kubectl delete pod nginx --timeout=30sDeployment
Deployment Manifest
A Deployment manages ReplicaSets and provides declarative rolling updates and rollbacks. spec.selector must match spec.template.metadata.labels. The template defines the pod spec that gets rolled out.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.25
ports:
- containerPort: 80Rolling Update Strategy
RollingUpdate (default) gradually replaces pods for zero downtime. maxSurge/maxUnavailable control rollout speed and availability. Recreate kills all pods before creating new ones — used when the app cannot run two versions at once (e.g. single-volume writers).
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25% # extra pods above replicas during update
maxUnavailable: 25% # pods allowed down during update
# alternative: recreate (kills all, then starts new)
# strategy:
# type: Recreate
selector:
matchLabels:
app: nginx
template:Rollback
Each change to the pod template creates a ReplicaSet revision. undo flips traffic back to a previous ReplicaSet. pause lets you do multiple fixes between canary increments. Use --record (deprecated) or annotations to track change causes.
# view rollout history
kubectl rollout history deployment/nginx
# see details of a specific revision
kubectl rollout history deployment/nginx --revision=2
# undo to the previous revision
kubectl rollout undo deployment/nginx
# undo to a specific revision
kubectl rollout undo deployment/nginx --to-revision=2
# pause/resume a rollout (for canary pauses)
kubectl rollout pause deployment/nginx
kubectl rollout resume deployment/nginx
# check rollout status
kubectl rollout status deployment/nginxScale & HPA
HPA scales replicas based on CPU/memory or custom metrics. Requires metrics-server for CPU/mem. minReplicas and maxReplicas bound scaling. The v2 API supports multiple metrics and behaviors (scale-down stabilization).
# scale manually
kubectl scale deployment/nginx --replicas=5
# horizontal pod autoscaler (CPU target)
kubectl autoscale deployment nginx --min=2 --max=10 --cpu-percent=80
# list HPA
kubectl get hpa
# HPA manifest (v2 with custom metrics)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: nginx
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 80Update Image & Restart
set image triggers a new ReplicaSet and rolling update. rollout restart is the cleanest way to pull the latest image or refresh config — it creates a new ReplicaSet with the same spec. edit opens your $EDITOR for ad-hoc fixes.
# update container image
kubectl set image deployment/nginx nginx=nginx:1.26
# update from a specific container
kubectl set image deployment/nginx container=nginx:1.26 --record
# trigger a rollout restart (same image, recreate pods)
kubectl rollout restart deployment/nginx
# edit a deployment in editor
kubectl edit deployment/nginx
# patch a single field
kubectl patch deployment/nginx -p '{"spec":{"replicas":5}}'
# apply a manifest change
kubectl apply -f nginx-deployment.yamlDeployment Status
A Deployment owns ReplicaSets; only one is active (replicas>0). Old ReplicaSets scale to zero for rollback. pod-template-hash labels each pod with its revision. Check status.conditions for Available, Progressing, and ReplicaFailure signals.
# deployment status
kubectl get deployment nginx
# watch rollout
kubectl rollout status deployment/nginx
# show ReplicaSets owned by the deployment
kubectl get rs -l app=nginx
# show pods across all revisions
kubectl get pods -l app=nginx --show-labels
# pod-template-hash label distinguishes revisions
# kubectl get pods -l app=nginx,pod-template-hash=xxxx
# conditions: Available, Progressing, ReplicaFailure
kubectl get deployment nginx -o jsonpath='{.status.conditions}'Service
Service Types
A Service provides a stable IP/DNS that load-balances across a set of pods. ClusterIP is the default (in-cluster only). NodePort and LoadBalancer expose externally. ExternalName is a DNS alias, not a proxy.
# four service types:
# - ClusterIP (default): reachable inside the cluster
# - NodePort: exposed on each node's IP at a static port
# - LoadBalancer: cloud-provisioned external LB
# - ExternalName: DNS CNAME to an external host
# quick create
kubectl expose deployment nginx --port=80 --type=ClusterIP
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl expose deployment nginx --port=80 --type=LoadBalancer
# external name (no proxying, pure DNS)
kubectl create service externalname my-svc --external-name=db.example.comClusterIP Service
ClusterIP gives a stable virtual IP and DNS name within the cluster. The selector matches pod labels; kube-proxy programs iptables/IPVS to load-balance across endpoints. Resolve via <svc>.<ns>.svc.cluster.local or just <svc>.
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
type: ClusterIP
selector:
app: nginx
ports:
- name: http
port: 80 # service port
targetPort: 80 # container port
protocol: TCP
# DNS: nginx.default.svc.cluster.local
# short names also work: nginx, nginx.default
# kube-dns / CoreDNS resolves the service VIPNodePort Service
NodePort opens the same static port (30000-32767) on every node. Traffic to any node:nodePort reaches a pod (kube-proxy may redirect to a pod on another node). Useful when you can't provision a cloud LB; pair with an external LB for HA.
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
type: NodePort
selector:
app: nginx
ports:
- port: 80
targetPort: 80
nodePort: 30080 # optional, 30000-32767 range
# reachable at <any-node-ip>:30080
# also still reachable via clusterIP:80
# get assigned nodePort
kubectl get svc nginx -o jsonpath='{.spec.ports[*].nodePort}'LoadBalancer Service
LoadBalancer asks the cloud provider to provision an external load balancer pointing at the nodes. externalTrafficPolicy: Local preserves the client source IP (no SNAT) but skips cross-node load balancing. STATUS shows the assigned external IP once ready.
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
type: LoadBalancer
selector:
app: nginx
ports:
- port: 80
targetPort: 80
# cloud-specific annotations
# annotations:
# service.beta.kubernetes.io/aws-load-balancer-type: nlb
# service.beta.kubernetes.io/azure-load-balancer-internal: "true"
# externalTrafficPolicy: Local preserves client IP
externalTrafficPolicy: Local
# status.ingress shows the external IP/hostnameHeadless Service
A headless service (clusterIP: None) has no VIP — DNS returns the pod IPs directly. Required by StatefulSet so each pod gets a stable DNS name (pod-0.svc, pod-1.svc). Also used by clients that want to discover all backends themselves.
apiVersion: v1
kind: Service
metadata:
name: nginx-headless
spec:
clusterIP: None # makes it headless
selector:
app: nginx
ports:
- port: 80
targetPort: 80
# DNS returns pod IPs directly (not a VIP)
# A records: nginx-headless.default.svc.cluster.local
# -> 10.1.0.5, 10.1.0.6, 10.1.0.7
# used by StatefulSet for stable pod DNS:
# <pod-name>.<svc-name>.default.svc.cluster.localEndpoints & EndpointSlices
Endpoints/EndpointSlices list the pod IPs that actually receive traffic — only Ready pods appear. If a service has no endpoints, the pods aren't matching the selector or aren't Ready. EndpointSlices (v1) scale better for huge services and are the default.
# view endpoints backing a service
kubectl get endpoints nginx
# endpoint slices (newer, more scalable)
kubectl get endpointslices -l kubernetes.io/service-name=nginx
# endpoints track pods that match the selector
# AND pass readiness checks
kubectl describe endpoints nginx
# a service with no ready pods has no endpoints
# - check pod readiness if traffic is refused
# - EndpointSlices are grouped by address type (IPv4/IPv6)
# manually managed endpoints (selector-less service)ConfigMap
Create ConfigMap
ConfigMaps store non-sensitive configuration as key-value pairs. --from-literal for inline values, --from-file to embed whole files (the filename becomes the key). Useful for decoupling config from images without rebuilding.
# from literal values
kubectl create configmap app-config \
--from-literal=LOG_LEVEL=debug \
--from-literal=ENABLE_FEATURE=true
# from a file (key = filename, value = contents)
kubectl create configmap app-config --from-file=app.properties
# from a directory (one key per file)
kubectl create configmap app-config --from-file=configs/
# from an env file
kubectl create configmap app-config --from-env-file=app.env
# from a file with a custom key
kubectl create configmap app-config --from-file=custom-key=app.propertiesConfigMap Manifest
data holds string values (each key becomes either an env var or a file when mounted). Multi-line strings use YAML block scalars (|). binaryData accepts base64 for binary blobs. Total size limit is 1 MiB per ConfigMap.
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
LOG_LEVEL: "debug"
ENABLE_FEATURE: "true"
application.yml: |
server:
port: 8080
logging:
level: debug
binaryData:
# base64-encoded binary values
logo.png: iVBORw0KGgoAAAANSUhEUg...Consume as Environment Variables
envFrom injects every ConfigMap key as an env var in one shot. env with configMapKeyRef picks individual keys. optional: true lets the pod start even if the ConfigMap doesn't exist. Updates to the ConfigMap do NOT refresh env vars in running pods.
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: myapp:1.0
env:
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: app-config
key: LOG_LEVEL
envFrom:
- configMapRef:
name: app-config # injects all keys as env vars
optional: true # pod starts even if CM missingConsume as Volume
Mounting a ConfigMap as a volume projects each key as a file. Updates to the ConfigMap are eventually reflected in mounted files (configured by kubelet sync period) — but apps must re-read them. subPath mounts pin a single key and do NOT update.
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: myapp:1.0
volumeMounts:
- name: config
mountPath: /etc/config
readOnly: true
volumes:
- name: config
configMap:
name: app-config
items: # optional: pick subset
- key: application.yml
path: app.yml # rename file
defaultMode: 0644
# each ConfigMap key becomes a file under /etc/configUpdate & Reload
Mounted ConfigMap files refresh automatically (after kubelet sync), but env vars are immutable for the pod's life — restart pods to pick up env changes. rollout restart is the cleanest way to propagate config updates to a Deployment.
# edit the configmap
kubectl edit configmap app-config
# patch a single key
kubectl patch configmap app-config \
--type merge -p '{"data":{"LOG_LEVEL":"info"}}'
# mounted files update within ~kubelet sync period (default 60s)
# but env vars do NOT update in running pods
# force pods to pick up new config by rolling them
kubectl rollout restart deployment/app
# verify the new value on a pod
kubectl exec app -- printenv LOG_LEVELImmutable ConfigMap
Setting immutable: true freezes the ConfigMap — any change requires delete + recreate. This drastically reduces kubelet-apiserver watch traffic for large, stable configs (a major scalability win at scale). Common in production for app configs that are versioned with the image.
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
immutable: true
data:
LOG_LEVEL: "debug"
# once immutable, the only change allowed is delete + recreate
# benefits: kubelet does not watch it, lower apiserver load
# useful for large, stable configs in production
# make it mutable again
kubectl patch configmap app-config -p '{"immutable":false}'
# (this fails if currently immutable — must delete instead)Secret
Create Secret
Secrets store sensitive data base64-encoded (NOT encrypted at rest by default — enable encryption at rest for production). Use generic/Opaque for arbitrary data, docker-registry for image pulls, tls for Ingress TLS. Never commit Secret YAML to git.
# generic (opaque) from literals
kubectl create secret generic db-creds \
--from-literal=username=admin \
--from-literal=password='S3cur3!'
# from a file (e.g. SSH key, certificate)
kubectl create secret generic ssh-key --from-file=ssh-privatekey=~/.ssh/id_rsa
# docker registry secret (for private image registries)
kubectl create secret docker-registry regcred \
--docker-server=registry.example.com \
--docker-username=user \
--docker-password=pass \
[email protected]
# TLS secret from cert/key files
kubectl create secret tls my-tls \
--cert=fullchain.pem \
--key=privkey.pemSecret Manifest
data values must be base64-encoded (echo -n 'value' | base64). stringData is a convenience that accepts plaintext and is merged into data on apply. type Opaque is the default; specialized types (kubernetes.io/tls, kubernetes.io/dockerconfigjson) validate the keys.
apiVersion: v1
kind: Secret
metadata:
name: db-creds
type: Opaque
data:
username: YWRtaW4= # base64 of "admin"
password: UzNjdXIzIQ== # base64 of "S3cur3!"
# stringData accepts plaintext (auto-encoded on apply)
stringData:
api-key: "live-key-xyz"
# encode on the command line:
# echo -n 'admin' | base64
# decode:
# echo 'YWRtaW4=' | base64 -dUse Secret as Environment Variable
Consuming secrets as env vars is identical to ConfigMaps but uses secretKeyRef/secretRef. Note that env vars are visible via /proc/1/environ and to any process in the container — prefer volume mounts for highly sensitive data.
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: myapp:1.0
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-creds
key: password
envFrom:
- secretRef:
name: db-creds # injects all keys
optional: trueUse Secret as Volume
Mounted secrets are projected as files into a tmpfs (memory) volume — never written to node disk. defaultMode 0400 restricts access. Mounted secrets DO update when the Secret changes (after kubelet sync), unlike env-var secrets.
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: myapp:1.0
volumeMounts:
- name: secrets
mountPath: /etc/secrets
readOnly: true
volumes:
- name: secrets
secret:
secretName: db-creds
defaultMode: 0400 # restrict permissions
items:
- key: password
path: db-password # written to /etc/secrets/db-password
# memory-backed tmpfs, never written to disk on the nodeImagePullSecret
imagePullSecrets authenticate to private registries when pulling images. Attaching to a ServiceAccount applies it to every pod using that SA — the preferred pattern. The docker-registry secret type produces the right .dockerconfigjson payload automatically.
# attach to a pod
apiVersion: v1
kind: Pod
metadata:
name: private-app
spec:
imagePullSecrets:
- name: regcred
containers:
- name: app
image: registry.example.com/myapp:1.0
# attach to a service account (auto-applied to all pods using it)
kubectl patch serviceaccount default \
-p '{"imagePullSecrets":[{"name":"regcred"}]}'
# verify
kubectl get sa default -o jsonpath='{.imagePullSecrets}'Secret Types
Specialized secret types enable validation and integration: tls for Ingress, dockerconfigjson for image pulls, service-account-token auto-mounted into pods. Using the right type ensures the keys are validated and consumed by the right controllers.
# Opaque (default) — arbitrary key-value
type: Opaque
# kubernetes.io/service-account-token — auto for SAs
type: kubernetes.io/service-account-token
# kubernetes.io/dockerconfigjson — registry auth
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: <base64>
# kubernetes.io/tls — for Ingress TLS
type: kubernetes.io/tls
data:
tls.crt: <base64>
tls.key: <base64>
# kubernetes.io/basic-auth
type: kubernetes.io/basic-auth
stringData:
username: admin
password: passNamespace
Create Namespace
Namespaces partition a cluster for multiple teams or environments (dev/stage/prod). Names of resources must be unique within a namespace. Some resources (Nodes, PVs, StorageClasses) are cluster-scoped and not namespaced.
# imperative
kubectl create namespace dev
# declarative
apiVersion: v1
kind: Namespace
metadata:
name: dev
labels:
environment: development
team: platform
# apply
kubectl apply -f namespace.yaml
# list namespaces
kubectl get namespaces
# set default namespace for current context
kubectl config set-context --current --namespace=devResourceQuota
ResourceQuota caps aggregate resource usage in a namespace — total CPU/memory requests/limits, object counts (pods, PVCs, services, deployments). Essential for multi-tenant clusters to prevent one team from starving others. Pods without explicit requests are rejected once quota is set.
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
namespace: dev
spec:
hard:
requests.cpu: "10"
requests.memory: 20Gi
limits.cpu: "20"
limits.memory: 40Gi
pods: "50"
services.loadbalancers: "2"
persistentvolumeclaims: "10"
count/deployments.apps: "5"
# limits total aggregate resource use in the namespaceLimitRange
LimitRange sets per-resource defaults and bounds: default requests/limits for pods that don't specify them, plus min/max to constrain outrageous values. Pairs with ResourceQuota — without default requests, quota would reject every pod that forgets to set them.
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
namespace: dev
spec:
limits:
- type: Container
default: # default limits (if unset)
cpu: 500m
memory: 512Mi
defaultRequest: # default requests (if unset)
cpu: 100m
memory: 128Mi
max: # max allowed limits
cpu: "2"
memory: 2Gi
min: # min allowed requests
cpu: 50m
memory: 64Mi
- type: PersistentVolumeClaim
max:
storage: 100Gi
min:
storage: 1GiDelete Namespace & Finalizers
Deleting a namespace cascades all its resources. A namespace stuck in Terminating usually has a finalizer that can't complete — an unavailable APIService (e.g. metrics-server) or a CRD whose controller is gone. Removing the finalizer forces completion but can leak external resources.
# delete a namespace (cascades all resources in it)
kubectl delete namespace dev
# if stuck in Terminating, inspect finalizers
kubectl get namespace dev -o yaml
# remove finalizer to force completion (DANGEROUS)
kubectl patch namespace dev -p '{"metadata":{"finalizers":null}}'
# resources that block deletion:
# - APIService unavailable (metrics-server removed)
# - custom resources whose CRD is gone
# - PVCs with attached PVs
# check what's still in the namespace
kubectl api-resources --verbs=list --namespaced -o name \
| xargs -n 1 kubectl get -n dev --ignore-not-foundDefault Namespaces
kube-system runs the control plane and add-ons (DNS, kube-proxy, CNI). kube-public holds bootstrap data readable by unauthenticated users. kube-node-lease stores node heartbeats as Lease objects for performance. default is where your workloads land if you don't specify a namespace.
# built-in namespaces
kubectl get namespaces
# NAME STATUS AGE
# default Active 30d <- user workloads land here
# kube-system Active 30d <- control plane / addons
# kube-public Active 30d <- publicly readable (bootstrap)
# kube-node-lease Active 30d <- node heartbeats (leases)
# kube-system holds:
# - kube-dns / CoreDNS
# - kube-proxy
# - metrics-server
# - CNI, CSI, cloud provider controllers
# kube-public is readable by all (used for cluster info discovery)Namespace-scoped vs Cluster-scoped
Most workload resources are namespaced; infrastructure (nodes, PVs, StorageClasses, ClusterRoles) is cluster-scoped. A Service can only target pods in its own namespace. -A is the shorthand for --all-namespaces — essential for cluster-wide visibility.
# which resources are namespaced?
kubectl api-resources --namespaced=true
kubectl api-resources --namespaced=false
# namespaced: pods, services, deployments, configmaps, secrets
# cluster-scoped: nodes, namespaces, PVs, StorageClasses, ClusterRoles
# access across namespaces:
# - a Service selector only matches pods in the SAME namespace
# - cross-namespace refs require special controllers (e.g. external-dns)
# list resources across all namespaces
kubectl get pods --all-namespaces
kubectl get pods -A # shorthandLabels & Selectors
Label Syntax
Labels are key-value pairs used to identify and group resources. Keys can have an optional prefix (a DNS subdomain, e.g. app.kubernetes.io/). Labels are meant for querying/filtering, unlike annotations which are for arbitrary non-identifying metadata.
# labels are key=value pairs
# key: prefix (optional DNS subdomain) + name
# value: string, up to 63 chars
metadata:
labels:
app: nginx
tier: frontend
env: prod
app.kubernetes.io/name: nginx
app.kubernetes.io/version: "1.25"
# view labels
kubectl get pods --show-labels
# add/overwrite a label
kubectl label pod nginx env=prod
kubectl label pod nginx env=staging --overwrite
# remove a label (trailing dash)
kubectl label pod nginx env-Equality & Set Selectors
Two selector flavors: equality-based (=, ==, !=) and set-based (in, notin, exists, !exists). Multiple comma-separated conditions are ANDed. Services and Deployments use equality-based selectors in YAML; kubectl -l supports both.
# equality-based: = == !=
kubectl get pods -l app=nginx
kubectl get pods -l env!=dev
kubectl get pods -l 'app=nginx,env=prod' # AND of conditions
# set-based: in, notin, exists
kubectl get pods -l 'env in (prod,staging)'
kubectl get pods -l 'env notin (dev)'
kubectl get pods -l 'tier' # key exists
kubectl get pods -l '!canary' # key does not exist
# combine equality and set-based
kubectl get pods -l 'app=nginx,env in (prod,staging)'Labeling Resources
label mutates identifying metadata — use it for anything selectors should match on. annotate is for non-identifying metadata (owner, ticket, description, tooling hints). Nodes are commonly labeled with topology or hardware hints to drive scheduling.
# label a pod
kubectl label pod nginx app=nginx tier=web
# label all pods matching a selector
kubectl label pods -l app=oldapp app=newapp --overwrite
# label a node (for scheduling hints)
kubectl label node node-1 disktype=ssd
# remove a node label
kubectl label node node-1 disktype-
# list nodes by label
kubectl get nodes -l disktype=ssd
# annotate (non-identifying metadata)
kubectl annotate pod nginx owner=alice
kubectl annotate pod nginx description="web server"Selectors in Manifests
In YAML, selectors use matchLabels (equality) and matchExpressions (set-based with In/NotIn/Exists/DoesNotExist). For Deployments the selector is immutable after creation — and the pod template labels must match it. Services only support matchLabels.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels: # equality-based
app: nginx
matchExpressions: # set-based (optional)
- key: env
operator: In
values: [prod, staging]
- key: tier
operator: Exists
template:
metadata:
labels:
app: nginx # MUST match selector
env: prod
tier: webAnnotations
Annotations hold non-identifying metadata used by tools — kubectl apply stores the last-applied config, cert-manager reads issuer hints, Prometheus reads scrape config, controllers store rollout state. Unlike labels, annotations values can be large JSON blobs.
metadata:
annotations:
kubernetes.io/change-cause: "bump nginx to 1.26"
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1",...}
cert-manager.io/cluster-issuer: "letsencrypt-prod"
# view annotations
kubectl get pod nginx -o jsonpath='{.metadata.annotations}'
# annotate
kubectl annotate pod nginx owner=team-platform
# remove an annotation (trailing dash)
kubectl annotate pod nginx owner-Recommended Labels
The app.kubernetes.io/* labels are a shared vocabulary so tools (Helm, operators, dashboards) can group all resources of an application. name+instance uniquely identify a release; component/part-of describe hierarchy; managed-by/created-by track ownership.
# shared recommended labels (app.kubernetes.io/*)
metadata:
labels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx-prod
app.kubernetes.io/version: "1.25"
app.kubernetes.io/component: web
app.kubernetes.io/part-of: storefront
app.kubernetes.io/managed-by: helm
app.kubernetes.io/created-by: controller-manager
# Helm applies these automatically; kubectl doesn't.
# They unify how tooling identifies an application's resources
# across Deployments, Services, Ingresses, ConfigMaps, etc.ReplicaSet
ReplicaSet Manifest
A ReplicaSet maintains a stable set of replica pods matching a selector — it recreates pods when they're deleted or fail. You almost always use a Deployment instead, which wraps a ReplicaSet and adds rolling updates and rollbacks.
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.25
# kubectl get rs
# kubectl describe rs nginxHow ReplicaSet Works
The ReplicaSet controller reconciles desired vs actual pod count using the selector. Each pod gets an ownerReference pointing at the RS — that's why kubectl get pod shows 'Owned by ReplicaSet/nginx'. Scaling just changes the desired count.
# the controller loop:
# 1. count pods matching selector
# 2. if fewer than replicas -> create from template
# 3. if more than replicas -> delete excess (arbitrary order)
# owned-by reference on each pod
kubectl get pod -l app=nginx -o jsonpath='{.items[*].metadata.ownerReferences}'
# if you delete a pod, the RS creates a new one immediately
kubectl delete pod <pod-name>
kubectl get pods -l app=nginx
# scale by editing replicas
kubectl scale rs nginx --replicas=5ReplicaSet vs ReplicationController
ReplicationController (the original v1 primitive) only supports equality selectors and is deprecated. ReplicaSet (apps/v1) adds set-based selectors. In practice, you should use Deployment — it manages ReplicaSets for you and provides rolling updates/rollbacks.
# ReplicationController (v1, legacy): equality-only selectors
# ReplicaSet (apps/v1, modern): set-based selectors (in, notin, exists)
# ReplicationController is deprecated; do not use.
apiVersion: v1
kind: ReplicationController # legacy
# ...
# ReplicaSet is the successor:
apiVersion: apps/v1
kind: ReplicaSet # modern
# ...
# Deployment wraps ReplicaSet and is the recommended primitive:
apiVersion: apps/v1
kind: Deployment # use this in practiceManual Scaling
kubectl scale changes the replicas field. If an HPA targets the same resource, it will override manual scaling on the next reconcile. Prefer kubectl scale for one-offs and HPA for ongoing elastic workloads.
# scale a replicaset directly
kubectl scale rs nginx --replicas=5
# scale via patch
kubectl patch rs nginx -p '{"spec":{"replicas":5}}'
# scale multiple resources at once
kubectl scale deployment nginx deployment web --replicas=3
# scale by referencing a file
kubectl scale --replicas=5 -f nginx.yaml
# the autoscaler (HPA) will override manual scaling
# if it targets the same resourceTemplate Changes & Adoption
Unlike a Deployment, a bare ReplicaSet does not roll out changes to the pod template — only newly created pods (after a delete/scale-up) get the new spec. A ReplicaSet will adopt any orphaned pod matching its selector, which can cause surprising behavior.
# changing the pod template does NOT update existing pods
# (unlike a Deployment, a bare ReplicaSet won't roll out)
kubectl edit rs nginx # change image -> new pods use it, old ones don't
# delete old pods to force them onto the new template
kubectl delete pods -l app=nginx
# a ReplicaSet will ADOPT existing pods that match its selector
# even pods created independently — useful (and dangerous)
# orphaned pods (no owner) matching the selector are taken over
kubectl get pods -l app=nginx --show-kindRemoving Pods from a ReplicaSet
To remove a pod without the ReplicaSet recreating it, either scale the RS to 0 first, or strip the ownerReferences from the pod so the RS no longer owns it. Useful for debugging a specific instance or extracting logs before cleanup.
# detaching a pod from its ReplicaSet (so it isn't replaced)
kubectl patch pod <pod-name> \
-p '{"metadata":{"ownerReferences":[]}}'
# now deleting this pod won't trigger a replacement,
# because the RS no longer owns it
# or scale the RS to 0 first, then delete
kubectl scale rs nginx --replicas=0
kubectl delete pod <pod-name>
# useful for debugging a specific failing pod instance
# without the RS immediately recreating itStatefulSet
StatefulSet Manifest
StatefulSet gives pods a stable identity (name, hostname, storage) and ordered deploy/scale. Requires a headless serviceName. volumeClaimTemplates creates a dedicated PVC per pod — so even if pod-0 is rescheduled, it reattaches the same volume.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
spec:
serviceName: mysql # required: headless service
replicas: 3
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:8
ports:
- containerPort: 3306
volumeMounts:
- name: data
mountPath: /var/lib/mysql
volumeClaimTemplates: # one PVC per pod
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10GiStable Identity
Each pod gets a stable ordinal name (mysql-0, mysql-1, …) and a stable DNS name via the headless service. When a node fails, pod-0 is rescheduled elsewhere but keeps its name and reattaches its volume — critical for clustered databases that depend on stable membership.
# pods are named <statefulset-name>-<ordinal>
# mysql-0, mysql-1, mysql-2
kubectl get pods -l app=mysql
# each pod has stable DNS (via the headless service):
# mysql-0.mysql.default.svc.cluster.local
# mysql-1.mysql.default.svc.cluster.local
# stable hostname inside the pod:
kubectl exec mysql-0 -- hostname
# -> mysql-0
# ordinal index also exposed:
kubectl exec mysql-0 -- sh -c 'echo $HOSTNAME'
# -> mysql-0
# pod identity persists across reschedulesStable Storage
volumeClaimTemplates generates one PVC per pod with a predictable name (data-mysql-0, …). PVCs survive pod and StatefulSet deletion — this prevents data loss but means you must explicitly delete PVCs to reclaim storage. The same volume always reattaches to the same ordinal pod.
# one PVC per pod, named <volumeClaimTemplate-name>-<pod-name>
kubectl get pvc
# NAME STATUS VOLUME CAPACITY ...
# data-mysql-0 Bound pvc-aaa 10Gi
# data-mysql-1 Bound pvc-bbb 10Gi
# data-mysql-2 Bound pvc-ccc 10Gi
# when pod-0 is rescheduled, it re-binds data-mysql-0
# deleting the StatefulSet does NOT delete PVCs by default
# (protects from accidental data loss)
# to actually delete the data:
kubectl delete pvc data-mysql-0 data-mysql-1 data-mysql-2Ordered Deployment & Pod Management
OrderedReady (default) creates pods strictly in order 0→1→2 and deletes in reverse — required by systems that need a primary to bootstrap before replicas. Parallel creates/deletes all pods simultaneously — faster for symmetric sharded stores (Cassandra, Elasticsearch).
spec:
podManagementPolicy: OrderedReady # default: strict order
# podManagementPolicy: Parallel # create/delete all at once
replicas: 3
# OrderedReady: create pod-0, then pod-1, then pod-2 (each waits for Ready)
# deletion is reverse: pod-2, pod-1, pod-0
# Parallel: all pods created/deleted simultaneously
# identity and storage remain stable either way
# check policy
kubectl get statefulset mysql -o jsonpath='{.spec.podManagementPolicy}'
# parallel is great for sharded stores that don't need bootstrap orderRolling Updates
StatefulSet rolling updates go in reverse ordinal order (N-1 down to 0). partition lets you canary the highest-ordinal pods first. OnDelete pauses updates until you manually delete a pod — useful for controlled migrations. Rollouts are slower than Deployments due to ordering.
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 2 # only pods with ordinal >= 2 update
maxUnavailable: 1 # (default 1) allowed down
# type: OnDelete -> only update when pod is manually deleted
# partition enables canaries: set partition=N to update
# only the top N pods; lower to 0 to roll out to all.
kubectl rollout status statefulset/mysql
# waits for all pods to be updated
# StatefulSets also support rollout history/undo,
# but only when the change is in the pod templateHeadless Service Association
The headless service (clusterIP: None) referenced by spec.serviceName is what makes stable per-pod DNS work. Without it the StatefulSet won't create pods. Clients resolve the service name for round-robin, or pod-0.svc to pin a primary. Don't forget to create the Service.
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
clusterIP: None # headless
selector:
app: mysql
ports:
- port: 3306
---
# StatefulSet references this service via spec.serviceName
# this gives each pod a stable DNS:
# mysql-0.mysql.default.svc.cluster.local
# clients can:
# - resolve mysql -> any pod IP (load-balanced by DNS round-robin)
# - resolve mysql-0.mysql -> pod-0's IP (pin to a specific replica)
# required: serviceName MUST match the headless service nameDaemonSet
DaemonSet Manifest
A DaemonSet runs exactly one pod per node (matching its scheduling constraints). Perfect for node-level agents: log collectors, monitoring, storage, CNI, ingress controllers. New nodes automatically get a pod; deleted nodes have theirs garbage-collected.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: log-agent
namespace: kube-system
labels:
app: log-agent
spec:
selector:
matchLabels:
app: log-agent
template:
metadata:
labels:
app: log-agent
spec:
containers:
- name: agent
image: fluent-bit:2.2
resources:
requests:
cpu: 100m
memory: 128Mi
# one pod per (matching) node automaticallyNode Scheduling
nodeSelector is the simplest constraint (all key=value must match). nodeAffinity adds In/NotIn/Exists operators and preferred/required rules. DaemonSets respect these like any pod — combined with tolerations you can run agents on tainted control-plane nodes too.
spec:
template:
spec:
nodeSelector: # simplest: must match all
disktype: ssd
# more flexible affinity
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values: [linux]
# only run on nodes labeled role=logging
nodeSelector:
role: logging
tolerations: # tolerate tainted nodes
- key: node-role.kubernetes.io/control-plane
operator: ExistsUpdate Strategy
RollingUpdate (default) updates pods node by node, bounded by maxUnavailable (can be a number or %). OnDelete pauses updates until you manually delete each pod — useful when node-by-node control is required. maxSurge defaults to 0 (DaemonSets rarely want extra pods).
spec:
updateStrategy:
type: RollingUpdate # default
rollingUpdate:
maxUnavailable: 1 # how many nodes update at once
maxSurge: 0 # (default 0) no extra pods
# type: OnDelete # only update when old pod is deleted
# trigger an update (e.g. new image)
kubectl set image daemonset/log-agent agent=fluent-bit:2.3
# watch rollout
kubectl rollout status daemonset/log-agent -n kube-system
# rollout restart (recreate pods on every node)
kubectl rollout restart daemonset/log-agent -n kube-systemTaints & Tolerations
Taints repel pods; tolerations let specific pods ignore a taint. Control-plane nodes are tainted NoSchedule so user workloads don't land there — node-level agents tolerate it to run everywhere. NoExecute also evicts non-tolerating pods already running.
# inspect a node's taints
kubectl describe node node-1 | grep Taints
# Taints: node-role.kubernetes.io/control-plane:NoSchedule
# tolerate control-plane so the agent runs there too
spec:
template:
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node.kubernetes.io/disk-pressure
operator: Exists
effect: NoExecute
# taint effects: NoSchedule | NoExecute | PreferNoScheduleDaemonSet Use Cases
DaemonSets are the standard deployment shape for node-level infrastructure. They frequently need privileged access (hostNetwork, hostPID, hostPath) to do their job — grant these deliberately and only to trusted images. Never use a DaemonSet for app workloads that should scale horizontally.
# classic DaemonSet workloads (node-level agents):
# - log collection: fluent-bit, filebeat, promtail
# - monitoring: node-exporter, datadog-agent
# - networking: kube-proxy, CNI plugins, ingress controllers
# - storage: CSI node plugins, Rook/Ceph
# - security: Falco, Tetragon, runtime security
# - GPU/FPGA drivers: nvidia-device-plugin
# storage / device plugins often need:
spec:
template:
spec:
hostNetwork: true # use node network namespace
hostPID: true # see node processes
hostPath: # mount node paths
- path: /var/log
- path: /var/lib/docker/containersDaemonSet Status
DESIRED equals the number of nodes matching the scheduling constraints; CURRENT/READY show how many pods exist and are passing readiness. -o wide is the easiest way to see which node each pod landed on. DaemonSets support rollout history/undo just like Deployments.
# desired = number of matching nodes
kubectl get daemonset log-agent -n kube-system
# NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
# log-agent 5 5 5 5 5
# show which nodes a DaemonSet pod landed on
kubectl get pods -n kube-system -l app=log-agent -o wide
# rollout history
kubectl rollout history daemonset/log-agent -n kube-system
# rollback
kubectl rollout undo daemonset/log-agent -n kube-system
# check a per-node pod's logs
kubectl logs -n kube-system log-agent-abcdeJob & CronJob
Job Manifest
A Job runs pods until a specified number of them complete successfully (completions). restartPolicy must be Never or OnFailure (not Always — that's for long-running workloads). backoffLimit caps retries; activeDeadlineSeconds caps total runtime.
apiVersion: batch/v1
kind: Job
metadata:
name: pi
spec:
completions: 1 # how many successful pods needed
parallelism: 1 # how many run concurrently
backoffLimit: 6 # retries before marking Failed
activeDeadlineSeconds: 600 # hard timeout
template:
spec:
restartPolicy: Never # OnFailure | Never
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
# kubectl get jobs
# kubectl logs job/piJob Completions & Parallelism
completions = how many successful pods satisfy the Job; parallelism = how many run concurrently. Indexed Jobs assign each pod an ordinal (env var JOB_COMPLETION_INDEX) — great for sharded batch processing where each pod owns one slice. NonIndexed (default) just needs N successes from anywhere.
spec:
completions: 5 # 5 successful pods total
parallelism: 2 # 2 pods run at a time
backoffLimit: 4
# default completionMode: NonIndexed (any 5 successes)
completionMode: Indexed
# Indexed: pods get 0..N-1 in JOB_COMPLETION_INDEX env,
# useful for sharded work (each pod handles one shard)
template:
spec:
restartPolicy: OnFailure
containers:
- name: worker
image: worker:1.0
# set via kubectl
kubectl create job pi --image=perl -- perl -Mbignum=bpi -wle "print bpi(2000)"Job Backoff & Timeouts
backoffLimit retries failed pods (exponential backoff). activeDeadlineSeconds is the hard ceiling — if hit, the Job is marked Failed even with retries left. ttlSecondsAfterFinished auto-cleans completed Jobs. podFailurePolicy (1.31+) lets you fail fast on certain exit codes or ignore voluntary evictions.
spec:
backoffLimit: 6 # default 6, retries on failure
activeDeadlineSeconds: 3600 # hard wall, even if retries remain
ttlSecondsAfterFinished: 86400 # auto-delete Job 1 day after done
startingDeadlineSeconds: 300 # (CronJob only) miss window
# pod failure handling (v1.31+):
podFailurePolicy:
rules:
- action: FailJob
onExitCodes:
operator: In
values: [42] # exit 42 -> don't retry, fail fast
- action: Count
onPodConditions:
- type: DisruptionTarget # don't count voluntary evictions
template:CronJob Manifest
A CronJob creates Jobs on a cron schedule (5 fields: minute hour day month weekday). concurrencyPolicy controls overlapping runs: Forbid skips a new run if the previous is still going; Replace kills the old and starts new. timeZone (1.27+) removes the need for UTC arithmetic.
apiVersion: batch/v1
kind: CronJob
metadata:
name: backup
spec:
schedule: "0 2 * * *" # 5-field cron: 02:00 daily
timeZone: "Asia/Shanghai"
concurrencyPolicy: Forbid # Allow | Forbid | Replace
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
startingDeadlineSeconds: 200
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: OnFailure
containers:
- name: backup
image: backup:1.0
# kubectl get cronjob
# kubectl get jobsCronJob Suspend & History
suspend: true pauses a CronJob without deleting it — handy during incidents. History limits keep the Job list manageable (default 3 success / 1 failure). kubectl create job --from=cronjob/X runs the template immediately, perfect for manual triggers or backfilling missed runs.
spec:
schedule: "*/5 * * * *"
suspend: true # pause without deleting the CronJob
successfulJobsHistoryLimit: 3 # default 3
failedJobsHistoryLimit: 1 # default 1
# toggle suspend
kubectl patch cronjob backup -p '{"spec":{"suspend":true}}'
kubectl patch cronjob backup -p '{"spec":{"suspend":false}}'
# trigger a one-off run immediately
kubectl create job --from=cronjob/backup manual-backup
# clean up old jobs manually
kubectl delete jobs -l job-name=backupCronJob Schedule & Edge Cases
Standard 5-field cron. When both day-of-month and day-of-week are restricted, cron fires on EITHER match (OR semantics) — a classic gotcha. If a CronJob's scheduled time is missed beyond startingDeadlineSeconds, that run is skipped rather than backfilled.
# cron syntax (5 fields):
# ┌───────────── minute (0-59)
# │ ┌───────────── hour (0-23)
# │ │ ┌───────────── day of month (1-31)
# │ │ │ ┌───────────── month (1-12)
# │ │ │ │ ┌───────────── day of week (0-6, Sun=0)
# "*/5 * * * *" every 5 minutes
# "0 2 * * *" daily at 02:00
# "0 0 * * 0" weekly Sunday midnight
# "0 0 1 * *" monthly on the 1st
# "30 3-5 * * *" 03:30, 04:30, 05:30
# gotchas:
# - day-of-month AND day-of-week both set => OR (not AND)
# - if startingDeadlineSeconds is missed, the run is skipped
# - control-plane restart can skip a tick if the schedule is missed
# - cron interprets literal fields; prefer timeZone over UTC mathIngress
Ingress Manifest
Ingress exposes HTTP/HTTPS routes from outside the cluster to Services, managed by an Ingress controller (nginx, traefik, ALB, …). ingressClassName selects which controller handles it. Each rule maps a host + path to a service+port backend.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: 80
- path: /api
pathType: Prefix
backend:
service:
name: api
port:
number: 8080Ingress with TLS
tls entries bind a TLS secret (type kubernetes.io/tls, holding tls.crt + tls.key) to one or more hosts. The Ingress controller terminates TLS and proxies plain HTTP to the backend. cert-manager can auto-issue and rotate Let's Encrypt certs based on the cluster-issuer annotation.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-tls
spec:
ingressClassName: nginx
tls:
- hosts:
- app.example.com
- www.example.com
secretName: web-tls-secret # kubernetes.io/tls secret
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: 80
# auto-issue certs with cert-manager:
# annotations:
# cert-manager.io/cluster-issuer: letsencrypt-prodIngress Rules & Default Backend
defaultBackend handles traffic that matches no rule (no host or unmatched path). Multiple rules let one Ingress route many virtual hosts. Each rule can have multiple path backends — a single Ingress can front a whole application's routing topology.
spec:
ingressClassName: nginx
defaultBackend: # catch-all for unmatched traffic
service:
name: default-svc
port:
number: 80
rules:
- host: app.example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service: { name: api, port: { number: 8080 } }
- host: blog.example.com # different host -> different svc
http:
paths:
- path: /
pathType: Prefix
backend:
service: { name: blog, port: { number: 80 } }IngressClass
An IngressClass declares a controller (e.g. k8s.io/ingress-nginx). Ingresses reference it via spec.ingressClassName so multiple controllers can coexist. Exactly one class can be marked default with the is-default-class annotation — new Ingresses without an explicit class use it.
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
# an Ingress picks a class via spec.ingressClassName
# (replaces the deprecated kubernetes.io/ingress.class annotation)
# multiple IngressClasses in one cluster:
# - nginx (community ingress-nginx)
# - traefik (Traefik controller)
# - alb (AWS ALB)
# only one IngressClass can be marked defaultPath Types
Exact matches the path verbatim; Prefix matches whole path segments (/api matches /api and /api/foo but not /apix). ImplementationSpecific defers to the controller — e.g. nginx regex. Always prefer Prefix or Exact for portable behavior; ImplementationSpecific ties you to one controller.
spec:
rules:
- host: app.example.com
http:
paths:
- path: /api/v1
pathType: Exact # only /api/v1 (exact match)
backend: ...
- path: /api
pathType: Prefix # /api, /api/, /api/x (prefix match)
backend: ...
- path: /old
pathType: ImplementationSpecific # controller decides
backend: ...
# Prefix matches path segments, not raw string prefix:
# /api matches /api, /api/, /api/x but NOT /apix
# Exact matches the path verbatim
# ImplementationSpecific defers to the controllerAnnotations (Nginx Controller)
Controller-specific annotations configure behavior beyond the standard Ingress spec. ingress-nginx has dozens: rewrite-target, ssl-redirect, body-size, timeouts, CORS, basic auth, configuration snippets. They're controller-specific — porting to another controller means rewriting them.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web
annotations:
kubernetes.io/ingress.class: nginx # legacy form
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com"
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Frame-Options: DENY";
spec:
rules:
- host: app.example.com
http:
paths:
- path: /v1(/|$)(.*)
pathType: ImplementationSpecific
backend:
service: { name: api, port: { number: 8080 } }Volume (PV/PVC)
PersistentVolume Manifest
A PersistentVolume (PV) is a cluster-scoped piece of storage provisioned by an admin or dynamically by a StorageClass. accessModes describe how it can be mounted. reclaimPolicy controls what happens to the storage when the PVC is released.
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-1
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce # RWO: one node R/W
persistentVolumeReclaimPolicy: Retain # Retain | Delete | Recycle
storageClassName: manual
hostPath: # for dev only (single node)
path: /mnt/data
# nfs:
# server: 10.0.0.1
# path: /export/data
# csi:
# driver: ebs.csi.aws.com
# volumeHandle: vol-xxxxPersistentVolumeClaim Manifest
A PersistentVolumeClaim (PVC) is a namespaced request for storage. The controller binds it to a matching PV (accessModes, sufficient capacity, same StorageClass). With a default StorageClass, leaving storageClassName empty triggers dynamic provisioning.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: manual
resources:
requests:
storage: 5Gi
volumeMode: Filesystem # Filesystem | Block
selector: # optional: filter PVs by labels
matchLabels:
disktype: ssd
# kubectl get pvc
# STATUS: Pending -> Bound once a matching PV is foundAccess Modes
AccessModes describe mount semantics — but actual support depends on the storage backend. EBS/GCE PD/Azure Disk are RWO only. RWX (multi-node read/write) needs NFS, CephFS, or a distributed filesystem. RWOP (1.27+) restricts to one pod even on the same node.
# four access modes (support is driver-specific):
# - ReadWriteOnce (RWO): mounted R/W by ONE node
# - ReadOnlyMany (ROX): mounted R/O by MANY nodes
# - ReadWriteMany (RWX): mounted R/W by MANY nodes
# - ReadWriteOncePod (RWOP): mounted R/W by ONE pod (1.27+)
# examples by backend:
# AWS EBS, GCE PD, Azure Disk: RWO only
# NFS: RWO, ROX, RWX
# CephFS, Portworx: RWO, RWX
# Block volumes: RWO
# RWX is required for pods on different nodes
# to share the same volume (e.g. read-only assets)Reclaim Policy
Retain keeps the data for safety — you must manually wipe and rebind the PV. Delete (default for dynamic provisioning) wipes both PV and underlying storage when the PVC goes. Recycle is deprecated. A Released PV with Retain needs its claimRef cleared before rebinding.
# what happens when a PVC is deleted:
# - Retain: PV stays, data preserved; PV must be manually
# reclaimed (delete data, then delete PV) before reuse
# - Delete: PV and the underlying storage are deleted (dynamic only)
# - Recycle: deprecated; ran scrub on the volume
# change policy on an existing PV
kubectl patch pv pv-1 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
# a released PV (PVC deleted, policy Retain) is stuck:
# phase=Released, claimRef still set
# to reuse: clear claimRef
kubectl patch pv pv-1 --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'PVC in a Pod
Mount a PVC like any volume via persistentVolumeClaim.claimName. subPath mounts a subdirectory — handy for sharing one volume across pods/paths. RWO volumes can be mounted by multiple pods but only on the same node; cross-node mounting causes multi-attach errors.
apiVersion: v1
kind: Pod
metadata:
name: db
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: data-claim
containers:
- name: db
image: postgres:16
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
subPath: pgdata # mount a subdirectory of the volume
# a PVC is namespaced and bound to ONE PV
# RWO PVCs can be mounted by multiple pods on the SAME node
# (e.g. one writer, several readers on that node)Static vs Dynamic Provisioning
Static provisioning requires admins to pre-create PVs — fine for NFS but painful at scale. Dynamic provisioning lets a PVC trigger on-demand volume creation via a StorageClass's provisioner. Modern clusters almost always use dynamic provisioning; static is reserved for shared/legacy storage.
# STATIC: admin pre-creates PVs; PVCs bind to them.
# Good for: pre-existing NFS exports, manual capacity planning.
# DYNAMIC: PVC requests storage; StorageClass provisions a PV
# automatically (AWS EBS, GCE PD, Ceph, local-path, etc.)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dynamic-claim
spec:
storageClassName: ebs-sc # triggers dynamic provisioning
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
# no PV exists -> provisioner creates one -> binds to PVC
# kubectl get pvc dynamic-claim -> Bound
# kubectl get pv -> shows the dynamically-created PVStorageClass
StorageClass Manifest
A StorageClass describes a 'flavor' of dynamically provisioned storage: the provisioner (CSI driver), parameters (disk type, encryption, fsType), and policies. PVCs reference it via spec.storageClassName. One cluster typically has several classes (fast SSD, bulk HDD, NFS, …).
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
type: gp3
fsType: ext4
encrypted: "true"
reclaimPolicy: Delete # Retain | Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
mountOptions:
- noatime
# kubectl get storageclassDefault StorageClass
Exactly one StorageClass can be the default (annotation is-default-class: true). PVCs that omit storageClassName automatically use it — convenient for users who don't care which backend they get. Use kubectl get sc to see which class is marked (default).
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
# a PVC with no storageClassName uses the default
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: auto
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
# (no storageClassName set -> uses 'standard')Provisioners
kubernetes.io/no-provisioner means static binding only (no dynamic creation). All real dynamic provisioning uses CSI drivers — cloud providers ship their own (ebs.csi.aws.com, etc.), and self-hosted options (Longhorn, Rook/Ceph, local-path) cover bare metal. Parameters are entirely provisioner-specific.
# built-in (no external driver): only static binding
provisioner: kubernetes.io/no-provisioner
# cloud CSI drivers (dynamic):
provisioner: ebs.csi.aws.com # AWS EBS
provisioner: pd.csi.storage.gke.io # GCE PD
provisioner: disk.csi.azure.com # Azure Disk
provisioner: nfs.csi.k8s.io # NFS
# popular self-hosted:
provisioner: driver.longhorn.io # Longhorn
provisioner: rook-ceph.ceph.com # Rook/Ceph
provisioner: rancher.io/local-path # local-path (dev)
# parameters are provisioner-specific
# (e.g. AWS: type=gp3|iops=3000|encrypted=true)Volume Binding Mode
Immediate binds the PVC the moment it's created — risky for zonal disks because the pod might land in a different zone. WaitForFirstConsumer defers binding until a pod actually consumes the PVC, so the volume lands in the right zone. Always use WaitForFirstConsumer for zonal cloud disks.
spec:
volumeBindingMode: WaitForFirstConsumer
# ^ default for many CSI drivers
# alternatives:
# volumeBindingMode: Immediate
# Immediate: PVC binds/provisions as soon as created
# risk: volume created in zone A, pod scheduled in zone B
# -> multi-attach / cross-zone latency issues
# WaitForFirstConsumer: delays binding until a pod using the PVC
# is scheduled. The PV is then created in the pod's zone.
# REQUIRED for topology-constrained backends (EBS, etc.)
# topology constraints come from the CSI driver via
# AllowedTopologiesReclaim Policy per StorageClass
StorageClass.reclaimPolicy sets the policy of every PV it dynamically provisions. Delete (default) destroys the volume when the PVC is deleted — efficient but dangerous for production data. To save data, switch the PV to Retain before deleting the PVC, or use a Retain-class from the start.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-temp
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete # default for dynamic volumes
# reclaimPolicy: Retain # keep data after PVC delete
# the StorageClass policy becomes the PV's policy
# when a PV is dynamically provisioned
# to preserve data, either:
# 1. use a Retain StorageClass
# 2. patch the dynamically-created PV to Retain before deleting PVC:
kubectl patch pv pvc-xxxx -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
# then delete PVC safely; data survivesVolume Expansion
Setting allowVolumeExpansion: true lets users grow a PVC by editing spec.resources.requests.storage. The CSI driver expands the underlying volume, then kubelet expands the filesystem on next mount. Shrinking is never supported. Most cloud CSI drivers support online expansion (no unmount needed).
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-sc
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true # must be true to resize
# then resize by editing the PVC's requested size
kubectl patch pvc data-claim -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
# status.conditions shows Resizing / FileSystemResizePending
kubectl describe pvc data-claim
# requirements:
# - StorageClass.allowVolumeExpansion = true
# - driver supports online expansion (most CSI drivers do)
# - shrinking is NOT supported (only grow)
# - FilesystemResize happens after the PV is expandedHelm
Helm Install & Release
Helm packages Kubernetes manifests as a 'chart' and manages them as a 'release'. install creates a release; upgrade applies changes; upgrade --install is idempotent. Releases are tracked by a Secret/ConfigMap in the namespace — that's how rollback works.
# add and update a chart repo
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
# search for charts
helm search repo nginx
helm search hub wordpress # searches Artifact Hub
# install a chart (creates a release)
helm install my-nginx bitnami/nginx --namespace web --create-namespace
# upgrade an existing release
helm upgrade my-nginx bitnami/nginx --set replicaCount=3
# upgrade + install in one idempotent command
helm upgrade --install my-nginx bitnami/nginx -f values.yaml
# uninstall
helm uninstall my-nginx -n web
# list releases
helm list -n web
helm list --all-namespacesChart Structure
A chart is a directory of templated YAML plus defaults. Chart.yaml holds metadata; values.yaml holds defaults; templates/ holds Go templates rendered with those values. helm create scaffolds a working chart. helm lint catches schema/template errors before publishing.
my-chart/
├── Chart.yaml # chart metadata (name, version, apiVersion)
├── values.yaml # default values (overridable)
├── charts/ # chart dependencies (subcharts)
├── templates/ # Go-templated YAML
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── _helpers.tpl # named templates (partials)
│ └── NOTES.txt # post-install notes
├── templates.yaml # (optional) CRDs to install before templates
└── README.md
# create a new chart skeleton
helm create my-chart
# lint a chart
helm lint my-chart
# package into a .tgz
helm package my-chartValues & Overrides
values.yaml holds chart defaults. Override with --set (for scalars) or -f (for whole files). helm show values lists every key a chart accepts. helm template renders manifests without touching the cluster — essential for diffing what a release will actually apply.
# values.yaml (chart defaults)
image:
repository: nginx
tag: "1.25"
replicaCount: 2
service:
type: ClusterIP
port: 80
# override on the command line
helm upgrade --install web my-chart \
--set replicaCount=5 \
--set image.tag=1.26 \
--set service.type=LoadBalancer
# override from a file
helm upgrade --install web my-chart -f prod-values.yaml
# see all values a chart supports
helm show values bitnami/nginx
# rendered output (debug, no install)
helm template web my-chart -f prod-values.yamlTemplates & Built-in Objects
Templates use Go template syntax with Sprig functions. .Release.Name makes every install unique; .Values exposes chart values; .Chart pulls from Chart.yaml. The default function supplies fallbacks. Render-safe naming via .Release.Name lets one chart run many releases in the same namespace.
# built-in template objects:
# .Values chart values (after overrides)
# .Release release metadata (Name, Namespace, Service, Revision, IsInstall, IsUpgrade)
# .Chart Chart.yaml contents
# .Files files in the chart
# .Capabilities cluster capabilities (KubeVersion, APIVersions)
# .Template current template info
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-app
labels:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount | default 1 }}
selector:
matchLabels:
app: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ .Release.Name }}Helm Repository
Helm 3 supports both classic HTTP chart repos and OCI registries (helm push/pull oci://…). OCI is the modern, signing-friendly path. helm repo update refreshes indexes; always run it before upgrade to get the latest chart versions. Artifact Hub (helm search hub) indexes public charts.
# add a chart repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add jetstack https://charts.jetstack.io
# list configured repos
helm repo list
# update repo indexes
helm repo update
# remove a repo
helm repo remove bitnami
# push to OCI registry (Helm 3+)
helm registry login registry.example.com
helm package my-chart
helm push my-chart-0.1.0.tgz oci://registry.example.com/charts
# pull an OCI chart
helm pull oci://registry.example.com/charts/my-chart --version 0.1.0Rollback & History
Every install/upgrade creates a numbered revision stored as a Secret. helm rollback flips to a previous revision instantly — Helm's killer feature vs plain kubectl apply. helm get values shows what was actually deployed, which is invaluable for debugging drifted state.
# view release history
helm history my-nginx -n web
# rollback to a previous revision
helm rollback my-nginx 1 -n web
# get rendered manifests of a deployed release
helm get manifest my-nginx -n web
# get the values used at install/upgrade
helm get values my-nginx -n web
helm get values my-nginx -n web --all # incl. chart defaults
# get all info (hooks, manifest, values, notes)
helm get all my-nginx -n web
# wait for resources to be ready on install/upgrade
helm upgrade --install my-nginx bitnami/nginx --wait --timeout 5mNetworkPolicy
NetworkPolicy Manifest
A NetworkPolicy is a firewall rule for pods. An empty podSelector selects all pods in the namespace; an empty rules list denies all traffic of that type. NetworkPolicies are additive — a pod is allowed if ANY policy permits it. They require a supporting CNI; plain flannel ignores them.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: prod
spec:
podSelector: {} # selects ALL pods in the namespace
policyTypes:
- Ingress
- Egress
# no ingress/egress rules => deny all traffic of that type
# effect: pods in 'prod' can't send or receive ANY traffic
# until another policy explicitly allows it
# REQUIRES a CNI that supports NetworkPolicy
# (Calico, Cilium, Weave, kube-router; NOT flannel alone)Ingress Rules
Ingress rules allow traffic INTO selected pods. from entries are OR'd; ports are OR'd. podSelector matches pods in the same namespace; namespaceSelector matches whole namespaces (use the auto-added kubernetes.io/metadata.name label). Combining both in one item ANDs them.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-web
namespace: prod
spec:
podSelector:
matchLabels:
app: web
policyTypes:
- Ingress
ingress:
- from:
- podSelector: # pods in same namespace
matchLabels:
app: gateway
- namespaceSelector: # whole other namespace
matchLabels:
kubernetes.io/metadata.name: monitoring
- ipBlock: # external CIDR
cidr: 10.0.0.0/8
ports:
- protocol: TCP
port: 8080Egress Rules
Egress rules control what selected pods can talk TO. A common gotcha: a default-deny egress breaks DNS resolution and pods can't reach kube-dns. Always allow UDP/TCP 53 to the kube-system namespace before denying everything else. ipBlock.except carves out exceptions.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-egress-dns
namespace: prod
spec:
podSelector:
matchLabels:
app: web
policyTypes:
- Egress
egress:
- to:
- namespaceSelector: {} # any namespace (kube-dns)
ports:
- protocol: UDP
port: 53
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8 # allow internet, deny internal
ports:
- protocol: TCP
port: 443
# WARNING: egress deny-all blocks DNS and breaks pods.
# Always allow kube-dns first.Selectors & Namespace Selection
A bare podSelector matches pods in the policy's own namespace. A namespaceSelector matches whole namespaces. Specifying BOTH podSelector AND namespaceSelector in one item ANDs them (pods with label X in namespace Y) — placing them as separate items ORs them. ipBlock matches external IPs.
spec:
podSelector: # selects pods in THIS namespace
matchLabels:
app: web
ingress:
- from:
# SAME namespace, matching pods
- podSelector:
matchLabels:
app: gateway
# DIFFERENT namespace (whole namespace)
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
# BOTH: pods in another namespace with specific labels
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: backend
podSelector:
matchLabels:
app: api
# external IP range
- ipBlock:
cidr: 192.168.1.0/24Default Policies (Deny/Allow All)
Default-deny policies (empty rules) lockdown a namespace so only explicit allow-lists open holes. Apply deny-all ingress + deny-all egress as a baseline, then add targeted allow policies. An empty ingress: [{}] entry allows all ingress — the escape hatch for opt-out.
# --- DEFAULT DENY INGRESS (in a namespace) ---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: prod
spec:
podSelector: {}
policyTypes: [Ingress]
---
# --- DEFAULT DENY EGRESS ---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: prod
spec:
podSelector: {}
policyTypes: [Egress]
---
# --- ALLOW ALL INGRESS (whitelist opt-out) ---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-ingress
namespace: prod
spec:
podSelector: {}
policyTypes: [Ingress]
ingress:
- {} # allow from anywherePolicy Types & Default Behavior
NetworkPolicies are additive — once any policy selects a pod, that traffic type becomes default-deny for the pod unless an explicit rule allows it. With no policies at all, everything is open. List policies matching a pod's labels to see what governs its traffic.
spec:
policyTypes:
- Ingress
- Egress
# if omitted, defaults are inferred:
# - Ingress is always present (a policy always affects ingress)
# - Egress is implied if any egress rule exists
# key behaviors:
# - policies are ADDITIVE: a pod is allowed if ANY matching
# policy permits the traffic
# - if NO policy selects a pod => all traffic allowed (default open)
# - if at least one policy selects a pod => default deny for that type
# unless an explicit rule allows it
# inspect effective policies for a pod
kubectl get networkpolicy -n prod -l app=webRBAC
Role (Namespaced)
A Role grants permissions on namespaced resources within a single namespace. rules list apiGroups, resources, and verbs. pods/log is a subresource; pods/exec requires the create verb (because exec creates a subresource). Use '*' to wildcard any value.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: dev
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
# a Role grants permissions WITHIN one namespace
# verbs: get, list, watch, create, update, patch, delete, deletecollection, "*"
# apiGroups: "" (core), "apps", "batch", "rbac.authorization.k8s.io", ...RoleBinding
A RoleBinding grants a Role (or a ClusterRole, scoped to one namespace) to subjects: Users, Groups, or ServiceAccounts. The binding lives in the namespace where the permissions apply. roleRef is immutable — to change the referenced Role, delete and recreate the binding.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: dev
subjects:
- kind: User
name: alice
apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
name: my-sa
namespace: dev
- kind: Group
name: dev-team
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
# binds the Role to users, groups, or service accounts
# RoleBinding can also reference a ClusterRole (scoped to its namespace)ClusterRole & ClusterRoleBinding
A ClusterRole grants permissions on cluster-scoped resources (nodes, namespaces, PVs) — Roles can't touch these. ClusterRoles can also be bound in any namespace via a RoleBinding for reuse (define once, bind many). ClusterRoleBinding grants a ClusterRole cluster-wide.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: node-reader
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: read-nodes
subjects:
- kind: ServiceAccount
name: monitor
namespace: kube-system
roleRef:
kind: ClusterRole
name: node-reader
apiGroup: rbac.authorization.k8s.io
# ClusterRole grants cluster-scoped resources (nodes, PVs, namespaces)
# or can be reused across namespaces via a RoleBindingVerbs, Resources & API Groups
apiGroups group resources — core is '' (empty string); apps has Deployments/StatefulSets; batch has Jobs/CronJobs. Subresources (pods/log, pods/exec, pods/portforward) are separate permissions. resourceNames restricts a rule to specific instances. nonResourceURLs covers /healthz, /metrics, etc.
rules:
- apiGroups: [""] # core API group
resources: ["pods", "services", "configmaps", "secrets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets"]
verbs: ["*"]
- apiGroups: ["batch"]
resources: ["jobs", "cronjobs"]
verbs: ["create", "delete", "get"]
- apiGroups: [""]
resources: ["pods/log", "pods/exec", "pods/portforward"]
verbs: ["get", "create"]
- nonResourceURLs: ["/healthz", "/healthz/*"] # non-resource URLs
verbs: ["get"]
# resourceNames: restrict to specific instances
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["my-config"]
verbs: ["get"]ServiceAccount
A ServiceAccount is an identity for pods. Assign it via serviceAccountName; the token auto-mounts at /var/run/secrets/kubernetes.io/serviceaccount. Since 1.24, SAs don't have permanent token Secrets — use kubectl create token for short-lived, time-bound tokens. automountServiceAccountToken: false opts out.
# create a service account
kubectl create serviceaccount my-sa -n dev
# assign a RoleBinding (see RoleBinding example)
# mount the SA in a pod
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
serviceAccountName: my-sa
automountServiceAccountToken: false # opt out of token mount
containers:
- name: app
image: myapp:1.0
# short-lived tokens (1.24+) - no permanent secret
kubectl create token my-sa -n dev --duration=1h
# list SAs
kubectl get sa -n devkubectl auth can-i
kubectl auth can-i checks RBAC permissions for the current (or impersonated) user — essential for debugging 'forbidden' errors. --list shows every allowed verb/resource. --as impersonates a user/SA for testing. Built-in roles (view, edit, admin, cluster-admin) cover common needs.
# check if you can do something
kubectl auth can-i create pods -n dev
kubectl auth can-i delete deployments -n prod
# check on behalf of another user/SA
kubectl auth can-i list secrets --as=system:serviceaccount:dev:my-sa -n dev
# check all allowed verbs on a resource
kubectl auth can-i --list --as=alice -n dev
# whoami (1.25+)
kubectl auth whoami
# reconcile: detect over-privileged bindings
kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide
# common built-in cluster roles:
# view, edit, admin, cluster-adminProbes (Liveness/Readiness/Startup)
Liveness Probe
A liveness probe tells kubelet when to restart a container. If it fails failureThreshold consecutive times, kubelet kills the container and the restart policy applies. Use it to recover from deadlocks or unrecoverable states — not to check dependencies (that's readiness).
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: myapp:1.0
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
successThreshold: 1
# if the probe fails failureThreshold times in a row,
# kubelet kills and restarts the container (per restartPolicy)Readiness Probe
A readiness probe controls whether the pod receives traffic from Services. A failing readiness probe removes the pod's IP from endpoints — but does NOT restart it. Use it for warm-up (don't route until ready) and to drain traffic during transient failures or graceful shutdown.
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: myapp:1.0
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
# a failing readiness probe REMOVES the pod from service endpoints,
# but does NOT restart the container.
# use to drain traffic during deploys, warm-up, or transient outages.Startup Probe
A startup probe (1.18+) is for slow-to-start apps (Java, legacy servers). While it runs, liveness and readiness are disabled — so a slow boot doesn't get killed by liveness. Once startup succeeds, liveness/readiness activate. Use a long failureThreshold × periodSeconds window for the boot.
apiVersion: v1
kind: Pod
metadata:
name: slow-app
spec:
containers:
- name: app
image: myapp:1.0
startupProbe:
httpGet:
path: /startup
port: 8080
failureThreshold: 30
periodSeconds: 10 # up to 300s to start
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
# while the startup probe runs, liveness/readiness are disabled.
# once startup succeeds, liveness/readiness take over.Probe Handlers
Three handler types: httpGet (success = HTTP 2xx/3xx), tcpSocket (success = TCP connect), exec (success = exit 0). gRPC probe (1.27+ GA) is the clean option for gRPC services. Prefer httpGet — it lets the app signal 'healthy but busy' (503) without restarting. exec is fragile (needs shell/binaries in image).
# three handler types:
# 1. HTTP GET (most common) - success = 200-399
livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: X-Custom
value: probe
scheme: HTTP
# 2. TCP socket - success = port open
livenessProbe:
tcpSocket:
port: 3306
# 3. Exec - success = exit code 0
livenessProbe:
exec:
command:
- /bin/sh
- -c
- "pg_isready -h localhost"
# gRPC (1.24+, alpha -> GA): use grpc probe
livenessProbe:
grpc:
port: 9090
service: my.package.HealthProbe Parameters
initialDelaySeconds delays the first probe — but a startup probe is a cleaner way to handle slow boots. periodSeconds is the cadence; failureThreshold × periodSeconds is how long a pod can be unhealthy before action. Tighten these for fast failover, loosen for tolerance. terminationGracePeriodSeconds caps shutdown.
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 15 # wait before first probe (after start)
periodSeconds: 10 # probe every 10s
timeoutSeconds: 2 # max time per probe
successThreshold: 1 # consecutive successes to mark OK
failureThreshold: 3 # consecutive failures to mark failed
terminationGracePeriodSeconds: 30 # (liveness) grace before SIGKILL
# tuning tips:
# - set initialDelaySeconds > typical startup time
# - OR use a startup probe instead (cleaner)
# - shorter period = faster reaction, more load
# - higher failureThreshold = more tolerant of blipsProbe Failure Behavior
Liveness failure kills+restarts the container in place (not rescheduled). Readiness failure just removes the pod from Services — no restart. Startup failure behaves like liveness. Flapping liveness often comes from probes that depend on slow downstream services — keep health checks local and cheap.
# --- LIVENESS FAIL ---
# container killed -> restarted (per restartPolicy: Always/OnFailure)
# pod stays on the same node (NOT rescheduled)
# CrashLoopBackOff if it keeps failing
# --- READINESS FAIL ---
# pod stays running, but is REMOVED from Service endpoints
# no traffic until it succeeds again
# NOT restarted by kubelet
# --- STARTUP FAIL ---
# container killed and restarted (like liveness)
# liveness/readiness never activate
# after failureThreshold × periodSeconds, pod is stuck restarting
# check probe results:
kubectl describe pod app | grep -A5 "Last State"
kubectl get pod app -o jsonpath='{.status.containerStatuses[*].state}'
# common cause of flapping: liveness probe hitting a slow dependencyResources & Advanced kubectl
Requests & Limits
requests are what the scheduler reserves (a pod is scheduled only if a node has spare requests) — set them to your steady-state needs. limits are hard ceilings: CPU is throttled, memory is OOMKilled. cpu is in cores (500m = 0.5), memory in bytes (Mi/Gi). Always set both for production.
spec:
containers:
- name: app
image: myapp:1.0
resources:
requests: # what the scheduler guarantees (reservation)
cpu: 250m # 250 millicores = 0.25 core
memory: 256Mi # 256 mebibytes
ephemeral-storage: 1Gi
limits: # hard ceiling (enforced)
cpu: "1" # 1 core
memory: 512Mi
ephemeral-storage: 2Gi
# cpu limit: throttled (cfs quota)
# memory limit: OOMKilled if exceeded
# ephemeral-storage limit: evicted if exceeded
# request only (no limit): best-effort burstLimitRange Defaults
LimitRange applies default requests/limits to pods that don't specify them — critical when a ResourceQuota requires requests. max/min bound how large/small a single container can go. maxLimitRequestRatio caps how much a container can burst above its request, preventing noisy neighbors.
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
namespace: prod
spec:
limits:
- type: Container
default: # applied as LIMITS when not set
cpu: 500m
memory: 512Mi
defaultRequest: # applied as REQUESTS when not set
cpu: 100m
memory: 128Mi
max: # max allowed LIMITS
cpu: "4"
memory: 8Gi
min: # min allowed REQUESTS
cpu: 50m
memory: 64Mi
maxLimitRequestRatio: # limit/request ratio cap (burst cap)
cpu: "4"
# a pod with no resources gets default/defaultRequest applied
# a pod with limit=4cpu, request=50m would breach maxLimitRequestRatioHorizontal Pod Autoscaler
HPA scales replicas based on metrics (CPU/memory via metrics-server, or custom/external via Prometheus Adapter). behavior (v2) tunes aggressiveness: stabilizationWindowSeconds prevents flapping, policies cap rate per minute. Scale up is usually fast/aggressive; scale down is intentionally slow.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web
minReplicas: 3
maxReplicas: 50
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization # avg across pods
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
behavior: # (v2) scale up/down tuning
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50 # max 50% down per minute
periodSeconds: 60
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 60
# requires metrics-server for CPU/memory
kubectl get hpa web -wPod Disruption Budget
A PodDisruptionBudget limits how many pods of a set can be voluntarily evicted at once — protecting availability during node drains and cluster-autoscaler actions. Use minAvailable or maxUnavailable (not both). It does NOT protect against involuntary disruptions (hardware failure) — only replicas do.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-pdb
namespace: prod
spec:
minAvailable: 2 # at least 2 pods always available
# OR
# maxUnavailable: 1 # at most 1 down at a time
selector:
matchLabels:
app: web
# PDB protects against VOLUNTARY disruptions:
# - kubectl drain (node maintenance)
# - cluster autoscaler evictions
# - kubectl delete pod (controller-managed)
# It does NOT prevent INVOLUNTARY disruptions (node failure, kernel panic).
# check PDB status
kubectl get pdb -n prod
kubectl describe pdb web-pdb -n prodTaints, Tolerations & Affinity
Taints repel pods; tolerations let specific pods ignore them — use to dedicate nodes (GPU, ingress, control-plane). nodeAffinity prefers/requires certain nodes. podAntiAffinity spreads replicas across topology domains (hosts, zones) for HA. Remove a taint with a trailing dash on the same key+effect.
# taint a node
kubectl taint nodes node-1 dedicated=gpu:NoSchedule
# tolerate the taint so a pod can schedule there
spec:
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
# prefer specific nodes
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: disktype
operator: In
values: [ssd]
podAntiAffinity: # spread replicas across nodes
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app: web
# remove a taint
kubectl taint nodes node-1 dedicated=gpu:NoSchedule-Advanced kubectl (patch/wait/top/debug)
patch supports JSON, strategic-merge, and merge types. wait blocks until a condition is met — perfect for scripts. top shows live CPU/memory (needs metrics-server). kubectl debug attaches an ephemeral container to a running pod (great for distroless images) or opens a shell on a node.
# JSON patch (precise field ops)
kubectl patch deployment web --type json -p \
'[{"op":"replace","path":"/spec/replicas","value":3}]'
# strategic merge patch (default for k8s objects)
kubectl patch svc web -p '{"spec":{"type":"NodePort"}}'
# wait for a condition
kubectl wait --for=condition=Ready pod/web --timeout=60s
kubectl wait --for=delete pod/web --timeout=30s
# resource usage (needs metrics-server)
kubectl top nodes
kubectl top pods -n prod
kubectl top pod web --containers
# ephemeral debug container (1.25+)
kubectl debug -it web --image=busybox --target=web
# debug a node
kubectl debug node/node-1 -it --image=ubuntu
# run a one-off pod
kubectl run tmp --rm -it --image=alpine -- sh
# explain a field
kubectl explain pod.spec.containers.resources관련 Kubernetes 스니펫
Copy-paste ready code for common tasks.
Pod Manifest
Define a standalone Pod with a container, env, and probe.
Deployment & Rollout
Manage a replicated, rolling-updated workload.
Service Types
Expose Pods via ClusterIP, NodePort, and LoadBalancer services.
ConfigMap & Secret
Inject configuration and sensitive data into containers.
Ingress Routing
Route external HTTP traffic to services by host and path.
Labels & Selectors
Tag resources and query them with selectors.
kubectl Basics
Common kubectl commands for everyday operations.
Namespaces
Partition a cluster into isolated virtual clusters.
Was this helpful?