Kubernetes (K8S)

Kubernetes reference skill: kubectl commands for pods, deployments, services, networking, RBAC, Helm, and cluster troubleshooting.

Kubernetes (K8S)

Kubernetes reference skill: kubectl commands for pods, deployments, services, networking, RBAC, Helm, observability, GitOps, security, and cluster troubleshooting.

What this skill covers

Operating Kubernetes (K8s) from basic pod management through deployment strategies, networking, and security to cluster administration — the kubectl and helm commands and YAML manifests needed for day-to-day operations and cluster troubleshooting. It now also covers the ecosystem around the cluster: observability (Prometheus, Grafana, OpenTelemetry), GitOps delivery (Flux, Argo CD), security scanning and runtime detection (Trivy, Falco), and a recommended learning path including preparation for the CKA/CKAD/CKS certifications.

When to use it

Setting up or administering a Kubernetes cluster (nodes, namespaces, RBAC)
Deploying, scaling, or rolling back an application using Deployments
Debugging a pod stuck in Pending, CrashLoopBackOff, or ImagePullBackOff
Exposing a service internally or externally (Services, Ingress, DNS)
Managing configuration and secrets (ConfigMaps, Secrets, TLS, registry credentials)
Installing or upgrading applications with Helm charts (including template debugging)
Investigating a node in NotReady state or under resource pressure
Writing or reviewing Kubernetes YAML manifests (Pod, Deployment, Service)
Setting up monitoring and alerting (Prometheus, Grafana) or instrumenting applications (OpenTelemetry)
Adopting a GitOps workflow (Flux, Argo CD) or progressive delivery (Argo Rollouts)
Scanning images and manifests (Trivy) and runtime threat detection (Falco)
Preparing for the CKA / CKAD / CKS certifications

Core concepts

Kubernetes architecture

Control Plane: API Server, etcd, Controller Manager, Scheduler
Worker Nodes: Kubelet, Kube Proxy, Container Runtime
Pods: The smallest deployable unit containing one or more containers
Services: Network abstraction for pods
Deployments: Declarative way of managing pod replicas
ConfigMaps/Secrets: Managing configuration and sensitive data

Tip for understanding the architecture: Work through Kubernetes the Hard Way once — a manual cluster bootstrap (certificates, etcd, API server, kubelet) with no kubeadm. After one pass you will never forget what the control plane actually does: for example, that only the API server talks to etcd, and that the kubelet itself watches for pods assigned to its node.

The CNCF ecosystem

Cloud-native tools have three maturity tiers at the CNCF:

Graduated — a safe production choice (Kubernetes, Prometheus, Helm, Envoy, Falco, Flux, Argo)
Incubating — proven but still evolving projects
Sandbox — experimental projects

The maturity tier is a useful filter: when someone proposes a new tool, its CNCF tier tells you a lot about community health and long-term viability.

Pod management

Basic pod commands

# Get pods
kubectl get pods
kubectl get pods -o wide          # Detailed view
kubectl get pods -w               # Watch mode
kubectl get pods -o yaml          # YAML output

# Pod operations
kubectl describe pod <pod-name>   # Detailed information
kubectl logs <pod-name>           # View logs
kubectl logs -f <pod-name>        # Follow logs
kubectl logs --previous <pod-name>  # Logs from crashed container
kubectl exec -it <pod-name> -- /bin/bash  # Execute into pod

# Debug distroless/minimal images (no shell inside)
kubectl debug -it <pod-name> --image=busybox --target=<container-name>

# Pod lifecycle
kubectl delete pod <pod-name>     # Delete pod
kubectl edit pod <pod-name>       # Edit pod

Creating pods

# Create pod from image
kubectl run <pod-name> --image=<image-name>

# Create pod with port and expose as service
kubectl run <pod-name> --image=<image-name> --port=<port> --expose

# Generate pod YAML
kubectl run <pod-name> --image=<image-name> --dry-run=client -o yaml > pod.yaml

Node management

Node operations

# Get nodes
kubectl get nodes
kubectl get nodes -o wide
kubectl describe node <node-name>

# Node maintenance
kubectl drain <node-name> --ignore-daemonsets  # Safely evict pods
kubectl cordon <node-name>        # Mark as unschedulable
kubectl uncordon <node-name>      # Allow scheduling

Creating resources

Applying manifests

# Apply single file
kubectl apply -f <file>.yaml

# Apply multiple files
kubectl apply -f <file1>.yaml -f <file2>.yaml

# Apply directory
kubectl apply -f ./<directory>/

# Apply from URL
kubectl apply -f https://<url>

# Diff before apply
kubectl diff -f <file>.yaml

Imperative resource creation

# Create deployment
kubectl create deployment <name> --image=<image>
kubectl create deployment <name> --image=<image> --dry-run=client -o yaml > deployment.yaml

# Create service
kubectl create service <type> <name> --tcp=<port>:<target-port>
kubectl create service <type> <name> --tcp=<port>:<target-port> --dry-run=client -o yaml > service.yaml

# Expose existing deployment
kubectl expose deployment <name> --type=<type> --port=<port> --target-port=<target-port>

Configuration management

# Create ConfigMap
kubectl create configmap <name> --from-literal=<key>=<value>
kubectl create configmap <name> --from-file=<file>
kubectl create configmap <name> --from-env-file=<file>

# Create Secret
kubectl create secret generic <name> --from-literal=<key>=<value>
kubectl create secret generic <name> --from-file=<file>

Monitoring and troubleshooting

Resource monitoring

# Node utilization
kubectl top nodes
kubectl top node <node-name>

# Pod utilization
kubectl top pods
kubectl top pods <pod-name>

Debugging commands

# Check pod status
kubectl get pods
kubectl describe pod <pod-name>

# Check logs
kubectl logs <pod-name>
kubectl logs <pod-name> -c <container-name>  # Multi-container pods

# Check events
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl events --for pod/<pod-name>            # Events for one object
kubectl get events --field-selector involvedObject.name=<pod-name>

# Port forwarding for debugging
kubectl port-forward pod/<pod-name> <local-port>:<pod-port>

Observability (Prometheus, Grafana, OpenTelemetry)

Observability is not optional — put it in place before you think you need it. Three layers form a single whole:

Prometheus — metrics

The de facto standard for metrics in Kubernetes: pull-based scraping, a time-series database, the PromQL query language, and Alertmanager for alerting.

Key concepts: counter vs. gauge vs. histogram, service discovery, recording rules. In K8s it is typically deployed via kube-prometheus-stack, which automatically scrapes the control plane, nodes, and annotated pods.

# Install kube-prometheus-stack via Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

# Access Prometheus / Grafana locally
kubectl port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 -n monitoring
kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring

Useful PromQL patterns:

# CPU throttling (a common problem caused by bad limits)
rate(container_cpu_cfs_throttled_seconds_total[5m])

# Pods close to their memory limit (OOMKill candidates)
container_memory_working_set_bytes / on(pod) kube_pod_container_resource_limits{resource="memory"} > 0.9

# Container restarts in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 0

Grafana — visualization

The visualization layer on top of Prometheus (plus Loki for logs and Tempo for traces). The goal is dashboards that surface problems, not dashboards that just look good:

RED method for services: Rate (request count), Errors (error rate), Duration (latency)
USE method for resources: Utilization, Saturation, Errors

Four panels someone actually looks at beat forty panels nobody reads. Hands-on tutorials: grafana.com/tutorials.

OpenTelemetry — instrumentation

The vendor-neutral standard for traces, metrics, and logs. One SDK/API for instrumenting code, the OTel Collector for receiving/processing/exporting telemetry — the backend (Prometheus, Jaeger, Datadog…) can be swapped without re-instrumenting applications.

OpenTelemetry won the standards war — learning vendor-specific agents is wasted time today. Start at opentelemetry.io/docs.

# OTel Collector via Helm
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector -n observability --create-namespace \
  --set mode=deployment

GitOps and delivery (Flux, Argo)

The GitOps principle

The Git repository is the single source of truth for cluster state. A GitOps controller in the cluster continuously reconciles the actual state with the declared one — drift from a manual kubectl apply is automatically reverted. No kubectl apply from a laptop into production.

Flux

A minimalist, composable GitOps controller (CNCF graduated). I recommend understanding Flux before Argo — it teaches the pure reconciliation model. Documentation: fluxcd.io/docs.

# Bootstrap Flux against a GitLab repo
flux bootstrap gitlab --owner=<group> --repository=<repo> --branch=main --path=clusters/production

# Check reconciliation status
flux get kustomizations
flux get helmreleases

# Force reconciliation
flux reconcile kustomization <name> --with-source

# Suspend/resume (e.g. during incident)
flux suspend kustomization <name>
flux resume kustomization <name>

The Argo project

Four tools in one project (argoproj.github.io):

Argo CD — GitOps with a strong UI, Flux’s main competitor
Argo Rollouts — progressive delivery: canary, blue-green, automated metric-based rollback (Prometheus integration)
Argo Workflows — DAG-based job orchestration, widespread in ML pipelines
Argo Events — event-driven triggers

# Argo CD basics
argocd app list
argocd app sync <app-name>
argocd app diff <app-name>
argocd app rollback <app-name>

# Argo Rollouts
kubectl argo rollouts get rollout <name> --watch
kubectl argo rollouts promote <name>
kubectl argo rollouts abort <name>

Flux vs. Argo CD: most teams pick one of the two. Flux is minimalist and composable (a good fit as infrastructure underneath a platform), Argo CD has a strong UI and multi-tenancy (a good fit when developers use GitOps through an interface too). Argo Rollouts is valuable regardless of which GitOps tool you choose.

Networking

Network policies

# Get network policies
kubectl get networkpolicies
kubectl describe networkpolicy <name>

# Apply network policy
kubectl apply -f network-policy.yaml

DNS and services

# Service DNS format
<service-name>.<namespace>.svc.cluster.local

# Test DNS resolution
kubectl run test-pod --image=busybox:1.36 --rm -it -- nslookup <service-name>

Deployment management

Deployment operations

# Get deployments
kubectl get deployments
kubectl get deployment <deployment-name>
kubectl describe deployment <deployment-name>

# Scale deployment
kubectl scale deployment <name> --replicas=<count>

# Update deployment
kubectl set image deployment/<name> <container>=<new-image>
kubectl rollout status deployment/<name>

# Rollback deployment
kubectl rollout undo deployment/<name>
kubectl rollout undo deployment/<name> --to-revision=<number>

Rolling updates

# Check rollout status
kubectl rollout status deployment/<name>

# Pause/resume rollout
kubectl rollout pause deployment/<name>
kubectl rollout resume deployment/<name>

# View rollout history
kubectl rollout history deployment/<name>

Note: For canary and blue-green deployments with automated metric-based rollback, see Argo Rollouts above — a native Deployment only supports rolling updates.

Service management

Service types

# ClusterIP (default)
kubectl create service clusterip <name> --tcp=<port>:<target-port>

# NodePort
kubectl create service nodeport <name> --tcp=<port>:<target-port> --node-port=<node-port>

# LoadBalancer
kubectl create service loadbalancer <name> --tcp=<port>:<target-port>

# ExternalName
kubectl create service externalname <name> --external-name=<external-name>

Service discovery

# Get services
kubectl get services
kubectl get svc
kubectl describe service <name>

# Test service connectivity
kubectl run test-pod --image=busybox:1.36 --rm -it -- wget <service-name>:<port>

Configuration and storage

Persistent volumes

# Get PV/PVC
kubectl get pv
kubectl get pvc

# Create PVC
kubectl apply -f pvc.yaml

# Check storage classes
kubectl get storageclass

Ingress management

# Get ingresses
kubectl get ingress
kubectl describe ingress <name>

# Create ingress
kubectl create ingress <name> --rule="<host>/<path>=<service>:<port>"

Advanced operations

Namespaces

# Get namespaces
kubectl get namespaces
kubectl get ns

# Create namespace
kubectl create namespace <name>

# Switch context to namespace
kubectl config set-context --current --namespace=<name>

# Get resources in all namespaces
kubectl get pods --all-namespaces

Labels and selectors

# Label resources
kubectl label pods <pod-name> app=web
kubectl label nodes <node-name> disktype=ssd

# Select with labels
kubectl get pods -l app=web
kubectl get pods -l 'app in (web,api)'

# Remove labels
kubectl label pods <pod-name> app-

Resource quotas

# Get resource quotas
kubectl get resourcequotas
kubectl describe resourcequota <name>

# Create resource quota
kubectl create quota <name> --hard=cpu=2,memory=4Gi,pods=10

Security

RBAC (Role-Based Access Control)

# Get roles/rolebindings
kubectl get roles
kubectl get rolebindings
kubectl get clusterroles
kubectl get clusterrolebindings

# Create service account
kubectl create serviceaccount <name>

# Bind role to user/serviceaccount
kubectl create rolebinding <name> --role=<role> --user=<user>

# Verify permissions
kubectl auth can-i create deployments --as=system:serviceaccount:<ns>:<sa>

Managing Secrets

# Get secrets
kubectl get secrets
kubectl describe secret <name>

# Create TLS secret
kubectl create secret tls <name> --cert=<cert-file> --key=<key-file>

# Create docker registry secret
kubectl create secret docker-registry <name> --docker-server=<server> --docker-username=<user> --docker-password=<pass>

Security contexts and Pod Security Standards

# Container-level security context
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

# Enforce Pod Security Standards on a namespace
kubectl label namespace <name> pod-security.kubernetes.io/enforce=restricted

Trivy — pre-deployment scanning (shift-left)

One tool for CVEs in images, IaC misconfigurations (K8s manifests, Dockerfile, Terraform), secrets detection, and SBOM generation. It drops into GitLab CI trivially — there is no excuse not to use it. Documentation: trivy.dev.

# Scan container image for CVEs
trivy image <image>:<tag>

# Scan K8s manifests / Helm chart for misconfigurations
trivy config ./manifests/
trivy config ./chart/

# Scan running cluster
trivy k8s --report summary

# Fail CI on HIGH/CRITICAL
trivy image --exit-code 1 --severity HIGH,CRITICAL <image>:<tag>

Example GitLab CI job:

container_scanning:
  image: aquasec/trivy:latest
  script:
    - trivy image --exit-code 1 --severity HIGH,CRITICAL "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

Falco — runtime threat detection

The other half of the security story: Trivy prevents deploying the known-bad, Falco watches what happens after deployment. Using eBPF it monitors syscalls and alerts on suspicious behavior: a shell spawned in a container, an unexpected outbound connection, sensitive file reads, privilege escalation. The tool you wish you had before an incident, not after. Documentation: falco.org.

# Install Falco via Helm
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco -n falco --create-namespace

# Watch alerts
kubectl logs -l app.kubernetes.io/name=falco -n falco -f

Helm (package manager)

You will use it whether you like it or not — practically everything third-party (kube-prometheus-stack, ingress-nginx, cert-manager, Falco…) is distributed as a Helm chart. Documentation: helm.sh/docs.

Basic operations

# Repositories
helm repo add <name> <url>
helm repo update
helm search repo <keyword>

# Install / upgrade / rollback
helm install <release-name> <chart-name> -n <namespace> --create-namespace
helm upgrade <release-name> <chart-name>
helm upgrade --install <release-name> <chart-name>   # Idempotent (CI/CD friendly)
helm rollback <release-name> <revision>

# Inspect
helm list -A
helm status <release-name>
helm history <release-name>

# Uninstall
helm uninstall <release-name>

Values and overrides

# Show default values of a chart
helm show values <chart-name> > values.yaml

# Install with custom values
helm install <release-name> <chart-name> -f values.yaml
helm install <release-name> <chart-name> --set image.tag=1.2.3

# Show values of a deployed release
helm get values <release-name>

Template debugging

# Render templates locally WITHOUT installing (key debugging tool)
helm template <release-name> <chart-name> -f values.yaml

# Dry run against the cluster (validates against API server)
helm install <release-name> <chart-name> --dry-run --debug

# Lint chart
helm lint ./chart/

Nobody loves Go templating in YAML, but helm template + helm lint turn chart debugging into a manageable discipline.

Troubleshooting common problems

Real-world team post-mortems at k8s.af (Kubernetes Failure Stories) teach more than any course — recurring patterns: bad resource limits (CPU throttling, OOMKill), DNS (it’s always DNS), certificate expiry, and broken upgrades.

Pod problems

# Pod stuck in Pending
kubectl describe pod <pod-name>  # Check events
kubectl get nodes                # Check node capacity

# Pod CrashLoopBackOff
kubectl logs --previous <pod-name>  # Logs from the crashed run
kubectl describe pod <pod-name>     # Check exit codes (137 = OOMKill)

# Pod in ImagePullBackOff
kubectl describe pod <pod-name>  # Check image pull errors
kubectl get secrets              # Check registry credentials

Service problems

# Service not accessible
kubectl get endpoints <service-name>  # Check if pods are selected
kubectl get pods -l <selector>        # Check pod labels

# DNS resolution issues
kubectl run test-pod --image=busybox:1.36 --rm -it -- nslookup <service-name>

Node problems

# Node NotReady
kubectl describe node <node-name>     # Check conditions

# Resource pressure
kubectl top nodes                     # Check resource usage
kubectl get pods -o wide              # Check pod distribution

Best practices

Resource limits: Always set CPU/memory requests and limits — and watch for throttling in Prometheus
Health checks: Implement readiness and liveness probes
Rolling updates: For zero-downtime deployments; use Argo Rollouts for canary/blue-green
Secrets management: Never store secrets in code or in a ConfigMap
Network policies: Implement network segmentation
Observability: Prometheus + Grafana (RED/USE dashboards) + OpenTelemetry instrumentation from day one
GitOps: Git as the source of truth (Flux or Argo CD), no manual kubectl apply into production
Backups: Regularly back up etcd and persistent data
Security: RBAC, security contexts, Pod Security Standards; Trivy in CI (shift-left) + Falco at runtime
Pinned image tags: Never latest — always a specific version or digest
Updates: Keep the cluster and applications up to date
Documentation: Document cluster configuration and processes

Example YAML manifests

Pod manifest

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    app: my-app
spec:
  containers:
  - name: my-container
    image: nginx:1.27-alpine   # Pinned tag, never :latest
    ports:
    - containerPort: 80
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 64Mi
    securityContext:
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]

Deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: nginx:1.27-alpine
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi

Service manifest

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP

Learning path and certifications

Recommended order:

Kubernetes the Hard Way — once, by hand, for a mental model of the control plane
roadmap.sh/kubernetes — the skill map; it exposes unknown unknowns and helps sequence your learning
Core kubectl practice — this handbook
Helm — packaging; you’ll need it for everything else
Prometheus + Grafana — observability from the start, not after the first incident
GitOps — Flux for the pure reconciliation model, then Argo (CD, Rollouts, Workflows, Events)
Trivy + Falco — build-time and runtime security
OpenTelemetry — application instrumentation with the vendor-neutral standard
killer.sh — the CKA/CKAD/CKS exam simulator; deliberately harder than the real exam (broken clusters, RBAC puzzles, etcd backup/restore); two sessions are free with a Linux Foundation exam registration

Ongoing: k8s.af (failure stories from real teams) and cncf.io/projects (a map of the ecosystem by maturity).

← Back to Homepage