Kubernetes Monitoring, Logging and Troubleshooting

TLDR: When a pod misbehaves, run kubectl get pods, then kubectl describe pod (its events), then kubectl logs (add --previous after a crash). The pod's status tells you which of a handful of problems you have: ImagePullBackOff, CrashLoopBackOff, Pending, or Running but not ready. kubectl debug adds a temporary container with tools to a running pod. For history and alerts, install Prometheus with the kube-prometheus-stack Helm chart and describe what to scrape with a ServiceMonitor.

In this part you crash podinfo on purpose and read the logs of the dead container, diagnose three broken Deployments, take a pod out of rotation through its readiness probe, debug a pod and a node with kubectl debug, then install Prometheus with Helm 4, scrape podinfo, and watch an alert fire.

Prerequisites

• The app after Part 8, with metrics-server installed.
• Helm 4 (installed in Part 2) and jq.

Four Kinds of Signals

Signal	Answers	Where it comes from
Events	What did Kubernetes do, and what failed?	Controllers, the scheduler and kubelets; kept about an hour
Logs	What did the app say?	Each container's standard output and error
Metrics	How much, how fast, how often, over time?	metrics-server for now, Prometheus for history
Traces	Where did one request spend its time?	The app, instrumented with OpenTelemetry

Events and logs are enough to fix most problems in front of you. Metrics and alerts tell you about the problems you are not looking at yet.

Logs

The kubelet captures what each container writes to standard output and standard error and stores it on the node. kubectl logs reads it through the API server:

kubectl logs deployment/podinfo picks one pod of the Deployment. To read all of them, select by label and add --prefix to see which line came from which pod. Other useful flags: -f to follow, --since=10m, --timestamps, -c <container> for multi-container pods, and --all-containers.

Logs of a Crashed Container

podinfo's /panic endpoint makes the process exit with code 255. Call it inside one pod, and the container dies, taking the curl that called it along:

The kubelet restarted the container, so plain kubectl logs shows the new one. --previous shows the logs of the container that died, and its last line says why. The kubelet keeps only the current and the previous container's logs, and only while the pod exists.

Where Logs Live, and Shipping Them

On the node, container logs are files under /var/log/pods/<namespace>_<pod>_<uid>/. The kubelet rotates them (10 MiB and 5 files per container by default), and they disappear with the pod. For history and search across pods, run a log agent as a DaemonSet that reads those files and ships them to a store:

• Agents: Fluent Bit, the OpenTelemetry Collector, Grafana Alloy or Vector. Promtail, which many Loki tutorials still use, reached its end of life on 2 March 2026; Grafana replaced it with Alloy.
• Stores: Grafana Loki, Elasticsearch or OpenSearch, or your cloud provider's logging service.

Log in JSON, as podinfo does, so the agent can parse fields such as level and request ID without regular expressions.

Events

Events are the cluster's own log. They expire after about an hour, so look at them while the problem is fresh:

kubectl events --for pod/POD_NAME  # one object, oldest first
kubectl events --types=Warning    # only problems, in the current namespace
kubectl events -A --watch         # everything, as it happens

A Troubleshooting Playbook

Most pod problems fall into one of four shapes. The STATUS column of kubectl get pods tells you which:

These three Deployments are each broken in one of the first three ways:

# Three Deployments that fail in three common ways
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-image
  labels:
    lab: broken
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bad-image
  template:
    metadata:
      labels:
        app: bad-image
        lab: broken
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.150.0 # a typo in the tag
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-config
  labels:
    lab: broken
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bad-config
  template:
    metadata:
      labels:
        app: bad-config
        lab: broken
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.15.0
          command: ['./podinfo']
          args: ['--random-delay-min=5', '--random-delay-max=1'] # an invalid combination
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: no-node
  labels:
    lab: broken
spec:
  replicas: 1
  selector:
    matchLabels:
      app: no-node
  template:
    metadata:
      labels:
        app: no-node
        lab: broken
    spec:
      nodeSelector:
        disktype: ssd # no node has this label
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.15.0

ImagePullBackOff. The events of the pod name the image and the registry's answer, not found:

kubectl describe pod -l app=bad-image | sed -n '/^Events:/,$p'

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  60s                default-scheduler  Successfully assigned demo/bad-image-76c5699b4d-vvz77 to k8s-guide-worker
  Normal   Pulling    22s (x3 over 60s)  kubelet            spec.containers{podinfo}: Pulling image "ghcr.io/stefanprodan/podinfo:6.150.0"
  Warning  Failed     22s (x3 over 59s)  kubelet            spec.containers{podinfo}: Failed to pull image "ghcr.io/stefanprodan/podinfo:6.150.0": rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/stefanprodan/podinfo:6.150.0": failed to resolve reference "ghcr.io/stefanprodan/podinfo:6.150.0": ghcr.io/stefanprodan/podinfo:6.150.0: not found
  Warning  Failed     22s (x3 over 59s)  kubelet            spec.containers{podinfo}: Error: ErrImagePull
  Normal   BackOff    8s (x3 over 59s)   kubelet            spec.containers{podinfo}: Back-off pulling image "ghcr.io/stefanprodan/podinfo:6.150.0"
  Warning  Failed     8s (x3 over 59s)   kubelet            spec.containers{podinfo}: Error: ImagePullBackOff

Check the tag for typos, check that the registry is reachable from the nodes, and for private registries add imagePullSecrets to the pod or its ServiceAccount. Kubernetes retries with growing delays, which is the "back-off".

CrashLoopBackOff. The container starts and exits; here STATUS shows Error between restarts. The logs say why, and the exit code (2 for a Go panic) says how:

The first two lines of the log are enough: two options contradict each other. If a crashing container prints nothing at all, check its last state: Reason: OOMKilled means the kernel's out-of-memory killer ended it, usually at its memory limit, sometimes because the node ran out (Part 8); exit code 137 without that reason means something else sent SIGKILL. Exit codes 127 and 126 mean the command in the pod spec does not exist or is not executable. CrashLoopBackOff with no logs covers more cases.

Pending. No node fits. The scheduler's event lists, per node, why not:

kubectl describe pod -l app=no-node | sed -n '/^Events:/,$p'

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  60s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

One node is the tainted control plane, and the two workers lack the disktype=ssd label. Other common reasons are Insufficient cpu or Insufficient memory (requests larger than any node's free capacity), a PVC that cannot bind, and a quota (in which case the pod is never created, and the event is on the ReplicaSet). Clean up:

kubectl delete -f 09-observability/broken.yaml

deployment.apps "bad-image" deleted from demo namespace
deployment.apps "bad-config" deleted from demo namespace
deployment.apps "no-node" deleted from demo namespace

Running but not ready. podinfo can switch its own readiness off. Do that in one pod and watch it leave the Service:

The pod keeps running (readiness failures never restart a container; liveness failures do), but its endpoint is marked ready=false, so the Service sends it no new connections. That is how a pod that is overloaded or warming up steps aside without being killed.

kubectl debug

Many production images contain only the app: no shell, no ps, no network tools. kubectl debug adds an ephemeral container with an image of your choice to a running pod. With --target, it shares the process namespace of the app's container, so you can see its processes:

The debug container sees podinfo as process 1, running as user 100 (busybox has no name for that ID), listening on ports 9898 and 8443. Add -it instead of a command to get an interactive shell. Ephemeral containers cannot be removed; they stay in the pod's spec until the pod is replaced. Two other modes are useful: kubectl debug $POD -it --copy-to=debug-copy --container=podinfo -- sh starts a copy of the pod with a different command, and kubectl debug node/<name> starts a pod on a node with the node's filesystem mounted at /host:

There are the log directories from above, one per pod on that node. --profile sets the debug container's privileges: general for most work, netadmin for network tools, sysadmin for full access.

Metrics with metrics-server

metrics-server keeps only the latest values, which is enough for kubectl top and the HPA:

kubectl top pods --containers

POD                        NAME      CPU(cores)   MEMORY(bytes)
cache-0                    valkey    1m           8Mi
podinfo-75bb4bb8cc-6h7vc   podinfo   1m           18Mi
podinfo-75bb4bb8cc-cjtwp   podinfo   1m           16Mi
podinfo-75bb4bb8cc-zjlzn   podinfo   1m           18Mi

For history, rates, percentiles and alerts, you need Prometheus.

Prometheus with the kube-prometheus-stack Chart

The kube-prometheus-stack Helm chart installs the Prometheus Operator, Prometheus, Alertmanager, Grafana, kube-state-metrics (metrics about Kubernetes objects) and node-exporter (metrics about nodes), with dashboards and alert rules for the cluster. Helm installs charts straight from OCI registries such as GitHub's. On kind, a few control plane components only listen on 127.0.0.1, so this values file turns off scraping them:

# Values for the kube-prometheus-stack chart on a kind cluster.
# kind binds these control plane components to 127.0.0.1, so Prometheus cannot scrape them.
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
kubeProxy:
  enabled: false

(The chart's notes go on with Grafana instructions, cut here.) Pin the chart version, as here, so an upgrade is a decision rather than an accident.

Scrape podinfo with a ServiceMonitor

The Prometheus Operator adds CRDs: a ServiceMonitor tells Prometheus to scrape the endpoints of matching Services, and a PrometheusRule holds alert and recording rules. This chart's Prometheus picks up objects labeled release: monitoring:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: podinfo
  labels:
    release: monitoring # the chart's Prometheus selects ServiceMonitors with this label
spec:
  selector:
    matchLabels:
      app: podinfo # scrape the endpoints of Services with this label
  endpoints:
    - port: http # the named Service port
      path: /metrics
      interval: 15s

The alert fires when more than 5 percent of podinfo's requests fail for a minute. podinfo counts its requests in http_requests_total with a status label:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: podinfo
  labels:
    release: monitoring
spec:
  groups:
    - name: podinfo
      rules:
        - alert: PodinfoHighErrorRate
          expr: |
            sum(rate(http_requests_total{job="podinfo", status=~"5.."}[2m]))
              / sum(rate(http_requests_total{job="podinfo"}[2m])) > 0.05
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: More than 5 percent of podinfo requests fail
            description: '{{ $value | humanizePercentage }} of podinfo requests returned a 5xx status over the last 2 minutes.'

Prometheus needs a minute or so to discover the new targets. Then all three pods are scraped (up=1). You can also open http://localhost:9090 in a browser and use the Prometheus UI.

Make an Alert Fire

podinfo's /status/500 endpoint answers with a 500 error. A small pod sends one failing and one good request every half second:

kubectl run errors --image=curlimages/curl:8.23.0 --restart=Never -- sh -c 'for i in $(seq 1 400); do curl -s -o /dev/null http://podinfo/status/500; curl -s -o /dev/null http://podinfo/; sleep 0.5; done'

pod/errors created

A few minutes later, the request rate by status and the alert:

About two requests per second fail and three succeed. The 3 per second of status 200 include the kubelets' liveness and readiness probes, which call podinfo too; the 503 and 202 series are left over from the readiness switch earlier. The alert moved from pending to firing after its for: 1m, and Prometheus sent it to Alertmanager, which routes alerts to Slack, email, PagerDuty and others once you configure a receiver. Clean up the error generator and the port-forward:

What to Watch

• For each app, the RED signals: request rate, errors and duration (latency percentiles from a histogram such as podinfo's http_request_duration_seconds).
• From kube-state-metrics: kube_pod_container_status_restarts_total (crash loops), kube_deployment_status_replicas_available against the desired count, and pods stuck in Pending.
• From the nodes: CPU, memory, disk space and the kubelet's pressure conditions.
• For the cluster: API server latency and errors, etcd health (on self-managed clusters), and certificate expiry.

Grafana comes with the chart and has dashboards for most of these. Its admin password is in the monitoring-grafana Secret; port-forward service/monitoring-grafana to port 3000 to open it. Alert on symptoms users feel (errors, latency, availability), not on every high CPU reading.

Traces with OpenTelemetry

Metrics tell you that requests are slow; a trace tells you where one request spent its time across services. Instrument apps with the OpenTelemetry SDK for their language and send traces with the OTLP protocol to an OpenTelemetry Collector, which forwards them to Jaeger, Grafana Tempo or a commercial backend. podinfo has OpenTelemetry built in: setting the PODINFO_OTEL_SERVICE_NAME and the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variables turns it on. The Collector can also receive logs and metrics, so many teams run one Collector DaemonSet for all three signals.

Common Mistakes

• Reading only kubectl logs after a crash. You see the new container. Use --previous.
• Not looking at events. Image pulls, scheduling, probes and quotas all report there, not in the app's logs.
• Logs only on the nodes. They vanish with the pod and rotate quickly; ship them somewhere.
• No resource requests, then wondering why the HPA or the dashboards show nothing useful. Percentages are relative to requests.
• Alerting on causes instead of symptoms. Page on errors and latency users see; keep CPU and memory alerts for tickets.
• Unpinned charts. helm install without --version installs whatever is newest today.

Summary

• get, describe and logs --previous find most problems; the pod's status names the kind of problem.
• kubectl debug adds a tool container to a running pod or a node without changing the image.
• Logs belong in a central store, shipped by a DaemonSet agent such as Fluent Bit, the OpenTelemetry Collector or Alloy.
• kube-prometheus-stack gives you Prometheus, Alertmanager and Grafana; ServiceMonitors and PrometheusRules are how apps plug in.

Keep Learning

• How to follow logs continuously
• Fix pods stuck in Terminating
• Prometheus and Grafana monitoring exercise
• Prometheus vs Datadog and the monitoring and observability quiz

Next, in Part 10, you lock the app and the cluster down.