Kubernetes Persistent Volumes, PVCs and StatefulSets

Keep data when pods restart: PersistentVolumeClaims, StorageClasses, access modes, StatefulSets with volumeClaimTemplates, volume expansion and backups.

TLDR: Files written inside a container disappear with the pod. To keep data, a pod mounts a PersistentVolumeClaim (PVC), a request for storage of a certain size and access mode. A StorageClass creates a matching PersistentVolume (PV) for it on demand, usually through a CSI driver. A StatefulSet gives each of its pods a stable name and its own PVC, which is what databases and caches need.

In this part you watch the cache lose its data when its pod restarts, create a PVC and see when and where its volume appears, move the cache to a StatefulSet that keeps its data, and learn the limits of node-local storage that will matter again in Part 8.

Prerequisites

The cache and the configured podinfo from Part 6. On a new cluster, apply 04-deployments/deployment.yaml and 05-networking/service.yaml, create the podinfo-tls Secret as Part 6 shows, then apply the files in 06-config/ (cache.yaml, configmap.yaml, deployment.yaml).

Data Without a Volume Is Lost

The cache from Part 6 is a Deployment without a volume, so Valkey keeps its data in the container's own filesystem. Write a key through podinfo, then replace the cache pod:

podinfo answers 404 for a key that does not exist, and also when it cannot reach the cache at all. The round trip with the check key proves that podinfo is connected to the new cache pod, so the 404 for greeting means the data is gone: the new pod started empty. If you get a 404 for check too, podinfo has not reconnected yet; wait a few seconds and try again. That is fine for a cache you can refill, and a disaster for a database.

Volumes, PersistentVolumes and Claims

Kubernetes separates what a pod needs from how storage is provided:

A volume in a pod spec is anything mounted into containers. Some live as long as the pod (emptyDir, configMap, secret, projected), others point at storage that outlives it (persistentVolumeClaim).
A PersistentVolumeClaim is a namespaced request: "1 GiB that one node can mount read-write". Apps ask for storage with claims and do not need to know the storage system.
A PersistentVolume is the cluster-wide object for a real piece of storage: a cloud disk, an NFS export, a directory on a node.
A StorageClass tells Kubernetes how to create PVs on demand (dynamic provisioning) through a provisioner. On cloud clusters that is a CSI driver, the plugin that talks to the storage system; kind uses a simpler provisioner. Most clusters have a default StorageClass, used by claims that name none.

hostPath volumes, which mount a directory of the node directly, appear in many old examples. Avoid them in apps: a pod can read or overwrite node files through them, the data stays on one node, and the restricted Pod Security level forbids them (Part 10).

Dynamic Provisioning on kind

kind ships Rancher's local-path provisioner (not a CSI driver) as its default StorageClass. It creates each volume as a hostPath directory on the node where the pod runs:

kubectl get storageclass

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  77s

WaitForFirstConsumer means the volume is created only when a pod that uses the claim is scheduled, on that pod's node. Immediate would create it right away, possibly in a zone or on a node where the pod cannot run. ALLOWVOLUMEEXPANSION false comes back below.

Create a claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scratch
spec:
  accessModes:
    - ReadWriteOnce # one node can mount it read-write
  resources:
    requests:
      storage: 1Gi
  # no storageClassName: the cluster's default StorageClass is used

Pending is expected here. Now start a pod that mounts the claim and writes a file:

apiVersion: v1
kind: Pod
metadata:
  name: writer
spec:
  containers:
    - name: writer
      image: busybox:1.38.0
      command: ['sh', '-c', 'date -u > /data/created-at.txt && sleep 3600']
      volumeMounts:
        - name: scratch
          mountPath: /data
  volumes:
    - name: scratch
      persistentVolumeClaim:
        claimName: scratch

The provisioner created a PV, named after the claim's UID, and bound the two. Node Affinity pins this PV to the node the pod landed on: a local volume exists on one node only, so any pod that uses it must run there. Cloud disks are pinned the same way to a zone.

When you delete the claim, the reclaim policy decides what happens to the volume. Delete, the default for dynamically provisioned volumes, deletes the PV and the data:

For data you care about, use a StorageClass with reclaimPolicy: Retain. The PV then stays after the claim is deleted, and an administrator decides whether to reuse or delete it.

Access Modes

Mode	Short	Meaning
`ReadWriteOnce`	RWO	Mounted read-write by pods on one node at a time
`ReadWriteOncePod`	RWOP	Mounted read-write by exactly one pod (stable since 1.29)
`ReadOnlyMany`	ROX	Mounted read-only on many nodes
`ReadWriteMany`	RWX	Mounted read-write on many nodes, for example NFS or a cloud file service

Block storage such as cloud disks supports RWO and RWOP; RWX needs a file system that many nodes can share, such as NFS, CephFS, Amazon EFS, Azure Files or Google Filestore. Use RWOP for databases, so that a second pod can never mount the same disk by mistake.

A StatefulSet for the Cache

A Deployment's pods are interchangeable, and they would all share one claim. A StatefulSet gives each pod:

a stable name, cache-0, cache-1 and so on, kept when the pod is replaced;
a stable DNS name through a headless Service, cache-0.cache-headless.demo.svc.cluster.local;
its own PVC from volumeClaimTemplates, data-cache-0 for cache-0, which the replacement pod mounts again;
ordered startup, scaling and updates, one pod at a time.

# A headless Service gives each StatefulSet pod a stable DNS name: cache-0.cache-headless
apiVersion: v1
kind: Service
metadata:
  name: cache-headless
  labels:
    app: cache
spec:
  clusterIP: None
  selector:
    app: cache
  ports:
    - name: valkey
      port: 6379
      targetPort: valkey
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cache
  labels:
    app: cache
spec:
  serviceName: cache-headless
  replicas: 1
  selector:
    matchLabels:
      app: cache
  # Keep the volumes when the StatefulSet is deleted or scaled down (the default)
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  template:
    metadata:
      labels:
        app: cache
    spec:
      containers:
        - name: valkey
          image: valkey/valkey:9.1.2-alpine
          # --appendonly yes writes every change to a log file in /data
          args: ['/etc/valkey/valkey.conf', '--appendonly', 'yes']
          ports:
            - name: valkey
              containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
            - name: config
              mountPath: /etc/valkey
              readOnly: true
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
      volumes:
        - name: config
          configMap:
            name: cache-config
  # One PersistentVolumeClaim per pod: data-cache-0, data-cache-1, ...
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ['ReadWriteOnce']
        resources:
          requests:
            storage: 1Gi

--appendonly yes makes Valkey log every write to a file under /data, which is now the volume. The existing cache Service from Part 6 selects app: cache, so podinfo reaches the StatefulSet's pod without any change. Replace the Deployment:

Check that podinfo reaches the new cache pod, write a key and read it back, delete the pod, and read the key again:

The StatefulSet recreated cache-0 with the same name, it mounted data-cache-0 again, and Valkey replayed its append-only file. Each pod also has a DNS name of its own, which clustered databases use to find each other:

kubectl exec deploy/podinfo -- nslookup cache-0.cache-headless.demo.svc.cluster.local

Server:		10.96.0.10
Address:	10.96.0.10:53

Name:	cache-0.cache-headless.demo.svc.cluster.local
Address: 10.244.2.13

Scaling a StatefulSet down does not delete its claims (persistentVolumeClaimRetentionPolicy controls that, and Retain is the default), so scaling back up brings the data back:

Note where cache-0 runs. Its volume is a directory on that node, so cache-0 can only ever run there; Part 8 drains that node and shows what happens:

Expanding a Volume

To grow a claim, raise resources.requests.storage. That only works when the StorageClass has allowVolumeExpansion: true and its CSI driver supports resizing. kind's local-path class does not:

kubectl patch pvc data-cache-0 -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'

Error from server (Forbidden): persistentvolumeclaims "data-cache-0" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

Cloud StorageClasses usually allow it, and the disk grows while the pod runs. Claims can only grow, never shrink. In a StatefulSet, volumeClaimTemplates cannot be changed in place, so you resize each existing PVC yourself and update the template for future pods by recreating the StatefulSet with kubectl delete statefulset cache --cascade=orphan, which keeps the pods and claims. To change a volume's performance, such as IOPS, without a new disk, use a VolumeAttributesClass (stable since 1.34) if your CSI driver supports it.

Storage on Managed Clusters

Managed services install their CSI driver, or offer it as an add-on (the EBS CSI driver on EKS clusters without Auto Mode), and usually a default StorageClass. On DigitalOcean Kubernetes, the default do-block-storage class creates a DigitalOcean Volume for each claim, billed while it exists. Here are StorageClasses for the other big providers (examples; each needs that provider's cluster):

# StorageClasses for managed clusters. Each needs that provider's CSI driver,
# which the managed service installs or offers as an add-on.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3 # Amazon EKS Auto Mode, as created in Part 2
provisioner: ebs.csi.eks.amazonaws.com # with the EBS CSI driver add-on instead: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: 'true'
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      - key: eks.amazonaws.com/compute-type
        values: ['auto']
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pd-balanced # Google Kubernetes Engine
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-balanced
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-premium # Azure Kubernetes Service
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_LRS
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Old tutorials use in-tree provisioners such as kubernetes.io/aws-ebs or kubernetes.io/gce-pd. Kubernetes migrated them to CSI drivers and has removed the in-tree plugins for the major clouds, so use the CSI provisioner names above.

Snapshots and Backups

A VolumeSnapshot asks the CSI driver for a point-in-time copy of a claim, and a new claim can start from it. It needs the snapshot CRDs, the snapshot controller and a CSI driver that supports snapshots, which most managed services provide (kind's local-path driver does not):

# Needs the snapshot CRDs and controller, and a CSI driver with snapshot support
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: cache-data-snapshot
spec:
  volumeSnapshotClassName: csi-snapclass # provided by your CSI driver
  source:
    persistentVolumeClaimName: data-cache-0
---
# A new claim that starts with the snapshot's data
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-cache-restore
spec:
  dataSource:
    name: cache-data-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes: ['ReadWriteOnce']
  resources:
    requests:
      storage: 1Gi

Snapshots are not a full backup plan. For databases, also take application-level backups (pg_dump, Valkey's RDB files) and store them outside the cluster. Velero backs up Kubernetes objects and volumes together and restores them into another cluster. For a production database on Kubernetes, use an operator such as CloudNativePG, which handles replicas, failover and backups, or a managed database outside the cluster.

Common Mistakes

No volume for data you need. Anything written to the container's filesystem or an emptyDir is gone when the pod is replaced.
Immediate binding in multi-zone clusters. The disk can end up in a zone where the pod cannot run. Use WaitForFirstConsumer.
reclaimPolicy: Delete for important data. Deleting the claim deletes the disk. Use Retain, snapshots and backups.
Sharing one RWO claim between replicas. A Deployment with several replicas and one RWO claim only works while all pods land on one node. Use a StatefulSet with one claim per pod.
Forgetting that local volumes pin pods to nodes. A pod with a node-local volume cannot move when its node is drained or dies.

Summary

Pods ask for storage with PersistentVolumeClaims; StorageClasses and their provisioners (usually CSI drivers) create PersistentVolumes on demand.
WaitForFirstConsumer creates the volume where the pod runs; the reclaim policy decides what happens when the claim goes.
A StatefulSet gives each pod a stable name, DNS name and its own claim, so its data survives restarts and rescheduling.
Expansion, snapshots and performance changes depend on the CSI driver; backups belong outside the cluster.

Keep Learning

Next, in Part 8, you give the pods CPU and memory budgets, scale them with load and drain a node, including the one this volume lives on.

Part of: Introduction to Kubernetes

Updated: 10/24/2026