Introduction

Kubernetes is a popular platform for deploying and managing containerized applications across a cluster of nodes. However, containers are ephemeral and stateless by nature, which means they do not persist any data after they are terminated. This poses a problem for applications that need to store and access data, such as databases, analytics, or web services. How can we ensure that the data is available and consistent, even when the containers are moved, scaled, or updated?

This is where Kubernetes storage comes in. Kubernetes storage is a set of features and resources that enable persistent and reliable data management for containerized applications. It allows us to attach external storage volumes to containers, provision and manage storage resources dynamically, and backup and restore data using snapshots. Kubernetes storage also supports different types of storage backends, such as local disks, cloud storage, or network-attached storage, to suit different use cases and performance requirements.

In this article, we will explore the key concepts of Kubernetes storage.

Kubernetes Storage Concepts

Kubernetes storage is based on a few key concepts that define how storage resources are created, allocated, and consumed by containers. These concepts are:

Volumes

A volume is a logical unit of storage that can be attached to a container and mounted as a directory. A volume can have different types, such as emptyDir, hostPath, nfs, gcePersistentDisk, awsElasticBlockStore, etc., depending on the underlying storage provider. A volume can be defined as part of a pod specification, and it is scoped to the pod’s lifetime. For example, the following pod specification defines a volume of type emptyDir and mounts it to the /data directory of the container:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
    volumeMounts:
    - name: my-volume
      mountPath: /data
  volumes:
  - name: my-volume
    emptyDir: {}

The volume will be accessible to the Pod, and any data written to /data by the container will be stored in the volume. However, it's important to note that the data stored in an emptyDir volume is not preserved when the pod is restarted or rescheduled. The data is lost when the pod is removed from a node.

Persistent Volumes (PV)

A persistent volume is a cluster-wide unit of storage that can outlive the pod that uses it. A persistent volume can have different types, such as csi, nfs, iscsi, local, etc., depending on the underlying storage provider. A persistent volume can be created and managed by an administrator or dynamically provisioned by Kubernetes. For example, the following persistent volume specification defines a volume of type gcePersistentDisk and size 10 GB:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  gcePersistentDisk:
    pdName: my-disk
    fsType: ext4

This creates a Persistent Volume with a storage capacity of 10 Gibibytes. The PersistentVolume is backed by a GCE persistent disk with an ext4 file system. The PersistentVolume can be mounted as read-write by a single Node.

Persistent Volume Claims (PVC)

A persistent volume claim is a request for a persistent volume by a user or an application. A persistent volume claim specifies the desired size, access mode, and storage class of the volume. Kubernetes matches the claim with an available persistent volume or dynamically provisions a new one if possible. A persistent volume claim can be defined as part of a pod specification, and it is scoped to the pod’s namespace. For example, the following persistent volume claim specification requests a volume of size 8 GB and access mode ReadWriteOnce:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi

This configuration creates a Persistent Volume Claim named my-pvc that requests a Persistent Volume with a storage capacity of 8 Gibibytes. The Persistent Volume that satisfies this claim will be mounted as read-write by a single Node.

Storage Classes

A storage class is a way to categorize and configure different types of storage resources in a cluster. A storage class defines the provisioner, parameters, and reclaim policy of the storage resources. A storage class can be used by a user or an application to request a specific type of storage when creating a persistent volume claim. For example, the following storage class specification defines a storage class named standard that uses the kubernetes.io/gce-pd provisioner and creates pd-standard disks:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
reclaimPolicy: Delete

This creates a Storage that dynamically provisions PersistentVolumes backed by GCE Persistent Disks of the pd-standard type. When a PersistentVolume is deleted, the underlying GCE Persistent Disk is also deleted.

Dynamic Provisioning

Dynamic provisioning is a feature that allows Kubernetes to automatically create and attach persistent volumes to pods based on the persistent volume claims and storage classes. Dynamic provisioning eliminates the need for manual creation and management of persistent volumes by administrators. Most cloud providers, including GCE, AWS, Azure, and others, support the feature of dynamic provisioning. For example, the following pod specification uses a persistent volume claim that references the standard storage class, which triggers the dynamic provisioning of a gcePersistentDisk volume:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
    volumeMounts:
    - name: my-volume
      mountPath: /data
  volumes:
  - name: my-volume
    persistentVolumeClaim:
      claimName: my-pvc

It's important to note that the gcePersistentDisk volume is dynamically provisioned only if no existing PersistentVolume satisfies the claim. If an existing PersistentVolume satisfies the claim, Kubernetes will use that volume instead of dynamically provisioning a new one.

Volume Snapshots

A volume snapshot is a point-in-time copy of a persistent volume. A volume snapshot can be used to backup and restore data, or to clone a volume for another application. A volume snapshot can be created and managed by a user or an application using the VolumeSnapshot and VolumeSnapshotContent resources. For example, the following volume snapshot specification creates a snapshot of the persistent volume claim my-pvc

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-snapshot
spec:
  source:
    persistentVolumeClaimName: my-pvc

The Volume Snapshot feature in Kubernetes is not activated by default in all Kubernetes deployments. It must be activated by the system administrator.

Container Storage Interface (CSI)

Container Storage Interface is a standard that defines an interface for container orchestrators, such as Kubernetes, to communicate with storage providers, such as GCE, AWS, Azure, etc. CSI enables the development and deployment of vendor-neutral storage plugins that can work with any container orchestrator. CSI also supports advanced features, such as volume resizing, volume snapshots, and volume cloning. For example, the following pod specification uses a CSI driver to mount a gcePersistentDisk volume:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
    volumeMounts:
    - name: my-volume
      mountPath: /data
  volumes:
  - name: my-volume
    csi:
      driver: pd.csi.storage.gke.io
      volumeHandle: my-disk

The CSI feature is also not enabled by default on all Kubernetes deployments. It must be enabled by the administrator.

Tips on How to Use Kubernetes Storage Effectively

• Choosing the right storage class: Depending on the use case and performance requirements of the application, it is important to choose the appropriate storage class for the persistent volume claims. Different storage classes may have different characteristics, such as availability, durability, throughput, latency, cost, etc. For example, for applications that need high performance and low latency, it may be better to use a storage class that uses SSD disks, such as pd-ssd for GCE or gp2 for AWS.

  For applications that need high availability and durability, it may be better to use a storage class that uses replicated or distributed storage, such as pd-standard for GCE or ebs for AWS. For applications that need cost-effective and scalable storage, it may be better to use a storage class that uses object storage, such as gcs for GCE or s3 for AWS.

• Optimizing performance and availability: To improve the performance and availability of the storage resources, some factors need to be considered, such as the access mode, the mount options, the topology, and the resource limits. For example, for applications that need concurrent read and write access to the same volume, it may be better to use a storage class that supports ReadWriteMany access mode, such as nfs or cephfs.

  For applications that need to optimize the file system performance, it may be better to use the appropriate mount options, such as noatime, nodiratime, or discard. For applications that need to ensure the affinity and anti-affinity of the pods and the volumes, it may be better to use topology-aware scheduling and provisioning features, such as volumeBindingMode, allowedTopologies, or topologySpreadConstraints. For applications that need to prevent the over-commitment and oversubscription of storage resources, it may be better to use the resource limits and requests, such as storage, iops, or bandwidth.

• Securing data: To protect the data from unauthorized access, modification, or deletion, some security measures need to be applied, such as encryption, authentication, and authorization. For example, for applications that need to encrypt the data at rest and in transit, it may be better to use a storage class that supports encryption, such as pd-encrypted for GCE or ebs-encrypted for AWS, or use a CSI driver that supports encryption, such as secrets-store.csi.k8s.io.

  For applications that need to authenticate access to the volumes, it may be better to use a storage class that supports authentication, such as iscsi or rbd, or use a CSI driver that supports authentication, such as vault.csi.k8s.io. For applications that need to authorize access to the volumes, it may be better to use the role-based access control (RBAC) and the pod security policy (PSP) features, such as serviceAccountName, volumeClaimTemplates, or allowedCSIDrivers.

• Backing up and restoring volumes: To ensure data recovery and continuity in case of disasters, failures, or errors, there are some backup and restore strategies that need to be implemented, such as volume snapshots, volume cloning, and application-level backup. For example, for applications that need to backup and restore the volumes at the block level, it may be better to use the volume snapshot and volume clone features, such as VolumeSnapshot, VolumeSnapshotContent, or dataSource.

  For applications that need to backup and restore the volumes at the file level, it may be better to use the application-level backup tools, such as velero, restic, or ark.

Conclusion

In this article, we have learned about the basics of Kubernetes storage, including the concepts of persistent volumes, persistent volume claims, storage classes, and dynamic provisioning. We have also seen how to use these resources to create and manage storage for stateful applications on Kubernetes.