Introduction
Kubernetes is a popular platform for deploying and managing containerized applications across a cluster of nodes. However, containers are ephemeral and stateless by nature, which means they do not persist any data after they are terminated. This poses a problem for applications that need to store and access data, such as databases, analytics, or web services. How can we ensure that the data is available and consistent, even when the containers are moved, scaled, or updated?
This is where Kubernetes storage comes in. Kubernetes storage is a set of features and resources that enable persistent and reliable data management for containerized applications. It allows us to attach external storage volumes to containers, provision and manage storage resources dynamically, and backup and restore data using snapshots. Kubernetes storage also supports different types of storage backends, such as local disks, cloud storage, or network-attached storage, to suit different use cases and performance requirements.
In this article, we will explore the key concepts of Kubernetes storage.
Kubernetes Storage Concepts
Kubernetes storage is based on a few key concepts that define how storage resources are created, allocated, and consumed by containers. These concepts are:
Volumes
A volume is a logical unit of storage that can be attached to a container and mounted as a directory. A volume can have different types, such as emptyDir
, hostPath
, nfs
, gcePersistentDisk
, awsElasticBlockStore
, etc., depending on the underlying storage provider. A volume can be defined as part of a pod specification, and it is scoped to the pod’s lifetime. For example, the following pod specification defines a volume of type emptyDir
and mounts it to the /data
directory of the container:
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: my-container
image: my-image
volumeMounts:
- name: my-volume
mountPath: /data
volumes:
- name: my-volume
emptyDir: {}
The volume will be accessible to the Pod, and any data written to /data
by the container will be stored in the volume. However, it's important to note that the data stored in an emptyDir
volume is not preserved when the pod is restarted or rescheduled. The data is lost when the pod is removed from a node.
Persistent Volumes (PV)
A persistent volume is a cluster-wide unit of storage that can outlive the pod that uses it. A persistent volume can have different types, such as csi
, nfs
, iscsi
, local
, etc., depending on the underlying storage provider. A persistent volume can be created and managed by an administrator or dynamically provisioned by Kubernetes. For example, the following persistent volume specification defines a volume of type gcePersistentDisk
and size 10 GB:
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: my-disk
fsType: ext4
This creates a Persistent Volume with a storage capacity of 10 Gibibytes. The PersistentVolume is backed by a GCE persistent disk with an ext4 file system. The PersistentVolume can be mounted as read-write by a single Node.
Persistent Volume Claims (PVC)
A persistent volume claim is a request for a persistent volume by a user or an application. A persistent volume claim specifies the desired size, access mode, and storage class of the volume. Kubernetes matches the claim with an available persistent volume or dynamically provisions a new one if possible. A persistent volume claim can be defined as part of a pod specification, and it is scoped to the pod’s namespace. For example, the following persistent volume claim specification requests a volume of size 8 GB and access mode ReadWriteOnce
:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
This configuration creates a Persistent Volume Claim named my-pvc
that requests a Persistent Volume with a storage capacity of 8 Gibibytes. The Persistent Volume that satisfies this claim will be mounted as read-write by a single Node.
Storage Classes
A storage class is a way to categorize and configure different types of storage resources in a cluster. A storage class defines the provisioner, parameters, and reclaim policy of the storage resources. A storage class can be used by a user or an application to request a specific type of storage when creating a persistent volume claim. For example, the following storage class specification defines a storage class named standard
that uses the kubernetes.io/gce-pd
provisioner and creates pd-standard
disks:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
reclaimPolicy: Delete
This creates a Storage that dynamically provisions PersistentVolumes backed by GCE Persistent Disks of the pd-standard
type. When a PersistentVolume is deleted, the underlying GCE Persistent Disk is also deleted.
Dynamic Provisioning
Dynamic provisioning is a feature that allows Kubernetes to automatically create and attach persistent volumes to pods based on the persistent volume claims and storage classes. Dynamic provisioning eliminates the need for manual creation and management of persistent volumes by administrators. Most cloud providers, including GCE, AWS, Azure, and others, support the feature of dynamic provisioning. For example, the following pod specification uses a persistent volume claim that references the standard
storage class, which triggers the dynamic provisioning of a gcePersistentDisk
volume:
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: my-container
image: my-image
volumeMounts:
- name: my-volume
mountPath: /data
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: my-pvc
It's important to note that the gcePersistentDisk
volume is dynamically provisioned only if no existing PersistentVolume
satisfies the claim. If an existing PersistentVolume
satisfies the claim, Kubernetes will use that volume instead of dynamically provisioning a new one.
Volume Snapshots
A volume snapshot is a point-in-time copy of a persistent volume. A volume snapshot can be used to backup and restore data, or to clone a volume for another application. A volume snapshot can be created and managed by a user or an application using the VolumeSnapshot
and VolumeSnapshotContent
resources. For example, the following volume snapshot specification creates a snapshot of the persistent volume claim my-pvc
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: my-snapshot
spec:
source:
persistentVolumeClaimName: my-pvc
The Volume Snapshot feature in Kubernetes is not activated by default in all Kubernetes deployments. It must be activated by the system administrator.
Container Storage Interface (CSI)
Container Storage Interface is a standard that defines an interface for container orchestrators, such as Kubernetes, to communicate with storage providers, such as GCE, AWS, Azure, etc. CSI enables the development and deployment of vendor-neutral storage plugins that can work with any container orchestrator. CSI also supports advanced features, such as volume resizing, volume snapshots, and volume cloning. For example, the following pod specification uses a CSI driver to mount a gcePersistentDisk
volume:
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: my-container
image: my-image
volumeMounts:
- name: my-volume
mountPath: /data
volumes:
- name: my-volume
csi:
driver: pd.csi.storage.gke.io
volumeHandle: my-disk
The CSI feature is also not enabled by default on all Kubernetes deployments. It must be enabled by the administrator.
Tips on How to Use Kubernetes Storage Effectively
Choosing the right storage class: Depending on the use case and performance requirements of the application, it is important to choose the appropriate storage class for the persistent volume claims. Different storage classes may have different characteristics, such as availability, durability, throughput, latency, cost, etc. For example, for applications that need high performance and low latency, it may be better to use a storage class that uses SSD disks, such as
pd-ssd
for GCE orgp2
for AWS.For applications that need high availability and durability, it may be better to use a storage class that uses replicated or distributed storage, such as
pd-standard
for GCE orebs
for AWS. For applications that need cost-effective and scalable storage, it may be better to use a storage class that uses object storage, such asgcs
for GCE ors3
for AWS.Optimizing performance and availability: To improve the performance and availability of the storage resources, some factors need to be considered, such as the access mode, the mount options, the topology, and the resource limits. For example, for applications that need concurrent read and write access to the same volume, it may be better to use a storage class that supports
ReadWriteMany
access mode, such asnfs
orcephfs
.For applications that need to optimize the file system performance, it may be better to use the appropriate mount options, such as
noatime
,nodiratime
, ordiscard
. For applications that need to ensure the affinity and anti-affinity of the pods and the volumes, it may be better to use topology-aware scheduling and provisioning features, such asvolumeBindingMode
,allowedTopologies
, ortopologySpreadConstraints
. For applications that need to prevent the over-commitment and oversubscription of storage resources, it may be better to use the resource limits and requests, such asstorage
,iops
, orbandwidth
.Securing data: To protect the data from unauthorized access, modification, or deletion, some security measures need to be applied, such as encryption, authentication, and authorization. For example, for applications that need to encrypt the data at rest and in transit, it may be better to use a storage class that supports encryption, such as
pd-encrypted
for GCE orebs-encrypted
for AWS, or use a CSI driver that supports encryption, such assecrets-store.csi.k8s.io
.For applications that need to authenticate access to the volumes, it may be better to use a storage class that supports authentication, such as
iscsi
orrbd
, or use a CSI driver that supports authentication, such asvault.csi.k8s.io
. For applications that need to authorize access to the volumes, it may be better to use the role-based access control (RBAC) and the pod security policy (PSP) features, such asserviceAccountName
,volumeClaimTemplates
, orallowedCSIDrivers
.Backing up and restoring volumes: To ensure data recovery and continuity in case of disasters, failures, or errors, there are some backup and restore strategies that need to be implemented, such as volume snapshots, volume cloning, and application-level backup. For example, for applications that need to backup and restore the volumes at the block level, it may be better to use the volume snapshot and volume clone features, such as
VolumeSnapshot
,VolumeSnapshotContent
, ordataSource
.For applications that need to backup and restore the volumes at the file level, it may be better to use the application-level backup tools, such as
velero
,restic
, orark
.
Conclusion
In this article, we have learned about the basics of Kubernetes storage, including the concepts of persistent volumes, persistent volume claims, storage classes, and dynamic provisioning. We have also seen how to use these resources to create and manage storage for stateful applications on Kubernetes.