loading...

Kubernetes – Persistent storage

So far, we only worked with workloads that we could start and stop at will, with no issue. However, real-world applications often carry state and record data that we prefer (even insist) not to lose. The transient nature of containers themselves can be a big challenge. If you recall our discussion of layered filesystems in Chapter 1, Introduction to Kubernetes, the top layer is writable. (It’s also frosting, which is delicious.) However, when the container dies, the data goes with it. The same is true for crashed containers that Kubernetes restarts.

This is where volumes or disks come into play. Volumes exist outside the container and are coupled to the pod, which allows us to save our important data across containers outages. Further more, if we have a volume at the pod level, data can be shared between containers in the same application stack and within the same pod. A volume itself on Kubernetes is a directory, which the Pod provides to the containers running on it. There are a number of different volume types available at spec.volumes, which we’ll explore, and they’re mounted into containers with the spec.containers.volumeMounts parameter.

To see all the types of volumes available, visit https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes.

Docker itself has some support for volumes, but Kubernetes gives us persistent storage that lasts beyond the lifetime of a single container. The volumes are tied to pods and live and die with those pods. Additionally, a pod can have multiple volumes from a variety of sources. Let’s take a look at some of these sources.

Temporary disks

One of the easiest ways to achieve improved persistence amid container crashes and data sharing within a pod is to use the emptydir volume. This volume type can be used with either the storage volumes of the node machine itself or an optional RAM disk for higher performance.

Again, we improve our persistence beyond a single container, but when a pod is removed, the data will be lost. A machine reboot will also clear any data from RAM-type disks. There may be times when we just need some shared temporary space or have containers that process data and hand it off to another container before they die. Whatever the case, here is a quick example of using this temporary disk with the RAM-backed option.

Open your favorite editor and create a storage-memory.yaml file and type the following code:

apiVersion: v1 
kind: Pod 
metadata: 
  name: memory-pd 
spec: 
  containers: 
  - image: nginx:latest 
    ports: 
    - containerPort: 80 
    name: memory-pd 
    volumeMounts: 
    - mountPath: /memory-pd 
      name: memory-volume 
  volumes: 
  - name: memory-volume 
    emptyDir: 
      medium: Memory 

The preceding example is probably second nature by now, but we will once again issue a create command followed by an exec command to see the folders in the container:

$ kubectl create -f storage-memory.yaml
$ kubectl exec memory-pd -- ls -lh | grep memory-pd

This will give us a Bash shell in the container itself. The ls command shows us a memory-pd folder at the top level. We use grep to filter the output, but you can run the command without | grep memory-pd to see all folders:

Temporary storage inside a container

Again, this folder is temporary as everything is stored in the node’s (minion’s) RAM. When the node gets restarted, all the files will be erased. We will look at a more permanent example next.

Cloud volumes

Let’s move on to something more robust. There are two types of PersistentVolumes that we’ll touch base with in order to explain how you can use AWS’s and GCE’s block storage engines to provide stateful storage for your Kubernetes cluster. Given that many companies have already made significant investment in cloud infrastructure, we’ll get you up and running with two key examples. You can consider these types of volumes or persistent volumes as storage classes. These are different from the emptyDir that we created before, as the contents of a GCE persistent disk or AWS EBS volume will persist even if a pod is removed. Looking ahead, this provides operators with the clever feature of being able to pre-populate data in these drives and can also be switched between pods.

GCE Persistent Disks

Let’s mount a gcePersistentDisk first. You can see more information about these drives here: https://cloud.google.com/compute/docs/disks/.

Google Persistent Disk is durable and high performance block storage for the Google Cloud Platform. Persistent Disk provides SSD and HDD storage, which can be attached to instances running in either Google Compute Engine or Google Container Engine. Storage volumes can be transparently resized, quickly backed up, and offer the ability to support simultaneous readers.

You’ll need to create a Persistent Disk using the GCE GUI, API, or CLI before we’re able to use it in our cluster, so let’s get started:

  1. From the console, in  Compute Engine, go to Disks. On this new screen, click on the Create Disk button. We’ll be presented with a screen similar to the following  GCE new persistent disk screenshot:

GCE new persistent disk
  1. Choose a name for this volume and give it a brief description. Make sure that Zone is the same as the nodes in your cluster. GCE Persistent Disks can only be attached to machines in the same zone.
  2. Enter mysite-volume-1 in the Name field. Choose a zone matching at least one node in your cluster. Choose None (blank disk) for Source type and give 10 (10 GB) as the value in Size (GB). Finally, click on Create:

The nice thing about Persistent Disks on GCE is that they allow for mounting to multiple machines (nodes in our case). However, when mounting to multiple machines, the volume must be in read-only mode. So, let’s first mount this to a single pod, so we can create some files. Use the following code to make a storage-gce.yaml file to create a pod that will mount the disk in read/write mode:

apiVersion: v1 
kind: Pod 
metadata: 
  name: test-gce 
spec: 
  containers: 
  - image: nginx:latest 
    ports: 
    - containerPort: 80 
    name: test-gce 
    volumeMounts: 
    - mountPath: /usr/share/nginx/html 
      name: gce-pd 
  volumes: 
  - name: gce-pd 
    gcePersistentDisk: 
      pdName: mysite-volume-1 
      fsType: ext4 

First, let’s issue a create command followed by a describe command to find out which node it is running on:

$ kubectl create -f storage-gce.yaml 
$ kubectl describe pod/test-gce

Note the node and save the pod IP address for later. Then, open an SSH session into that node:

Pod described with persistent disk

Type the following command:

$ gcloud compute --project "<Your project ID>" ssh --zone "<your gce zone>" "<Node running test-gce pod>"

Since we’ve already looked at the volume from inside the running container, let’s access it directly from the node (minion) itself this time. We will run a df command to see where it is mounted, but we will need to switch to root first:

$ sudo su -
$ df -h | grep mysite-volume-1

As you can see, the GCE volume is mounted directly to the node itself. We can use the mount path listed in the output of the earlier df command. Use cd to change to the folder now. Then, create a new file named index.html with your favorite editor:

$ cd /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mysite-volume-1
$ vi index.html

Enter a quaint message, such as Hello from my GCE PD!. Now, save the file and exit the editor. If you recall from the storage-gce.yaml file, the Persistent Disk is mounted directly to the nginx HTML directory. So, let’s test this out while we still have the SSH session open on the node. Do a simple curl command to the pod IP we wrote down earlier:

$ curl <Pod IP from Describe>

You should see Hello from my GCE PD! or whatever message you saved in the index.html file. In a real-world scenario, we can use the volume for an entire website or any other central storage. Let’s take a look at running a set of load balanced web servers all pointing to the same volume.

First, leave the SSH session with two exit commands. Before we proceed, we will need to remove our test-gce pod so that the volume can be mounted read-only across a number of nodes:

$ kubectl delete pod/test-gce

Now, we can create an ReplicationController that will run three web servers, all mounting the same Persistent Disk, as follows. Save the following code as the http-pd-controller.yaml file:

apiVersion: v1 
kind: ReplicationController 
metadata: 
  name: http-pd 
  labels: 
    name: http-pd 
spec: 
  replicas: 3 
  selector: 
    name: http-pd 
  template: 
    metadata: 
      name: http-pd 
      labels:
        name: http-pd
    spec: 
      containers: 
      - image: nginx:latest 
        ports: 
        - containerPort: 80 
        name: http-pd 
        volumeMounts: 
        - mountPath: /usr/share/nginx/html 
          name: gce-pd 
      volumes: 
      - name: gce-pd 
        gcePersistentDisk: 
          pdName: mysite-volume-1 
          fsType: ext4 
          readOnly: true 

Let’s also create an external service and save it as the http-pd-service.yaml file, so we can see it from outside the cluster:

apiVersion: v1 
kind: Service 
metadata: 
  name: http-pd 
  labels: 
    name: http-pd 
spec: 
  type: LoadBalancer 
  ports: 
  - name: http 
    protocol: TCP 
    port: 80 
  selector: 
    name: http-pd 

Go ahead and create these two resources now. Wait a few moments for the external IP to get assigned. After this, a describe command will give us the IP we can use in a browser:

$ kubectl describe service/http-pd

The following screenshot is the result of the preceding command:

K8s service with GCE PD shared across three pods

If you don’t see the LoadBalancer Ingress field yet, it probably needs more time to get assigned. Type the IP address from LoadBalancer Ingress into a browser, and you should see your familiar index.html file show up with the text we entered previously!

AWS Elastic Block Store

K8s also supports AWS Elastic Block Store (EBS) volumes. Like the GCE Persistent Disks, EBS volumes are required to be attached to an instance running in the same availability zone. A further limitation is that EBS can only be mounted to a single instance at one time. Similarly to before, you’ll need to create an EBS volume using API calls, the CLI, or you’ll need to log in to the GUI manually and create the volume referenced by volumeID. If you’re authorized in the AWS CLI, you can use the following command to create a volume:

$ aws ec2 create-volume --availability-zone=us-west-1a eu-west-1a --size=20 --volume-type=gp2

Make sure that your volume is created in the same region as your Kubernetes cluster!

For brevity, we will not walk through an AWS example, but a sample YAML file is included to get you started. Again, remember to create the EBS volume before your pod. Save the following code as the storage-aws.yaml file:

apiVersion: v1 
kind: Pod 
metadata: 
  name: test-aws 
spec: 
  containers: 
  - image: nginx:latest 
    ports: 
    - containerPort: 80 
    name: test-aws 
    volumeMounts: 
    - mountPath: /usr/share/nginx/html 
      name: aws-pd 
  volumes: 
  - name: aws-pd 
    awsElasticBlockStore: 
      volumeID: aws://<availability-zone>/<volume-id> 
      fsType: ext4 

Other storage options

Kubernetes supports a variety of other types of storage volumes. A full list can be found here: https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes.

Here are a few that may be of particular interest:

  • nfs: This type allows us to mount a Network File Share (NFS), which can be very useful for both persisting the data and sharing it across the infrastructure
  • gitrepo: As you might have guessed, this option clones a Git repository into a new and empty folder

PersistentVolumes and Storage Classes

Thus far, we’ve seen examples of directly provisioning the storage within our pod definitions. This works quite well if you have full control over your cluster and infrastructure, but at larger scales, application owners will want to use storage that is managed separately. Typically, a central IT team or the cloud provider will take care of the details behind provisioning storage and leave the application owners to worry about their primary concern, the application itself. This separation of concerns and duties in Kubernetes allows you to structure your engineering focus around a storage subsystem that can be managed by a distinct group of engineers.

In order to accommodate this, we need some way for the application to specify and request storage without being concerned with how that storage is provided. This is where PersistentVolumes and PersistentVolumeClaim come into play.

PersistentVolumes are similar to the volumes we created earlier, but they are provided by the cluster administrator and are not dependent on a particular pod. PersistentVolumes are a resource that’s provided to the cluster just like any other object. The Kubernetes API provides an interface for this object in the form of NFS, EBS Persistent Disks, or any other volume type described before. Once the volume has been created, you can use PersistentVolumeClaims to request storage for your applications.

PersistentVolumeClaims is an abstraction that allows users to specify the details of the storage needed. We can defined the amount of storage, as well as the access type, such as ReadWriteOnce (read and write by one node), ReadOnlyMany (read-only by multiple nodes), and ReadWriteMany (read and write by many nodes). The cluster operators are in charge of providing a wide variety of storage options for application operators in order to meet requirements across a number of different access modes, sizes, speeds, and durability without requiring the end users to know the details of that implementation. The modes supported by cluster operators is dependent on the backing storage provider. For example, we saw in the AWS aws-ebs example that mounting to multiple nodes was not an option, while with GCP Persistent Disks could be shared among several nodes in read-only mode. 

Additionally, Kubernetes provides two other methods for specifying certain groupings or types of storage volumes. The first is the use of selectors, as we have seen previously for pod selection. Here, labels can be applied to storage volumes and then claims can reference these labels to further filter the volume they are provided. Second, Kubernetes has the concept of StorageClass, which allows us specify a storage provisioner and parameters for the types of volumes it provisions.

PersistentVolumes and PersistentVolumeClaims have a life cycle that involves the following phases:

  • Provisioning
  • Static or dynamic
  • Binding
  • Using
  • Reclaiming
  • Delete, retain, or recycle

We will dive into Storage Classes in the next section, but here is a quick example of a PersistentVolumeClaim for illustration purposes. You can see in the annotations that we request 1Gi of storage in ReadWriteOnce mode with a StorageClass of solidstate and a label of aws-storage. Save the following code as the pvc-example.yaml file:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: demo-claim
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: ssd
  selector:
    matchLabels:
      release: "aws-storage"
  matchExpressions:
      - {key: environment, operator: In, values: [dev, stag, uat]}

As of Kubernetes version 1.8, there’s also alpha support for expanding PersistentVolumeClaim for gcePersistentDisk, awsElasticBlockStore, Cinder, glusterfs, and rbd volume claim types. These are similar to the thin provisioning that you may have seen with systems such as VMware, and they allow for resizing of a storage class via the allowVolumeExpansion field as long as you’re running either XFS or Ext3/Ext4 filesystems. Here’s a quick example of what that looks like:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: Cinder-volume-01
provisioner: kubernetes.io/cinder
parameters:
 resturl: "http://192.168.10.10:8080"
 restuser: ""
 secretNamespace: ""
 secretName: ""
allowVolumeExpansion: true

Dynamic volume provisioning

Now that we’ve explored how to build from volumes, storage classes, persistent volumes, and persistent volume claims, let’s take a look at how to make that all dynamic and take advantage of the built-in scaling of the cloud! Dynamic provisioning removes the need for pre-crafted storage; it relies on requests from application users instead. You use the StorageClass API object to create dynamic resources.

First, we can create a manifest that will define the type of storage class that we’ll use for our dynamic storage. We’ll use a vSphere example here to try out another storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: 
  name: durable-medium
provisioner: kubernetes.io/vsphere-volume
parameters:
  type: thin

Once we have the manifest, we can use this storage by including it as a class in a new PersistentVolumeClaim. You may remember this as volume.beta.kubernetes.io/storage-class in earlier, pre-1.6 versions of Kubernetes, but now you can simply include this property in the PersistentVolumeClaim object. Keep in mind that the value of storageClassName must match the available, dynamic StorageClass that the cluster operators have provided. Here’s an example of that:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: webtier-vclaim-01
spec:
 accessModes:
   - ReadWriteMany
 storageClassName: durable-medium
 resources:
   requests:
     storage: 20Gi

When this claim is removed, the storage is dynamically deleted. You can make this a cluster default by ensuring that the DefaultStorageClass admission controller is turned on, and after you ensure that one StorageClass object is set to default.

Comments are closed.

loading...