When deploying an application in a containers based environment, one of the usual challenges to tackle is how/where to store application state if your application happens to require it, which is usually the case.

Today I was tackling one of the recurrent issues while building a web application: an user should be able to upload a file (an image in this case) and retrieve it later. This obviously means storing the file in the server side.

Files inside a container are ephemeral, there’s no guarantee how long a container will be alive, and therefore local state inside the container is not an option (it’s usually not an option if you want to scale horizontally).

In this scenario, the state is represented by a binary file. While there’re multiples solutions for this use case, using a shared disk that can be mounted by a set of nodes is what I’m more used to.

But working with kubernetes, “mounting a shared disk in a set of nodes” is not that straight forward.

In this post I’ll go through the relevant kubernetes resources for tackling this problem, and specifically how to implement it on top of AWS using an EFS storage resource.

The basic. A kubernetes Volume

A Volume is a kubernetes storage resource attached to a Pod, and it lives as long as the Pod it’s attached to does.

Those are the usual scenarios where I’ve been using a Volume so far:

  • mount ConfigMap data using a configMap Volume type. This is handy for creating files in the container with the ConfigMap information.
  • share state between containers that are part of the same Pod using an emptyDir Volume type.
  • mount an external block storage, like Amazon EBS or Google Persistent Disk, using awsElasticBlockStore and gcePersistentDisk respectively. This is useful if you need to store data which availability is limited to one container at a time (no need to share state between pods), so data will nb. Note that the external resources must be created before you can use them in kubernetes via the cloud provider Web console or command line tool.

While the Volume is indeed convenient for the scenarios described above, there’s a big limitation: it can be mounted only in one Pod. Therefore, a Volume is not a good solution for my scenario, where I need binary files to be available in several Pods (to scale horizontally the solution).

The advanced. A kubernetes Persistent Volume

A Persistent Volume is a cluster resource on its own and has its own lifecycle. It represents a storage resource available to any Pod created in the cluster. Not being attached to a specific node/pod is one of the main differences with a Volume.

Similar to how memory and CPU resources can be configured in a Pod specification, a Pod storage requirements (Persistent Volume) can be defined using a PersistentVolumeClaim definition. There’re two attributes that can be configured: size and access mode (read, write).

Mind the difference between these two concepts: a Persistent Volume is a cluster resource (like nodes, memory, CPU), while and a Persistent Volume Claim is a set of requirements about the storage a Pod needs.

Last but not least concept is the StorageClass kind, which is used to describe a storage resource (similar to include metadata or define several profiles). A pod storage requirements can be configured either by defining size and access mode via PVC, or by defining the needs in more abstract terms, using a StorageClass.

A Persistent Volume can be provisioned dynamically by means of a StorageClass definition (using the parameter provisioner).

Steps to mount an EFS resource in a Pod

Back to my original problem, how can I mount a disk for sharing state (binary files) between Pods?

Running a kubernetes cluster in AWS, it seems like EFS is the natural choice.

Those are the steps I went through:

1. Create an EFS resource and make it available to kubernetes nodes using aws cli

An EFS resource can be created executing the following command:

aws efs create-file-system –creation-token efs-for-testing

The response is a JSON payload including a field named FileSystemId, which represents the unique identifier that should be used to manage the EFS volume. Let’s assume the FileSystemId is fs-testing.

EFS creation is an asynchronous process, and before managing it you need to make sure its LifeCycleState is available. The EFS state can be checked as follows:

aws efs describe-file-systems –file-system-id fs-testing

Once the EFS is available, next step is creating a mount target associated to it. A mount target acts as a virtual firewall, defining a subnet and a security group that is granted permissions to mount the EFS volume.

For creating the mount target you need the subnet-id and security-groups associated to your kubernetes cluster nodes. Usual scenario is that every node will share the same security group, while subnet id will differ based on the Availability Zone where the node is located:

aws ec2 describe-instances –filters <your-filters-to-retrieve-k8s-nodes>

Per each SubnetId and SecurityGroupId execute the following command:

aws efs create-mount-target \ –file-system-id fs-testing \ –subnet-id {SubnetId} \ –security-groups {SecurityGroupId}

2. Deploy the EFS provisioner

A Kubernetes deployment includes, by default, several Persistent Volume types, like AWSElasticBlockStore, GCEPersistentDisk, AzuleFile and NFS for naming a few. Each of them defines a specific provisioner that can be used to create a PV.

Furthermore, the kubernetes incubator external-storage repository holds additional Persistent Volumes that are not part of a Kubernetes default deployment, and here I found the answer to my specific need: the EFS provisioner.

The EFS provisioner is a deployment that runs a container with access to the AWS EFS resource. It acts as an EFS broker, allowing other pods to mount the EFS resource as a PV.

These are the definitions I used for deploying the EFS provisioner, even though you can find a very similar definitions in kubernetes-incubator github repository:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: efs-provisioner
data:
  file.system.id: '<<your-efs-id>>'
  aws.region: '<<your-region-id>>'
  provisioner.name: mycompany.com/aws-efs

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: efs-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: efs-provisioner
    spec:
      containers:
        - name: efs-provisioner
          image: quay.io/external_storage/efs-provisioner:latest
          env:
            - name: FILE_SYSTEM_ID
              valueFrom:
                configMapKeyRef:
                  name: efs-provisioner
                  key: file.system.id
            - name: AWS_REGION
              valueFrom:
                configMapKeyRef:
                  name: efs-provisioner
                  key: aws.region
            - name: PROVISIONER_NAME
              valueFrom:
                configMapKeyRef:
                  name: efs-provisioner
                  key: provisioner.name
          volumeMounts:
            - name: pv-volume
              mountPath: /persistentvolumes
      volumes:
        - name: pv-volume
          nfs:
            server: <<your-efs-id>>.efs.<<your-region-id>>.amazonaws.com
            path: /

kubectl apply -f efs-provisioner.yaml

3. Define the StorageClass kind

StorageClass is used as an intermediate step for connecting a PersistentVolumeClaim with a specific storage resource:

  • metadata.name field is used to refer to the resource.
  • provisioner is used to identify the provisioner (EFS provisioner in this case).

Important: An StorageClass definition cannot be updated.

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: aws-efs
provisioner: mycompany.com/aws-efs

kubectl apply -f storage-class.yaml

4. Define the PersistentVolumeClaim

The PVC definition connects access mode and size requirements with a specific StorageClass item. In this case, as EFS has unlimited storage, the size requested won’t have any real impact.

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: efs
spec:
  storageClassName: aws-efs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

kubectl apply -f pvc.yaml

As soon as you create the PVC, the EFS provisioner will get notified and will create a PV that matches the requirements. These are the EFS provisioner logs showing the PV creation:

I0928 11:03:45.897983       1 controller.go:987] provision "default/efs" class "aws-efs": started
I0928 11:03:45.900711       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"efs", UID:"2b56b224-c30e-11e8-abf5-023d3cfc37fe", APIVersion:"v1", ResourceVersion:"52345195", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/efs"
I0928 11:03:45.950090       1 controller.go:1087] provision "default/efs" class "aws-efs": volume "pvc-2b56b224-c30e-11e8-abf5-023d3cfc37fe" provisioned
I0928 11:03:45.950116       1 controller.go:1101] provision "default/efs" class "aws-efs": trying to save persistentvvolume "pvc-2b56b224-c30e-11e8-abf5-023d3cfc37fe"
I0928 11:03:45.956467       1 controller.go:1108] provision "default/efs" class "aws-efs": persistentvolume "pvc-2b56b224-c30e-11e8-abf5-023d3cfc37fe" saved
I0928 11:03:45.956498       1 controller.go:1149] provision "default/efs" class "aws-efs": succeeded
I0928 11:03:45.956643       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"efs", UID:"2b56b224-c30e-11e8-abf5-023d3cfc37fe", APIVersion:"v1", ResourceVersion:"52345195", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-2b56b224-c30e-11e8-abf5-023d3cfc37fe

And now you can retrieve both PV and PVC using kubectl:

kubectl get pv
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM         STORAGECLASS   REASON    AGE
persistentvolume/pvc-2b56b224-c30e-11e8-abf5-023d3cfc37fe   1Gi        RWX            Delete           Bound     default/efs       aws-efs              4m

kubectl get pvc
NAME                        STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/efs   Bound     pvc-2b56b224-c30e-11e8-abf5-023d3cfc37fe   1Gi        RWX            aws-efs        4m

5. Create a Deployment with 2 replicas and mount the Volume

Pods get access to the PV storage by defining the claim as a volume in the Pod definition. Claims must exist in the same namespace as the pods using the claim (StorageClass and PersistentVolume are global kinds in the cluster).

The snippet below is a basic Deployment example with 2 pods mouting a volume using a PVC. Each Pod will generate a single file in the shared folder and check that the folder has additional files, which would reflect that indeed the other Pod has created its file.

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: test-efs
spec:
  replicas: 2
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: test-efs
    spec:
      restartPolicy: Always
      containers:
      - name: test-pod
        image: gcr.io/google_containers/busybox:1.24
        command:
          - "sh"
        args:
          - '-c'
          - 'touch "${MEDIA_PATH}/${MY_POD_NAME}"; echo "File created, waiting a bit to ensure the other Pod had the time as well"; sleep 5; [[ $(ls -l "$MEDIA_PATH" | wc -l) -gt 1 ]] && (echo "Both pods generated the file!" && exit 0) || (echo "Unable to create both files in the shared folder" && exit 1)'
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MEDIA_PATH
            value: "/var/media/uploads"
        volumeMounts:
          - name: efs-pvc
            mountPath: "/var/media/uploads"
      volumes:
        - name: efs-pvc
          persistentVolumeClaim:
            claimName: efs

Checking the Pods logs we can see that the scenario is successfully validated:

kubetail --selector app=test-efs
Will tail 2 logs...
test-efs-546d6d7456-2fvgp
test-efs-546d6d7456-2gqx6
[test-efs-546d6d7456-2fvgp] File created, waiting a bit to ensure the other Pod had the time as well
[test-efs-546d6d7456-2gqx6] File created, waiting a bit to ensure the other Pod had the time as well
[test-efs-546d6d7456-2fvgp] Both pods generated the file!
[test-efs-546d6d7456-2gqx6] Both pods generated the file!

Conclusions

While not very obvious, once you interiorize the concepts around persistent storage, having a shared folder mounted in several Pods in a kubernetes cluster running in AWS is quite straight forward.

« Home