The Pod is stuck on ContainerCreating

The Pod is stuck on ContainerCreating
Photo by Tareq Shuvo / Unsplash

I am running a blog that is deployed in the Kubernetes cluster. And as with every self-hosted application, it requires a bit of maintenance from time to time. Every application has bugs and those bugs get fixed at some point. It is good to update an application regularly to apply all the necessary fixes. It is really important as it reduces the likelihood of the application being exploited. Plus probably there are new features as well.

Rolling update

Not hesitating too much I updated Helm Charts and triggered an update.

❯ helm upgrade ghost bitnami/ghost --namespace ghost-test --version 19.1.67 -f values.yml

New Pod creation has started.

❯ kubectl get pods 
NAME                     READY   STATUS              RESTARTS      AGE 
ghost-7d4f67c998-bbfxg   1/1     Running             5 (40m ago)   44h 
ghost-ff4b84cfd-hvwbz    0/1     ContainerCreating   0             44m

But my excitement fainted really quickly as I noticed that Pod got stuck in the ContainerCreating state indefinitely.

❯ kubectl get events 
2m11s       Normal    Scheduled                pod/ghost-ff4b84cfd-hvwbz        Successfully assigned ghost-test/ghost-ff4b84cfd-hvwbz to vm0102 
3s          Warning   FailedMount              pod/ghost-ff4b84cfd-hvwbz        MountVolume.MountDevice failed for volume "pvc-6cbd418d-b9bd-410c-90c5-822123c5df94" : rpc error: code = Internal desc = Volume pvc-6cbd418d-b9bd-410c-90c5-822123c5df94 still mounted on node vm0202 
8s          Warning   FailedMount              pod/ghost-ff4b84cfd-hvwbz        Unable to attach or mount volumes: unmounted volumes=[ghost-data], unattached volumes=[kube-api-access-9cqkw ghost-data]: timed out waiting for the condition 
2m11s       Normal    SuccessfulCreate         replicaset/ghost-ff4b84cfd       Created pod: ghost-ff4b84cfd-hvwbz

It turned out that the newly spawned Pod wanted to mount the volume that had been already mounted but on the older Pod.

❯ kubectl get pvc 
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE 
data-ghost-mysql-0   Bound    pvc-4ed9e42c-920a-4032-b915-a114061eef31   8Gi        RWO            openebs-cstor-csi-default   45h 
ghost                Bound    pvc-6cbd418d-b9bd-410c-90c5-822123c5df94   8Gi        RWO            openebs-cstor-csi-default   45h

Printing PersistentVolumeClaims showed the volumes were mounted in ReadWriteOnce (RWO) mode, so there was no way they could have been mounted on another node hosting the newly created Pod.

💡
My Kubernetes cluster has OpenEBS cStor installed and configured for Dynamic Volume provisioning.

One option would be not to scale the number of Pods during an update. So Terminate an old Pod and create a new one. But this would result in an outage of service for a while. So why not mount the volume in ReadWriteMany (RWX) mode?

The failed approach

I uninstalled ghost and then updated the values.yml file with the persistence section to force RWX.

❯ cat values.yml 
ghostUsername: 'kuba' 
ghostPassword: '12345abcde' 
ghostEmail: 'jakub@slys.dev' 
ghostBlogTitle: 'slys.dev' 
ghostHost: 'blog.slys' 
allowEmptyPassword: false 
ghostSkipInstall: false 
ingress: 
  enabled: true 
  hostname: 'blog.slys' 
  ingressClassName: 'nginx' 
service: 
  type: 'ClusterIP' 
startupProbe: 
  enabled: true 
  periodSeconds: 30 
  failureThreshold: 10 
image: 
  debug: true 
mysql: 
  auth: 
    password: '5S3MjHA0QQ' 
    rootPassword: 'k72H3TCqnA' 
persistence: 
  accessModes: 
    - ReadWriteMany

Applied helm again.

❯ helm upgrade --install ghost bitnami/ghost --namespace ghost-test --version 19.1.60 -f values.yml 
Release "ghost" does not exist. Installing it now. 
NAME: ghost 
LAST DEPLOYED: Tue Feb  7 12:32:14 2023 
NAMESPACE: ghost-test 
STATUS: deployed 
REVISION: 1 
TEST SUITE: None 
NOTES: 
CHART NAME: ghost 
CHART VERSION: 19.1.60 
APP VERSION: 5.31.0 
** Please be patient while the chart is being deployed ** 
1. Get the Ghost URL and associate its hostname to your cluster external IP: 
  export CLUSTER_IP=$(minikube ip) # On Minikube. 
  Use: `kubectl cluster-info` on others K8s clusters echo "Ghost URL: http://blog.slys" 
    echo "$CLUSTER_IP  blog.slys" | sudo tee -a /etc/hosts 
2. Get your Ghost login credentials by running: 
  echo Email:    jakub@slys.dev 
  echo Password: $(kubectl get secret --namespace ghost-test ghost -o jsonpath="{.data.ghost-password}" | base64 -d)

Pods started to create. However, after a moment I realized that created Pod was stuck, again.

❯ kubectl get events 
8s          Normal    Provisioning            persistentvolumeclaim/ghost                External provisioner is provisioning volume for claim "ghost-test/ghost" 
2s          Normal    ExternalProvisioning    persistentvolumeclaim/ghost                waiting for a volume to be created, either by external provisioner "cstor.csi.openebs.io" or manually created by system administrator 
8s          Warning   ProvisioningFailed      persistentvolumeclaim/ghost                failed to provision volume with StorageClass "openebs-cstor-csi-default": rpc error: code = InvalidArgument desc = only SINGLE_NODE_WRITER supported, unsupported access mode requested: MULTI_NODE_MULTI_WRITER

Printing events exposed an issue - the current provisioner is unable to satisfy the request.

Dynamic NFS Provisioner

The remedy to those stuck Pods was to install another dynamic volume provisioner in the cluster - the NFS provisioner.

❯ helm upgrade openebs openebs/openebs -n openebs \ 
  --set cstor.enabled=true \ 
  --set nfs-provisioner.enabled=true \ 
  --namespace openebs

After a while, a Pod with an NFS provisioner was up and running.

❯ kubectl get pods -n openebs 
NAME                                                              READY   STATUS    RESTARTS      AGE 
openebs-nfs-provisioner-787d694555-8k72g                          1/1     Running   0             24d

‌Also, a new StorageClass was installed.

❯ kubectl get sc 
NAME                                  PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE 
openebs-kernel-nfs                    openebs.io/nfsrwx      Delete          Immediate              false                  24d

The last bit was to update the values.yml file to point to the proper StorageClass.

❯ cat values.yml 
... 
persistence: 
  storageClass: 'openebs-kernel-nfs' 
  accessModes: 
    - ReadWriteMany

And then install it again.

❯ helm install ghost bitnami/ghost --namespace ghost-test --create-namespace --version 19.1.60 -f values.yml

Now, volume was provisioned with desired Access Mode - RWX, at last!

❯ kubectl get pvc 
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE 
data-ghost-mysql-0   Bound    pvc-647eb4c2-b113-4bde-bdf6-1276c83caad1   8Gi        RWO            openebs-cstor-csi-default   10m 
ghost                Bound    pvc-c0f9610e-b4d1-4fa3-b06e-e451113b7568   8Gi        RWX            openebs-kernel-nfs          10m

But still, the Pod was stuck...

❯ kubectl get pods 
NAME                     READY   STATUS    RESTARTS       AGE 
ghost-7d4f67c998-6cznw   0/1     Running   2 (108s ago)   4m58s 
ghost-mysql-0            1/1     Running   1 (2m ago)     4m58s

Describing that miserable Pod exposed yet another issue.

❯ kubectl describe pod ghost-7d4f67c998-6cznw 
Name:             ghost-7d4f67c998-6cznw 
Namespace:        ghost-test 
... 
Events: 
Type     Reason            Age    From               Message 
----     ------            ----   ----               ------- 
Warning  FailedMount       6m     kubelet            MountVolume.SetUp failed for volume "pvc-c0f9610e-b4d1-4fa3-b06e-e451113b7568" : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs 10.97.190.112:/ /var/lib/kubelet/pods/aa9e7549-c24d-4ca2-9d53-f63e98ef3a87/volumes/kubernetes.io~nfs/pvc-c0f9610e-b4d1-4fa3-b06e-e451113b7568 Output: mount.nfs: access denied by server while mounting 10.97.190.112:/ 
Normal   Pulled     3m14s (x3 over 5m59s)  kubelet  Container image "docker.io/bitnami/ghost:5.31.0-debian-11-r0" already present on machine

The Pod doesn't have enough privileges to mount NFS resources.

Making it right

Due to security reasons, we don't want Pods to be running in privileged mode as the root user. So outstanding issue to crack was granting proper privileges.

I had to add an annotation in values.yml for OpenEBS to provision volume with correct access rights for the user 1001.

persistence:
  storageClass: 'openebs-kernel-nfs'
  accessModes:
    - ReadWriteMany
  annotations:
    cas.openebs.io/config: |
      - name: FilePermissions
        data:
          GID: '1001'
          UID: '1001'
          mode: "g+s"

The last thing was to update the security context to run as the user 1001.

volumePermissions:
  enabled: true
  securityContext:
    runAsUser: 1001
    runAsGroup: 1001

And voila! Pod was up and running and the volume was provisioned in RWX mode.

❯ kubectl get pods 
NAME                    READY   STATUS    RESTARTS        AGE 
ghost-9bd7dbb8c-llc9p   1/1     Running   2 (4m52s ago)   8m1s 
ghost-mysql-0           1/1     Running   1 (5m1s ago)    8m1s
❯ kubectl get pvc 
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE 
data-ghost-mysql-0   Bound    pvc-c63e0252-f30b-455c-9d08-585dbe69cb17   8Gi        RWO            openebs-cstor-csi-default   9m15s 
ghost                Bound    pvc-0b6869e7-a258-43a5-b805-fa53e506d6e1   8Gi        RWX            openebs-kernel-nfs          9m15s

Now was the time to test out a rolling update.

❯ helm upgrade ghost bitnami/ghost --namespace ghost-test --version 19.1.66 -f values.yml

‌ We could see as before, a new Pod was created, but now it didn't get stuck in the ContainerCreating state.

❯ kubectl get pods 
NAME                     READY   STATUS    RESTARTS        AGE 
ghost-75559f9d44-zwt7m   0/1     Running   0               22s 
ghost-9bd7dbb8c-llc9p    1/1     Running   2 (8m34s ago)   11m 
ghost-mysql-0            0/1     Running   0               18s

Once the new Pod was initialized, the old one was Terminated.

❯ kubectl get pods 
NAME                     READY   STATUS    RESTARTS        AGE 
ghost-75559f9d44-zwt7m   1/1     Running   1 (2m23s ago)   3m39s 
ghost-mysql-0            1/1     Running   0               3m35s

Conclusion

RWX access mode can be very useful when deploying an application that is not designed to run natively in a cloud environment. In our example with the Ghost application, we can see that when performing an update, a new Pod is spawned. The new Pod wants to bind to the same PersistentVolume as an old Pod is bound to. In the case of RWX, it is pretty legal, so they will share the same volume. However, if RWO was selected, newly deployed would get stuck waiting to be bound, but that would never happen.

One last remark. Let's print PersistentVolumes.

❯ kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                  STORAGECLASS                REASON   AGE pvc-0b6869e7-a258-43a5-b805-fa53e506d6e1   8Gi        RWX            Delete           Bound    ghost-test/ghost                                       openebs-kernel-nfs                   18m pvc-c63e0252-f30b-455c-9d08-585dbe69cb17   8Gi        RWO            Delete           Bound    ghost-test/data-ghost-mysql-0                          openebs-cstor-csi-default            18m

We can see that pvc-0b6869e7-a258-43a5-b805-fa53e506d6e1 comes from ghost-test/ghost. However, pvc-f469b7a5-8fda-4823-8162-8786fe7d22a1 comes from openebs/nfs-pvc-0b6869e7-a258-43a5-b805-fa53e506d6e1. That means that the NFS provisioner uses cStor underneath to provision a volume.