C4W2_Lab_2 Deployment not ready FailedCreatePodSandBox

Deployment is not getting ready.

This is my configmap.yaml:

Output of kubectl describe pods:

Name:             tf-serving-deployment-568cd6d868-5228g
Namespace:        default
Priority:         0
Service Account:  default
Node:             minikube/192.168.59.100
Start Time:       Wed, 22 Feb 2023 19:28:19 +0530
Labels:           app=tf-serving
                  pod-template-hash=568cd6d868
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/tf-serving-deployment-568cd6d868
Containers:
  tf-serving:
    Container ID:   
    Image:          tensorflow/serving
    Image ID:       
    Port:           8501/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     10m
      memory:  64M
    Requests:
      cpu:     5m
      memory:  32M
    Environment Variables from:
      tfserving-configs  ConfigMap  Optional: false
    Environment:         <none>
    Mounts:
      /models/half_plus_two from tf-serving-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4pkb5 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tf-serving-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /var/tmp/saved_model_half_plus_two_cpu
    HostPathType:  Directory
  kube-api-access-4pkb5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               9m3s                  default-scheduler  Successfully assigned default/tf-serving-deployment-568cd6d868-5228g to minikube
  Normal   SandboxChanged          3m1s (x2 over 5m53s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  52s (x3 over 7m4s)    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "tf-serving-deployment-568cd6d868-5228g": operation timeout: context deadline exceeded

minikube schedules the sandbox again however it doesn’t work even after 30mins.

If anyone can let me know what the issue is, it’ll be helpful. Thanks in advance.

Please look at deployment.yaml where /models/half_plus_two is set as the mount path which maps to /var/tmp/saved_model_half_plus_two_cpu. So, don’t change anything in yaml/configmap.yaml

Done. Error remains still.

You’ve opened a bunch of yaml files. If you’re sure that you haven’t made any changes to the lab files, please zip the lab files and message it to me as an attachment. Here’s the community user guide to get started.

Do update the topic with your latest log information.

What’s your OS / hardware (RAM, HDD free space & CPU) ?

Ubuntu 20.04, SSD 200GB alloted, i7 9th gen, RAM 15.5GB

48GB free of total 200GB.

I’m having trouble reproducing this issue and so have asked other mentors / staff to look at your issue as well. Someone should get back to you.

I solved the issue by using docker as the vm-driver. This is the command which I used for starting minikube:

minikube start --mount=True --mount-string="/var/tmp:/var/tmp" --vm-driver=docker

Thanks for sharing. Do try to get it to run on virtualbox.

Here’s a note from the exercise:

Minikube is meant to run in a virtual machine (VM) so you will need virtualization software to act as the VM driver. While you can also specify docker as the VM driver, we found that it has limitations so it’s best to use Virtualbox instead.

1 Like