Distributed Multi-worker TensorFlow Training on Kubernetes: My quota has been exceeded

Course Q&A Machine Learning Engineering for Production(MLOps) Machine Learning Modeling Pipelines in Production
1

My quota has been exceeded for “Distributed Multi-worker TensorFlow Training on Kubernetes” lab. How could I solve this?

I was stuck on “Preparing TFJob” and I was doing /submitting this lab at least 4 times, may be that caused me exceeding quota.

Here is what I have done for “Preparing TFJob” part.

  1. Retrieve lab-files from github.
  2. Build the image and push it to the registry.
  3. Submit the job using kubectl. " kubectl apply -f tfjob.yaml"
  4. Check the status of all pods using “kubectl get pods”. Status doesn’t change to Running , it always show “ImagePullBackOff” till the labs end.

I got only 70% for this lab and it was out of quota.

Thank you so much for your precious Time.

2

Hi! You can ask for an extension of your lab time by using the Qwiklabs support channels. You can see it in the upper right corner.

qwiksupport
qwiksupport852×806 34.2 KB

Also remember to edit the image and args in the YAML file as described in the instructions. Maybe that’s the reason the pods’ status was not changing. You can browse (or use the search button on the upper right) for similar threads in this MLEP Course 3 category to get some hints on what might have gone wrong. Hope this helps!

1 Like
3

Greetings @chris.favila – i’ve run into the same issue. I had to restart this lab a few times, & did make the .yaml file match the stated requirements. I tried to redeploy but Kubernetes didn’t recognize the changes. Now I’ve hit my lab quota. The Help Center guidance states the following:

There is a limit to the number of times you can take a specific lab. The limit is in effect in the following situations:

The lab is a free lab.
You use a Qwiklabs Advantage subscription to take the lab (for labs that cost credits).
The limit is set by your deployment.

If you try to take a specific lab again, after you have reached the limit, the lab will not start. You get the error message “Sorry, your quota has been exceeded for this lab”.

If that’s the case, the implication is I’m SOL on receiving credit for this course. Is it possible to get a reset, or to receive credit for the work completed?

4

Ok, I just spun up a chat w/ tech support. they overrode the quota.

5

Hi Dan! Glad you found the Qwiklabs chat support useful. Remember to check your tfjobs.yaml file. That is usually the problem here. This lab should not take more than an hour to complete.

*Note: For next time, you may want to post queries like this as a separate topic. That will get more mentors and learners to weigh in. Also please avoid tagging mentors and staff. That will discourage other mentors and learners from answering your query. You might also wait longer in case the person you tagged is offline or busy. Hope you understand. Thank you!