C4_W3_Graded_Lab_1 : csvexamplegen did not work

Could only create AI platform pipeline instance. Could not complete the balance 3 exercises. Reason csvexamplegen component has not worked (pipeline image below). So it has not been able to get training data.
Please help where am I going wrong. Also

  1. Need guidance to select correct values in the pipeline runtime environment values for config.py Only changed PIPELINE_NAME to kubeflow-pipelines-1 and left the rest to default values. Should MODEL_NAME and DATA_ROOT_URI be changed? Need help.
  2. I’ve selected Python[conda env:root] as the kernel for the notebook. Should I select Python3?

I already have a warning flashing on this assignment TFX on Google Cloud AI Platform Pipelines as I have tried to complete this lab a few times. Want to get it right next time round.
Regards
Sanjoy

Hi Sanjoy! In case you haven’t resolved this yet:

  1. The pipeline name should have been defined in the notebook already. The variables you should fill out in the notebook are ENDPOINT and Run ID.
  2. Will ask the Qwiklabs team for clarification but if I remember correctly, I was able to complete the assignment by choosing Python 3.

Hope this helps!

Thanks a ton Chris.
Yes. I’ve figured them out thru several iterations! Upside is kwiklab have reset my lab quota as needed. They have been making changes also. For example cluster-1 was not featuring as default while configuring kubeflow pipeline till yesterday. Hence one had to check allow API access to be able to do it. But today cluster-1 came on its as pre-provisioned and check Allow API access was not even visible. So some bug where I get 4th task complete but 2nd and 3rd does not show complete. Hope they will correct and revert.
Regards
Sanjoy

Noted! Hope they’re able to debug soon so you can complete the lab. Thanks for sharing the updates!

Hi Chris
No response from kwiklabs so far. I’ve had 3 chat sessions on the same issue with promise to get back on mail:

We have checked the case. It looks like there might be some issue with the lab. We don’t want to guide you in the wrong direction. We need to test the lab in order to debug this case. Will this be okay for you if we update you via email after testing the lab?

The same response.

I’ve now sent a mail to kwiklab to help resolve what error I am making.

Please help. This is going on for last 10 odd days.
Regards
Sanjoy

Hi Sanjoy! I am now able to replicate the issue. I am able to complete all steps (and all show the green check mark so I am able to get the grade) but the pipeline is stuck in the first step (csvexamplegen, resolvernode, importernode). I am escalating this to Qwiklabs. Hope you’re also able to get the perfect score so you can wait for the resolution without affecting your progress. Thanks!

Thank you Chris. That’s great.

I am reattempting the lab now. In one of the earlier attempts I had managed to run all components of the pipeline. It created some 4 pipeline jobs to do so. Took about 45 minutes. But still the 2 tasks

Compile the kubeflow pipeline
Deploy the pipeline package to AI Platform pipelines

did not get green tick. I was stuck at 60/100 points and the lab closed.

I hope this time it will go thru like yours. Wish me luck!

Regards
Sanjoy

Hi Chris,
It’s the same 2nd and 3rd task not getting completed despite pipeline running successfully.

I have inserted endpoint and the pipeline name (kubeflow-pipelines-1) as the only 2 inputs in the notebook. Everything else is as per variables definition of the notebook.
Right now 3 components of the pipeline are running:

The other thing I notice is that pipeline is not creating any job while running this pipeline:

Where am I going wrong? Do I have to modify runner.py?

Qwiklab Support has not been able to provide any answer for last 10 odd days.

Also, the above 3 components are still running for last 55 minutes and not completed yet.

Regards
Sanjoy.

Hi Sanjoy. Please leave the pipeline name definition in the notebook as is. I think there is a mismatch with the name you’re using that’s why it’s not getting checked. You will only need to modify the ENDPOINT variable. I did that earlier and was able to get the grade even without the pipeline completely working. I also used conda env:root earlier instead of Python3 when prompted to open the notebook. Lastly, I waited for a minute after finishing the related notebook cells’ execution before the grader recognized it and gave the green check mark. Hope you will get the grade with these changes.

In addition to the reply above, I am getting the endpoint here for reference:

Finally!!!

Thank you for your patience.

Here is what happened.

In task 1, we create a kubeflow pipeline instance as default name provided

kubeflow-pipelines-1 .

However the rest of the assignment expects pipeline name to be

tfx_covertype_continuous_training

Thus it is looking for compiled pipeline file tfx_covertype_continuous_training.tar.gz in the /pipeline folder to mark it correct. This is probably hard-coded into grader. That’s why no other PIPELINE_NAME even though valid is acceptable.

Also as you have seen, the first 3 components of the pipeline - csvexamplegen, resolvernode, importernode just kept running and never finished in more than 1 hour! Something wrong here.

Not sure if access to cloud APIs were allowed in runner.py or pipeline.py:

Allow access to the following Cloud APIs https://www.googleapis.com/auth/cloud-platform

Or is it automatic.

Lastly in all fairness, input of endpoint was the only #todo in the notebook and with clear instruction how to get it. So nothing wrong here. However, creating a new pipeline instance for task 1, but not using it for next 3 tasks confused me.

Thank you for your help once again, Chris.

Regards
Sanjoy

1 Like

Hi Sanjoy! Glad it worked and thank you for the detailed explanation on your findings. I’ll investigate these and add to my report to Qwiklabs. Thanks again!