Week3: Graded External Tool: TFX on Google Cloud AI Platform Pipelines

Course Q&A Machine Learning Engineering for Production(MLOps) Deploying Machine Learning Models in Production
1

Hello,
I’m facing an issue with my lab. I have already deployed Kubeflow Pipeline but when I click on Check my progress button I receive this message:
Please deploy Kubeflow Pipelines with name ‘kubeflow-pipelines-1’. If already deployed, then wait until the pipeline completes.

Even after 30 minutes I have deployed, I got the same message. I want to know how to fix this issue.
Thanks.

3

Check Week3 2 here - This solved the same issue on my side

4

Even after enabling the checkbox, I am still getting the same error on Check my progress. I see the Pipeline is created in the convention given every time. but getting into same issue and unable to proceed. Can anyone advise further ?

5

Hi Suresh! If you’re sure that you’ve followed all instructions correctly (and you’re using the credentials provided by Qwiklabs as mentioned here), you can file a ticket through the Qwiklabs support channels so they can see what might be going on with your account. The support channels can be found on the upper right of the Qwiklabs page:

qwiksupport
qwiksupport852×806 34.2 KB

The chat support is usually responsive. You can describe the issue in detail and please be ready with some screenshots because they usually ask for that. If they also can’t help, feel free to repost here so we can escalate further. Thank you!

6

Thanks Chris. Will do. One clarification through. What exactly is clearing means in lab_excercise.ipynb below ?

Clear all the cells in the notebook (look for the Clear button on the notebook toolbar) and then Run the cells one by one.

7

Hi Suresh! I think that’s just a typo and I will report it to the Qwiklabs team. Tha lab is hosted on their end so we can’t make revisions directly. You can just disregard that “Clear button” and run the cells inside the notebook. Thanks for the reminder!

8

Chris, I reattempted the lab to capture the screenshots as requested by Qwiklabs support team. It worked properly this time. Not sure what happened earlier. Thanks for your help.

Regards,
Suresh

2 Likes
9

Stucked in the last step with error

CLI
Creating a run for pipeline: tfx_covertype_continuous_training
Detected Kubeflow.
Use --engine flag if you intend to use a different orchestrator.
Traceback (most recent call last):
File “/home/jupyter/.local/bin/tfx”, line 8, in
sys.exit(cli_group())
File “/home/jupyter/.local/lib/python3.7/site-packages/click/core.py”, line 829, in call
return self.main(*args, **kwargs)
File “/home/jupyter/.local/lib/python3.7/site-packages/click/core.py”, line 782, in main
rv = self.invoke(ctx)

OOOOO

File “/home/jupyter/.local/lib/python3.7/site-packages/kfp_server_api/rest.py”, line 224, in request
raise ApiException(http_resp=r)
kfp_server_api.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({‘Content-Type’: ‘application/json’, ‘Date’: ‘Fri, 29 Oct 2021 13:23:22 GMT’, ‘Vary’: ‘Origin’, ‘X-Content-Type-Options’: ‘nosniff’, ‘X-Frame-Options’: ‘SAMEORIGIN’, ‘X-Powered-By’: ‘Express’, ‘X-Xss-Protection’: ‘0’, ‘Transfer-Encoding’: ‘chunked’, ‘Set-Cookie’: ‘S=cloud_datalab_tunnel=u8MI_rb7nZrupzyW6F-k0nqGJiWeu9VgJOJanu0cTH0; Path=/; Max-Age=3600’})
HTTP response body: {“error”:“Failed to create a new run.: Failed to fetch workflow spec.: Invalid input error: Please provide a valid pipeline spec”,“message”:“Failed to create a new run.: Failed to fetch workflow spec.: Invalid input error: Please provide a valid pipeline spec”,“code”:3,“details”:[{“@type”:“type.googleapis.com/api.Error",“error_message”:"Please provide a valid pipeline spec”,“error_details”:“Failed to create a new run.: Failed to fetch workflow spec.: Invalid input error: Please provide a valid pipeline spec”}]}

10

Hi Bruno! Did you modify anything else in the notebook? In this exercise, you’re only required to modify the ENDPOINT variable. You will need to copy the URL in the host parameter as shown below and use that as the endpoint:

endpoint_tfxc4w3lab1
endpoint_tfxc4w3lab11300×1010 87.4 KB

RUN_ID should also be replaced but that part is not graded.

You should be able to run the rest of the cells as is. Hope this helps!

11 
raise ApiException(http_resp=r)

kfp_server_api.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({‘Content-Type’: ‘application/json’, ‘Date’: ‘Mon, 01 Nov 2021 11:29:01 GMT’, ‘Vary’: ‘Origin’, ‘X-Content-Type-Options’: ‘nosniff’, ‘X-Frame-Options’: ‘SAMEORIGIN’, ‘X-Powered-By’: ‘Express’, ‘X-Xss-Protection’: ‘0’, ‘Transfer-Encoding’: ‘chunked’, ‘Set-Cookie’: ‘S=cloud_datalab_tunnel=W_y4xFKBX5-RiPGJPKqhzN0YeXIXsrT-TZClhYQwnso; Path=/; Max-Age=3600’})
HTTP response body: {“error”:“Failed to create a new run.: Failed to fetch workflow spec.: Invalid input error: Please provide a valid pipeline spec”,“message”:“Failed to create a new run.: Failed to fetch workflow spec.: Invalid input error: Please provide a valid pipeline spec”,“code”:3,“details”:[{"@type":“type.googleapis.com/api.Error",“error_message”:"Please provide a valid pipeline spec”,“error_details”:“Failed to create a new run.: Failed to fetch workflow spec.: Invalid input error: Please provide a valid pipeline spec”}]}

12

Same error , I’ve tried different version of kfp , tfx == 0.25, endpoint with or without http / https…

13

Hi @chris.favila got “sorry your quotas has been exceeded for this lab”
Best Regards
Bruno