TFX on Google Cloud Vertex Pipelines

Hi!

I have an issue when completing Lab Simple TFX Pipeline for Vertex Pipelines

After I return to Verex AI → Piplines I see
my pipeline running but it takes forever to complete CsvExampleGen step (first step). Looks like something stuck.

Do you have any idea how to fix this?
Screenshot 2022-06-14 at 14.41.16

1 Like

solution to previous question: patience :slight_smile: - I had to restart lab couple of times, but finally evrything is in place
now I see all the artifacts in storage

but check progress is still failing (waited for 30 min!)

something is 100% wrong with load on servers

solved: after 4th restart!!!
probably something was wrong with servers

Hi Nazarii! Welcome to the community and thank you for the updates! Will monitor this lab and tell our partners if this happens often. I’ve done the lab several times but did not run into the failing checkpoint. It usually works after the suggested wait time. Hopefully, there was just a glitch when you tried the lab. Thank you for being patient!

Hi @chris.favila hope you are well. I am also doing this course and having a problem. I closed the lab because the ‘vertex_pipelines_simple.ipynb’ script was stuck in the last cell running for about 30min, and no pipeline visualization were created at Vertex Pipelines, so I closed the Lab and it wont let me reopen another session. This is the error I get:

'Lab failed on Thu, 23 Jun 2022 10:22:07 -0400: Failed to launch lab resource project_0: 2:Exception calling application: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = “failed to connect to all addresses” debug_error_string = “{“created”:”@16559939… Any credits or tokens you used to start this lab have been refunded. ’

and
‘Sorry, cannot start Lab. Please contact your trainer or administrator.’

Do you have any idea how to reset it or should I wait and try tomorrow again?

Thank you Chris.

Update:
It seems there is a problem with google cloud at the moment, thats what the help center live chat replied, and they will update via email they said.

1 Like

I had the same issue yesterday. It is working for me this morning.

2 Likes

Hi Gent! Sorry to hear about the previous issue and glad it’s working now! It seems this lab is failing intermittently from time to time. Will report to our partners once I gather more info. In the meantime, please do contact the Qwiklabs Support channels in case you run into access/session issues. They will be more capable there because they can enable it themselves. But if you run into errors/timeouts related to the commands, you can continue reporting here on Discourse. Thank you!

1 Like

I got the same situation as @Nazarii_Nyzhnyk 's.
I tried for many times with exactly the same steps, and I successfully run the lab at the forth/fifth time…

same here. wait for 46mins to complete

I have the same problem. I have tried to complete this lab for a third time now. Pipeline runs indefinitely. This only lab blocks me currently from completing the specialization.

Hi everyone! We will report this to Qwiklabs so they can check what might be wrong. Thank you!

Just to let others know… I also tried several times running the pipeline from this lab and it passed successfully on the 5th time (without the need to modify the notebook, library versions etc.). So as someone suggested most of us may need patience for this lab… although it does not change the fact that qwiklabs team should look into the issue.