C2_W3 Data Engineering - Exercise 6

Link to the classroom item you are referring to: https://www.coursera.org/learn/source-systems-data-ingestion-and-pipelines/programming/w1wyQ/assignment-3-testing-data-quality-with-great-expectations/submission

IMO, in exercise 6 checkpoint folder is not created in artifacts bucket/checkpoints/generated checkpoint.

The generated checkpoint is created a level higher:
artifacts bucket/generated checkpoint
So it is missing one level

A validator looks after:
artifacts bucket/checkpoints/generated checkpoint

I have create artifacts bucket/checkpoints/
to bypass the validator

But the files generated are one level higher. The one selected on lefy hand side of screen shot

Hello @konutech,
Could you check in the great_expectations .yml you are using the correct prefix (checkpoints/) for the checkpoint store:

  checkpoint_store:
    class_name: CheckpointStore
      ...
      prefix: checkpoints/ <------ADD THIS

For the last part you need to add files in docs to be graded:
checkpoint_result = checkpoint.run()
and finally:
context.build_data_docs()

Thank you kindly Georgios for the reply.
That might be a solution, that my be the case.

Let’s keep it for other students. I have already closed the lab related to Great Expectations.

Now I have similiar problem with .parquet files of C2W4. My DAGs were generated fine so I got 60% in total.

However, the validator states I have no files in my bucket when in reality I do have them. I did a sync earlier:

I know it is a diffrent topic. Sorry for putting it here. It is just informative

Wrong bucket :slight_smile: there are two buckets

Hello @konutech,
Did you run this command in the correct raw data bucket, thanks:

cd data
aws s3 sync work_zone s3://<RAW-DATA-BUCKET>/work_zone/
cd ..
1 Like

@konutech,
You should be able to see the dags now in airflow. Before there were not data in the raw data bucket to process. The dags bucket is only to save the dag files. Cheers

1 Like

I passed the Assignment. Thank you Georgios for quick help. Cheers!

1 Like