Data Pipeline vs ML Pipeline

Course Q&A Machine Learning Engineering for Production(MLOps) Introduction to Machine Learning in Production
1

I was of the belief that one must learn how to build Data Pipelines before they move on to building ML Pipelines, because I noticed we started building ML pipeline right from Week 1.

So far, I noticed that the course only talks about ML Pipelines, but why doesn’t it also talk about Data Pipelines?

Has it been discussed in a prerequisite course?

2

Hi @alhajee
Thank you for your suggestion.
It would be nice to be able to deepen the subject of data pipelines from the ground up before diving into the world of ML pipelines. However, I believe that this topic deserves a separate course that has not been planned at the moment. I believe that the approach followed by those who developed the course was to jump directly to the ML pipelines intended as a particular instance of Data Pipelines.
What are the more general topics that it would have been useful to present before jumping into the world of ML Pipelines?
Regards

2 Likes
3

Thank you @fabioantonini for your response.

I believe building data pipelines for use in production is not as straight forward as it seems because of the best practices that needs to be followed.

One of such best practices is choosing between Batch, Real-Time or One-Time Data Ingestion and Processing for the Data Pipeline.

Since production pipelines typically deal with a huge amount of data, it is not practical to ingest everything at once.

Unfortunately, this topic is not often discussed.

Additionally, most jobs openings I have seen always stress about being able to build ETL pipelines, which I believe is Data Pipeline.

4

@alhajee thanks a lot for your interesting contribution.
I can open an internal ticket to ask adding some more details about the topics you mentioned. Not sure it can be done shortly, but i agree with you that some more details could be helpful.
regards

2 Likes
5

Hi @alhajee
I have just realized that there is a specific chapter named ’ Batch Processing with ETL’ about ETL Pipelines in the Course 4 of the MLEP Specialization ‘Deploying Machine Learning Models in Production’.
Maybe it might be helpful to take a look.
Best regards
Fabio