Hello! Thank you very much for providing an interesting course about MlOps. I went through the topics, but I am confused how everything fits together for e2e ML Pipeline:
Lets say I have the following comopnents:
Apache Beam
Apache Spark
KFP Pipelines
TFX
Training Cluster
Serving cluster
I use apache beam+apache spark on N nodes to do data preprocessing. I use Training cluster to perform train on GPUs via Slurm, and kubernetes + docker for serving.
I can use TFX + KFP Pipelines to create my DAG. But how Do I specify where each individual node should be running? Afaik, KFP provides a logical view of the graph, but the real job is actually done somewhere else.
I see a lot of tutorial of using TFX+KFP/Airflow, but they do not cover the most important aspect: how do I translate logical view to physical ? Is there a really e2e example that shows how to run E2E pipeline with logical representation as well as physical?
By logical view I mean the DAG, by physical view I mean the mapping of a specific component to the set op nodes that it executes on, with their own environments. E.g. validation should be running on spark cluster, while training on another set of physical machines.