Framework agnostic ML Pipelines

Does TFX has a dependancy on TF? What if the devs are developing in other frameworks such as Torch?

How should we achieve ML pipelining that is agnostic to frameworks? Also how should we transit from dev pipeline to prod pipeline, should we even have a dev pipeline in the first place?

Hi @jax79sg

This is really a BIG question.
I can bring you my personal experience: mostly I have worked in the past with TF, then since one year I have started to work more and more with PyTorch.
TFX basically has dependencies on Apache Beam.
But, what I think is the most TF part important to know is tf.data and especially TFRecord file format.
It should possible to work limiting the code written in TF, but I would say that if you want to use it, definitively a TF knowledge is useful.

There are many pieces in the MLOps spaces that can be used without TF. For example:

  • MLflow, for experiment tracking and (some) project management
  • Apache Airflow, that enables you to design pipelines as DAG, in Python

TF has a wider ecosystem that PyTorch, but I understand why people don’t want to use it or prefer PyTorch.

I understand that this is not the definitive answer, but hope other people will add their thought to the conversation.

Hi, thanks for the response. Personally i think this whole area is still emerging. Besides the big players, who focused on the framework they are supporting (E.g. Google for Tensorflow). There are so many startups out there striving to solve the same problem. I would be keen to focus on those that are more framework agnostic as my organisation uses both, and more. ML Flow and Apache Airflow that you mentioned is definitely worth a look.

Hi @jax79sg
yes, it is definitively emerging, toolkits are maturing.
That’s the reason why, I think, Coursera has decided to start this specialization.
You, we, will learn some more. There will be new things in the future, we don’t have all the answers today, but we will have the cultural basis for mastering what will happen next.

Happy learning