Apache Beam vs TFX Transform :muscle:t4:

Course Q&A Machine Learning Engineering for Production(MLOps) Machine Learning Data Lifecycle in Production
1

I am new to this and would really appreciate some clarification. Is there a specific occasion when the TFX Transform component is preferable to Apache Beam? I notice that they are often used together in some Labs, but I am a bit confused about implementing either outside the Lab environment. Should I choose one over the other? What do they have in common?

1 Like
2

Hi @cpohagwu great question!!

I have more experience working with TFX transform, mainly beacuse I tend to focus more on ML portion of the ML life cycle, apache beam is a more general purpose platform and it can help to develop things in the broader expectrum, such as data engineering task.

My suggestion is keep using both and evaluate the difference between one and the other, usually there are preference on tools based on how much you learn and how well it integrates with your data.

Short answer:
TFX: ML oriented specially with TF pipelines
APACHE BEAN: general purpose platform, handling data processing tasks

I hope this helps!

2 Likes
3

Hi @pastorsoto , I really appreciate your response.

What I have noticed, at least in the Labs so far, is that the TFX Transform component is often used with other TFX components (ExampleGen, StatisticsGen, etc.) and with data that doesn’t require as much cleaning/filtering. I am wondering if there is an equivalent of these functions with tf.Transform(). For example, before preprocessing, I would like to convert some features to a specific DataType before developing a schema and also filter out some lines that don’t meet specific criteria. Is there an alternative way to do this using TFX components only?

4

Hi @cpohagwu yes! you could create the whole pipeline with only TFX components but it might be a bit challenging. I haven’t try but for sure is possible

1 Like