Is TFX Too Much for non-TF Applications?

We’re building out a product that requires training about 10GB total of data for about 200 models that we train each day. It’s time series. One model for each customer-category combination. We’re using GBM and XGBoost and maybe RF in an ensemble. It’s a lot to keep track of.

TFX is mentioned in C2_W2 for orchestration. Is TFX “too much,” too heavy weight to use for non-TF use cases? Does it play well in a heterogeneous environment that might include MLFlow and data observability tools?

Are there other orchestration tools that might be up for consideration for an environment with many small models trained daily?

Have you seen airflow ?

Yeah, I should look at Airflow. Thank you for mentioning it.