How to perform Model Parallel in Tensorflow?

Stav28 · January 28, 2024, 4:28pm

We have Mirror Strategy which uses Data Parallel. Is Model parallel available in TF? What about ZeRO Parallel technique?

TMosh · January 28, 2024, 5:55pm

I found this TensorFlow forum discussion by using an internet search. It may answer your question.

TMosh · January 28, 2024, 5:56pm

TensorFlow seems to prefer data parallelism.

Topic		Replies	Views
Distributed training - how exactly is model "split up" while using model parallelism? Optimize ML Models and Deploy Human-in-the-Loop Pi	3	571	July 23, 2021
C4W1 the second program Convolutional Neural Networks	1	496	August 5, 2022
About familiarizing with Keras models used in the Deep learning course Sequence Models general	2	11	July 8, 2024
Alternative way of defining model architecture Convolutional Neural Networks in TensorFlow week-3	3	354	December 2, 2023
C3w3 OneDeviceStrategy Machine Learning Modeling Pipelines in Production	6	585	July 11, 2021