Combining data parallelism with model parallelism

Week 1 quiz had a question about combining data parallelism with model parallelism. I don’t remember seeing this in any of the lecture videos of week 1. Where was this covered? Is this from the optional video (if so it shouldn’t be optional) or any linked reading materials?

Dear @kmnajih,

Welcome to the Community!

May I confirm if your concern has been resolved, or if you still require assistance?

–
Keep Learning AI with DeepLearning.AI - Girijesh

The course video Efficient multi-GPU compute strategies talks about different techniques to efficiently train large models.
Amongst these techniques, ZeRO (Zero Redundancy Optimizer) in addition to Data Parallelism across GPUs also shards Model parameters, gradients and optimizer states across GPUs thereby reducing redundant memory consumption on each GPU