# Combining data parallelism with model parallelism

**URL:** <https://community.deeplearning.ai/t/combining-data-parallelism-with-model-parallelism/531272>\
**Category:** Generative AI with Large Language Models\
**Tags:** quiz-help, week-module-1\
**Created:** [December 31, 2023, 12:54am UTC](https://community.deeplearning.ai/t/combining-data-parallelism-with-model-parallelism/531272 "2023-12-31T00:54:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmnajih](https://avatars.discourse-cdn.com/v4/letter/k/839c29/32.png) [@kmnajih](https://community.deeplearning.ai/u/kmnajih)\
**Post date:** [December 31, 2023, 12:54am UTC](https://community.deeplearning.ai/t/combining-data-parallelism-with-model-parallelism/531272/1 "2023-12-31T00:54:56Z")

</div>

Week 1 quiz had a question about combining data parallelism with model parallelism. I don’t remember seeing this in any of the lecture videos of week 1. Where was this covered? Is this from the optional video (if so it shouldn’t be optional) or any linked reading materials?

---

<div class="post-metadata">

**Author:** ![Girijesh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/girijesh/32/37283_2.png) [@Girijesh](https://community.deeplearning.ai/u/Girijesh)\
**Post date:** [July 30, 2025, 4:48pm UTC](https://community.deeplearning.ai/t/combining-data-parallelism-with-model-parallelism/531272/3 "2025-07-30T16:48:48Z")

</div>

Dear @kmnajih,

Welcome to the Community!

May I confirm if your concern has been resolved, or if you still require assistance?

–  
_Keep Learning AI with DeepLearning.AI_ - Girijesh

---

<div class="post-metadata">

**Author:** ![dhawalkapil](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/dhawalkapil/32/76303_2.png) [@dhawalkapil](https://community.deeplearning.ai/u/dhawalkapil)\
**Post date:** [July 30, 2025, 8:29pm UTC](https://community.deeplearning.ai/t/combining-data-parallelism-with-model-parallelism/531272/4 "2025-07-30T20:29:57Z")

</div>

The course video **[Efficient multi-GPU compute strategies](https://www.coursera.org/learn/generative-ai-with-llms/lecture/e8hbI/optional-video-efficient-multi-gpu-compute-strategies)** talks about different techniques to efficiently train large models.  
Amongst these techniques, ZeRO (Zero Redundancy Optimizer) in addition to Data Parallelism across GPUs also shards Model parameters, gradients and optimizer states across GPUs thereby reducing redundant memory consumption on each GPU
