# GPU Overhead in

**URL:** <https://community.deeplearning.ai/t/gpu-overhead-in/892275>\
**Category:** Transformers in Practice\
**Tags:** week-module-3, course-topic\
**Created:** [May 18, 2026, 4:10am UTC](https://community.deeplearning.ai/t/gpu-overhead-in/892275 "2026-05-18T04:10:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lijie\_Tu](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/lijie_tu/32/126736_2.png) [@Lijie\_Tu](https://community.deeplearning.ai/u/Lijie_Tu)\
**Post date:** [May 18, 2026, 4:10am UTC](https://community.deeplearning.ai/t/gpu-overhead-in/892275/1 "2026-05-18T04:10:34Z")

</div>

The test actually shows the smallest model has the worst throughput

```auto
Throughput comparison:

   qwen-0.5b (0.5B): 10.6 tok/s | 100 tokens in 9.45s
✓ Unloaded Qwen/Qwen2.5-0.5B-Instruct
Loading Qwen/Qwen2.5-1.5B-Instruct...
✓ Loaded Qwen/Qwen2.5-1.5B-Instruct
   qwen-1.5b (1.5B): 66.5 tok/s | 100 tokens in 1.50s
✓ Unloaded Qwen/Qwen2.5-1.5B-Instruct
Loading Qwen/Qwen2.5-3B-Instruct...
✓ Loaded Qwen/Qwen2.5-3B-Instruct
     qwen-3b ( 3B): 53.4 tok/s | 100 tokens in 1.87s
✓ Unloaded Qwen/Qwen2.5-3B-Instruct
Loading Qwen/Qwen2.5-7B-Instruct...
✓ Loaded Qwen/Qwen2.5-7B-Instruct
     qwen-7b ( 7B): 67.7 tok/s | 100 tokens in 1.48s
✓ Unloaded Qwen/Qwen2.5-7B-Instruct
Loading Qwen/Qwen2.5-0.5B-Instruct...
✓ Loaded Qwen/Qwen2.5-0.5B-Instruct

```

That’s the opposite of what should happen. It’s the first model executed in the loop, so it absorbed all the cold-start costs — CUDA context init, lazy kernel compilation, cuBLAS/cuDNN autotuning, allocator warmup, HF `from_pretrained` finalization. Notice the timing: 9.45s for 100 tokens vs ~1.5s for the others. That’s ~8s of one-time overhead, not steady-state decode time.

---

<div class="post-metadata">

**Author:** ![reinoudbosch](https://avatars.discourse-cdn.com/v4/letter/r/22d042/32.png) [@reinoudbosch](https://community.deeplearning.ai/u/reinoudbosch)\
**Post date:** [May 18, 2026, 4:41pm UTC](https://community.deeplearning.ai/t/gpu-overhead-in/892275/2 "2026-05-18T16:41:49Z")

</div>

Hi Lijie\_Tu,

Nice catch. To resolve this, a dummy model should be loaded first so that the initialization has no effect on the steady-state decode time. I’ll pass this on.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/dlai/original/1X/7ffdac4759e95c8945939943bd85b8aca342d961.png) [@system](https://community.deeplearning.ai/u/system)\
**Post date:** [June 17, 2026, 4:10am UTC](https://community.deeplearning.ai/t/gpu-overhead-in/892275/4 "2026-06-17T04:10:43Z")

</div>

This topic was automatically closed after 30 days. New replies are no longer allowed.