I found that the Encoder Block mentioned in PyTorch certificate course 3 and module 3, is different from the Encoder Block introduced in Lilian Weng’s Blog. The link to that Blog is here: Attention? Attention! | Lil'Log .
Does the order of LayerNorm, Multi-Head Attention, Residual Connection, and Feed Forward Network important? I suppose it does.
I attached two images here, the first one is Encoder Block illustration from Lilian Weng’s blog, and the second image is from PyTorch Course 3 Module 3. The order of LayerNorm is different in these two images.
Both orderings are valid, but they are not the same thing.
Lilian Weng’s diagram (and the original Vaswani et al. 2017 paper) uses what’s called Post-Norm: the sub-layer runs first, then the residual is added, then LayerNorm is applied.
The course slide uses Pre-Norm: LayerNorm is applied before feeding into the sub-layer, and the residual is added after.
\text{output} = x + \text{SubLayer}(\text{LayerNorm}(x))
You can see this in the code on the slide: x_norm = self.ln1(x) happens first, then x = x + attn_out.
So yes, the order matters. The key practical differences:
Post-Norm (original paper): needs careful learning-rate warmup to train stably. Without it, gradients in the early layers can explode.
Pre-Norm: training is more stable out of the box, often doesn’t need warmup. This is why most modern implementations (GPT-2 onward) default to Pre-Norm.
In terms of final performance, results are comparable when both are tuned properly. Pre-Norm just makes your life easier during training.
The course is teaching the Pre-Norm variant because it’s the more commonly used pattern in practice today. Lilian Weng’s blog is faithfully illustrating the original 2017 architecture. Neither is “wrong”; they’re two well-studied design choices.
