# Understanding Transformer Network

**URL:** <https://community.deeplearning.ai/t/understanding-transformer-network/23671>\
**Category:** Sequence Models\
**Tags:** coursera-platform\
**Created:** [July 27, 2021, 4:02pm UTC](https://community.deeplearning.ai/t/understanding-transformer-network/23671 "2021-07-27T16:02:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![piyush23](https://avatars.discourse-cdn.com/v4/letter/p/e36b37/32.png) [@piyush23](https://community.deeplearning.ai/u/piyush23)\
**Post date:** [July 27, 2021, 4:02pm UTC](https://community.deeplearning.ai/t/understanding-transformer-network/23671/1 "2021-07-27T16:02:42Z")

</div>

Hi,

I was going through lecture videos on Transformer Network of week 4. The first part of the combination i.e. self-attention signifies enriching the relationship between words through query key value concept. I want to understand, doesn’t the **attention model** already do that? I mean how it adds value here. Given the example, we are already taking into account the contribution of **'visite** in **‘l’Afrique’** using a simple attention model by incorporating alpha\<t,t’\>. What does q, k, v do differently that alpha doesn’t?

---

<div class="post-metadata">

**Author:** ![edwardyu](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/edwardyu/32/82_2.png) [@edwardyu](https://community.deeplearning.ai/u/edwardyu)\
**Post date:** [July 29, 2021, 3:56am UTC](https://community.deeplearning.ai/t/understanding-transformer-network/23671/2 "2021-07-29T03:56:58Z")

</div>

Hi @piyush23 ,

I’d like to share my understanding below, and welcome any different ideas.  
“ **Attention** ” per se, the concept of the two is the same, but different implementation and application. The attention model in week 3 is **additive attention**. Instead, the Transformer in week 4 uses **scaled dot-product attention**. The [Attention Is All You Need](https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf) paper also compares these two implements.  
In fact, you can think of attention model in the same way as q, k, v. For instance, alpha\<t, t’\> is calculated by q (i.e., S) and k (i.e., a\<t’\>), and then taking the attention weights (i.e., alpha\<t, t’\>) to multiply v (i.e., a\<t’\>).  
Of course, just as you know, apart from attention mechanism, the NMT architecture in week 3, 4 is different.