I want to understand real time analog example of Transformer vs Attention

[Sharad_Saxena1]

This is my first post, wanna say Hi you all, lets discuss and build.

[Mark_Nelson]

Hi Sharad,

Attention is a mechanism - its a way of computing which other tokens to focus on, and how much weight to place on each one.

The transformer is the architecture of the full model/system - that is built by stacking up layers of attention “heads”, feed-forward networks, normalization and so on, repeated many times, with an embedding layer at the start, and some task-specific head/layer at the end.

Image you are in a meeting with ten people, and you are trying to form an opinion or make a decision.

Attention would be you weighing in your mind how much each other person’s comments matters to what you are about to say. You might give a lot more weight to a detailed technical comment from an engineer, and less to a passing joke from someone else. That process of weighing and blending/weighting - that’s attention.

The entire meeting process, run in multiple rounds - pople talking, responding, asking questions, updating their opinions based on new information, someone summarizing, etc. - that whole process is like the Transformer.

Hope this helps!

-Mark