Transformers and Attention
Transformers process relationships among tokens using attention. This architecture made it practical to train models on long sequences at large scale.
Learning objectives
What you will be able to do
- Explain transformers and attention accurately in your own words.
- Recognise how the concept appears in real AI products and professional work.
- Identify an important limitation, risk or evaluation requirement.
- Apply the concept through a practical activity and knowledge check.
Core explanation
Build the right mental model
Attention calculates how strongly each token should consider other tokens when building a contextual representation. The word 'bank' should relate differently to 'river' than to 'loan'; attention helps represent that context.
Multiple attention heads can learn different relationships, while feed-forward layers transform each position. Positional information preserves order. Stacking many blocks creates increasingly rich representations.
Attention values are useful engineering signals but should not automatically be treated as a complete human-readable explanation of a model’s reasoning.
Self-attention
Tokens build contextual representations by weighting relationships with other tokens.
Attention head
One learned pattern of relationships within an attention layer.
Position
Information representing token order in a sequence.
Transformer block
Attention and feed-forward components repeated through the network.
Resolving meaning in a sentence
In 'The router dropped the packet because it was overloaded', attention helps connect 'it' with the router.
Apply your learning
Practical activity
- Write a sentence containing an ambiguous word.
- Circle the surrounding words required to determine its meaning.
- Sketch attention lines from the ambiguous word to those clues.
Write your answers in a learning journal. The value comes from connecting the concept to your own profession.
Knowledge check
Test your understanding
Q1What problem does self-attention address?
Answer: It helps each token incorporate relevant information from other tokens in the sequence.
Q2Does an attention map fully explain model reasoning?
Answer: No. It exposes part of model computation but is not a complete causal explanation.
If you cannot explain the answer without reading it, revisit the core explanation and example before continuing.
Continue with authoritative sources
Important industry resources
These links lead to official organisations, industry laboratories or established open-source learning projects.