Tokens, Context and Embeddings
Tokens are the model’s working units, context is the information available for the current response, and embeddings are numerical representations that place related meaning near each other.
Learning objectives
What you will be able to do
- Explain tokens, context and embeddings accurately in your own words.
- Recognise how the concept appears in real AI products and professional work.
- Identify an important limitation, risk or evaluation requirement.
- Apply the concept through a practical activity and knowledge check.
Core explanation
Build the right mental model
A token can be a word, part of a word, punctuation or code fragment. Models process token sequences, so token count affects context capacity, latency and cost.
The context window contains system instructions, conversation history, supplied documents and generated tokens. Information outside the window is not automatically available. More context is not always better: irrelevant material can distract the model.
Embeddings convert text or other data into vectors. Similar meanings tend to have nearby vectors, enabling semantic search, recommendation, clustering and retrieval-augmented generation.
Token
A unit of text processed by a language model.
Context window
The limited amount of information available during one generation.
Embedding
A vector representation capturing useful relationships or similarity.
Semantic search
Retrieval based on meaning rather than exact keyword matching.
Searching university regulations
Each paragraph is embedded; a student question is embedded; the closest paragraphs are retrieved for an answer.
Apply your learning
Practical activity
- Compare keyword search and semantic search for the query 'rules for repeating a failed assessment'.
- List useful exact keywords and related phrases an embedding should connect.
Write your answers in a learning journal. The value comes from connecting the concept to your own profession.
Knowledge check
Test your understanding
Q1Is one token always one word?
Answer: No. Tokens may be words, word pieces, punctuation or other fragments.
Q2What do embeddings enable?
Answer: Tasks based on similarity and meaning, including semantic search and retrieval.
If you cannot explain the answer without reading it, revisit the core explanation and example before continuing.
Continue with authoritative sources
Important industry resources
These links lead to official organisations, industry laboratories or established open-source learning projects.