Tokens, Context and Embeddings
Tokens are the model’s working units, context is the information available for the current response, and embeddings are numerical representations that place related meaning near each other.
Learning objectives
What you will be able to do
- Explain tokens, context and embeddings accurately in your own words.
- Recognise how the concept appears in real AI products and professional work.
- Identify an important limitation, risk or evaluation requirement.
- Apply the concept through a practical activity and knowledge check.
Core explanation
Build the right mental model
A token can be a word, part of a word, punctuation or code fragment. Models process token sequences, so token count affects context capacity, latency and cost.
The context window contains system instructions, conversation history, supplied documents and generated tokens. Information outside the window is not automatically available. More context is not always better: irrelevant material can distract the model.
Embeddings convert text or other data into vectors. Similar meanings tend to have nearby vectors, enabling semantic search, recommendation, clustering and retrieval-augmented generation.
Keep three different concepts separate
Tokenisation converts text into units a model can process. A token may correspond to a whole word, a word fragment, punctuation or another pattern, depending on the tokenizer. The same text can have different token counts for different models. Do not promise that a fixed number of words always fits a context window, especially across languages, code and unusual symbols.
Context is the information supplied for a particular response. It may include instructions, conversation messages, retrieved passages and space for output. An embedding is a numerical representation used to compare or process items. Token count measures neither truth nor semantic similarity. A short sentence can contain a wrong claim, while two differently worded passages can express similar ideas.
Budget context deliberately
Treat a context limit as a shared capacity, not an invitation to paste everything. Reserve space for instructions and the response, then choose relevant source material. A larger source collection often needs retrieval or summarisation rather than indiscriminate concatenation. When summarising, retain document identifiers and important qualifications so later answers can be checked.
The following budget uses invented limits, not specifications for any product. Suppose a system allows 8,000 total tokens for the input-and-output combination used here. Instructions use 700, conversation history 1,300, and the response reserve is 1,500. This leaves 4,500 for source passages. A 5,000-token attachment would exceed that plan even though it is smaller than the headline limit. Actual providers may apply separate limits, so check the chosen model’s documentation.
Use embeddings to find candidates, then inspect evidence
In a semantic search workflow, documents are divided into passages, each passage is represented numerically, and a query is represented using a compatible embedding system. A similarity comparison retrieves candidate passages. The retrieved passage is then read or supplied to a response generator. Search ranking is a relevance estimate, not a truth certificate.
Preparation changes the result. A passage that separates a rule from its exception can be misleading even when it contains the right keywords. Keeping a heading, date and nearby qualification with the passage helps interpretation. Use exact matching for identifiers when necessary: a course code or product version may be important even if another passage sounds semantically similar.
Token
A unit of text processed by a language model.
Context window
The limited amount of information available during one generation.
Embedding
A vector representation capturing useful relationships or similarity.
Semantic search
Retrieval based on meaning rather than exact keyword matching.
Worked example: retrieval for an assessment question
Fictional question: “Can I redo an unsuccessful practical assessment?” Candidate A says, “Students may request a reassessment after an unsuccessful practical, subject to approval.” Candidate B says, “Practice exercises may be repeated without limit.” Candidate C says, “Final assessment dates are published on the timetable.” A should be inspected first because it addresses reassessment; B is about practice, which is a different entitlement.
For an arithmetic illustration of vector comparison, use a unit query vector (1, 0), candidate A (0.8, 0.6) and candidate B (0, 1). Both candidates have length one. Their cosine similarities to the query are 0.8 and 0 respectively. These invented two-dimensional numbers show the calculation; real embedding coordinates are learned and should not be interpreted as named meaning axes.
A response must preserve “may request” and “subject to approval”. Changing this to “you are automatically entitled to redo it” would be a generation error after successful retrieval. The complete workflow needs two checks: retrieve the relevant evidence and accurately express its conditions.
Apply your learning
Practical activity
- Spend 5 minutes making four fictional policy cards: reassessment, practice, attendance and equipment loans. Give each a heading and one exception.
- Spend 5 minutes writing three queries with different wording from the cards. Predict the most relevant card and identify a plausible but wrong match.
- Spend 5 minutes applying the invented 8,000-token budget above. Reduce the response reserve to 1,000 and recalculate the available passage space.
- Spend 5 minutes drafting a grounded answer to one query and checking that its exception remains visible.
Compare with an explained example answer
With the response reserve reduced to 1,000, the passage budget becomes 8,000 − 700 − 1,300 − 1,000 = 5,000 tokens. The arithmetic does not establish a real model limit; it illustrates capacity planning.
For “Can I try the practical again?”, choose the reassessment card rather than the practice card and preserve the approval condition. The wrong match is useful evidence for improving retrieval queries or passage organisation, not a reason to assume all semantic search fails.
Write your answers in a learning journal. The value comes from connecting the concept to your own profession.
Knowledge check
Test your understanding
Q1Why is a token limit not a fixed word limit?
Answer: Tokenisation depends on the model and text. Words, punctuation, language and code can be segmented differently.
Q2Does the highest similarity score prove a passage is authoritative?
Answer: No. It ranks candidates according to a representation and metric. Authority, date, scope and the actual supporting clause require inspection.
Q3Why keep a rule and its exception together when making passages?
Answer: Retrieving only the rule can cause an answer to overstate permission. Preserving context reduces that particular failure mode.
If you cannot explain the answer without reading it, revisit the core explanation and example before continuing.
Continue with authoritative sources
Important industry resources
These links lead to official organisations, industry laboratories or established open-source learning projects.