Multimodal AI Systems
Multimodal AI works across more than one form of information—such as text, images, audio and video—within one system or coordinated workflow.
Learning objectives
What you will be able to do
- Explain multimodal ai systems accurately in your own words.
- Recognise how the concept appears in real AI products and professional work.
- Identify an important limitation, risk or evaluation requirement.
- Apply the concept through a practical activity and knowledge check.
Core explanation
Build the right mental model
A multimodal model can describe an image, answer questions about a chart, transcribe speech, compare documents with photographs or generate images from text. Representations from different modalities are aligned so the system can connect them.
Multimodal capability creates new accessibility and productivity benefits, but also new errors. Small text in an image, unusual accents, visual ambiguity and missing context can cause confident mistakes.
Evaluation must reflect each modality and the complete task. A model that transcribes well may still summarise poorly; a model that recognises objects may misunderstand the relationship among them.
Map the information before choosing the model
A multimodal task uses more than one kind of information: for example, a chart image plus a written question, or speech plus a transcript and meeting agenda. Start by asking what each input contributes. The chart may show a trend, the caption may identify the unit, and the question may specify which comparison matters. If the unit is missing from every input, the system should not invent it.
Not every system accepts or produces every modality. An image-and-text model may generate text but not audio; a speech pipeline may combine several specialised components. Describe the actual input and output path instead of labelling an application “multimodal” and assuming universal capabilities. File types, image resolution, audio quality and available context all affect the task.
Separate extraction, interpretation and communication
For a chart task, extraction identifies labels and values, interpretation compares them, and communication explains the result. A wrong answer can originate at any stage. If the image is read as 80 instead of 30, the later arithmetic may be correct for the wrong numbers. If extraction is correct but the model confuses percent with percentage points, the interpretation fails. These errors need different fixes.
Ask for an intermediate transcription when exact values matter. Compare it with the source before requesting conclusions. For small or obscured text, obtain a clearer original or a data table. Repeatedly asking a model to guess a blurred label is not a verification method. Keep uncertainty visible, particularly when the input cannot support an exact number.
Provide an accessible result, not just another image
A useful chart description states the purpose, axes, units and important relationships. A short alternative text may identify the chart, while a longer description or data table can carry the detailed information. W3C guidance on complex images emphasises conveying their essential content in text. An attractive screenshot alone is not an accessible substitute for the underlying values.
Protect people represented in images and recordings. Remove unneeded personal details from a practice example, and use fictional data where possible. An image can contain names, faces, account numbers or location clues that are easy to overlook. Do not infer a person’s ability, intent or health from an image just because a model offers an interpretation.
Modality
A form of information such as text, vision, audio or sensor data.
Alignment
Learning connections between representations from different modalities.
Fusion
Combining information from several modalities for one output.
Accessibility
Using cross-modal conversion to support different user needs.
Worked example: a chart description with a unit trap
Imagine a fictional bar chart labelled “Workshop completion rate”. Its three bars are Week 1: 40%, Week 2: 55%, and Week 3: 70%. The vertical axis is percent and the horizontal axis is week. First transcribe those labels and values. Then calculate the difference between Week 1 and Week 3: 70 − 40 = 30 percentage points.
A draft description saying “completion increased by 30%” is ambiguous or incorrect if intended as relative growth. Relative growth is (70 − 40) ÷ 40 × 100 = 75%. A clear description is: “The completion rate rises from 40% in Week 1 to 70% in Week 3, an increase of 30 percentage points.” This communicates the direct chart comparison without implying a cause.
The chart does not state how many learners participated or whether the same learners were counted each week. It therefore cannot establish that a teaching intervention caused the increase. A complete accessible version would include the three-value table and these limitations, not just “an upward trend”.
If the Week 2 label is unreadable, preserve Week 1 and Week 3 where legible and mark Week 2 as unreadable. Do not fill in 55% from an imagined smooth trend. In this exercise 55% is explicitly supplied; in a real image it would need to be observed or obtained from source data.
Apply your learning
Practical activity
- Spend 5 minutes drawing a fictional three-bar chart with values 20%, 35% and 50%. Include title, axes and units.
- Spend 5 minutes writing a transcription and a concise description. You can compare it with an AI draft if an approved tool is available.
- Spend 5 minutes checking the difference in percentage points and relative percentage change.
- Spend 5 minutes writing a text alternative and listing two conclusions the chart cannot support.
Compare with an explained example answer
The first-to-last difference is 50 − 20 = 30 percentage points. The relative increase is 30 ÷ 20 × 100 = 150%. These express different comparisons; neither should silently replace the other.
An appropriate description states the three values and the increase while avoiding a causal claim. The chart alone does not reveal sample size or why the rates changed. A text table makes the exact values available to someone who cannot inspect the drawing.
Write your answers in a learning journal. The value comes from connecting the concept to your own profession.
Knowledge check
Test your understanding
Q1Why inspect an extracted table before accepting a chart explanation?
Answer: A transcription error can propagate into otherwise correct arithmetic. Checking extraction isolates the failure before interpretation.
Q2What is the difference between 30 percentage points and 30 percent growth?
Answer: Percentage points subtract two percentages. Relative growth divides the change by the starting value before converting to percent.
Q3What should happen when a chart label cannot be read?
Answer: Mark it uncertain or unreadable and seek a better source. A plausible guessed value is not evidence.
If you cannot explain the answer without reading it, revisit the core explanation and example before continuing.
Continue with authoritative sources
Important industry resources
These links lead to official organisations, industry laboratories or established open-source learning projects.