Evaluating AI System Quality
AI quality is multidimensional. Accuracy matters, but so do reliability, fairness, safety, latency, cost, security and usefulness in the real workflow.
Learning objectives
What you will be able to do
- Explain evaluating ai system quality accurately in your own words.
- Recognise how the concept appears in real AI products and professional work.
- Identify an important limitation, risk or evaluation requirement.
- Apply the concept through a practical activity and knowledge check.
Core explanation
Build the right mental model
Start with a baseline and a test set representing intended users and conditions. Select task metrics: precision and recall for classification, numerical error for regression, retrieval quality for search, or rubric-based scoring for generated content.
Generative systems require evaluation sets with realistic prompts, expected evidence, prohibited behaviour and edge cases. Automated scoring can help, but expert human review remains important.
Evaluation continues after release. Monitor data drift, failure reports, subgroup performance and operational outcomes. A model can remain technically available while its real-world quality declines.
Define success before inspecting a score
A quality evaluation begins with the intended use, the population of cases and the consequences of mistakes. “The model is 95% accurate” is incomplete without knowing which task, dataset and threshold produced that number. A model used to suggest a draft may have a different acceptance standard from one that takes an irreversible action. The same model can be suitable in one workflow and unsuitable in another.
Write an evaluation plan before choosing the final system. Include normal cases, difficult cases and situations where the system should decline to answer. Use representative examples that were not used to tune the prompt or model. If every disappointing test becomes a development example, maintain a separate held-back set for the eventual assessment.
Use more than one lens
For a classifier, a confusion matrix separates true positives, false positives, false negatives and true negatives. Precision asks what fraction of positive predictions were correct; recall asks what fraction of actual positives were found. Accuracy measures overall agreement. These quantities answer different questions and can move differently when a decision threshold changes.
For a text assistant, define a rubric that refers to observable evidence: supported factual claims, required information retained, readable wording and compliance with exclusions. A single subjective “good” score hides disagreements. Keep examples of acceptable and unacceptable outputs so reviewers can apply the criteria consistently. Human evaluation also needs clear instructions and a process for resolving disagreements.
Evaluate the complete system and its failure modes
Quality includes latency, cost, data handling, accessibility and the user’s ability to correct a mistake. A factual answer delivered after a deadline can still fail the task. A model can be accurate on average while performing poorly for a particular language or input format. Examine relevant subgroups and edge cases without treating small samples as precise population estimates.
An evaluation is evidence under stated conditions, not a permanent certificate. Inputs, tools, prompts and model versions can change. Record those conditions and repeat relevant checks after a material change. Monitor real failures through an appropriate feedback process and define when a system should be paused or rolled back. NIST’s risk-management approach places measurement within an ongoing management process.
Baseline
A simple existing method against which improvement is measured.
Task metric
A numerical measure linked to the intended capability.
Robustness
Stable performance under variation, noise or difficult conditions.
Monitoring
Ongoing measurement after deployment.
Worked example: why high accuracy can hide missed cases
Use 100 fictional support messages. Ten truly need urgent review and 90 do not. A system flags eight messages: six are genuinely urgent and two are not. It misses four urgent messages. The counts are TP = 6, FP = 2, FN = 4 and TN = 88; they total 100.
Accuracy = (6 + 88) ÷ 100 = 94%. Precision = 6 ÷ (6 + 2) = 75%. Recall = 6 ÷ (6 + 4) = 60%. The system gets most cases right overall, yet misses four out of ten urgent cases. Whether that is acceptable depends on the review process and the cost of missing urgency.
A baseline that never flags a message obtains 90% accuracy because 90 messages are not urgent, but it finds none of the urgent cases. Comparing only 94% with 90% conceals the operational question. Examine missed urgent cases and false alarms, and consider whether a human review queue can handle a different threshold.
These are invented counts for learning, not a tested model or a recommended threshold. A real study would need enough representative cases, reliable labels and uncertainty analysis before operational claims could be justified.
Apply your learning
Practical activity
- Spend 5 minutes drawing a confusion matrix for 200 fictional items: 20 actual positives, 15 true positives and 5 false positives. Fill the remaining cells.
- Spend 5 minutes calculating accuracy, precision and recall.
- Spend 5 minutes writing a three-criterion rubric for a fictional FAQ answer, including an “insufficient information” case.
- Spend 5 minutes writing a release decision that names one missing piece of evidence rather than treating the small exercise as certification.
Compare with an explained example answer
FN = 20 − 15 = 5. There are 180 actual negatives, so TN = 180 − 5 = 175. Accuracy = 190 ÷ 200 = 95%; precision = 15 ÷ 20 = 75%; recall = 15 ÷ 20 = 75%. Equal precision and recall here result from the chosen counts, not a general rule.
A useful FAQ rubric checks that claims match the source, required conditions are retained, and missing information is acknowledged. A defensible release note could say the arithmetic is checked but representative cases and reviewer agreement are still required. This states the boundary of the evidence without inventing validation.
Write your answers in a learning journal. The value comes from connecting the concept to your own profession.
Knowledge check
Test your understanding
Q1Why can a high-accuracy model miss many important cases?
Answer: If important cases are rare, correctly handling the majority dominates accuracy. Recall makes missed positives visible.
Q2What is the denominator of precision?
Answer: All predicted positives: true positives plus false positives. Recall instead uses all actual positives.
Q3Why re-evaluate after changing a prompt or tool?
Answer: The system’s behaviour may change even if the underlying model stays the same. Previous results apply to the tested configuration, not every future configuration.
If you cannot explain the answer without reading it, revisit the core explanation and example before continuing.
Continue with authoritative sources
Important industry resources
These links lead to official organisations, industry laboratories or established open-source learning projects.