Training, Validation and Testing Data
Training, validation and test sets answer different questions. Mixing their roles produces misleading results.
Learning objectives
What you will be able to do
- Explain training, validation and testing data accurately in your own words.
- Recognise how the concept appears in real AI products and professional work.
- Identify an important limitation, risk or evaluation requirement.
- Apply the concept through a practical activity and knowledge check.
Core explanation
Build the right mental model
The training set is used to fit model parameters. The validation set guides choices such as model type, features and hyperparameters. The test set is held back until the end to estimate performance on unseen data.
Data leakage occurs when information unavailable in real use influences training or evaluation. Leakage can create impressive test scores that disappear after deployment.
Random splitting is not always suitable. Time-based systems often need earlier data for training and later data for testing. Groups such as patients, customers or devices may need to remain entirely within one split.
Training set
Examples used directly to fit model parameters.
Validation set
Examples used to compare choices during development.
Test set
Held-back examples used for final performance estimation.
Data leakage
Information improperly crosses boundaries and inflates evaluation results.
Forecasting demand
A team randomly mixes future sales records into training while predicting earlier periods.
Apply your learning
Practical activity
- Design a split for a dataset from your field.
- State the unit that must not cross splits.
- Identify one feature that could leak future information.
Write your answers in a learning journal. The value comes from connecting the concept to your own profession.
Knowledge check
Test your understanding
Q1Why should the test set be held back?
Answer: Repeatedly using it to make development choices causes overfitting to the test and weakens its independence.
Q2When is chronological splitting appropriate?
Answer: When the model will predict future events and real-world patterns may change over time.
If you cannot explain the answer without reading it, revisit the core explanation and example before continuing.
Continue with authoritative sources
Important industry resources
These links lead to official organisations, industry laboratories or established open-source learning projects.