Training, Validation and Testing Data
Training, validation and test sets answer different questions. Mixing their roles produces misleading results.
Learning objectives
What you will be able to do
- Explain training, validation and testing data accurately in your own words.
- Recognise how the concept appears in real AI products and professional work.
- Identify an important limitation, risk or evaluation requirement.
- Apply the concept through a practical activity and knowledge check.
Core explanation
Build the right mental model
The training set is used to fit model parameters. The validation set guides choices such as model type, features and hyperparameters. The test set is held back until the end to estimate performance on unseen data.
Data leakage occurs when information unavailable in real use influences training or evaluation. Leakage can create impressive test scores that disappear after deployment.
Random splitting is not always suitable. Time-based systems often need earlier data for training and later data for testing. Groups such as patients, customers or devices may need to remain entirely within one split.
Give each dataset a distinct job
Training data is used to fit model parameters. Validation data helps compare design choices such as model complexity or a decision threshold. Test data is reserved for a final assessment of the selected approach. If you repeatedly examine test performance and change the system in response, the test has become part of development rather than an untouched check.
There is no split ratio that is correct for every task. Dataset size, rare outcomes, repeated individuals and time structure all matter. A random row split can be appropriate for some independent examples, but misleading when rows are closely related. Decide what “new case” means in the intended use before deciding how to partition the data.
Keep related examples on the right side of the boundary
Suppose ten recordings come from each speaker. If recordings from the same speaker appear in both training and test sets, a model may exploit speaker-specific patterns. When the intended task is performance on new speakers, group all recordings from one speaker into one split. Similar issues occur with repeated patients, devices, documents and near-duplicate images.
For forecasting, a future-facing evaluation often requires time ordering: fit on earlier data and evaluate on later data. Mixing future observations into training can make the historical test unlike the real prediction problem. The split must reflect the conditions under which the model will actually be used, not merely produce convenient set sizes.
Fit preparation steps using training information
Data leakage can occur before the model is trained. For example, calculating a scaling mean using the entire dataset allows information from evaluation cases to influence preparation. Fit data-dependent preprocessing on training data, then apply the fitted transformation to validation and test data. The same principle applies to selecting features using their relationship with the target.
Document the split rule, any grouping key, the time boundary and the preprocessing sequence. Check duplicate examples and inspect whether each subset represents the relevant task. Keep the final test’s purpose explicit. A single headline score is hard to interpret without knowing which cases were kept separate and how the system was selected.
Training set
Examples used directly to fit model parameters.
Validation set
Examples used to compare choices during development.
Test set
Held-back examples used for final performance estimation.
Data leakage
Information improperly crosses boundaries and inflates evaluation results.
Design a grouped evaluation
A fictional dataset contains 100 devices, each contributing ten readings, for 1,000 rows. The goal is to predict on previously unseen devices. Choose 70 devices for training, 15 for validation and 15 for testing. This produces 700, 150 and 150 rows, with no device shared between sets.
A random row split with the same row counts could place readings from nearly every device in all three sets. That might test new readings from familiar devices rather than generalisation to new devices. Both evaluations can answer legitimate questions, but they are not interchangeable. State which question you intend to answer.
If a normalisation mean is needed, calculate it using the 700 training rows only. Choose model settings using validation results. After selecting the approach, assess the 150 test rows and report the split design with the result. These counts illustrate a workflow, not a recommendation that 70/15/15 is universally optimal.
Apply your learning
Practical activity
- Spend five minutes designing a split for 60 fictional learners, each with five practice attempts, when evaluating performance on new learners.
- Spend five minutes choose counts for training, validation and test learners and calculate their row totals.
- Spend five minutes identify two ways information could leak across the boundary.
- Spend five minutes write a short evaluation protocol explaining what remains untouched until final assessment.
Compare with an explained example answer
One possible allocation is 42 learners for training, nine for validation and nine for testing, giving 210, 45 and 45 rows. All attempts by one learner stay together. Other allocations may be reasonable; the grouping principle is the key requirement.
Leaks include putting attempts by the same learner into different sets or choosing features after inspecting test labels. The protocol should say that preprocessing is fitted on training data, design choices use validation data, and test outcomes are not repeatedly used to select the model.
Write your answers in a learning journal. The value comes from connecting the concept to your own profession.
Knowledge check
Test your understanding
Q1Can you tune a threshold repeatedly on the test set and still call it untouched?
Answer: No. Those decisions use test information, so it is no longer independent of model selection.
Q2Why might a row-level random split be misleading?
Answer: Related rows can reveal information about the same entity across sets, testing a different generalisation problem from the intended one.
Q3Where should a scaling mean be fitted?
Answer: On the training data. Apply that fitted transformation to the other sets without recalculating it using their values.
If you cannot explain the answer without reading it, revisit the core explanation and example before continuing.
Continue with authoritative sources
Important industry resources
These links lead to official organisations, industry laboratories or established open-source learning projects.