Neural Networks in Plain Language
A neural network is a flexible mathematical function built from connected layers. During training, it adjusts many numerical weights so that inputs produce useful outputs.
Learning objectives
What you will be able to do
- Explain neural networks in plain language accurately in your own words.
- Recognise how the concept appears in real AI products and professional work.
- Identify an important limitation, risk or evaluation requirement.
- Apply the concept through a practical activity and knowledge check.
Core explanation
Build the right mental model
Each unit combines incoming values, weights and a bias, then applies an activation function. Layers allow the network to build increasingly useful representations—for example, edges, shapes and objects in an image.
Backpropagation efficiently calculates the gradient of the loss with respect to each weight. An optimiser then makes small weight updates. Repeating this process aims to reduce training loss; improvement on new examples must be checked separately and is not guaranteed.
The brain metaphor is historically influential but limited. Artificial neural networks are engineered mathematical systems, not replicas of human cognition.
Follow information through a small network
A neural network transforms numerical inputs through connected layers. A typical node combines inputs using weights, adds a bias and applies an activation function. The output of one layer becomes input to the next. This is a mathematical construction; the word neuron is an analogy, not a claim that the software reproduces a biological brain.
Weights determine how strongly inputs contribute to a calculation. A bias shifts the calculation’s starting point. During training, these values are adjusted to improve an objective. During a forward pass, their current values are used to produce an output. Understanding a tiny forward pass is a useful first step before considering networks with millions of parameters.
See why activation functions matter
A sequence of linear transformations without nonlinear activations can collapse into a single linear transformation. Nonlinear activations let a network represent more complex relationships. A common example is ReLU, defined as max(0, z): it keeps a positive input and replaces a negative input with zero. Other activation functions behave differently and suit different parts of a model.
The final layer must match the task. A model predicting a number might use a numerical output directly. A classifier might turn output scores into category probabilities with an appropriate transformation. Do not read an arbitrary hidden-node value as a probability, a confidence level or an explanation of the model’s reasoning.
Separate learning capacity from reliable performance
More parameters can give a network capacity to fit more complicated patterns, but capacity alone does not ensure useful generalisation. Data quality, objective choice, training and evaluation still matter. A flexible network can fit accidental details in a small dataset. Comparing performance on genuinely unseen examples helps reveal that problem.
Backpropagation calculates how parameter changes affect loss, using the chain rule through the network. An optimiser uses those gradients to update parameters. The forward pass, loss calculation, backward calculation and update are different stages. Calling the whole process “the network thinks” hides the operations that can actually be inspected and tested.
Weight
A learned value controlling the influence of one signal.
Activation
A non-linear transformation enabling complex patterns.
Layer
A stage that transforms one representation into another.
Backpropagation
A method for calculating how parameters should change to reduce error.
Calculate a two-node forward pass
Let the input be x₁ = 2 and x₂ = 1. Hidden node h₁ uses z₁ = 0.5x₁ + 1x₂ − 1, giving 1 + 1 − 1 = 1. Applying ReLU gives h₁ = 1. Hidden node h₂ uses z₂ = −1x₁ + 1x₂ + 0, giving −2 + 1 = −1; ReLU gives h₂ = 0.
The output is y = 2h₁ + h₂ + 0.5, so y = 2 × 1 + 0 + 0.5 = 2.5. This output is simply a number in our invented network. It is not automatically a probability. We chose all parameters for illustration; no training has taken place.
Now change x₂ to 3 while keeping x₁ = 2. Then z₁ = 3 and z₂ = 1, so both activations are positive. The output becomes 2 × 3 + 1 + 0.5 = 7.5. Tracking the intermediate values shows exactly how the same network responds to a changed input.
Apply your learning
Practical activity
- Spend five minutes drawing the two inputs, two hidden nodes and one output with their weights.
- Spend five minutes calculating the output for x₁ = 1 and x₂ = 0.
- Spend five minutes repeating that case without ReLU and comparing the result.
- Spend five minutes explaining why neither calculation demonstrates that the network has learned a useful real task.
Compare with an explained example answer
With x₁ = 1 and x₂ = 0, z₁ = −0.5 and z₂ = −1. ReLU makes both zero, so the output is 0.5. Without ReLU the output would be 2 × (−0.5) + (−1) + 0.5 = −1.5.
The different outputs show the activation’s effect. They do not show predictive quality because no target, dataset, training objective or held-out evaluation has been supplied. A correct forward calculation establishes that you followed the specified network, not that its parameters solve a useful problem.
Write your answers in a learning journal. The value comes from connecting the concept to your own profession.
Knowledge check
Test your understanding
Q1What does a bias do?
Answer: It adds an offset to a node’s weighted input sum, allowing the calculation to shift independently of the input values.
Q2Why use nonlinear activations?
Answer: They allow compositions that cannot be reduced to a single linear transformation, enabling more complex relationships.
Q3Is a hidden activation an explanation or probability?
Answer: Not automatically. Its interpretation depends on the model and task; a numerical value alone does not establish either.
If you cannot explain the answer without reading it, revisit the core explanation and example before continuing.
Continue with authoritative sources
Important industry resources
These links lead to official organisations, industry laboratories or established open-source learning projects.