Each neuron also has a bias - a number that shifts the output up or down, like adjusting the baseline volume before any signal arrives.

When a neural network learns, it is really just adjusting thousands or millions of these weights and biases until it finds the combination that gives the best predictions.

How a Neural Network Learns

Learning happens in a cycle with four steps:

Step 1: Forward Pass

Data flows through the network from input to output. Each neuron multiplies its inputs by its weights, adds its bias, and passes the result through an activation function (which decides whether the neuron should "fire" or stay quiet). The network produces a prediction.

Step 2: Calculate the Error

The prediction is compared to the correct answer. The difference is the error (also called loss). A prediction of "7" when the answer is "3" produces a large error; a prediction of "3" produces a small one.

Step 3: Backpropagation

The error is sent backwards through the network. Each weight learns how much it contributed to the mistake. This is the mathematical magic of backpropagation - it figures out which knobs to turn and by how much.

Step 4: Update Weights

The weights and biases are adjusted slightly to reduce the error. Then the cycle repeats with the next piece of data.

Visual Walkthrough: Classifying a Handwritten Digit

Let us trace how a neural network classifies a handwritten "5" from the MNIST dataset:

1. Input: The 28×28 pixel image is flattened into 784 numbers (pixel brightness values from 0 to 255). These enter the 784 input neurons.

2. Hidden layers: The first hidden layer might detect simple edges and curves. The second hidden layer combines those into shapes like loops and strokes. Deeper layers recognise digit-like patterns.

3. Output: The 10 output neurons produce probabilities. The network might output:

   • Digit 3: 5% confident
   • Digit 5: 89% confident
   • Digit 8: 4% confident
   • All others: less than 1%

4. Decision: The network picks the digit with the highest probability - 5. Correct!

5. If wrong: Backpropagation adjusts the weights so next time, the correct digit gets a higher score.

Activation Functions: The Gatekeepers

Not every signal should pass through a neuron at full strength. Activation functions act as gatekeepers that decide whether and how much a neuron should fire.

The most common activation function today is ReLU (Rectified Linear Unit). It has a simple rule: if the input is positive, let it through unchanged; if negative, output zero. This simplicity makes it fast to compute while still allowing the network to learn complex patterns.

Without activation functions, no matter how many layers you stack, the network could only learn simple linear relationships - like drawing straight lines through data. Activation functions give neural networks the ability to learn curves, boundaries, and intricate patterns.

Why "Deep" Learning?

When a neural network has many hidden layers, it is called a deep neural network, and training it is called deep learning. Depth allows the network to learn hierarchies of features:

• Layer 1: Detects edges and gradients.
• Layer 2: Combines edges into textures and simple shapes.
• Layer 3: Recognises parts of objects (eyes, wheels, letters).
• Layer 4+: Identifies whole objects and scenes.

This layered approach is why deep learning excels at complex tasks like image recognition, language understanding, and game playing.

Key Takeaways

• Neural networks are inspired by the brain's neurons and synapses.
• They have three layer types: input, hidden, and output.
• Weights and biases are the adjustable knobs that control learning.
• Learning follows a cycle: forward pass → error → backpropagation → update.
• Deep learning uses many hidden layers to learn complex patterns.

In the next lesson, we will zoom in on the training process - how you actually teach a neural network to get smarter over time.