• High training error AND high test error
• Model predictions cluster around a mean regardless of input
• Learning curves show both training and validation error are high and close together

Causes of Underfitting

• Model is too simple for the complexity of the data (e.g., linear model for non-linear data)
• Too few training epochs (model hasn't had time to learn)
• Too aggressive regularisation (see below)
• Important features are missing from the input

📉 Overfitting: Too Complex, Memorising Noise

Overfitting occurs when your model learns the training data too well — including its noise and random variation — and fails to generalise to new examples.

A Visual Example

Using a 15-degree polynomial to fit the same house price data:

# Overfitting: overly complex model
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

# Degree-15 polynomial — wildly complex for this problem
model = make_pipeline(PolynomialFeatures(15), LinearRegression())
model.fit(X_train, y_train)

train_score = model.score(X_train, y_train)   # 0.99 — looks amazing!
test_score  = model.score(X_test, y_test)     # 0.43 — terrible on new data

The polynomial has twisted itself into knots to pass through every training point — including the noisy outliers. It has memorised the training set rather than learning the underlying pattern. On unseen data, it's useless.

Signs of Overfitting

• Very low training error, but much higher test error (a large generalisation gap)
• Model performance degrades significantly on any new data
• The model is surprisingly sensitive to small changes in input

🔀 Train / Validation / Test Split

A fundamental tool for detecting overfitting is splitting your data into three sets:

from sklearn.model_selection import train_test_split

# First split: hold out 20% as the final test set
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Second split: hold out 20% of remaining as validation set
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25, random_state=42
)

# Result: 60% train / 20% validation / 20% test

• Training set: what the model learns from
• Validation set: what you use to tune hyperparameters and detect overfitting during development
• Test set: the final, held-out evaluation — touch it only once, at the very end

The validation set is your early warning system. If training accuracy keeps improving but validation accuracy plateaus or drops, you're overfitting.

🛡️ Fixes for Overfitting

1. Regularisation

Regularisation adds a penalty to the loss function that discourages the model from learning overly complex patterns.

L2 Regularisation (Ridge) penalises large weights:

from sklearn.linear_model import Ridge

model = Ridge(alpha=1.0)  # alpha controls regularisation strength
model.fit(X_train, y_train)

L1 Regularisation (Lasso) can drive some weights all the way to zero, performing feature selection:

from sklearn.linear_model import Lasso

model = Lasso(alpha=0.1)
model.fit(X_train, y_train)

2. Dropout (Neural Networks)

During training, randomly "drop out" (set to zero) a proportion of neurons:

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Dropout(p=0.5),   # 50% of neurons randomly deactivated during training
    nn.Linear(64, 1)
)

This prevents neurons from co-adapting and forces the network to learn more robust, distributed representations.

3. Early Stopping

Monitor validation loss during training and stop when it starts to increase:

from tensorflow.keras.callbacks import EarlyStopping

early_stop = EarlyStopping(
    monitor='val_loss',
    patience=5,          # stop after 5 epochs without improvement
    restore_best_weights=True
)

model.fit(X_train, y_train,
          validation_data=(X_val, y_val),
          callbacks=[early_stop],
          epochs=1000)

4. More Training Data

More data makes it harder for the model to memorise noise — there's simply too much to fit exactly. When data collection is expensive, data augmentation (creating modified copies of existing examples) can help.

5. Simpler Architecture

Sometimes the right fix is simply choosing a less complex model for the problem.

🔄 Cross-Validation

With small datasets, a single train/val split might be misleading due to randomness. K-fold cross-validation gives a more reliable estimate:

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100)

# 5-fold cross-validation
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')

print(f"Mean accuracy: {scores.mean():.3f} ± {scores.std():.3f}")
# Mean accuracy: 0.847 ± 0.023

The data is split into 5 folds; the model trains on 4 and validates on 1, rotating each time. The final score is the average across all 5 — much more reliable than a single split.

Key Takeaways

• Underfitting (high bias): the model is too simple; it performs poorly on both training and test data — fix by using a more complex model or adding features
• Overfitting (high variance): the model is too complex; it performs great on training data but poorly on test data — fix by regularising, adding data, or simplifying the model
• The bias-variance tradeoff is the fundamental tension: reducing one tends to increase the other; your goal is to find the sweet spot
• Always split data into train / validation / test sets — the test set should only be touched once, at the end
• Key fixes for overfitting: L1/L2 regularisation, dropout, early stopping, more data, simpler architecture
• Cross-validation gives a more reliable performance estimate when data is limited, by averaging results across multiple train/validation splits