Bias-Variance Tradeoff
What
Two sources of prediction error that pull in opposite directions:
- Bias: error from overly simple assumptions → model misses patterns → underfitting
- Variance: error from oversensitivity to training data → model memorizes noise → overfitting
Total Error = Bias² + Variance + Irreducible Noise
How to diagnose
| Symptom | Train error | Test error | Problem | Fix |
|---|---|---|---|---|
| Underfitting | High | High | High bias | More complex model, more features |
| Overfitting | Low | High | High variance | More data, regularization, simpler model |
| Good fit | Low | Low (similar) | Balanced | Ship it |
Model complexity spectrum
Simple ←————————————————————→ Complex
Linear Regression → Decision Tree → Deep Neural Net
High bias High variance
Low variance Low bias
Practical fixes
Reduce overfitting (high variance)
- More training data
- Regularization (L1, L2, dropout)
- Simpler model (fewer parameters, shallower trees)
- Cross-Validation to detect it early
- Early stopping (deep learning)
- Feature selection (remove noisy features)
Reduce underfitting (high bias)
- More complex model
- More/better features (Feature Engineering)
- Less regularization
- Train longer (deep learning)