Scaling Laws for Neural Language Models

Kaplan et al. (2020)

Read paper

Why It Matters

Loss scales as power law with model size, data, and compute. Empirical justification for the scaling paradigm driving modern LLM development.

Key Ideas

1. Language model loss follows predictable power-law trends as model size, dataset size, and compute grow.
2. Larger models can be more sample-efficient, so scale changes training efficiency as well as final capability.
3. Compute allocation should be based on measured scaling relationships rather than intuition alone.
4. The paper makes frontier model planning feel more like curve-fitting engineering than guesswork.

Notes

• Later papers revised the exact compute-optimal balance, but this work established the central fact that scaling is regular and forecastable.
• It is foundational for capability planning and budget decisions in large-model training.