Technology · chips, models and the machines that train them · 1950 to today

The Curve

Data through

Gordon Moore, seated, with Robert Noyce at Intel in 1970

Gordon Moore and Robert Noyce at Intel, 1970. Photo: Intel Free Press, CC BY-SA 2.0.

1965

"The number of transistors on a chip will double every two years."

Gordon Moore, writing in Electronics magazine, working from five data points. He called it a guess.

It has held for sixty years.

It is the floor the rest of this story stands on. Scroll on.

↓

Wen Tsing Chow, portrait from the Air Force Space and Missile Pioneers biography

July 1969

The Saturn V that carried Apollo 11 was steered by an IBM computer in the rocket's instrument ring. My uncle, Wen Tsing Chow, was the scientific advisor on its design. He had already invented programmable read-only memory and built the first solid-state computer to fly in space, for the Atlas missile. I grew up hearing about this, and it still amazes me.

The guidance computer inside the spacecraft itself ran on integrated circuits carrying six transistors each. That is the dot on the left.

Portrait: U.S. Air Force, Space and Missile Pioneers, 2004.

1971 to 2019

Two years after the Moon landing, Intel shipped the 4004, some 2,300 transistors on one chip and the first microprocessor anyone could buy. Then Moore's guess simply kept coming true: the line follows the record holder, chip by chip, and doubles about every two years for five decades.

Zoom out

Pull the frame back to today and the same line goes vertical. Everything in the dashed box is the chart you were just looking at. The record chip now carries hundreds of billions of transistors; the Apollo chip's six would not register as a pixel.

Rows of cabinets of the Summit supercomputer at Oak Ridge National Laboratory — Summit, the world's fastest computer when it was switched on in 2018, at Oak Ridge National Laboratory. Photo: OLCF/ORNL, Carlos Jones, CC BY 2.0.

The turn

What is less understood is that AI is not just riding Moore's curve; it is building new exponentials on top of it. To see that, the y-axis has changed to a logarithmic scale, which turns the doubling of transistors into a straight line. Every line from here on is also divided by its own 2012 level, the year AlexNet made GPUs the engine of AI, so the AI curves can be read against that steady doubling. On this axis a straight line is a constant doubling time, and steeper is faster.

Training compute

The compute used to train the largest AI models has doubled every since 2012. Moore's Law manages a doubling every . That is roughly AI doublings for every one the chips deliver; the rest is money and scale, more accelerators running for longer.

Accelerators

An accelerator is the chip that does the training: a GPU or a purpose-built part like a TPU, measured here by how many arithmetic operations one chip can do per second at the precision used for training. Its throughput has doubled every since 2008, faster than transistor counts, because these chips also grew larger, more specialized and better fed with memory. Faster chips explain part of the compute line; the rest is buying more of them and running them longer.

Parameters

Parameters are the basic units of learning in a large language model, the numbers it adjusts as it trains, and they are often compared to the synaptic connections in a brain. Many things determine how capable a model is, but parameter count is the figure most often used to compare the relative power of models: GPT-3 had 175 billion, the largest open models today have a few trillion, and the record has doubled every since 2012. With all four lines in place the picture is complete: Moore's Law, the exponential everyone knows, is the slowest line on the chart.

Explore it

Below, the same data with every control: switch layers, go linear or log, brush any period for its doubling time, and click any point for its source.

Moore's Law holds that the power of a computer chip, measured by the transistors packed onto it, doubles about every two years. For sixty years it has proven stubbornly true. So what does AI's own growth look like against it - and how much of AI's exponent does the chip curve actually explain? Every line here is a real chip, model or accelerator, and every point has a source.

Frontier AI training compute, doubling time since 2012

—

Four exponentials

Hover to read · click a point to pin it and see its source · Esc clears · drag the strip to zoom · double-click it to reset

The largest training runs on record

Estimated total compute used to train each model, in floating-point operations. Frontier labs stopped publishing these figures around 2023, so many of the largest are inferred from hardware, duration and cost; the confidence label is Epoch's.

What all this compute buys is a separate question, and it gets its own plot: How Fast the Tests Fall tracks the benchmarks these models are measured against, and how quickly each one is solved.

Reading this page

Transistors are per-chip counts for CPUs, systems-on-chip and GPUs from public specifications, as compiled on Wikipedia's Transistor count page. Multi-die packages count as one chip; wafer-scale parts are excluded. Training compute, parameters and accelerator throughput come from Epoch AI's open datasets, Notable AI Models and Machine Learning Hardware, both CC-BY 4.0. Throughput is dense FP16/BF16 tensor performance per accelerator, the figure that matters for training.

The record line for each layer follows the highest value published up to that date. Doubling times are least-squares fits to the record line, in log space, over the period named; the fit needs a few points and at least a year. The comparison view divides each line by its own 2012 record so the four pass through one point and their slopes can be compared; 2012 is the year AlexNet made GPUs the engine of AI. The parameters layer leaves out a handful of pre-2010 systems whose published "parameter" counts are n-gram tables or kernel machines rather than learned weights. Milestones are context, not causes. This page re-fetches its sources and rebuilds itself on a schedule; the stamp at the top is the last run.