← Research

The Law of the Vital Few

Testing Zipf's Law across 20 real-world datasets

In 1935, the linguist George Kingsley Zipf noticed something peculiar about English: the second most common word (of) appears about half as often as the first (the). The third most common word appears about a third as often. The fourth, a quarter. Plot word frequency against rank on logarithmic axes and you get a straight line with slope roughly −1.

This alone would be a curiosity. But the same pattern appears in city populations, wealth distributions, earthquake magnitudes, website traffic, species abundance, and dozens of other unrelated domains. A power law hiding in plain sight, connecting linguistics to seismology to economics. The question is: is this universality real, or is it an artefact of how we look at data?

I collected 20 datasets from across the sciences, humanities, and internet, and tested each one for power law behaviour using proper statistical methods — not just eyeballing a log-log plot, but maximum likelihood estimation, Kolmogorov-Smirnov goodness-of-fit tests, and log-likelihood ratio comparisons against the lognormal alternative.

The Rank-Frequency Plot

The defining feature of Zipf's Law is the log-log plot: rank on the x-axis, frequency (or size, or count) on the y-axis. If the data follows a power law, the result is a straight line. The slope of that line is the Zipf exponent s: a value of s = 1 gives the classic Zipf distribution where the kth item has frequency proportional to 1/k.

Select a dataset below to see its rank-frequency distribution. The gold line shows the best-fit power law. Pay attention to where the data deviates from the fit — the tails tell the story.

Some observations jump out immediately. Word frequencies produce the cleanest power laws — Moby Dick's word distribution has R² = 0.987, and the Google Web Corpus achieves 0.986 across 100,000 words. The Zipf exponents cluster around 1.1–1.5, slightly steeper than the ideal s = 1.

City populations (s = 1.83) and baby names (s = 2.22) follow power laws but with steeper exponents — the big cities and popular names dominate even more than Zipf's law would predict. Earthquake magnitudes, ranked by size, appear nearly flat (s = 0.07) because they're already on a logarithmic scale — the Gutenberg-Richter law is a power law in energy, not magnitude.

The Exponent Spectrum

If everything followed exactly the same power law, every dataset would have the same exponent. They don't — and the variation is itself interesting. The exponent tells you how concentrated the distribution is: small s means the leaders barely dominate (GitHub stars: s = 0.49), while large s means extreme concentration (city populations: s = 1.83).

The language datasets cluster tightly around s = 1.1–1.6 — a remarkably narrow band given the diversity of texts (English vs Spanish, single book vs trillion-word corpus, characters vs words). This is the heart of Zipf's original observation: something about how humans use language produces a consistent power law, independent of the specific language or text.

Datasets from the physical sciences (earthquakes, asteroids, rivers) and the internet (GitHub stars, Wikipedia, website traffic) have lower exponents, typically s < 1. Economic and demographic data (cities, names, companies) tends higher, above s = 1.5. The exponent isn't universal — but the form of the distribution is.

The Universality Test

The real question isn't whether individual datasets are well-fit by power laws — it's whether they all follow the same law. If we normalise each dataset (dividing values by the maximum, ranks by the total count), do they collapse onto a single curve?

They don't collapse perfectly — the different exponents fan the curves out like a hand of cards. But the shape is consistent: convex on log-log axes, with the same gradual roll-off. This is what universality looks like in practice. Not identical behaviour, but a shared functional form across domains that have nothing to do with each other.

Is It Really a Power Law?

The dirty secret of power law research is that many things that look like power laws on a log-log plot are actually better described by lognormal distributions, stretched exponentials, or other alternatives. As Clauset, Shalizi, and Newman showed in their influential 2009 paper, the standard practice of fitting a line to a log-log plot is statistically naive — it will find "power laws" in almost anything.

Proper testing requires three steps: estimate the exponent using maximum likelihood (not OLS), compute the Kolmogorov-Smirnov statistic for goodness of fit, and compare the power law fit to alternatives using a log-likelihood ratio test. I applied this methodology to all 20 datasets.

The results are striking: every dataset in our sample favours the power law over the lognormal according to the log-likelihood ratio test. But the quality of fit varies enormously. Word frequencies achieve KS statistics below 0.03 — nearly perfect power laws. City populations and baby names, despite looking linear on log-log plots, have subtle curvature that pushes KS above 0.05.

The datasets with the fewest items (letter frequencies, word lengths) fit worst, which is partly a sample-size effect. But even the large datasets show systematic deviations. Pure power laws are idealisations — the real world is always more complicated. What's remarkable is how well the idealisation works as a first approximation.

Why Is Everything a Power Law?

There is no single explanation. Power laws can arise from:

Preferential attachment (the rich get richer): new connections in a network go preferentially to already-popular nodes. This explains website traffic, citation counts, and social media followers. A page with 10,000 links is more likely to get the 10,001st than a page with 10.

Multiplicative processes: when growth is proportional to current size (like compound interest), the distribution of outcomes converges to a power law. This may explain city populations, company sizes, and wealth distributions.

Optimisation under constraints: Zipf himself argued that word frequencies reflect an optimisation between speaker effort (favouring few, common words) and listener comprehension (favouring many, specific words). The power law is the equilibrium.

Self-organised criticality: systems that naturally evolve toward a critical state — like sandpiles, earthquakes, or forest fires — produce power law distributions of event sizes without any tuning.

The diversity of mechanisms is itself the point. Power laws aren't one phenomenon — they're a common mathematical form that many different processes converge to. Like the normal distribution for sums, or the exponential for waiting times, the power law is an attractor in the space of distributions.

The Shape of Inequality

One way to understand the Zipf exponent is as a measure of inequality. When s = 0, everything is equally sized. When s = 1, the largest item is twice the second-largest, three times the third-largest. When s = 2, the largest is four times the second, nine times the third.

The Pareto principle ("80/20 rule") is a special case of Zipf's law. When s ≈ 1.16, the top 20% of items account for 80% of the total. Many real-world distributions fall near this value — the concentration of wealth, website traffic, and species abundance all hover around this threshold.

Methodology

Data was collected from 20 sources spanning 9 domains: language (word/character frequencies in English, Spanish, and Chinese), geography (city and country populations, river lengths), demographics (baby names, surnames), internet (GitHub stars, Wikipedia pageviews, Instagram followers), economics (company revenues), geophysics (earthquake magnitudes), astronomy (asteroid diameters), biology (species per family), and computing (file sizes).

For each dataset, I computed: (1) the OLS Zipf exponent s and R² from log-log regression; (2) the MLE power law exponent α and optimal x_min following Clauset, Shalizi & Newman (2009); (3) the KS statistic for power law goodness of fit; (4) lognormal parameters μ and σ via MLE; and (5) the log-likelihood ratio R comparing power law to lognormal (positive R favours power law).

The OLS exponent s (slope of log-log regression) is reported for comparability with the Zipf literature, but should not be used for formal hypothesis testing. The MLE exponent α is the statistically rigorous estimate and relates to s approximately as α ≈ 1 + 1/s for rank-ordered data.