Introductory statistics textbooks usually point out Simpson’s paradox, an interesting phenomenon that’s usually illustrated with a story from the University of California, Berkeley. The story goes something like this:
In 1973, UC Berkeley was sued for gender bias, because their graduate school admission figures showed obvious bias against women:1
| Applicants | Admitted | |
|---|---|---|
| Men | 8442 | 44% |
| Women | 4321 | 35% |
Men were much more successful in admissions than women, leading Berkeley to be “one of the first universities to be sued for sexual discrimination”. (The difference is statistically significant with p ≈ 10-26!) The lawsuit failed, however, when statisticians examined each department separately. Graduate departments have independent admissions systems, so it makes sense to check them separately—and when you do, there appears to be a bias in favor of women.
How does this happen? The simple explanation is that women tended to apply to the departments that are the hardest to get into, and men tended to apply to departments that were easier to get into. (Humanities departments tended to have less research funding to support graduate students, while science and engineer departments were awash with money.) So women were rejected more than men. Presumably, the bias wasn’t at Berkeley but earlier in women’s education, when other biases led them to different fields of study than men.
Now, this example has been analyzed to death in many places: on Wikipedia, in various blogs, in many textbooks (including my own book), and pretty much everywhere else. I’m not going to present a new analysis of the data or of Simpson’s paradox.
I just want to point out something simpler: There never was a lawsuit!
A Wall Street Journal interview with Peter Bickel, one of the statisticians involved in the original study, makes clear that Berkeley was never sued—it was merely afraid of being sued:
Simpson’s Paradox has fooled many. In the fall of 1973, for instance, the University of California, Berkeley’s graduate division admitted about 44% of male applicants and 35% of female applicants. That raised eyebrows among school officials, who feared bias and asked Peter Bickel, now a professor emeritus of statistics at Berkeley, to analyze the data.
“The associate dean of the graduate school thought that the university might be sued,” Mr. Bickel says.
When Mr. Bickel and his colleagues scrutinized the data, they found little evidence of gender bias. Instead, they discovered that more women had applied to departments that admitted a small percentage of applicants, like English, than to departments that admitted a large percentage of applicants, like mechanical engineering.
The core paradox matches the usual story, but no lawsuit was involved. I’ve done some digging and I haven’t been able to find the original source of the mythical lawsuit—perhaps an early textbook or journal article author misheard the original story, wrote about a lawsuit, and authors ever since have copied the story unchanged.
A scan through Google Books reveals this urban legend has infected many recent books, and undoubtedly many older ones:
It’s also present in the scientific literature:
Now, this isn’t the first time scientists have unwittingly propagated a myth through decades of the literature. The best example is possibly the century-old myth that spinach is a rich dietary source of iron: even the myth is a myth, as it turns out the common explanation (that German chemists measuring the iron content of spinach had misplaced a decimal point) is also a myth, propagated through the literature by scientists copying references without checking up on their provenance.2
It seems Simpson’s paradox has experienced a similar problem. Someone, perhaps back in the 1970s or 1980s, before the story was easily Googleable, wrote about a lawsuit to spice up their example, and the story has been repeated ever since.3 I only detected the problem because I am the type of nerd who wonders “Is the court opinion in this case available online? I’d love to read what the judge thought of the statistics”, and so I started hunting for a nonexistent court case.
Perhaps when we use stories to illustrate common statistical errors, we should make sure our stories are not in error as well.
Bickel, P. J., Hammel, E. A., & O’Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398–404. http://doi.org/10.1126/science.187.4175.398↩
Rekdal, O. B. (2014). Academic urban legends. Social Studies of Science, 44(4), 638–654. http://doi.org/10.1177/0306312714535679↩
Kudos to the anonymous Wikipedian who noticed that none of the Simpson’s Paradox article’s sources could confirm a lawsuit and removed the mention.↩