Judea Pearl
and Dana Mackenzie
THE BOOK OF WHY
The New Science of Cause and Effect
Students are never allowed: With possibly one exception: if we have performed a randomized controlled trial, as discussed in Chapter 4.
then the opposite is true: In other words, when evaluating an intervention in a causal model, we make the minimum changes possible to enforce its immediate effect. So we “break” the model where it comes to A but not B.
We should thank the language: I should also mention here that counterfactuals allow us to talk about causality in individual cases: What would have happened to Mr. Smith, who was not vaccinated and died of smallpox, if he had been vaccinated? Such questions, the backbone of personalized medicine, cannot be answered from rung-two information.
Yet we can answer: To be more precise, in geometry, undefined terms like “point” and “line” are primitives. The primitive in causal inference is the relation of “listening to,” indicated by an arrow.
And now the algebraic magic: For anyone who takes the trouble to read Wright’s paper, let me warn you that he does not compute his path coefficients in grams per day. He computes them in “standard units” and then converts to grams per day at the end.
“Cigarette smoking is causally related”: The evidence for women was less clear at that time, primarily because women had smoked much less than men in the early decades of the century.
And Abraham drew near: As before, I have used the King James translation but made small changes to align it more closely with the Hebrew.
The ease and familiarity of such: The 2013 Joint Statistical Meetings dedicated a whole session to the topic “Causal Inference as a Missing Data Problem”—Rubin’s traditional mantra. One provocative paper at that session was titled “What Is Not a Missing Data Problem?” This title sums up my thoughts precisely.
This difference in commitment: Readers who are seeing this distinction for the first time should not feel alone; there are well over 100,000 regression analysts in the United States who are confused by this very issue, together with most authors of statistical textbooks. Things will only change when readers of this book take those authors to task.
Unfortunately, Rubin does not consider: “Pearl’s work is clearly interesting, and many researchers find his arguments that path diagrams are a natural and convenient way to express assumptions about causal structures appealing. In our own work, perhaps influenced by the type of examples arising in social and medical sciences, we have not found this approach to aid the drawing of causal inferences” (Imbens and Rubin 2013, p. 25).
One obstacle I faced was cyclic models: These are models with arrows that form a loop. I have avoided discussing them in this book, but such models are quite important in economics, for example.
Even today modern-day economists: Between 1995 and 1998, I presented the following toy puzzle to hundreds of econometrics students and faculty across the United States:
Consider the classical supply-and-demand equations that every economics student solves in Economics 101.
1. What is the expected value of the demand Q if the price is reported to be P = p0?
2. What is the expected value of the demand Q if the price is set to P = p0?
3. Given that the current price is P = p0, what would the expected value of the demand Q be if we were to set the price at P = p1?
The reader should recognize these queries as coming from the three levels of the Ladder of Causation: predictions, actions, and counterfactuals. As I expected, respondents had no trouble answering question 1, one person (a distinguished professor) was able to solve question 2, and nobody managed to answer question 3.
The Model Penal Code expresses: This is a set of standard legal principles proposed by the American Law Institute in 1962 to bring uniformity to the various state legal codes. It does not have full legal force in any state, but according to Wikipedia, as of 2016, more than two-thirds of the states have enacted parts of the Model Penal Code.
Those sailors who had eaten: The reason is that polar bear livers do contain vitamin C.
“On the Inadequacy of the Partial”: The title refers to partial correlation, a standard method of controlling for a confounder that we discussed in Chapter 7.
here is how to define the NIE: In the original delivery room, NIE was expressed using nested subscripts, as in Y(0,M1). I hope the reader will find the mixture of counterfactual subscripts and do-operators above more transparent.
In that year researchers identified: To be technically correct it should be called a “single nucleotide polymorphism,” or SNP. It is a single letter in the genetic code, while a gene is more like a word or a sentence. However, in order not to burden the reader with unfamiliar terminology, I will simply refer to it as a gene.
UK | USA | Canada | Ireland | Australia
India | New Zealand | South Africa
Penguin Books is part of the Penguin Random House group of companies whose addresses can be found at global.penguinrandomhouse.com.
First published in the United States of America by Basic Books, an imprint of Perseus Books, LLC 2018
First published in Great Britain by Allen Lane 2018
Copyright © Judea Pearl and Dana Mackenzie, 2018
The moral rights of the authors have been asserted
Cover design by Richard Green
Author photograph: Dana Mackenzie © Kay Mackenzie
ISBN: 978-0-241-24264-3
To Ruth
Almost two decades ago, when I wrote the preface to my book Causality (2000), I made a rather daring remark that friends advised me to tone down. “Causality has undergone a major transformation,” I wrote, “from a concept shrouded in mystery into a mathematical object with well-defined semantics and well-founded logic. Paradoxes and controversies have been resolved, slippery concepts have been explicated, and practical problems relying on causal information that long were regarded as either metaphysical or unmanageable can now be solved using elementary mathematics. Put simply, causality has been mathematized.”
Reading this passage today, I feel I was somewhat shortsighted. What I described as a “transformation” turned out to be a “revolution” that has changed the thinking in many of the sciences. Many now call it “the Causal Revolution,” and the excitement that it has generated in research circles is spilling over to education and applications. I believe the time is ripe to share it with a broader audience.
This book strives to fulfill a three-pronged mission: first, to lay before you in nonmathematical language the intellectual content of the Causal Revolution and how it is affecting our lives as well as our future; second, to share with you some of the heroic journeys, both successful and failed, that scientists have embarked on when confronted by critical cause-effect questions.
Finally, returning the Causal Revolution to its womb in artificial intelligence, I aim to describe to you how robots can be constructed that learn to communicate in our mother tongue—the language of cause and effect. This new generation of robots should explain to us why things happened, why they responded the way they did, and why nature operates one way and not another. More ambitiously, they should also teach us about ourselves: why our mind clicks the way it does and what it means to think rationally about cause and effect, credit and regret, intent and responsibility.
When I write equations, I have a very clear idea of who my readers are. Not so when I write for the general public—an entirely new adventure for me. Strange, but this new experience has been one of the most rewarding educational trips of my life. The need to shape ideas in your language, to guess your background, your questions, and your reactions, did more to sharpen my understanding of causality than all the equations I have written prior to writing this book.
For this I will forever be grateful to you. I hope you are as excited as I am to see the results.
Judea Pearl
Los Angeles, October 2017