the signal and the noise
Posted by halshop on 4 January 2013
Nate Silver has made a name for himself in recent years, largely as the founder of Five Thirty Eight, a blog that uses statistics to discuss and predict the outcome of elections and other political issues. His book, The Signal and the Noise: Why So Many Predictions Fail — But Some Don’t, is an extended discussion in context of the ideas behind his methods. Whether exploring the statistics of gambling, sports, the recent housing bubble, the stock market, weather reports, hurricanes, disease, or anything else, Silver’s thoroughly researched writing is almost always approachable and compelling, more narrative than demonstration of technique.
But I think the real point of the book is to suggest that Bayesian probability is an important, perhaps the best, way to understand the world. The main Bayesian idea is to start with some assumption of how likely an event is and then, as new information is acquired, modify the chance of the event as often as necessary, coming closer and closer to the truth. This explicitly probabilistic view of the world expects you to make predictions and to test them against what happens. If you refuse to do this, you are either dishonest, don’t recognize the biases you bring to the way you see the world, don’t believe in your own assessment of the likelihood of an event, can’t or won’t see the world probabilistically, or some combination of these. One proof of Silver’s methods is that he correctly predicted the outcome for every state in the nation in the last two presidential elections.
Lest you think Silver is bombastic or trying to force an ideology on the reader, let me assure you: on the contrary, the writing is almost humble in its willingness to question itself and tries hard to present the evidence and let you decide what seems right to you — an especially good example of this is the chapter on global warming, in which Silver, who appears to believe that global warming exists and is a problem, acknowledges the strength of the skeptical arguments and responds to them respectfully.
As a math teacher, I appreciated the wealth of examples and the deep conversation about probability, statistics, assumptions, models, uncertainty, and heuristics. Any reader would enjoy the book for its careful and clear handling of complex topics.