Home » math » the signal and the noise

the signal and the noise

images.jpg

Nate Silver has made a name for himself in recent years, largely as the founder of Five Thirty Eight, a blog that uses statistics to discuss and predict the outcome of elections and other political issues. His book, The Signal and the Noise: Why So Many Predictions Fail — But Some Don’t, is an extended discussion in context of the ideas behind his methods. Whether exploring the statistics of gambling, sports, the recent housing bubble, the stock market, weather reports, hurricanes, disease, or anything else, Silver’s thoroughly researched writing is almost always approachable and compelling, more narrative than demonstration of technique.

But I think the real point of the book is to suggest that Bayesian probability is an important, perhaps the best, way to understand the world. The main Bayesian idea is to start with some assumption of how likely an event is and then, as new information is acquired, modify the chance of the event as often as necessary, coming closer and closer to the truth. This explicitly probabilistic view of the world expects you to make predictions and to test them against what happens. If you refuse to do this, you are either dishonest, don’t recognize the biases you bring to the way you see the world, don’t believe in your own assessment of the likelihood of an event, can’t or won’t see the world probabilistically, or some combination of these. One proof of Silver’s methods is that he correctly predicted the outcome for every state in the nation in the last two presidential elections.

Lest you think Silver is bombastic or trying to force an ideology on the reader, let me assure you: on the contrary, the writing is almost humble in its willingness to question itself and tries hard to present the evidence and let you decide what seems right to you — an especially good example of this is the chapter on global warming, in which Silver, who appears to believe that global warming exists and is a problem, acknowledges the strength of the skeptical arguments and responds to them respectfully.

As a math teacher, I appreciated the wealth of examples and the deep conversation about probability, statistics, assumptions, models, uncertainty, and heuristics. Any reader would enjoy the book for its careful and clear handling of complex topics.

Advertisements

2 Comments

  1. johnmmartin says:

    You make me want to read Silver’s book – after I get through all this post-apocalyptic fiction I got in celebration of a holiday tracing some of its roots to a Christian celebration of renewal and birth of new possibility. Does silver, and do statisticians in general talk about truth. This seems a jarring concept. I would expect something more like “predicted observations.”

    • halshop says:

      Truth doesn’t seem “jarring” to me, but I take your point.

      Silver’s emphasis is on predicting — so the iterative process of adjusting probabilities as you receive more information is designed to increase your accuracy, not necessarily find the truth. However, accuracy for statisticians means something like “the degree to which the result of a measurement, calculation, or specification conforms to the correct value,” where the “correct value” is agreed upon (usually by measurement, though sometimes through a judicial process) only after the event being predicted has occurred. So the predicted value of an event before it occurs is accurate if it is the same or close (enough) to the measured value of the event after it occurs. While there is no way to know if the measured value of an event after it occurs is the “Truth,” most people accepted it as such.

      BTW — nice juxtaposition of the post-apocalyptic fiction with the Christmas holiday.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: