The Signal and the Noise by Nate Silverexplains how data is used to make predictions in many different fields; why some of those have failed to show meaningful advances in accuracy; why a few have; and what we can learn from both the successes and failures. The book is not an easy read. There’s nothing in the book that is overly hard to understand, at least not when Silver explains it. The problem is that he covers such a broad landscape and he does so in such great detail that the overall experience is somewhat overwhelming.
Even with being more than a decade old, The Signal and the Noise is still worth reading. There are a few helpful over-arching concepts that Silver introduces that will continue to be relevant, a bunch of real-world examples that demonstrate those concepts, and a few general suggestions that will continue to be helpful as experts in any field seek to use data to make hard predictions about the future.
In my full review linked below, I briefly describe three over-arching concepts: signal/noise, uncertainty/probability/confidence, and Bayes theorem. If you want to know what each of these concepts reference, click the link below. These three concepts are foundational to the book, but they are mostly presented through a series of detailed examples from different industries. The author has professionally applied data analysis in a variety of different fields. He started his career as a pricing analyst for KPMG, built a system to predict the performance of Major League Baseball players (which he sold to Baseball Prospectus), became a professional gambler, accurately predicted the 2008 presidential and Senate elections, and turned that into a successful business called FiveThirtyEight (which he sold to Disney/ESPN/ABC).
Each chapter in The Signal and the Noise digs into a different domain and the challenges and opportunities for applying data analysis to make accurate predictions in that domain. I can’t possibly do justice in summarizing these detailed analyses, but I can tell you the domains covered: economic forecasting, political punditry, baseball analysis, weather forecasting, earthquake prediction, health epidemic management, chess and poker playing, stock market investing, climate change warning, geo-political intelligence.
The final chapter in The Signal and the Noise is titled “Conclusion” and helps the reader to pull together the important lessons taught throughout the book. “This book is less about what we know than about the difference between what we know and what we think we know. It recommends a strategy so that we might close that gap. The strategy requires one giant leap and then some small steps forward. The leap is into the Bayesian way of thinking about prediction and probability.” The small steps are best summarized by the sub-section titles that follow in the chapter: think probabilistically, know where you’re coming from, try, and err.