A thought-provoking and startlingly insightful reworking of the science of prediction In Prediction Revisited: The Importance of Observation, a team of renowned experts in the field of data-driven investing delivers a ground-breaking reassessment of the delicate science of prediction for anyone who relies on data to contemplate the future. The book reveals why standard approaches to prediction based on classical statistics fail to address the complexities of social dynamics, and it provides an alternative method based on the intuitive notion of relevance. The authors describe, both conceptually and with mathematical precision, how relevance plays a central role in forming predictions from observed experience. Moreover, they propose a new and more nuanced measure of a prediction’s reliability. Prediction Revisited also offers: Clarifications of commonly accepted but less commonly understood notions of statistics Insight into the efficacy of traditional prediction models in a variety of fields Colorful biographical sketches of some of the key prediction scientists throughout history Mutually supporting conceptual and mathematical descriptions of the key insights and methods discussed within With its strikingly fresh perspective grounded in scientific rigor, Prediction Revisited is sure to earn its place as an indispensable resource for data scientists, researchers, investors, and anyone else who aspires to predict the future from the data-driven lessons of the past.

Table of Contents:
Timeline of Innovations ix Essential Concepts xi Preface xv 1 Introduction 1 Relevance 2 Informativeness 3 Similarity 4 Roadmap 4 2 Observing Information 7 Observing Information Conceptually 7 Central Tendency 8 Spread 9 Information Theory 10 The Strong Pull of Normality 14 A Constant of Convenience 17 Key Takeaways 18 Observing Information Mathematically 20 Average 20 Spread 21 Information Distance 24 Observing Information Applied 26 Appendix 2.1: On the Inflection Point of the Normal Distribution 32 References 39 3 Co-occurrence 41 Co-occurrence Conceptually 41 Correlation as an Information-Weighted Average of Co-occurrence 46 Pairs of Pairs 49 Across Many Attributes 50 Key Takeaways 52 Co-occurrence Mathematically 54 The Covariance Matrix 58 Co-occurrence Applied 59 References 66 4 Relevance 67 Relevance Conceptually 67 Informativeness 68 Similarity 72 Relevance and Prediction 73 How Much Have You Regressed? 74 Partial Sample Regression 76 Asymmetry 80 Sensitivity 86 Memory and Bias 87 Key Takeaways 88 Relevance Mathematically 90 Prediction 95 Equivalence to Linear Regression 97 Partial Sample Regression 100 Asymmetry 102 Relevance Applied 107 Appendix 4.1: Predicting Binary Outcomes 114 Predicting Binary Outcomes Conceptually 114 Predicting Binary Outcomes Mathematically 116 References 121 5 Fit 123 Fit Conceptually 123 Failing Gracefully 125 Why Fit Varies 126 Avoiding Bias 129 Precision 130 Focus 133 Key Takeaways 134 Fit Mathematically 136 Components of Fit 138 Precision 139 Fit Applied 143 6 Reliability 149 Reliability Conceptually 149 Key Takeaways 153 Reliability Mathematically 155 Reliability Applied 163 References 168 7 Toward Complexity 169 Toward Complexity Conceptually 169 Learning by Example 170 Expanding on Relevance 171 Key Takeaways 175 Toward Complexity Mathematically 177 Complexity Applied 183 References 183 8 Foundations of Relevance 185 Observations and Relevance: A Brief Review of the Main Insights 186 Spread 187 Co-occurrence 187 Relevance 188 Asymmetry 188 Fit and Reliability 189 Partial Sample Regression and Machine Learning Algorithms 189 Abraham de Moivre (1667–1754) 190 Pierre-Simon Laplace (1749–1827) 192 Carl Friedrich Gauss (1777–1853) 193 Francis Galton (1822–1911) 195 Karl Pearson (1857–1936) 197 Ronald Fisher (1890–1962) 199 Prasanta Chandra Mahalanobis (1893–1972) 200 Claude Shannon (1916–2001) 202 References 206 Concluding Thoughts 209 Perspective 209 Insights 210 Prescriptions 210 Index 211