Rise of Super Crunchers
Story of Orley Ashenfelter prediction of wine quality using average growing season temperature and amount of winter rainfall
Examples of Super Crunching
Amazon.com, Netflix recommendations
eHarmony - prediction of marital happiness
Casinos computing your pain point
Loss of privacy (past and future)
Super Crunching to help consumers: Farecast, Zillow.com
Regressions can also give confidence of its predictions.
Power of randomized trials
Generating new data using randomized trails.
A/B testing: Offermatica, Google AdWords
Randomized testing of government policy
Heather Ross tests the idea of Negative Income Tax. Larry Katz tests the idea of giving additional money to given the unemployed job-search assistance.
Do longer sentences increase or decrease the change that a prisoner will commit another crime? Not much effect either way.
Progresa Program for Education Health and Nutrition started by Mexican President Ernesto Zedillo
Don Berwick’s 100,000 Lives Campaign
Old myths die hard.
Physicians not doing patient specific research.
Isabel: Diagnostic-decision support software
Experts vs Equations
Equations are typically more accurate.
Humans are usually over confident of our own knowledge. Emotional biases.
Human predictions as input to equations. Human intuition needed to select the factors to consider in the equation.
Rise of cheap storage and digitization of data.
Pitfalls of Super Crunching
Evidence in favor of Direction instruction (DI)
Epagogix can predict the success of a movie based on its script.
Using Super Crunching to discriminate (finding factors that can predict the race)
John Lott: errors in Super Crunching, open data policy
The Future of Intuition and Expertise
The future belongs to Super Crunchers who can work back and forth and back again between his intuitions and numbers.