Super Crunchers

Rise of Super Crunchers

Story of Orley Ashenfelter prediction of wine quality using average growing season temperature and amount of winter rainfall

Examples of Super Crunching

Amazon.com, Netflix recommendations

Daily Me.

eHarmony - prediction of marital happiness

Casinos computing your pain point

Loss of privacy (past and future)

Super Crunching to help consumers: Farecast, Zillow.com

Regressions can also give confidence of its predictions.

PageRank

Power of randomized trials

Generating new data using randomized trails.

A/B testing: Offermatica, Google AdWords

Randomized testing of government policy

Heather Ross tests the idea of Negative Income Tax. Larry Katz tests the idea of giving additional money to given the unemployed job-search assistance.

Do longer sentences increase or decrease the change that a prisoner will commit another crime? Not much effect either way.

Progresa Program for Education Health and Nutrition started by Mexican President Ernesto Zedillo

Evidence-Based Medicine

Don Berwick’s 100,000 Lives Campaign

Old myths die hard.

Physicians not doing patient specific research.

Isabel: Diagnostic-decision support software

Experts vs Equations

Equations are typically more accurate.

Humans are usually over confident of our own knowledge. Emotional biases.

Human predictions as input to equations. Human intuition needed to select the factors to consider in the equation.

Why Now?

Rise of cheap storage and digitization of data.

Pitfalls of Super Crunching

Evidence in favor of Direction instruction (DI)

Epagogix can predict the success of a movie based on its script.

Using Super Crunching to discriminate (finding factors that can predict the race)

John Lott: errors in Super Crunching, open data policy

The Future of Intuition and Expertise

The future belongs to Super Crunchers who can work back and forth and back again between his intuitions and numbers.