Hacker News Books

Superforecasting: The Art and Science of Prediction - Philip E. Tetlock, Dan Gardner

Funny how that works. I'm just getting into the book Superforecasting: The Art and Science of Prediction by Tetlock and Gardner and came across this passage:

Galen is an extreme example but he is the sort of figure who pops up repeatedly in the history of medicine. They are men (always men) of strong convinction and profound trust in their own judgement. They embrace treaments, develop bold theories for why they work, denounce rivals as quacks and charlatans, and spread their insights with evangelical passion.

So even if Theranos is a complete flop, Holmes may still prove a groundbreaker in her hubris. Tetlock and Gardner sum up the problem by quoting Richard Feynman:

What medicine lacked was doubt. "Doubt is not a fearful thing," Feynman observed, "but a thing of very great value." It's what propels science forward.

Less so, I guess, visionary entrepreneurs.

Interestingly, Tetlock's 'superforecasters' predict a 30% chance of more than 350,000 coronavirus deaths in the US: https://goodjudgment.io/covid/dashboard/

This is up from 17% on March 21

Edit: got the title wrong

[0] https://en.wikipedia.org/wiki/Superforecasting:_The_Art_and_...

[1] http://mason.gmu.edu/~rhanson/futarchy.html

It's an enlightening read as they describe some of the processes used to hone intuited estimates using an outward and inward looking processes. I'm going to have to look into what you mean by using intuition to judge independence. Any good sources on that?

[1]: https://en.m.wikipedia.org/wiki/Superforecasting

I voted Remain at the time, but stumbling across Cummings' blogs and subsequently reading about Tetlock [0] + David Deutsch's arguments [1] has left me in favour of Leave (although I don't have strongly nailed down views on this).

This project is a great idea to improve the effectiveness of government by creating tools that will help ministers and officials make better decisions in the face of complex systems. Will it work? I'm optimistic but there will surely be unknown unknowns that could derail progress in addition to plain old politics.

[0] - Superforecasting is a great book: https://www.amazon.co.uk/Superforecasting-Science-Prediction...

[1] - https://www.youtube.com/watch?v=xdtssXITXuE

Also, Timothy Gowers (Fields Medallist) has an interesting piece in favour Remain [2], although he overlooks the fact that differences in institutional design of UK vs EU mean that sovereignty has significant implications beyond just sovereignty for sovereignty's sake.

[2] - https://gowers.wordpress.com/2016/06/02/6172/

Edit: Would love to hear counterarguments from downvoters btw. I don't mean that in a hostile way, I've changed my mind on this topic many times in the past 3 years and I'd be more than happy to be exposed to more good arguments against Brexit and Cummings' ideas.

Practice is also good. If you're not used to probabilistic thinking you'll need to develop that intuition and calibration.

Anything about how to think about things better is going to be useful. There's a Coursera course called Model Thinking that is might be useful. Just being curious about things in general and pushing yourself outside of your normal areas of competency/interest.

It might seem weird, but I find Twitter to be pretty essential these days. There are a lot of smart people freely sharing information and some don't mind answering questions.

Aside: I was pretty into the roguelikes that descend from the Moria/Angband line for the better part of a decade. I'd frequently get to know people in that community and, after some light googling, realise they were famous in some other field.

I learned about Toby Ord through a terrific game he wrote called Sil [0], which is a nearly-unrecognizable fork of NPPAngband that sticks tightly to the theme of the Silmarillion. It's a very tight game, and it oozes atmosphere--especially neat considering there is no sound, and only an ASCII-character terminal-like display.

This was also how I first discovered Nick Bostrom and "Superintelligence"; Toby Ord was a postdoc of Bostrom's at the time, and following the chain I stumbled into this work.

I guess roguelikes might appeal to rationalists in particular; there's a lot to reason about in there, but sometimes random things happen and there's just nothing you can do :)

[0] http://www.amirrorclear.net/flowers/game/sil/index.html

This whole section of the author's blog is very interesting. It goes into the US Intelligence Community's interest in superforecasters and the Good Judgement Project, a kind of elite tournament of forecasters. It talks about how superforecasters estimate by overcoming cognitive biases and questioning assumptions. And then it ultimately points out though that there are limits to forecasting and low probability events do occur all the time.

https://commoncog.com/blog/the-forecasting-series/

Red Famine: Stalin's War on Ukraine, 1921-1933 - by Anne Applebaum.

Superforecasting: The Art and Science of Prediction by Phillip Tetlock.

The Sports Gene: Inside the Science of Extraordinary Athletic Performance by David Epstein.

- Hillbilly Elegy by JD Vance

- Tools of Titan by Tim Ferriss

- Competing Against Luck by Clayton Christensen

- Scrum: A Breathtakingly Brief and Agile Introduction by Chris Sims

- Build Better Products by Laura Klein

- Capital in the Twenty-first Century by Thomas Picketty

- Shoe Dog by Phil Knight

- Lean Customer Development by Cindy Alvarez

- Impossible to Inevitable by Aaron Ross & Jason Lemkin

- Grit by Angela Duckworth

- Love Sense by Sue Johnson

- Thinking, Fast and Slow by Daniel Kahneman

- Working Effectively With Legacy Code by Michael Feathers

- Smarter Faster Better by Charles Duhigg

- Sprint by Jake Knapp

- Fooled by Randomness by Nassim Taleb

- Becoming a Supple Leopard by Kelly Starrett

- Superforecasting by Philip Tetlock

- The Inner Game Of Tennis by Timothy Gallwey

- Design Sprint by Richard Banfield

- The Structure of Scientific Revolutions by Thomas Kuhn

- The Signal and the Noise by Nate Silver

- Advanced Swift by Chris Eidoff

- Siddhartha by Hermann Hesse

Some of these books are older and had been on my list for awhile. Some were released this year. Most of these books are very good. I usually stop reading bad books by the end of the first chapter.

https://twitter.com/PTetlock

On the subject of predictions and credibility, when the question "what's your brilliant startup idea" comes up, one of my half-joking responses: an ESPN site for CNBC and other cable news sites where talking heads spend all data talking about the prediction performance of other talking heads and making predictions about future prediction performance of those talking heads.

I'd be curious how many people subscribe to both intersectionalism and rationalism, I suspect it's very hard to do.

I recently read an interesting take on this from Philip Telock in Superforecasting (unrelated book): Belief systems are like jenga blocks, we arrive at conclusions from myriad assumptions all stacked upon each other. It's hard to change core beliefs once they're tied to a bunch of other downstream beliefs (like blocks lower in the tower), especially if those blocks involve group identity.

All this to say I suspect this thread will consist of people talking past one another because everyone's belief frameworks are very different, and language means slightly different things. E.g. the word "sexist" could refer to either the behaviors, both explicit or implicit (micro-aggressions), or the entire system as measured by outcome, depending on who you ask.

Kim Zetter - Countdown to Zero (on Stuxnet virus and how it was smuggled into the nuclear facility; very interesting)

Gary Kasparov - Winter is Coming (we should consider Russia a dictatorship by now; though until recently, western politicians treated it as a democratic partner country)

Mark Goodman - Future crimes (wide spanning book on crime in the age of the internet)

Philip E. Tetlock - Superforecasting (how amateurs can consistently beat domain professionals in forecasting all kind of stuff)

Venkat Subramaniam - Programming Concurrency on the JVM (good overview of your options (diy with locking / akka / clojure & STM))

Thinking, problem solving related:

Superforecasting by Philip Tetlock: accurate forecasting

Thinking fast and slow by Daniel Kahneman: how to avoid bias

Misbehaving: like thinking fast and slow but more hilarious

The checklist manifesto by Atul Gawande: the power of simple process

From Darwin to Munger by Peter Bevelin: lots of mental models to add to your latticework

Business management:

The Outsiders by William Thorndike: capital allocation

The hard thing about hard things by Ben Horowitz: some mental models for managers facing the real-life struggles of startups

Zero to One by Peter Thiel and Blake masters: for the chapter on what kinds of business are always going to be tough (i.e. ones in perfectly competitive industries)

Worldview:

The Better Angels of Our Nature: Why violence has declined

The making of modern economics by Mark Skousen (audiobook): explains various economic ideas through telling the history of the fathers of those ideas.

Investing:

You can be a stock market genius by Joel Greenblatt: where to look for undervaluation

The Essays of Warren Buffett by Lawrence Cunningham: Buffett's thoughts in Buffett's words, neatly categorised by topic

Competition Demystified by Bruce Greenwald: how to identify a high quality business

- Statistics Done Wrong by Alex Reinhart. Plenty of gotchas with real world examples from academia. Well written and easy to read.

- The Circle by Dave Eggers. This one was scary. About imaginary corporation (a blend of Facebook and Google and Amazon) and probably not too distant future. If you liked Black Mirrors, you will love this.

- Brave New World by Huxley, Aldous. Classic novel with interesting thoughts about engineered society, where every human is assigned class, purpose in the society and feature at birth.

- Hatching Twitter: A True Story of Money, Power, Friendship, and Betrayal by Bilton, Nick. Read this book in a weekend, really well written and well researched about the inception of Twitter.

- Superforecasting: The Art and Science of Prediction by Tetlock, Philip E. A study on people with above average ability to forecast feature events (mostly geo-political). Talks about measuring predictions and improving them.

- The Black Swan by Nassim Nicholas Taleb. Brilliant book about overlooking rare events which have dramatic consequences because 'it's unlikely to happen'.

https://www.amazon.com/Superforecasting-Science-Prediction-P...

A similar effect can be found with Grit, Fogg’s Behavior Model, Superforecasting, and most Gladwel books.

On the coding side, I’ve only noticed this with Pragmatic Programmer, Clean Code, and maybe Phoenix/Unicorn project. Could I don’t read enough of those or they’re too focused on specific technologies instead of broad ideas … or I get too much of my technical reading from blogs and twitter. Those do get repetitive and you quickly find common patterns, but no titles to refer to.

EDIT: the intent here was to expose overconfidence and vague predictions, not pay fan service to ethereum or suggest an actual bet. if anybody is interested in how to make proper predictions, I recommend the books by Philip Tetlock, especially the latest called Superforecasting [1].

EDIT2: I wasn't aware my views are so controversial, so here is some more background: If somebody was convinced something couldn't happen, he'd assign a probability of 0 to that event. If that person wanted to act according to her believes, taking on bets, no matter the odds, would have positive expected utility. Since almost nobody takes on such bets, it suggests that we generally over-exaggerate when we say things like "impossible" or "sorry for you loss", hence we are being overconfident.
The other is vagueness. By not being clear about what exactly we are predicting, we're leaving the door open to back out of it later. In fact, Tetlock has found that, by making vague predictions, experts could later convince themselves (and others) that they were "close", skewing their sense of accuracy. Unfortunately, when subject to a prediction tournament with strict rules, they would score no better than random [2].

[1]: http://www.amazon.com/Superforecasting-The-Art-Science-Predi...

[2]: http://www.amazon.com/Expert-Political-Judgment-Good-Know/dp...

It explains that we naturally trust people who sound smart, well-informed, and CONFIDENT. We don't want to hear uncertainty, probabilities, or the other signs of someone who thinks in a careful quantitative way. We want to accept a cognitively simple answer, then move on. This is what we find comfortable.

However this is a good way to select people who are terrible at making actual predictions. They appear to predict, but often with sufficient weasel words that it is hard afterwards to say whether it was violated. (The book gives real examples.) But if you put them in a setting where they can be tested, they perform worse than uninformed monkeys. And the part of the future that they are worst at predicting is exactly what they were supposed to be experts at!

The book Superforecasting walks through how this was demonstrated, and the discovery that there are people you will never see on CNN or Fox news who are really good at forecasting. A fact that is extremely interesting to various TLA agencies (one of whom paid for the research in question).

The long and short of it? Bayes' Theorem actually works in the real world. The revolution that started with quants on wall street, analytics in baseball and Nate Silver in politics is still ongoing.

When you are done with the book and have processed it, hopefully you will understand why the author said in response to an audience question after a talk, Here’s my long-term prediction for Long Now. When the Long Now audience of 2515 looks back on the audience of 2015, their level of contempt for how we go about judging political debate will be roughly comparable to the level of contempt we have for the 1692 Salem witch trials.

Hopefully the contempt that some of us have for how talking heads in January and February of 2020 dismissed Coronavirus is a step on the path to that future.

The signal and the noise - Nate Silver;

Black Swan - Nassim Nicholas Taleb;

Antifragile - Nassim Nicholas Taleb;

1984 - Orwell;

Man's search for meaning - Viktor Frankl;

Diplomacy - Henry Kissinger (not only international politics but also deep-thinking strategy that can be used anywhere);

Meditations - Marcus Aurelius;

Superforecasting - Philip Tetlock;

Propaganda - Edward Bernays;

Pitch anything - Oren Klaff;

Guns, Germs and Steel - Jared Diamond;

How to win friends and influence people& Stop worrying (both by Dale Carnegie);

The Selfish Gene - Richard Dawkins;

Trust - Francis Fukuyama;

Philip Tetlock has written on this topic for years. Two of his books are Expert Political Judgment and Superforecasting.

[1]: https://en.wikipedia.org/wiki/The_Good_Judgment_Project

Edit: So to directly answer your question, rather than AI experts, I'd prefer technology experts (AI or otherwise) with a track record of well-calibrated predictions.

What I'm referring to is the "now-cast", but his other two definitions both seem to shy away from saying "this is flat-out the probability we think of the election".

The point is, you can redefine or choose a definition of probability if you want, but if it's less useful than the normal definition (and confusing to people!) then people are free to criticize your work on that basis.

And there's a very useful, testable, mathematical definition of probability that allows us to equally assess everyone's predicting ability, and Nate Silver is dodging it.

If you're interested in this subject, there's a non-mathematical discussion somewhere in Tetlock's book Superforecasting which is interesting in general.

The book Superforecasting (2015) uses Larry Kudlow as an example of 'hedgehog forecaster' that is consistently wrong.

Not only is Kudlow consistently wrong in forecasting, he also fails nowcasting. When Financial crisis was unfolding Kudlow did not realize that something was going wrong.

>The National Bureau of Economic Research later designated December 2007 as the official start of the Great Recession of 2007–9. As the months passed, the economy weakened and worries grew, but Kudlow did not budge. There is no recession and there will be no recession, he insisted. When the White House said the same in April 2008, Kudlow wrote, “President George W. Bush may turn out to be the top economic forecaster in the country.”20 Through the spring and into summer, the economy worsened but Kudlow denied it. “We are in a mental recession, not an actual recession,”21 he wrote, a theme he kept repeating until September 15, when Lehman Brothers filed for bankruptcy, Wall Street was thrown into chaos, the global financial system froze, and people the world over felt like passengers in a plunging jet, eyes wide, fingers digging into armrests.