Oct 3·edited Oct 3

This may interest if you haven't seen it already:


Tetlock is referring to https://static1.squarespace.com/static/635693acf15a3e2a14a56a4a/t/64abffe3f024747dd0e38d71/1688993798938/XPT.pdf and specifically to p51:

Superforecasters and AI domain experts disagreed about the risk of extinction due to AI, but both groups assigned meaningful probability to the outcome and predicted that AI was the most likely cause of extinction among the options included in the XPT. Superforecasters were more skeptical than domain experts: the median superforecaster gave a 0.38% risk of extinction due to AI by 2100, while the median AI domain expert gave a 3.9% risk of extinction.


we observed little convergence in the groups’ forecasts, even after tournament participants were exposed to each others’ rationales and placed on teams where they could interact.

Expand full comment

This is probably the first episode I have any real gripe with. Parts of the episode really rely on this X years to Y outcome framing but in order to get that you were extremely credulous about the words of people with a strong vested interest in overstating AI development. There was no real attempt to broach the self-driving car problem: why should I trust the timelines of an industry that told me truckers would be obsolete four years ago? I think a large chunk of AI skepticism comes from the fact that the same industry has burned people that trusted their maximalist predictions again and again and again. At a certain point you overpredict things so often that I really need a reason to listen to your prediction. The boy who cried wolf cried before and all that.

This is the harshest criticism I can possibly levy: this came off a bit like reading the American Society of Civil Engineers Infrastructure Report Card - ya you can speculate but why should I trust people that routinely overstate things and who have a vested interest IN overstating things?

Expand full comment

I enjoyed this episode, but your priors seemed very different to the other topics you've discussed. The broad argument seems to follow Pascal's Wager; there's no evidence that something bad will happen, but the hypothesised bad outcome is severe enough that considerations of expected value (https://en.wikipedia.org/wiki/Expected_value) mean we should act anyway.

How does this differ from the other cases of 'no evidence' that you've discussed? E.g. one could argue that there's no reliable evidence for placebos/breastfeeding/psychedelics/whatever,* but they are likely to do more good than harm, so we should promote them out of an 'abundance of caution'. My impression is you'd be very skeptical of such takes!

*I do mean cases with 'no evidence this is beneficial' rather than 'evidence this is not beneficial'. I haven't gone back through the episodes, so I hope those 3 examples belong on the list!

Expand full comment

The episode did a fair job canvassing the mainstream positions about X risk, but I confess I find many of the mainstream positions indistinguishable from quackery.

The bulk of X-risk worries seem to rest on a deductive argument: an AI with sufficient capabilities could destroy humanity, even if unwittingly; rapid advances in AI continue to produce more and more of these capabilities; and therefore, if we care about human wellbeing, we ought to be seriously worried.

At the outset, it’s worth noting this kind of deductive apocalypticism smacks of Malthusianism. For the unfamiliar reader, Malthus raised a significant kerfuffle among his peers with a deductive argument he published in 1798: population is growing exponentially; food production is growing linearly; and therefore, considering food production can’t catch up, we ought to be worried.

Yet, he was entirely wrong. His prediction failed to account for the human ingenuity and adaptation which rendered the problem moot by means of technological advances in food production.

Of course, Malthus is hardly the only worry-wart with a deductive gotcha even in modern history alone: Y2K, peak oil, global cooling, overpopulation — each turned out to be a farce fueled by little more than a penchant for catastrophizing and a failure of imagination.

While it’s certainly not a decisive blow to point out the suspicious shape of X-risk arguments, I do think it bolts the burden of proof firmly to those demanding we worry. Hitchens Dictum is instructive here: “that which can be asserted without evidence can be dismissed without evidence.”

With a high bar for evidence in mind, then, I turn to three arguments that weigh against the X-worriers I feel are missed in the mainstream conversation.


Despite the theory of climate change having a distinctively apocalyptic flavor, skeptics were won over because any sensible reading of the theory entails that gradual increases in warming will be associated with gradually increasing harms, a prediction which has in fact been confirmed by multiple lines of evidence.

By analogy, we might ask: If the hypothesis that an AI with sufficient capabilities poses an existential risk to our species, shouldn’t we expect to find that as we pile on capabilities and distribute them widely, we observe associated harms?

Put another way, if superintelligent AI is going to be so awful, not-quite-superintelligent AI should be pretty bad too, right?

By my lights that seems exactly right and yet, to my knowledge, despite having widely deployed fantastical new AI capabilities to a huge swathe of the global population, we have observed zero real world harms, with the exception of the odd Bard conversation that’s gone off the rails.

So X-riskers, where’s the beef?

The reply will perhaps be the real X-risk occurs on a fast or near-vertical “takeoff,” a scenario in which an AI suddenly leaps into a sufficiently dangerous category of intelligence, rendering the harms undetectable to any previous, gradual intelligence gains. While that’s conceptually possible of course, it also stinks of unfalsifiability. And are we not already drowning in unfalsifiable apocalyptic claims, from the political to the religious to name only two? What makes the X-risk claims different from, or more believable than, say the Jones apocalyptic cult, if they cannot be subject to confirmation?


Hang on — we do have solid evidence that experts close to these systems are worried about X-risk. Isn’t that something?

I must confess, I find the way this line of “evidence” is used to be mind-bogglingly anti-scientific. Yes, of course, it’s true that many in the field are worried about X-risk. But what credibility, if any, does that actually lend to the veracity of X-risk claims?

How would we feel about a 1798 survey showing 10% of scientists and economists agreed with Malthus’ overpopulation fears? The simple fact is the 10% would have just been plain wrong. How do we know that isn’t the case here?

Whatever these prediction popularity contest studies are good for, surely they are not a reliable means of evaluating the underlying predictions themselves. Yet, they are often cited as evidence in favor of X-risk.

I cannot help but also point out the mechanism of measurement used in these studies is quite suspicious. They aggregating knee-jerk, gut-level expert intuitions about the risk of a future event that is typically measured in orders of magnitude. But do we really think an expert who imagines there to be a small risk could meaningfully distinguish between two or four orders of magnitude when the risk falls below 1%? I am extremely skeptical — and yet it is often said that even if the experts are off by some order of magnitude it’s a risk worth considering.

Why might these experts be wrong?


Malthus did not account for human ingenuity or adaptability and by my lights, neither does the X-risk hand wringing. I would be much more concerned about AI safety, for example, if we were not awash in public debate, regulatory frameworks, fail-safes and redundancies, and technologies being developed to steer AI. The reality is AI systems are always deployed into an ecosystem of constantly evolving processes and institutions which have already displayed resilience.

To put a fine point on it, by my lights the relevant question for X-risk is not whether an AI could extinguish our species. It is whether there is good evidence to think that despite all of the safeguards with which AI has and will be deployed, humanity will be unable to mitigate the risk.

Consider the fears about an explosion in misinformation as the result of recent advances in Generative AI. Despite ChatGPT being the most successful product in human history, having garnered nearly a billion users, to my knowledge there has been nothing close to a successful wide-scale misinformation campaign using generative AI. If there were an attempt by a malicious AI or malicious actor using AI, our existing social media channels have (imperfect but basically effective) mechanisms to stop the spread of misinformation.

I suspect many of the safeguards our institutions currently have in place that are supposed to be at risk to an AI takeover are similarly more resilient than imagined. The burden of proof here again seems to be on the X-risk worrier to show otherwise.

To be fair, I don’t think these are knock-down-drag-out arguments that definitively establish X-risk has no merit. And I’m open to the evidence — if it can be shown that harms really do obtain on an AI takeoff timeline despite our checks and balances, then so be it. But I do think if I'm right, there is reason to be extremely skeptical of X-risk claims, and the scientifically informed approach would be to reserve judgment until X-riskers can marshal actual evidence to support their hypothesis.

Should that evidence fail to materialize, I regret to say the long tail of history will record the X-risk debacle not for its prescience but for its commentary on the perennial human susceptibility to apocalypticism, even in “rational” quarters.

Expand full comment

Really interesting episode, I liked the change of format. I had mostly thought that the fear of AI was overblown sci-fi stuff, but this really makes the concerns tangible.

It made me think of Asimov's three laws of robotics, which I don't think were mentioned. Not sure if they are realistic in this context, but he clearly predicted a need for some sort of safeguard.

Expand full comment