Superforecasters and AI domain experts disagreed about the risk of extinction due to AI, but both groups assigned meaningful probability to the outcome and predicted that AI was the most likely cause of extinction among the options included in the XPT. Superforecasters were more skeptical than domain experts: the median superforecaster gave a 0.38% risk of extinction due to AI by 2100, while the median AI domain expert gave a 3.9% risk of extinction.
[...]
we observed little convergence in the groups’ forecasts, even after tournament participants were exposed to each others’ rationales and placed on teams where they could interact.
This is probably the first episode I have any real gripe with. Parts of the episode really rely on this X years to Y outcome framing but in order to get that you were extremely credulous about the words of people with a strong vested interest in overstating AI development. There was no real attempt to broach the self-driving car problem: why should I trust the timelines of an industry that told me truckers would be obsolete four years ago? I think a large chunk of AI skepticism comes from the fact that the same industry has burned people that trusted their maximalist predictions again and again and again. At a certain point you overpredict things so often that I really need a reason to listen to your prediction. The boy who cried wolf cried before and all that.
This is the harshest criticism I can possibly levy: this came off a bit like reading the American Society of Civil Engineers Infrastructure Report Card - ya you can speculate but why should I trust people that routinely overstate things and who have a vested interest IN overstating things?
Here's my response. First, I get that people in AI have an incentive to overstate how soon it's coming, but it's just not true that people in an industry consistently undershoot on timelines. I gave the Rutherford/nuclear power example in the episode, I think, but a more relevant one: AI specialists thought it would AI becoming superhuman at Go wouldn't happen until the late 2020s or even late 21st century; DeepMind cracked it in 2016.
But anyway, I don't think the "X years to Y outcome" framing is key at all, and if I made it seem like it was then I've failed a bit: the point is just that people in the AI industry think that general AI could come pretty soon. They might be wrong – and as you say, maybe they're likely to be undershooting – but it seems pretty plausible to me that something really powerful could be along in the next 50 years. Maybe it won't! But anyone who tells me it *definitely* won't immediately seems wildly overconfident to me.
(I also don't think AI has overpromised particularly! The progress in recent years has been incredible. Sure, some people made overexcitable claims about autonomous vehicle progress, and it does turn out that's been a bit harder than thought, but AI capabilities have absolutely exploded in the last 10 years.)
And – as I tried to say in the episode – all I'm really interested in getting across is that *this is plausible*. It's really obvious that when you get a system to optimise for some metric, like education systems optimising for exam grades, then the system will maximise those and not necessarily what you care about, like educational progress or children's flourishing or whatever. That's just Goodhart's law. The concern is that superintelligent AI will the same only more so: incredibly powerful at optimising for some goal you give it, but that goal might not be quite what you actually wanted, with disastrous consequences.
I get that it sounds weird, but I've spoken to lots of people in AI who think it's a very realistic danger. And you can say "well AI scientists want to talk up AI capabilities," but that's not obvious to me (you could equally argue that AI scientists would want to downplay the possibility of AI killing everyone: you don't tend to see nuclear scientists overstating the death toll of Chernobyl to make nuclear sound more powerful than it is).
Having followed this debate for six years now, my position is this: there are a lot of serious, senior people in AI, including people in academia who have less obvious financial stake in the system, and people like Geoff Hinton who've quit industry specifically to speak about this stuff, who say 1) that superintelligent AI might be coming fairly soon and 2) that AI could be catastrophically dangerous. There are also some senior people in AI who think that's a silly thing to worry about, although they never seem to address the specific concerns raised. As someone without the technical knowledge but who's followed the arguments, I don't think you can dismiss a small, but real, possibility – maybe one or two percent! But that's still way too high! – of something very very bad happening, and I am very keen that people don't dismiss it as sci-fi because it sounds a bit like The Terminator.
Sorry, that was a way longer post than I intended to write!
That's all totally fair. I think part of the frusteration is the demands that we take this one particular existential risk seriously. It feels weirdly technophobic to have these demands (not saying y'all are necessarily demanding it, but people in AI are) when high-level climate doomers, nuclear preppers, and people afraid of meteors are firmly relegated to loon status. That kinda sums up my beliefs on AI. Yes it's possible but also a lot of things are possible and when you are relegated to a single planet it frankly isn't THAT hard to have existential threats.
I’d really recommend reading Toby Ord’s The Precipice on extinction risks. And Stuart Russell’s human compatible on AI. I’m very convinced that AI and bioengineered pandemics are actual existential risks and the other things basically aren’t
Finally got round to listening to this episode so this comment is very late.
I wanted to comment on the energy consumption of large AI models as a limiting factor. I rather see what Tom described as a limitation of human brains (that they have very low energy consumption and are rather small) as an advantage. Perhaps the first sub-challenge (I forget the technical term Tom mentioned) for an AI wanting to take over the world is to build it's own power station (probably using nuclear fusion). Tom mentioned Moore's Law and in a rather throw away comment admitted it might not be still true. I rather suspect a radical breakthrough on hardware design would be needed for the AI apocalypse to be a threat, as the models may get cleverer but as the consume more and more electricity that becomes the limiting factor.
I think even if some of the pessimists have no financial incentive anymore, if following their advice will lead to centralisation of the industry, that still means we should take their arguments with more salt than we otherwise would.
I enjoyed this episode, but your priors seemed very different to the other topics you've discussed. The broad argument seems to follow Pascal's Wager; there's no evidence that something bad will happen, but the hypothesised bad outcome is severe enough that considerations of expected value (https://en.wikipedia.org/wiki/Expected_value) mean we should act anyway.
How does this differ from the other cases of 'no evidence' that you've discussed? E.g. one could argue that there's no reliable evidence for placebos/breastfeeding/psychedelics/whatever,* but they are likely to do more good than harm, so we should promote them out of an 'abundance of caution'. My impression is you'd be very skeptical of such takes!
*I do mean cases with 'no evidence this is beneficial' rather than 'evidence this is not beneficial'. I haven't gone back through the episodes, so I hope those 3 examples belong on the list!
I get your point, but I don't actually know what my position is on *what we should do*. My only point in the podcast is that *this is a realistic thing to worry about*, that there are people who know an awful lot about the topic and are serious not-crazy types who think there's a non-trivial chance it could kill everyone, and that there's a pretty well-established model for how and why.
How the cost-benefit analysis/expected value thing on, say, AI research regulation or bombing AI data centres shakes out i don't know (to be clear, I don't think we should bomb any AI date centres). But I am trying to make the case that extreme certainty that there ISN'T something to worry about is unwarranted. What we do after that point, god knows.
I guess the difference between that and, say, breastfeeding is that no one says "it's obviously unrealistic to even imagine that breastfeeding could be [good/bad]". There's a complicated scientific question, and on balance we think it comes out as saying the benefits are probably overstated. But if we'd looked at it and the evidence (somehow) said "there's probably no benefit, but we can't rule out about a non-trivial risk that not breastfeeding will literally kill everyone" then I'd be much more keen for people to look into that very closely. And maybe promote breastfeeding in the meantime until we can be sure either way! The severity of the outcome does matter in expected-value calculations, after all.
(Pascal's Wager is a fascinating thought experiment, but since it was first formulated by Nick Bostrom, one of the main AI-risk exponents, I think we can safely say the AI-riskers have considered it.)
The episode did a fair job canvassing the mainstream positions about X risk, but I confess I find many of the mainstream positions indistinguishable from quackery.
The bulk of X-risk worries seem to rest on a deductive argument: an AI with sufficient capabilities could destroy humanity, even if unwittingly; rapid advances in AI continue to produce more and more of these capabilities; and therefore, if we care about human wellbeing, we ought to be seriously worried.
At the outset, it’s worth noting this kind of deductive apocalypticism smacks of Malthusianism. For the unfamiliar reader, Malthus raised a significant kerfuffle among his peers with a deductive argument he published in 1798: population is growing exponentially; food production is growing linearly; and therefore, considering food production can’t catch up, we ought to be worried.
Yet, he was entirely wrong. His prediction failed to account for the human ingenuity and adaptation which rendered the problem moot by means of technological advances in food production.
Of course, Malthus is hardly the only worry-wart with a deductive gotcha even in modern history alone: Y2K, peak oil, global cooling, overpopulation — each turned out to be a farce fueled by little more than a penchant for catastrophizing and a failure of imagination.
While it’s certainly not a decisive blow to point out the suspicious shape of X-risk arguments, I do think it bolts the burden of proof firmly to those demanding we worry. Hitchens Dictum is instructive here: “that which can be asserted without evidence can be dismissed without evidence.”
With a high bar for evidence in mind, then, I turn to three arguments that weigh against the X-worriers I feel are missed in the mainstream conversation.
1. WHERE’S THE BEEF?
Despite the theory of climate change having a distinctively apocalyptic flavor, skeptics were won over because any sensible reading of the theory entails that gradual increases in warming will be associated with gradually increasing harms, a prediction which has in fact been confirmed by multiple lines of evidence.
By analogy, we might ask: If the hypothesis that an AI with sufficient capabilities poses an existential risk to our species, shouldn’t we expect to find that as we pile on capabilities and distribute them widely, we observe associated harms?
Put another way, if superintelligent AI is going to be so awful, not-quite-superintelligent AI should be pretty bad too, right?
By my lights that seems exactly right and yet, to my knowledge, despite having widely deployed fantastical new AI capabilities to a huge swathe of the global population, we have observed zero real world harms, with the exception of the odd Bard conversation that’s gone off the rails.
So X-riskers, where’s the beef?
The reply will perhaps be the real X-risk occurs on a fast or near-vertical “takeoff,” a scenario in which an AI suddenly leaps into a sufficiently dangerous category of intelligence, rendering the harms undetectable to any previous, gradual intelligence gains. While that’s conceptually possible of course, it also stinks of unfalsifiability. And are we not already drowning in unfalsifiable apocalyptic claims, from the political to the religious to name only two? What makes the X-risk claims different from, or more believable than, say the Jones apocalyptic cult, if they cannot be subject to confirmation?
2. % RISK MEASURES FEAR, NOT FACTS
Hang on — we do have solid evidence that experts close to these systems are worried about X-risk. Isn’t that something?
I must confess, I find the way this line of “evidence” is used to be mind-bogglingly anti-scientific. Yes, of course, it’s true that many in the field are worried about X-risk. But what credibility, if any, does that actually lend to the veracity of X-risk claims?
How would we feel about a 1798 survey showing 10% of scientists and economists agreed with Malthus’ overpopulation fears? The simple fact is the 10% would have just been plain wrong. How do we know that isn’t the case here?
Whatever these prediction popularity contest studies are good for, surely they are not a reliable means of evaluating the underlying predictions themselves. Yet, they are often cited as evidence in favor of X-risk.
I cannot help but also point out the mechanism of measurement used in these studies is quite suspicious. They aggregating knee-jerk, gut-level expert intuitions about the risk of a future event that is typically measured in orders of magnitude. But do we really think an expert who imagines there to be a small risk could meaningfully distinguish between two or four orders of magnitude when the risk falls below 1%? I am extremely skeptical — and yet it is often said that even if the experts are off by some order of magnitude it’s a risk worth considering.
Why might these experts be wrong?
3. AI IS NOT DEPLOYED IN A VACUUM
Malthus did not account for human ingenuity or adaptability and by my lights, neither does the X-risk hand wringing. I would be much more concerned about AI safety, for example, if we were not awash in public debate, regulatory frameworks, fail-safes and redundancies, and technologies being developed to steer AI. The reality is AI systems are always deployed into an ecosystem of constantly evolving processes and institutions which have already displayed resilience.
To put a fine point on it, by my lights the relevant question for X-risk is not whether an AI could extinguish our species. It is whether there is good evidence to think that despite all of the safeguards with which AI has and will be deployed, humanity will be unable to mitigate the risk.
Consider the fears about an explosion in misinformation as the result of recent advances in Generative AI. Despite ChatGPT being the most successful product in human history, having garnered nearly a billion users, to my knowledge there has been nothing close to a successful wide-scale misinformation campaign using generative AI. If there were an attempt by a malicious AI or malicious actor using AI, our existing social media channels have (imperfect but basically effective) mechanisms to stop the spread of misinformation.
I suspect many of the safeguards our institutions currently have in place that are supposed to be at risk to an AI takeover are similarly more resilient than imagined. The burden of proof here again seems to be on the X-risk worrier to show otherwise.
—
To be fair, I don’t think these are knock-down-drag-out arguments that definitively establish X-risk has no merit. And I’m open to the evidence — if it can be shown that harms really do obtain on an AI takeoff timeline despite our checks and balances, then so be it. But I do think if I'm right, there is reason to be extremely skeptical of X-risk claims, and the scientifically informed approach would be to reserve judgment until X-riskers can marshal actual evidence to support their hypothesis.
Should that evidence fail to materialize, I regret to say the long tail of history will record the X-risk debacle not for its prescience but for its commentary on the perennial human susceptibility to apocalypticism, even in “rational” quarters.
Really interesting episode, I liked the change of format. I had mostly thought that the fear of AI was overblown sci-fi stuff, but this really makes the concerns tangible.
It made me think of Asimov's three laws of robotics, which I don't think were mentioned. Not sure if they are realistic in this context, but he clearly predicted a need for some sort of safeguard.
Thank you! It's something that I wanted to do because exactly, if you come to it cold it sounds like crazy sci-fi, but actually there are reasonable concerns and the people who laugh it off never seem to me to have engaged with the topic.
Re Azimov's laws: as I'm sure you'll remember from I, Robot etc, about 80% of the narrative point of the laws is showing how they break down in complicated situations. But also, it's not straightforward to put them into a robot: How do you define "harm", "inaction" etc? And with existing neural nets you don't "program" rules in like that, you train behaviours – almost grow them – so it's a very different process.
Thanks for the response! I think I have a pretty primitive understanding of how these programs work. I just bought the audiobook version of your book, which I'm sure it will shed even more light on the topic.
This may interest if you haven't seen it already:
https://twitter.com/PTetlock/status/1649749012029792256
Tetlock is referring to https://static1.squarespace.com/static/635693acf15a3e2a14a56a4a/t/64abffe3f024747dd0e38d71/1688993798938/XPT.pdf and specifically to p51:
Superforecasters and AI domain experts disagreed about the risk of extinction due to AI, but both groups assigned meaningful probability to the outcome and predicted that AI was the most likely cause of extinction among the options included in the XPT. Superforecasters were more skeptical than domain experts: the median superforecaster gave a 0.38% risk of extinction due to AI by 2100, while the median AI domain expert gave a 3.9% risk of extinction.
[...]
we observed little convergence in the groups’ forecasts, even after tournament participants were exposed to each others’ rationales and placed on teams where they could interact.
This is probably the first episode I have any real gripe with. Parts of the episode really rely on this X years to Y outcome framing but in order to get that you were extremely credulous about the words of people with a strong vested interest in overstating AI development. There was no real attempt to broach the self-driving car problem: why should I trust the timelines of an industry that told me truckers would be obsolete four years ago? I think a large chunk of AI skepticism comes from the fact that the same industry has burned people that trusted their maximalist predictions again and again and again. At a certain point you overpredict things so often that I really need a reason to listen to your prediction. The boy who cried wolf cried before and all that.
This is the harshest criticism I can possibly levy: this came off a bit like reading the American Society of Civil Engineers Infrastructure Report Card - ya you can speculate but why should I trust people that routinely overstate things and who have a vested interest IN overstating things?
Hey Joseph! Totally understandable concerns.
Here's my response. First, I get that people in AI have an incentive to overstate how soon it's coming, but it's just not true that people in an industry consistently undershoot on timelines. I gave the Rutherford/nuclear power example in the episode, I think, but a more relevant one: AI specialists thought it would AI becoming superhuman at Go wouldn't happen until the late 2020s or even late 21st century; DeepMind cracked it in 2016.
But anyway, I don't think the "X years to Y outcome" framing is key at all, and if I made it seem like it was then I've failed a bit: the point is just that people in the AI industry think that general AI could come pretty soon. They might be wrong – and as you say, maybe they're likely to be undershooting – but it seems pretty plausible to me that something really powerful could be along in the next 50 years. Maybe it won't! But anyone who tells me it *definitely* won't immediately seems wildly overconfident to me.
(I also don't think AI has overpromised particularly! The progress in recent years has been incredible. Sure, some people made overexcitable claims about autonomous vehicle progress, and it does turn out that's been a bit harder than thought, but AI capabilities have absolutely exploded in the last 10 years.)
And – as I tried to say in the episode – all I'm really interested in getting across is that *this is plausible*. It's really obvious that when you get a system to optimise for some metric, like education systems optimising for exam grades, then the system will maximise those and not necessarily what you care about, like educational progress or children's flourishing or whatever. That's just Goodhart's law. The concern is that superintelligent AI will the same only more so: incredibly powerful at optimising for some goal you give it, but that goal might not be quite what you actually wanted, with disastrous consequences.
I get that it sounds weird, but I've spoken to lots of people in AI who think it's a very realistic danger. And you can say "well AI scientists want to talk up AI capabilities," but that's not obvious to me (you could equally argue that AI scientists would want to downplay the possibility of AI killing everyone: you don't tend to see nuclear scientists overstating the death toll of Chernobyl to make nuclear sound more powerful than it is).
Having followed this debate for six years now, my position is this: there are a lot of serious, senior people in AI, including people in academia who have less obvious financial stake in the system, and people like Geoff Hinton who've quit industry specifically to speak about this stuff, who say 1) that superintelligent AI might be coming fairly soon and 2) that AI could be catastrophically dangerous. There are also some senior people in AI who think that's a silly thing to worry about, although they never seem to address the specific concerns raised. As someone without the technical knowledge but who's followed the arguments, I don't think you can dismiss a small, but real, possibility – maybe one or two percent! But that's still way too high! – of something very very bad happening, and I am very keen that people don't dismiss it as sci-fi because it sounds a bit like The Terminator.
Sorry, that was a way longer post than I intended to write!
That's all totally fair. I think part of the frusteration is the demands that we take this one particular existential risk seriously. It feels weirdly technophobic to have these demands (not saying y'all are necessarily demanding it, but people in AI are) when high-level climate doomers, nuclear preppers, and people afraid of meteors are firmly relegated to loon status. That kinda sums up my beliefs on AI. Yes it's possible but also a lot of things are possible and when you are relegated to a single planet it frankly isn't THAT hard to have existential threats.
I’d really recommend reading Toby Ord’s The Precipice on extinction risks. And Stuart Russell’s human compatible on AI. I’m very convinced that AI and bioengineered pandemics are actual existential risks and the other things basically aren’t
Finally got round to listening to this episode so this comment is very late.
I wanted to comment on the energy consumption of large AI models as a limiting factor. I rather see what Tom described as a limitation of human brains (that they have very low energy consumption and are rather small) as an advantage. Perhaps the first sub-challenge (I forget the technical term Tom mentioned) for an AI wanting to take over the world is to build it's own power station (probably using nuclear fusion). Tom mentioned Moore's Law and in a rather throw away comment admitted it might not be still true. I rather suspect a radical breakthrough on hardware design would be needed for the AI apocalypse to be a threat, as the models may get cleverer but as the consume more and more electricity that becomes the limiting factor.
I think even if some of the pessimists have no financial incentive anymore, if following their advice will lead to centralisation of the industry, that still means we should take their arguments with more salt than we otherwise would.
I enjoyed this episode, but your priors seemed very different to the other topics you've discussed. The broad argument seems to follow Pascal's Wager; there's no evidence that something bad will happen, but the hypothesised bad outcome is severe enough that considerations of expected value (https://en.wikipedia.org/wiki/Expected_value) mean we should act anyway.
How does this differ from the other cases of 'no evidence' that you've discussed? E.g. one could argue that there's no reliable evidence for placebos/breastfeeding/psychedelics/whatever,* but they are likely to do more good than harm, so we should promote them out of an 'abundance of caution'. My impression is you'd be very skeptical of such takes!
*I do mean cases with 'no evidence this is beneficial' rather than 'evidence this is not beneficial'. I haven't gone back through the episodes, so I hope those 3 examples belong on the list!
Hi Mohan! Thanks for listening.
I get your point, but I don't actually know what my position is on *what we should do*. My only point in the podcast is that *this is a realistic thing to worry about*, that there are people who know an awful lot about the topic and are serious not-crazy types who think there's a non-trivial chance it could kill everyone, and that there's a pretty well-established model for how and why.
How the cost-benefit analysis/expected value thing on, say, AI research regulation or bombing AI data centres shakes out i don't know (to be clear, I don't think we should bomb any AI date centres). But I am trying to make the case that extreme certainty that there ISN'T something to worry about is unwarranted. What we do after that point, god knows.
I guess the difference between that and, say, breastfeeding is that no one says "it's obviously unrealistic to even imagine that breastfeeding could be [good/bad]". There's a complicated scientific question, and on balance we think it comes out as saying the benefits are probably overstated. But if we'd looked at it and the evidence (somehow) said "there's probably no benefit, but we can't rule out about a non-trivial risk that not breastfeeding will literally kill everyone" then I'd be much more keen for people to look into that very closely. And maybe promote breastfeeding in the meantime until we can be sure either way! The severity of the outcome does matter in expected-value calculations, after all.
(Pascal's Wager is a fascinating thought experiment, but since it was first formulated by Nick Bostrom, one of the main AI-risk exponents, I think we can safely say the AI-riskers have considered it.)
Thanks again for listening!
Tom
>there are people who know an awful lot about the topic and are serious not-crazy types who think there's a non-trivial chance it could kill everyone,
You're basically saying that we should trust AI doomers more than those other groups because they have more cultural prestige.
Fair enough! & thanks for the detailed reply. Looking forward to the next episode.
The episode did a fair job canvassing the mainstream positions about X risk, but I confess I find many of the mainstream positions indistinguishable from quackery.
The bulk of X-risk worries seem to rest on a deductive argument: an AI with sufficient capabilities could destroy humanity, even if unwittingly; rapid advances in AI continue to produce more and more of these capabilities; and therefore, if we care about human wellbeing, we ought to be seriously worried.
At the outset, it’s worth noting this kind of deductive apocalypticism smacks of Malthusianism. For the unfamiliar reader, Malthus raised a significant kerfuffle among his peers with a deductive argument he published in 1798: population is growing exponentially; food production is growing linearly; and therefore, considering food production can’t catch up, we ought to be worried.
Yet, he was entirely wrong. His prediction failed to account for the human ingenuity and adaptation which rendered the problem moot by means of technological advances in food production.
Of course, Malthus is hardly the only worry-wart with a deductive gotcha even in modern history alone: Y2K, peak oil, global cooling, overpopulation — each turned out to be a farce fueled by little more than a penchant for catastrophizing and a failure of imagination.
While it’s certainly not a decisive blow to point out the suspicious shape of X-risk arguments, I do think it bolts the burden of proof firmly to those demanding we worry. Hitchens Dictum is instructive here: “that which can be asserted without evidence can be dismissed without evidence.”
With a high bar for evidence in mind, then, I turn to three arguments that weigh against the X-worriers I feel are missed in the mainstream conversation.
1. WHERE’S THE BEEF?
Despite the theory of climate change having a distinctively apocalyptic flavor, skeptics were won over because any sensible reading of the theory entails that gradual increases in warming will be associated with gradually increasing harms, a prediction which has in fact been confirmed by multiple lines of evidence.
By analogy, we might ask: If the hypothesis that an AI with sufficient capabilities poses an existential risk to our species, shouldn’t we expect to find that as we pile on capabilities and distribute them widely, we observe associated harms?
Put another way, if superintelligent AI is going to be so awful, not-quite-superintelligent AI should be pretty bad too, right?
By my lights that seems exactly right and yet, to my knowledge, despite having widely deployed fantastical new AI capabilities to a huge swathe of the global population, we have observed zero real world harms, with the exception of the odd Bard conversation that’s gone off the rails.
So X-riskers, where’s the beef?
The reply will perhaps be the real X-risk occurs on a fast or near-vertical “takeoff,” a scenario in which an AI suddenly leaps into a sufficiently dangerous category of intelligence, rendering the harms undetectable to any previous, gradual intelligence gains. While that’s conceptually possible of course, it also stinks of unfalsifiability. And are we not already drowning in unfalsifiable apocalyptic claims, from the political to the religious to name only two? What makes the X-risk claims different from, or more believable than, say the Jones apocalyptic cult, if they cannot be subject to confirmation?
2. % RISK MEASURES FEAR, NOT FACTS
Hang on — we do have solid evidence that experts close to these systems are worried about X-risk. Isn’t that something?
I must confess, I find the way this line of “evidence” is used to be mind-bogglingly anti-scientific. Yes, of course, it’s true that many in the field are worried about X-risk. But what credibility, if any, does that actually lend to the veracity of X-risk claims?
How would we feel about a 1798 survey showing 10% of scientists and economists agreed with Malthus’ overpopulation fears? The simple fact is the 10% would have just been plain wrong. How do we know that isn’t the case here?
Whatever these prediction popularity contest studies are good for, surely they are not a reliable means of evaluating the underlying predictions themselves. Yet, they are often cited as evidence in favor of X-risk.
I cannot help but also point out the mechanism of measurement used in these studies is quite suspicious. They aggregating knee-jerk, gut-level expert intuitions about the risk of a future event that is typically measured in orders of magnitude. But do we really think an expert who imagines there to be a small risk could meaningfully distinguish between two or four orders of magnitude when the risk falls below 1%? I am extremely skeptical — and yet it is often said that even if the experts are off by some order of magnitude it’s a risk worth considering.
Why might these experts be wrong?
3. AI IS NOT DEPLOYED IN A VACUUM
Malthus did not account for human ingenuity or adaptability and by my lights, neither does the X-risk hand wringing. I would be much more concerned about AI safety, for example, if we were not awash in public debate, regulatory frameworks, fail-safes and redundancies, and technologies being developed to steer AI. The reality is AI systems are always deployed into an ecosystem of constantly evolving processes and institutions which have already displayed resilience.
To put a fine point on it, by my lights the relevant question for X-risk is not whether an AI could extinguish our species. It is whether there is good evidence to think that despite all of the safeguards with which AI has and will be deployed, humanity will be unable to mitigate the risk.
Consider the fears about an explosion in misinformation as the result of recent advances in Generative AI. Despite ChatGPT being the most successful product in human history, having garnered nearly a billion users, to my knowledge there has been nothing close to a successful wide-scale misinformation campaign using generative AI. If there were an attempt by a malicious AI or malicious actor using AI, our existing social media channels have (imperfect but basically effective) mechanisms to stop the spread of misinformation.
I suspect many of the safeguards our institutions currently have in place that are supposed to be at risk to an AI takeover are similarly more resilient than imagined. The burden of proof here again seems to be on the X-risk worrier to show otherwise.
—
To be fair, I don’t think these are knock-down-drag-out arguments that definitively establish X-risk has no merit. And I’m open to the evidence — if it can be shown that harms really do obtain on an AI takeoff timeline despite our checks and balances, then so be it. But I do think if I'm right, there is reason to be extremely skeptical of X-risk claims, and the scientifically informed approach would be to reserve judgment until X-riskers can marshal actual evidence to support their hypothesis.
Should that evidence fail to materialize, I regret to say the long tail of history will record the X-risk debacle not for its prescience but for its commentary on the perennial human susceptibility to apocalypticism, even in “rational” quarters.
Really interesting episode, I liked the change of format. I had mostly thought that the fear of AI was overblown sci-fi stuff, but this really makes the concerns tangible.
It made me think of Asimov's three laws of robotics, which I don't think were mentioned. Not sure if they are realistic in this context, but he clearly predicted a need for some sort of safeguard.
Thank you! It's something that I wanted to do because exactly, if you come to it cold it sounds like crazy sci-fi, but actually there are reasonable concerns and the people who laugh it off never seem to me to have engaged with the topic.
Re Azimov's laws: as I'm sure you'll remember from I, Robot etc, about 80% of the narrative point of the laws is showing how they break down in complicated situations. But also, it's not straightforward to put them into a robot: How do you define "harm", "inaction" etc? And with existing neural nets you don't "program" rules in like that, you train behaviours – almost grow them – so it's a very different process.
Thanks for the response! I think I have a pretty primitive understanding of how these programs work. I just bought the audiobook version of your book, which I'm sure it will shed even more light on the topic.