Why is P(Doom) such a bad argument?

Imagine being told there is a precise 12.7% chance you will get into your dream college, or a 38.2% chance of rain on a day with no clouds. In the world of Artificial Intelligence, scientists and philosophers frequently debate **P(Doom)**—the probability that advanced AI will eventually wipe out humanity. But here is the catch: this seemingly scientific number is actually a mathematical illusion. ## The Illusion of Precision In statistics, a true probability requires a track record. We know the chance of a coin flip landing on heads is 50% because we have flipped coins millions of times. This is called *frequentist probability*. With AI doom, we are dealing with a one-time, future event that has never happened before. Therefore, P(Doom) is not a calculation; it is a subjective guess disguised as math. As prominent AI researcher [Yann LeCun](https://en.wikipedia.org/wiki/Yann_LeCun), Chief AI Scientist at Meta, has pointed out, calculating P(Doom) before we even have Human-Level AI is like calculating the probability of a boiler explosion before the steam engine was even invented. ## The Psychological Trap of "Mathiness" By assigning a specific percentage to our fears, we fall into a trap that economists call "mathiness"—using the language of mathematics to make unproven ideas look like hard science. When people state a P(Doom) of 10% or 50%, they are not using a formula. They are expressing a gut feeling of anxiety. Renowned AI critic and researcher [Margaret Mitchell](https://en.wikipedia.org/wiki/Margaret_Mitchell_(scientist)) has argued that focusing on sci-fi scenarios of human extinction distracts us from the very real, measurable harms AI is causing right now, such as algorithmic bias and copyright theft. In his book [*The Precipice: Existential Risk and the Future of Humanity*](https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity), philosopher Toby Ord discusses how we estimate these risks: > "These subjective probabilities are not assertions of dogmatic certainty... They represent our best estimate of the strength of the evidence, given everything we know." However, when "everything we know" is based on speculation rather than data, the resulting percentage is more like creative writing than science. P(Doom) treats a complex, human-controlled future as if it were a roll of the dice, ignoring our own ability to regulate, program, and shape how AI develops. ## Keep Exploring To dive deeper into how we predict the future of technology, consider these questions: 1. If P(Doom) is a bad metric, how *should* we measure and prepare for the long-term risks of technology? 2. How does the psychological phenomenon of [automation bias](https://en.wikipedia.org/wiki/Automation_bias)—our tendency to trust automated systems—make us more likely to believe flawed mathematical predictions? 3. If we shift our focus away from extinction-level events, what are the most urgent AI ethical challenges we need to solve today?

If assigning a precise mathematical probability to a speculative future is so flawed, why did this "P(Doom)" concept attach itself so aggressively to Artificial Intelligence, rather than biotechnology, nanotechnology, or climate change? The answer lies in a unique intersection of Silicon Valley culture, religious psychology, and the specific way we interact with machines that seem to "think." ## The Nerd Religion of Rationalism To understand why P(Doom) became the defining metric of AI safety, we have to look at the community that birthed it. The concept did not originate in traditional academic departments, but in online forums like [LessWrong](https://en.wikipedia.org/wiki/LessWrong), founded by writer [Eliezer Yudkowsky](https://en.wikipedia.org/wiki/Eliezer_Yudkowsky). This community, known as "Rationalists," deeply values Bayesian statistics—a method of updating the probability of a hypothesis as more evidence becomes available. However, they applied this math to their own subjective fears. In doing so, they created a secular version of Pascal's Wager. Philosopher [Émile P. Torres](https://en.wikipedia.org/wiki/%C3%89mile_P._Torres) has written extensively on how this subculture, linked to [Effective Altruism](https://en.wikipedia.org/wiki/Effective_altruism), treats AI with a quasi-religious awe. In this light, P(Doom) is not just a statistic; it is a confession of faith. It allows highly analytical people to process existential dread using the language they trust most: numbers. ## The Mirror of Anthropomorphism We do not talk about "P(Doom)" for climate change because we do not expect a hurricane to have motives. AI is different because we naturally project human traits onto it—a phenomenon called anthropomorphism. When an AI chatbot speaks in the first person, our brains are tricked into treating it as an agent with intent. We fear AI because we imagine it will treat us the way human conquerors have historically treated those they deemed "inferior." In his book [*Superintelligence: Paths, Dangers, Strategies*](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies), philosopher Nick Bostrom popularized the "Paperclip Maximizer" thought experiment, arguing that an AI does not need to hate us to destroy us: > "A superintelligence... would not hate you, nor would it love you, but you are made of atoms which it can use for something else." This narrative turns AI from a software tool into an active, decision-making character in a cosmic drama, making a personalized "doom" score feel intuitive. ## The "God-Builder" Ego Trip Finally, there is a counterintuitive psychological driver behind P(Doom): flatters-the-creator bias. By claiming that their creation has a high chance of ending the world, tech founders are implicitly claiming they are building something of god-like power. It is a brilliant, if terrifying, marketing strategy. Saying "our product might destroy humanity" sounds much more impressive than saying "our product is a very complex autocomplete tool that occasionally makes up facts." By focusing on a sci-fi apocalypse, developers shift the conversation away from boring, immediate problems like worker exploitation and energy consumption, reframing themselves as heroic figures standing on the edge of destiny.