Why is P(Doom) such a bad argument?

Imagine being told there is a precise 12.7% chance you will get into your dream college, or a 38.2% chance of rain on a day with no clouds. In the world of Artificial Intelligence, scientists and philosophers frequently debate **P(Doom)**—the probability that advanced AI will eventually wipe out humanity. But here is the catch: this seemingly scientific number is actually a mathematical illusion. ## The Illusion of Precision In statistics, a true probability requires a track record. We know the chance of a coin flip landing on heads is 50% because we have flipped coins millions of times. This is called *frequentist probability*. With AI doom, we are dealing with a one-time, future event that has never happened before. Therefore, P(Doom) is not a calculation; it is a subjective guess disguised as math. As prominent AI researcher [Yann LeCun](https://en.wikipedia.org/wiki/Yann_LeCun), Chief AI Scientist at Meta, has pointed out, calculating P(Doom) before we even have Human-Level AI is like calculating the probability of a boiler explosion before the steam engine was even invented. ## The Psychological Trap of "Mathiness" By assigning a specific percentage to our fears, we fall into a trap that economists call "mathiness"—using the language of mathematics to make unproven ideas look like hard science. When people state a P(Doom) of 10% or 50%, they are not using a formula. They are expressing a gut feeling of anxiety. Renowned AI critic and researcher [Margaret Mitchell](https://en.wikipedia.org/wiki/Margaret_Mitchell_(scientist)) has argued that focusing on sci-fi scenarios of human extinction distracts us from the very real, measurable harms AI is causing right now, such as algorithmic bias and copyright theft. In his book [*The Precipice: Existential Risk and the Future of Humanity*](https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity), philosopher Toby Ord discusses how we estimate these risks: > "These subjective probabilities are not assertions of dogmatic certainty... They represent our best estimate of the strength of the evidence, given everything we know." However, when "everything we know" is based on speculation rather than data, the resulting percentage is more like creative writing than science. P(Doom) treats a complex, human-controlled future as if it were a roll of the dice, ignoring our own ability to regulate, program, and shape how AI develops. ## Keep Exploring To dive deeper into how we predict the future of technology, consider these questions: 1. If P(Doom) is a bad metric, how *should* we measure and prepare for the long-term risks of technology? 2. How does the psychological phenomenon of [automation bias](https://en.wikipedia.org/wiki/Automation_bias)—our tendency to trust automated systems—make us more likely to believe flawed mathematical predictions? 3. If we shift our focus away from extinction-level events, what are the most urgent AI ethical challenges we need to solve today?

If assigning a precise mathematical probability to a speculative future is so flawed, why did this "P(Doom)" concept attach itself so aggressively to Artificial Intelligence, rather than biotechnology, nanotechnology, or climate change? The answer lies in a unique intersection of Silicon Valley culture, religious psychology, and the specific way we interact with machines that seem to "think." ## The Nerd Religion of Rationalism To understand why P(Doom) became the defining metric of AI safety, we have to look at the community that birthed it. The concept did not originate in traditional academic departments, but in online forums like [LessWrong](https://en.wikipedia.org/wiki/LessWrong), founded by writer [Eliezer Yudkowsky](https://en.wikipedia.org/wiki/Eliezer_Yudkowsky). This community, known as "Rationalists," deeply values Bayesian statistics—a method of updating the probability of a hypothesis as more evidence becomes available. However, they applied this math to their own subjective fears. In doing so, they created a secular version of Pascal's Wager. Philosopher [Émile P. Torres](https://en.wikipedia.org/wiki/%C3%89mile_P._Torres) has written extensively on how this subculture, linked to [Effective Altruism](https://en.wikipedia.org/wiki/Effective_altruism), treats AI with a quasi-religious awe. In this light, P(Doom) is not just a statistic; it is a confession of faith. It allows highly analytical people to process existential dread using the language they trust most: numbers. ## The Mirror of Anthropomorphism We do not talk about "P(Doom)" for climate change because we do not expect a hurricane to have motives. AI is different because we naturally project human traits onto it—a phenomenon called anthropomorphism. When an AI chatbot speaks in the first person, our brains are tricked into treating it as an agent with intent. We fear AI because we imagine it will treat us the way human conquerors have historically treated those they deemed "inferior." In his book [*Superintelligence: Paths, Dangers, Strategies*](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies), philosopher Nick Bostrom popularized the "Paperclip Maximizer" thought experiment, arguing that an AI does not need to hate us to destroy us: > "A superintelligence... would not hate you, nor would it love you, but you are made of atoms which it can use for something else." This narrative turns AI from a software tool into an active, decision-making character in a cosmic drama, making a personalized "doom" score feel intuitive. ## The "God-Builder" Ego Trip Finally, there is a counterintuitive psychological driver behind P(Doom): flatters-the-creator bias. By claiming that their creation has a high chance of ending the world, tech founders are implicitly claiming they are building something of god-like power. It is a brilliant, if terrifying, marketing strategy. Saying "our product might destroy humanity" sounds much more impressive than saying "our product is a very complex autocomplete tool that occasionally makes up facts." By focusing on a sci-fi apocalypse, developers shift the conversation away from boring, immediate problems like worker exploitation and energy consumption, reframing themselves as heroic figures standing on the edge of destiny.

Imagine you are offered a lottery ticket. It costs you almost nothing to hold, but if your numbers come up, the payout is infinite wealth. If you lose, you lose nothing. Would you take the ticket? In the 17th century, the French mathematician, physicist, and philosopher Blaise Pascal argued that human life presents us with this exact choice when it comes to belief in God. Found in his posthumously published work, the [*Pensées*](https://en.wikipedia.org/wiki/Pens%C3 mees), Pascal shifted the debate about God away from difficult metaphysical proofs and toward something brand new: practical risk management. ## The Logic of the Bet Pascal asked us to view belief in God not as a matter of pure reason, but as a decision made under uncertainty. He pointed out that reason alone cannot definitively prove or disprove God's existence. However, you cannot remain neutral; by living your life, you are effectively betting your soul. Pascal mapped out the decision using four possible outcomes: 1. **You believe in God, and God exists:** You gain infinite happiness (Heaven). 2. **You believe in God, and God does not exist:** You lose very little (a few finite pleasures or time spent praying). 3. **You do not believe, and God does not exist:** You gain very little (a bit of personal autonomy). 4. **You do not believe, and God exists:** You face infinite loss or eternal suffering. > "Let us weigh the gain and the loss in wagering that God is. Let us estimate these two chances. If you gain, you gain all; if you lose, you lose nothing. Wager, then, without hesitation that He is." — Blaise Pascal, *Pensées* In modern terms, Pascal created the very first decision matrix, laying the groundwork for [Decision Theory](https://plato.stanford.edu/entries/decision-theory/). Because an infinite reward multiplied by even a tiny probability equals infinity, the expected value of believing in God is infinitely higher than the alternative. ## The Flaws in the Math While the logic seems bulletproof at first glance, philosophers and theologians have argued against Pascal's wager for centuries. The most famous counterargument is the **Many-Gods Problem**. Philosopher Denis Diderot famously noted that a Muslim or a Hindu could use Pascal's exact logic to defend their own faith. If there are thousands of potential gods who punish disbelief in them, how do you know which "lottery ticket" to buy? Picking the wrong god might lead to the same infinite loss Pascal feared. Another major critique focuses on **sincerity**. Can you force yourself to genuinely believe something just because it is profitable? Critics argue that an all-knowing deity would easily spot someone who is merely feigning faith for a safety net. Pascal anticipated this, suggesting that if you act like a believer—attending services and adopting religious habits—true belief will naturally follow over time. ## Questions to Ponder Pascal’s Wager forces us to ask deep questions about why we believe what we do: - Is it rational to believe something based on the *consequences* of the belief rather than the *evidence* for it? - If a decision offers an infinite reward, does that make it rational to ignore extremely low probabilities?