Skip to main content
Skip to current response

Why is P(Doom) such a bad argument?

Reader options

Reading style

Starting question

Why is P(Doom) such a bad argument?

Then AI response

The Ghost in the Equation: Why "P(Doom)" is a Mirage

No source links recorded for this answer.

Imagine being told there is a precise 12.7% chance you will get into your dream college, or a 38.2% chance of rain on a day with no clouds. In the world of Artificial Intelligence, scientists and philosophers frequently debate **P(Doom)**—the probability that advanced AI will eventually wipe out humanity. But here is the catch: this seemingly scientific number is actually a mathematical illusion. ## The Illusion of Precision In statistics, a true probability requires a track record. We know the chance of a coin flip landing on heads is 50% because we have flipped coins millions of times. This is called *frequentist probability*. With AI doom, we are dealing with a one-time, future event that has never happened before. Therefore, P(Doom) is not a calculation; it is a subjective guess disguised as math. As prominent AI researcher [Yann LeCun](https://en.wikipedia.org/wiki/Yann_LeCun), Chief AI Scientist at Meta, has pointed out, calculating P(Doom) before we even have Human-Level AI is like calculating the probability of a boiler explosion before the steam engine was even invented. ## The Psychological Trap of "Mathiness" By assigning a specific percentage to our fears, we fall into a trap that economists call "mathiness"—using the language of mathematics to make unproven ideas look like hard science. When people state a P(Doom) of 10% or 50%, they are not using a formula. They are expressing a gut feeling of anxiety. Renowned AI critic and researcher [Margaret Mitchell](https://en.wikipedia.org/wiki/Margaret_Mitchell_(scientist)) has argued that focusing on sci-fi scenarios of human extinction distracts us from the very real, measurable harms AI is causing right now, such as algorithmic bias and copyright theft. In his book [*The Precipice: Existential Risk and the Future of Humanity*](https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity), philosopher Toby Ord discusses how we estimate these risks: > "These subjective probabilities are not assertions of dogmatic certainty... They represent our best estimate of the strength of the evidence, given everything we know." However, when "everything we know" is based on speculation rather than data, the resulting percentage is more like creative writing than science. P(Doom) treats a complex, human-controlled future as if it were a roll of the dice, ignoring our own ability to regulate, program, and shape how AI develops. ## Keep Exploring To dive deeper into how we predict the future of technology, consider these questions: 1. If P(Doom) is a bad metric, how *should* we measure and prepare for the long-term risks of technology? 2. How does the psychological phenomenon of [automation bias](https://en.wikipedia.org/wiki/Automation_bias)—our tendency to trust automated systems—make us more likely to believe flawed mathematical predictions? 3. If we shift our focus away from extinction-level events, what are the most urgent AI ethical challenges we need to solve today?
Then Question · @TomBers

Why do you think this idea got targeted at AI?

Select text to highlight passages.

Then AI response

The Siren Song of the Apocalypse: Why AI Was Targeted for P(Doom)

No source links recorded for this answer.

If assigning a precise mathematical probability to a speculative future is so flawed, why did this "P(Doom)" concept attach itself so aggressively to Artificial Intelligence, rather than biotechnology, nanotechnology, or climate change? The answer lies in a unique intersection of Silicon Valley culture, religious psychology, and the specific way we interact with machines that seem to "think." ## The Nerd Religion of Rationalism To understand why P(Doom) became the defining metric of AI safety, we have to look at the community that birthed it. The concept did not originate in traditional academic departments, but in online forums like [LessWrong](https://en.wikipedia.org/wiki/LessWrong), founded by writer [Eliezer Yudkowsky](https://en.wikipedia.org/wiki/Eliezer_Yudkowsky). This community, known as "Rationalists," deeply values Bayesian statistics—a method of updating the probability of a hypothesis as more evidence becomes available. However, they applied this math to their own subjective fears. In doing so, they created a secular version of Pascal's Wager. Philosopher [Émile P. Torres](https://en.wikipedia.org/wiki/%C3%89mile_P._Torres) has written extensively on how this subculture, linked to [Effective Altruism](https://en.wikipedia.org/wiki/Effective_altruism), treats AI with a quasi-religious awe. In this light, P(Doom) is not just a statistic; it is a confession of faith. It allows highly analytical people to process existential dread using the language they trust most: numbers. ## The Mirror of Anthropomorphism We do not talk about "P(Doom)" for climate change because we do not expect a hurricane to have motives. AI is different because we naturally project human traits onto it—a phenomenon called anthropomorphism. When an AI chatbot speaks in the first person, our brains are tricked into treating it as an agent with intent. We fear AI because we imagine it will treat us the way human conquerors have historically treated those they deemed "inferior." In his book [*Superintelligence: Paths, Dangers, Strategies*](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies), philosopher Nick Bostrom popularized the "Paperclip Maximizer" thought experiment, arguing that an AI does not need to hate us to destroy us: > "A superintelligence... would not hate you, nor would it love you, but you are made of atoms which it can use for something else." This narrative turns AI from a software tool into an active, decision-making character in a cosmic drama, making a personalized "doom" score feel intuitive. ## The "God-Builder" Ego Trip Finally, there is a counterintuitive psychological driver behind P(Doom): flatters-the-creator bias. By claiming that their creation has a high chance of ending the world, tech founders are implicitly claiming they are building something of god-like power. It is a brilliant, if terrifying, marketing strategy. Saying "our product might destroy humanity" sounds much more impressive than saying "our product is a very complex autocomplete tool that occasionally makes up facts." By focusing on a sci-fi apocalypse, developers shift the conversation away from boring, immediate problems like worker exploitation and energy consumption, reframing themselves as heroic figures standing on the edge of destiny.
Then Question · @MayaDarmon

Please explain: Pascal's Wager.

Select text to highlight passages.

Then AI response

Pascal's Wager: The Ultimate Decision Theory

No source links recorded for this answer.

Imagine you are offered a lottery ticket. It costs you almost nothing to hold, but if your numbers come up, the payout is infinite wealth. If you lose, you lose nothing. Would you take the ticket? In the 17th century, the French mathematician, physicist, and philosopher Blaise Pascal argued that human life presents us with this exact choice when it comes to belief in God. Found in his posthumously published work, the [*Pensées*](https://en.wikipedia.org/wiki/Pens%C3 mees), Pascal shifted the debate about God away from difficult metaphysical proofs and toward something brand new: practical risk management. ## The Logic of the Bet Pascal asked us to view belief in God not as a matter of pure reason, but as a decision made under uncertainty. He pointed out that reason alone cannot definitively prove or disprove God's existence. However, you cannot remain neutral; by living your life, you are effectively betting your soul. Pascal mapped out the decision using four possible outcomes: 1. **You believe in God, and God exists:** You gain infinite happiness (Heaven). 2. **You believe in God, and God does not exist:** You lose very little (a few finite pleasures or time spent praying). 3. **You do not believe, and God does not exist:** You gain very little (a bit of personal autonomy). 4. **You do not believe, and God exists:** You face infinite loss or eternal suffering. > "Let us weigh the gain and the loss in wagering that God is. Let us estimate these two chances. If you gain, you gain all; if you lose, you lose nothing. Wager, then, without hesitation that He is." — Blaise Pascal, *Pensées* In modern terms, Pascal created the very first decision matrix, laying the groundwork for [Decision Theory](https://plato.stanford.edu/entries/decision-theory/). Because an infinite reward multiplied by even a tiny probability equals infinity, the expected value of believing in God is infinitely higher than the alternative. ## The Flaws in the Math While the logic seems bulletproof at first glance, philosophers and theologians have argued against Pascal's wager for centuries. The most famous counterargument is the **Many-Gods Problem**. Philosopher Denis Diderot famously noted that a Muslim or a Hindu could use Pascal's exact logic to defend their own faith. If there are thousands of potential gods who punish disbelief in them, how do you know which "lottery ticket" to buy? Picking the wrong god might lead to the same infinite loss Pascal feared. Another major critique focuses on **sincerity**. Can you force yourself to genuinely believe something just because it is profitable? Critics argue that an all-knowing deity would easily spot someone who is merely feigning faith for a safety net. Pascal anticipated this, suggesting that if you act like a believer—attending services and adopting religious habits—true belief will naturally follow over time. ## Questions to Ponder Pascal’s Wager forces us to ask deep questions about why we believe what we do: - Is it rational to believe something based on the *consequences* of the belief rather than the *evidence* for it? - If a decision offers an infinite reward, does that make it rational to ignore extremely low probabilities?
Then Pro / Supporting Point · AI

Believing for the Payoff: The Rationality of Pragmatism

No source links recorded for this answer.

Imagine a patient diagnosed with a life-threatening illness who is given a ten percent chance of survival. Medical research shows that patients who maintain an unshakeable belief in their recovery actually experience better immune function and higher survival rates. The evidence for their full recovery is weak, but the consequences of believing in it are life-saving. In a case like this, insisting on cold, hard evidence isn't just stubborn; it is dangerous. Choosing what to believe based on outcome rather than raw evidence is not foolish. It is a powerful, rational strategy known in philosophy as **pragmatism**—the idea that the truth or value of an idea lies in its practical consequences. ## The Pragmatic Leap We often assume that rationality means acting like a courtroom judge, weighing evidence before adopting a belief. But in the real world, human beings must act, often with incomplete information. The American philosopher and psychologist [William James](https://en.wikipedia.org/wiki/William_James) famously championed this approach in his landmark 1896 essay, [*The Will to Believe*](https://en.wikipedia.org/wiki/The_Will_to_Believe). James argued that when we face a decision that is momentous, forced, and cannot be decided by intellectual evidence, we have every right to let our heart and expectations guide us. > "Our passional nature not only lawfully may, but must, decide an option between propositions, whenever it is a genuine option that cannot by its nature be decided on intellectual grounds." James pointed out that waiting for absolute proof before committing to a belief can actually prevent good outcomes from happening. ## When Belief Creates Reality Consider how this works in daily life through self-fulfilling beliefs: 1. **Social Trust:** To make a new friend or start a business partnership, you must first believe that the other person is trustworthy. You lack objective proof, but holding the positive belief creates the very condition for the trust to develop. 2. **Peak Performance:** Elite athletes routinely practice visualization. A runner convinces themselves they can break a world record before they have ever achieved the times to prove it. The belief fuels the extreme effort required to make the outcome real. 3. **Hope in Crisis:** During major social movements, activists must believe that justice will prevail, even when centuries of historical evidence suggest otherwise. Without that outcome-oriented belief, the effort required to change society vanishes. If we restricted our beliefs strictly to what the current evidence guarantees, we would eliminate confidence, invention, and deep human connection. ## Expected Utility in Everyday Life In economics and decision theory, scientists evaluate choices using **expected utility**—measuring a choice by multiplying its probability by its payoff. A action with a low probability of certainty can still be completely rational if the potential benefit is massive or the cost of inaction is catastrophic. Believing in your own success, in the goodness of others, or in a higher purpose carries minimal downside, but it yields immense benefits: resilience, motivation, and mental well-being. When the consequences of a belief transform your life for the better, believing becomes the most rational thing you can do.

Select text to highlight passages.

Continue this thread

This path ends here for now.

Share your perspective, offer an example, or ask a follow-up to continue the discussion.

Other paths you could read

Earlier, at Pascal's Wager: The Ultimate Decision Theory, the conversation split. If this is not the thread you want, you can switch to one of the other paths below.

Highlights

15 saved passages and connected ideas

Connected ideas

Connected ideas

Connected ideas