Why is P(Doom) such a bad argument?

“In the world of Artificial Intelligence, scientists and philosophers frequently debate P(Doom)—the probability that advanced AI will eventually wipe out humanity.”

“Address Present Vulnerabilities: Pivot our resources toward tangible, immediate safeguards. In climate policy, this means building resilient infrastructure today. In AI safety, as computer scientist Timnit Gebruen.wikipedia.org argues, it means addressing the immediate, measurable impacts of data exploitation and algorithmic bias right now, rather than obsessing over sci-fi apocalypses.”

“Shift from Risk to Fragility: Stop asking "What are the odds of doom?" Start asking "How fragile is our system if things go wrong?" A bank that requires an accurate probability model to survive is inherently fragile. An AI system whose safety relies on predicting its future intelligence is inherently dangerous.”

“Deciding to act as if something is true because of a potential reward is not the same as genuinely believing it.”

“By testing the limits of infinite rewards, we learn that pure mathematics cannot replace sound judgment. When infinity enters the equation, even the smallest probability can hijack our decisions unless we set rational boundaries.”

“If a decision offers an infinite reward, does that make it rational to ignore extremely low probabilities?”

“Is it rational to believe something based on the consequences of the belief rather than the evidence for it?”

“Applying a neat percentage to the likelihood of a market crash gives investors a false sense of security, making the eventual crash much worse.”

“"We forecast risk paths using past data, assuming the future will look like the past... We build toy models that ignore wild uncertainty, mistaking the map for the territory."”

“By focusing on a sci-fi apocalypse, developers shift the conversation away from boring, immediate problems like worker exploitation and energy consumption, reframing themselves as heroic figures standing on the edge of destiny.”

“It allows highly analytical people to process existential dread using the language they trust most: numbers.”

“Pascal's Wager.”

“P(Doom) treats a complex, human-controlled future as if it were a roll of the dice, ignoring our own ability to regulate, program, and shape how AI develops.”

“By assigning a specific percentage to our fears, we fall into a trap that economists call "mathiness"—using the language of mathematics to make unproven ideas look like hard science.”

“With AI doom, we are dealing with a one-time, future event that has never happened before. Therefore, P(Doom) is not a calculation; it is a subjective guess disguised as math”