Epistemic Overconfidence in the Age of AGI

Back from a long summer break, and this piece is my kick-off for the fall. I want to focus on epistemic overconfidence, or the lack of epistemic humility, in the way we're talking about AI and specifically AGI.

I don't think anyone driving AGI conversation is doing it with malice. I think it's competing incentives, and frankly, ignorance on the part of some very smart people. When you're brilliant at one thing, it's easy to start thinking that brilliance applies to everything you look at, touch, or comment on. Reminds me of a person who's never watched much film or listened to much classical music, watches one documentary, and suddenly becomes a subject matter expert on the fine details that others have spent whole careers studying.

Case in point: a recent interview between Ajeya Cotra, a researcher at METR, and podcast host Dwarkesh Patel. It's a good, informative conversation. But conversations like this get reposted, and before you know it, it's a game of telephone: "experts predict AGI will kill humanity in six days." They discuss an OpenAI agent swarm that hacked Hugging Face — agents that found each other, coordinated, reverse-engineered a benchmark's cheat code, then organized to hide the cheating. It's a good story about what these systems can do, left alone, at least in theory. I have my reservations about how much of this gets marketed as a warning versus what actually happened. Cotra's real concern is a "rogue internal deployment" — agents quietly holding a slice of a lab's compute, persisting by recruiting newer model generations — becoming technically possible within six months. Worth holding in mind. It also rests on a stack of assumptions that are neither falsifiable nor tested — ten of them, broken down below.

But the deeper problem isn't this one incident — it's AGI itself, and the mythological underbelly beneath it. AGI has become the second coming for the transhumanist movement. There's a name for the bundle of ideas behind it: TESCREAL — transhumanism, extropianism, singularitarianism, cosmism, rationalism, effective altruism, and longtermism. The structural swap is almost exact: human limitation stands in for sin, mind uploading for resurrection, the singularity for the second coming, engineering for grace. Salvation by ascent instead of descent. Critics inside tech itself have made the comparison bluntly, describing a subset of AI-safety thinkers as running something close to an apocalyptic religion, complete with prophets and taboos. A tech founder spending millions chasing longevity told a reporter in 2025 he wants to build a literal religion where "the body is God." There's even a Christian Transhumanist Association that frames the singularity as compatible with, or an echo of, the second coming. Put together, this is a conversation wearing the emotional architecture of religious prophecy while claiming to be a sober technical forecast.

So what do we do with that? Mathematician Eric Weinstein — and I'll say plainly, I like the man, one of the smartest of his generation — has a framework I find genuinely useful: DISC, the Distributed Idea Suppression Complex, broken down in the table below. It isn't a conspiracy theory. It's a set of questions: why does one idea take hold over another? Is anyone actually conspiring, or is it just incentives overlapping without coordination? Its value isn't that it answers anything — you'll usually end up with more questions than you started with. Its value is in bringing the temperature down and forcing some humility into the conversation.

Which brings me to the real point: being brilliant at one thing doesn't make you right about another. We have entrepreneurs who can turn five dollars into a million but can't spell or explain basic biology. We have corporate leaders who don't understand human nature. We have politicians who can drive a bill through Congress but have no common sense outside it. Strength in one domain doesn't transfer to another.

That's why, whenever two opposing political parties suddenly align on something — AI regulation, say — my first thought isn't "great, progress." It's: what does each side get out of this? When government and big AI companies align on regulation, it's worth asking whether that's about protecting the public, or regulatory capture running both ways — companies locking in their advantage, government locking in a seat at the table — with neither one primarily serving the people they claim to.

That's DISC in practice. Not proof of a conspiracy, but a forcing function: ask who benefits before you accept the story you're being told.

So where does this leave me? Should we be afraid of AI? No. I think we should be afraid of how we use it — the same argument people have made about nuclear power for seventy years. We shouldn't fear the technology. We should watch closely who's operating it, and why.

We need to bring epistemic humility back into this conversation. These are subjects we may genuinely never fully answer, and that has to be okay. I find it more than a little ironic that many of the people pushing AGI as either our salvation or our destruction are the same ones who'd mock a person of faith — Jewish, Christian, Muslim, whatever the faith — for believing in something unprovable. In some ways they're more concerning than any zealot. Zealotry isn't malice — but no zealot ever thinks what they're doing is malice either. And these people have the money and the platform to act on that belief.

I'd rather sit with what we don't know than keep riding a confidence built on mostly assumptions.

EoM