Some Thoughts on Experts

Stanley Milgram's famous obedience experiments demonstrated how powerfully people defer to perceived authority figures. In his original study, 65% of participants were willing to administer what they believed were potentially lethal electric shocks when instructed by an authority figure wearing a lab coat. When the experimenter wore ordinary clothes, obedience dropped dramatically to just 20%. Milgram's findings weren't just about authority—they were about our dangerous readiness to defer critical judgment when a symbol of expertise appears. The lab coat served as a powerful symbol that significantly increased compliance, creating a sense of credibility that made participants view the experimenter as more knowledgeable and trustworthy.

Historian Niall Ferguson has documented that Weimar Germany possessed what was arguably the world's most advanced educational system of its time. According to Ferguson, "100 years ago, in the 1920s, by far the best universities in the world were in Germany," with institutions like Heidelberg, Tübingen, and Marburg dominating nearly every academic field. More than a quarter of all Nobel Prizes in sciences between 1901 and 1940 were awarded to Germans, and ambitious academics worldwide needed German credentials to be taken seriously.

Germany's intellectual brilliance didn't inoculate it against catastrophe. In fact, its cultural deference to academic and bureaucratic authority may have paved the way for complicity. This historical example provides a sobering illustration of my central argument: even the most intellectually accomplished societies remain vulnerable when highly educated individuals abdicate critical thinking in favor of blindly following self-proclaimed experts and authoritarian voices.

At issue is highly educated people feeling entitled to remain unchallenged by those with less education or different opinions. The problem starts at home, moves to communities, enters the business world, and eventually reaches government. If we discourage people from challenging ideas just because an expert or self-proclaimed expert says so, we're in trouble. Dissent is the quintessential characteristic of being American, and the cornerstone of a free society.

Why is healthy skepticism or reluctance to accept authority for authority's sake good for society? Research in the field of expert prediction provides empirical evidence for my skepticism.

David Epstein, author of "Range: Why Generalists Triumph in a Specialized World," noted on the "World of DaaS" podcast that research has shown experts often get things wrong more frequently than generalists. Epstein specifically references Philip Tetlock's book "Expert Political Judgment: How Good Is It? How Can We Know?" published in 2005. The research suggested an inverse relationship between fame and accuracy—the more famous some experts are, the more they get things wrong. That's problematic because we listen to these experts who get significant airtime, creating a vicious cycle of wrong forecasting and an audience that is misled.

Tetlock divided his subjects into two groups: "foxes" and "hedgehogs." A fox is defined as a thinker who knows many things, is skeptical of grand theories, adaptable, and ready to adjust ideas based on actual events. A hedgehog knows one big thing, adheres to a grand theory, and expresses views with great confidence. Foxes, who use diverse experiences and remain adaptable in their thinking, consistently outperform hedgehogs in forecasting accuracy. Foxes' diverse thinking, better calibration of probabilities, willingness to learn from mistakes, and open-mindedness contributed to their superior predictions.

Paul Krugman's 1998 prediction about the internet is a perfect example of hedgehog thinking that I never tire of highlighting. Krugman stated: "The growth of the Internet will slow drastically, as the flaw in 'Metcalfe's law'—which states that the number of potential connections in a network is proportional to the square of the number of participants—becomes apparent: most people have nothing to say to each other! By 2005 or so, it will become clear that the Internet's impact on the economy has been no greater than the fax machine's." Krugman was awarded the Nobel Prize in Economics in 2008.

After years of experience, decades of study, and just simple gut feeling, I’ve developed five heuristics I apply when listening to “experts.” (Note, I use these on myself when I am dismissive of arguments against a position I feel strongly about).

First, a true expert should never fear or be annoyed by challenges to their ideas or position. They should welcome those challenges. Richard Feynman exemplified this throughout his career. When a graduate student pointed out a potential flaw in one of his lectures, rather than becoming defensive, Feynman responded, "That's a good point. Let me think about that." He then worked through the problem on the blackboard, acknowledging the validity of the challenge. Feynman famously said, "The first principle is that you must not fool yourself—and you are the easiest person to fool," highlighting his belief that experts should actively seek challenges to their thinking.

Second, an expert would never dismiss challenges as conspiratorial. Instead, they should welcome the opportunity to demonstrate their knowledge while acknowledging its limits. Hans Rosling, the renowned Swedish physician and statistician who gained fame for his data-driven presentations on global development, exemplifies this principle. Despite achieving celebrity status through his TED talks and bestselling book "Factfulness," Rosling actively engaged with his critics rather than dismissing them.

When challenged by environmental scientists and population researchers who argued his optimistic views on global development overlooked serious resource constraints and environmental degradation, Rosling didn't characterize their critiques as conspiracies against progress. Instead, he acknowledged the legitimate concerns while maintaining his evidence-based perspective on improvement trends. In debates with critics like Christian Berggren and the Ehrlichs (noted population alarmist), Rosling demonstrated intellectual honesty by engaging with their substantive arguments rather than questioning their motives or dismissing them. This willingness to address criticism on its merits rather than deflecting through ad hominem attacks demonstrates how true experts maintain their credibility by engaging substantively with challenges to their work.

Third, an expert should never rely on "because I said so." They must provide substantial evidence supporting their position—not just broad claims. Thomas Piketty's approach in "Capital in the Twenty-First Century" demonstrates this principle. Rather than making broad claims about inequality based solely on his authority, Piketty assembled an unprecedented database spanning three centuries and over 20 countries. He made his complete datasets publicly available online for verification and scrutiny. When challenged on his conclusions about wealth concentration, Piketty didn't resort to his credentials but pointed critics to specific data points and methodologies, allowing others to test his conclusions independently.

Fourth, experts should willingly acknowledge when they've been wrong. Alan Greenspan demonstrated this in his testimony before Congress in October 2008 during the financial crisis. After decades of advocating for financial deregulation based on his belief that markets would self-regulate effectively, Greenspan admitted he had been wrong. He stated: "I made a mistake in presuming that the self-interests of organizations, specifically banks and others, were such that they were best capable of protecting their own shareholders and their equity in the firms." Greenspan's admission shattered decades of ideological certainty. That's real intellectual integrity.

Fifth, an expert should not cherry-pick data and evidence. Steven Koonin, author of "Unsettled: What Climate Science Tells Us, What It Doesn't, and Why It Matters," demonstrates this principle in his approach to climate science. Rather than selecting only data that supports a particular conclusion, Koonin examines the full range of evidence, including contradictory findings. He analyzes the complete record of temperature data, sea level measurements, and extreme weather events, noting both alarming and non-alarming trends. Koonin differentiates between confirmed scientific findings and projections that contain greater uncertainty, providing readers with a comprehensive view that acknowledges areas of both consensus and ongoing debate. By refusing to cherry-pick only data points that support a predetermined conclusion, Koonin exemplifies how experts should present evidence holistically, even when it complicates the narrative.

The heuristics for evaluating expertise become particularly vital in technology domains, where the pace of change and complexity often create fertile ground for expert overconfidence. Technology leaders facing decisions about artificial intelligence, cybersecurity, or digital transformation are especially vulnerable to the siren call of expertise—the reassuring voice that promises certainty in uncertain terrain. Yet this is precisely where blind deference proves most dangerous. We've witnessed this repeatedly: Y2K hysteria, blockchain evangelism, metaverse proclamations, and countless enterprise software implementations that became costly failures because leaders subordinated their judgment to self-proclaimed experts making absolutist claims. Technology domains are especially prone to expert capture because their complexity intimidates many decision-makers into abdication. The result? Organizations implement solutions nobody fully understands or needs, pursue digital transformations without clear purpose, and chase AI implementations because experts declare them essential, not because they solve real problems. Leaders who apply the five heuristics when evaluating technological expertise will discover that the most valuable technology advisors acknowledge limitations, present balanced evidence, welcome challenges to their thinking, demonstrate willingness to change position when warranted, and never resort to appeals to their own authority or credentials as substitutes for substantive explanation. This approach doesn't diminish the value of specialized knowledge, but rather prevents the kind of expert worship that leads to technological decisions divorced from business reality and human needs.

Authors Note: My expressed views are personal and are not the opinions of my employer, AWS