Good People, Bad Systems: Insights From A Complex Systems Approach

Nicky Case approaches complex systems through interactive visualization, highlighting three fundamental concepts: feedback loops (self-reinforcing or self-balancing cycles), attractors (stable states systems naturally gravitate toward), and emergence (when simple components create unexpected collective behaviors). Rather than attempting to predict precise outcomes in chaotic systems, Case suggests we "fight loops with loops" and "move the hills instead of the balls"—redesigning systems' underlying structures rather than merely pushing against them.

In the Hillis-Dyson dialogue, Hillis argues that simple systems naturally evolve into complex entities with their own goals, with corporations and nation-states already functioning as superhuman intelligences operating beyond individual human control. These organizations leverage technology to coordinate at unprecedented scales, developing capabilities no single person could possess. Their conversation examines whether today's technological changes represent a critical evolutionary shift, while debating who controls these emergent systems and how their goals might align with humanity's interests.

These influences shaped my master's thesis proposal: "The Ethical Paradox in Organizations: Why Good People Enable Bad Systems." While focusing on ethical breakdowns, my hypothesis extends to culture's inability to scale. At best, this limitation creates an existential crisis in organizations (and perhaps countries), where culture—the very element that drove initial success—fails to scale effectively. Consequently, shared values and goals cease to drive the organization, allowing bureaucracy, gatekeeping, and mediocrity to dominate.

I believe that organizations, as they grow, exhibit emergent behaviors similar to ant colonies or beehives. Like the individual members of these social insect communities, corporate members remain unaware of their organizations' emergent behaviors—behaviors that include defensive, offensive, predatory, territorial, and reproductive patterns. These emergent properties arise not from individual intention but from the interaction of simpler components and rules.

So why do seemingly ethical individuals collectively participate in unethical organizational behavior? This disturbing paradox has manifested repeatedly in corporate scandals, religious institutions, government agencies, and other large organizations. The Enron scandal, where thousands of employees gradually became complicit in massive fraud, and the Catholic Church's decades-long concealment of abuse illustrate how organizational structures can transform ethical individuals into participants or enablers of misconduct.

This isn't simply about isolated "bad apples"—these are cases where organizational structures and network dynamics transformed initially ethical individuals into participants or enablers of collective misconduct. The question becomes: what properties of organizational networks cause ethical principles to become suppressed or redirected over time?

Ecological predator-prey relationships provide a powerful lens for understanding organizational ethics. Just as predator and prey populations regulate each other through complex feedback loops, organizational networks contain interacting forces that shape ethical behavior:

• Prey (Ethical Behaviors): Represents honest communication, transparency, accountability, and the flow of truthful information within the organization. These "prey" behaviors are essential for organizational health but vulnerable to "predation" under certain conditions.
• Predators (Self-Protective Mechanisms): Represents defensive behaviors that develop to conceal improper conduct, deflect accountability, and shield individuals or groups from the consequences of unethical actions. These include information suppression, scapegoating, plausible deniability structures, and retribution against whistleblowers.

In a healthy ecosystem, predator and prey populations exist in dynamic equilibrium. Similarly, in healthy organizations, mechanisms exist to maintain balance between ethical transparency and reasonable self-protection. However, this balance becomes increasingly difficult to maintain as organizations grow beyond certain thresholds.

I found the predator-prey analogy to be perfect for my thesis where prey represent ethical behavior and predators represent unethical bad players. This is based on the research paper "The Origins and Evolution of Predator-Prey Theory" by Alan A. Berryman, published in the journal Ecology in October 1992.

This paper provides a comprehensive historical overview of predator-prey mathematical models and how they evolved from simple logistic equations to more sophisticated ratio-dependent approaches. The theoretical framework Berryman outlines serves as an excellent foundation for modeling how ethical and unethical behaviors interact within systems, particularly how different modeling approaches might help explain the persistence or collapse of ethical standards under various conditions.

While I'll spare you the detailed mathematical equations, the conceptual model operates on these principles:

Think of ethical behaviors in an organization like a population of helpful creatures, while protective mechanisms are like predators that feed on transparency. In smaller organizations (under 150 people), ethical behaviors can flourish because everyone knows each other and cultural values spread easily through personal relationships. As organizations grow larger, their ability to maintain a cohesive ethical culture naturally weakens.

The model shows that as organization size increases:

• The "carrying capacity" for ethical behaviors decreases (due to weakening cultural transmission)
• The "hunting efficiency" of predatory mechanisms increases (as information silos form and accountability diffuses)
• The "reproduction rate" of protective behaviors accelerates (as incentive structures become more complex)

This creates a mathematical tipping point—a bifurcation in systems theory terms—where beyond a certain size threshold, the system naturally shifts from an ethical equilibrium state to one where protective mechanisms dominate and ethical behaviors become scarce. This tipping point helps explain why we observe so many large organizations experiencing ethical failures despite being composed of individually ethical people.

Research by anthropologist Robin Dunbar suggests humans can maintain stable social relationships with approximately 150 people—now known as "Dunbar's number." Beyond this threshold, our ability to maintain cohesive cultural transmission weakens significantly. Dunbar proposed that this cognitive limit represents the maximum group size where people can maintain personal relationships in which each person knows who everyone else is and how they relate to each other.

In business contexts, this limitation has profound implications. As noted in Roland Berger's research, when groups exceed 150 members, they become increasingly unstable and prone to fragmentation, regardless of their common purpose or identity. Without strong personal relationships, organizational culture becomes fragile and susceptible to ethical deterioration.

Real-world applications of this principle are evident in how some forward-thinking companies structure themselves. For example, manufacturing company W.L. Gore (makers of Gore-Tex) famously caps its factory sizes at 150 people because founder Bill Gore observed that beyond that number, "people began to lose a sense of community" and could no longer "remember faces and names of all the people in the building."

Dunbar's number directly impacts the predator-prey dynamics of organizational ethics:

• Below Dunbar's number (~150 people), ethical norms are transmitted efficiently through direct relationships. Unethical behavior is quickly identified and addressed through social pressure and shared values. In predator-prey terms, prey (ethical behaviors) have plentiful "hiding places" and can "reproduce" (spread) efficiently.
• Beyond Dunbar's number, direct relationships cannot maintain ethical culture across the organization. Formal systems must replace informal norms, creating opportunities for protective behaviors to develop and spread. Predatory mechanisms become more efficient as personal accountability weakens and information silos form.

This connection explains why organizations often begin with strong ethical cultures when small but experience ethical deterioration as they grow. The mathematical modeling suggests this isn't merely coincidental—it's a predictable outcome of fundamental human cognitive limitations interacting with network dynamics.

The Wells Fargo account fraud scandal illustrates how organizational dynamics can transform ethical individuals into participants in systemic misconduct. Between 2002 and 2016, Wells Fargo employees created millions of fraudulent savings and checking accounts without customer consent, leading to billions in fines and settlements.

Former employees described the environment as a "grind-house" with coworkers "cracking under pressure." Many reported that sales goals were impossible to meet, and incentives for compensation and ongoing employment encouraged gaming the system.

What's particularly revealing is that Wells Fargo had multiple controls in place to prevent abuse, including an ethics program and a whistleblower hotline to notify senior management of violations. Yet these safeguards failed to prevent widespread misconduct.

Viewing this through the predator-prey framework:

• Prey (Ethical Behaviors): Initially present throughout the organization but gradually suppressed
• Predators (Protective Mechanisms): Managers ignoring complaints, employees fearing job loss, leadership deflecting accountability
• Hunting Grounds: The diffuse structure of Wells Fargo with its 268,000 employees created fertile territory for predatory mechanisms
• Extinction Event: Ethical information flow was effectively "hunted to scarcity" in retail banking divisions

The Wells Fargo case demonstrates how predator-prey dynamics can overwhelm formal ethical controls when organizations grow beyond the size where cultural values can effectively transmit.

The Volkswagen emissions scandal provides another example of how organizational culture can enable systemic ethical failures. In 2015, it was revealed that Volkswagen had installed "defeat devices" in 11 million diesel vehicles worldwide to cheat on emissions tests, allowing cars to emit up to 40 times the legal limit of nitrogen oxides during normal driving while passing tests in laboratory conditions.

According to analysts, Volkswagen had developed "a culture of fear, pressure, and obedience, where employees were expected to meet unrealistic targets" set by leadership. The scandal wasn't the work of a few rogue engineers but rather reflected systemic failures in the organization's ethical culture.

Shannon Vallor, chair of the department of philosophy at Santa Clara University noted, "This can't be the act of a few rogue engineers...The implications go all the way up the corporate ladder." The sophisticated nature of the fraud required coordination across multiple departments and hierarchical levels.

Through the predator-prey lens:

• Prey (Ethical Behaviors): Engineers' professional standards and honest reporting
• Predators (Protective Mechanisms): Fear-based culture, hierarchical pressure, compartmentalized information
• Efficient Hunting: VW's size (640,000+ employees) and structure made it easy for predatory mechanisms to suppress ethical concerns
• Emergent Behavior: No single individual designed the entire fraud; it emerged from system dynamics

In both the Wells Fargo and Volkswagen cases, the organization's size exceeded the threshold where cultural values could effectively transmit, creating environments where predatory mechanisms naturally gained advantage over ethical behaviors.

Several factors contribute to the transformation of ethical individuals into participants or enablers of unethical organizational behavior:

• Diffusion of Responsibility: As organizations grow, accountability becomes increasingly diffuse. Individual actions seem inconsequential in the larger system. This diffusion weakens the "immune response" against unethical behaviors.
• Information Silos: Larger organizations develop compartmentalized information flows, preventing employees from seeing the full impact of their actions or recognizing systematic problems. These silos create "hunting blinds" from which predatory behaviors can operate undetected.
• Pressure from Leadership: Autocratic leadership styles and unrealistic expectations create environments where ethical corners are cut to meet targets. This functions as an environmental pressure that selects for protective rather than ethical behaviors. This is the leadership style that praises principles and tenets but doesn't believe in them nor practices them.
• Self-Protective Cultures: Many organizations develop cultures of secrecy and cover-up that prioritize protecting the institution over ethical behavior — functioning as "breeding grounds" for predatory mechanisms.
• Psychological Distance: In large organizations, decision-makers experience "psychological distance" from the victims of misconduct, making it easier to rationalize unethical choices. This distance weakens the natural inhibitions against harmful behaviors.
• Motivated Blindness: Leaders and employees alike develop "motivated blindness" where they overlook ethical problems because acknowledging them would hinder success. This functions as an adaptive mechanism that shields predatory behaviors.
• Ethical Fading: As organizations grow, ethics becomes increasingly framed as a compliance issue rather than a core aspect of organizational identity, causing ethical considerations to "fade" from decision-making. This represents the ecosystem shifting from one dominated by ethical behaviors to one dominated by protective mechanisms.

Each of these factors can be understood as amplifying the natural advantage that predatory mechanisms gain as organizations grow beyond Dunbar's number. They represent specific manifestations of the mathematical relationships described in the predator-prey model.

I recognize several limitations in my predator-prey analogy that merit acknowledgment:

First, my analogy doesn't adequately account for incentives. While I mention pressure from leadership and unrealistic expectations, the model doesn't fully integrate how compensation structures, promotion criteria, and other formal incentives fundamentally shape the "ecosystem." In ecological systems, predators hunt from natural instinct, but in organizations, carefully designed incentive systems can dramatically alter behavior patterns in ways not captured by the natural predator-prey relationship.

Second, I believe my model's focus on ethical/unethical behavior is too narrow. As I noted earlier, the same dynamics could apply to bureaucracy, mediocrity, or other organizational dysfunctions. The predator-prey framework might better serve as a model for how any positive organizational quality (innovation, efficiency, transparency) can be systematically undermined by opposing forces as organizations scale, rather than specifically ethics.

Third, I recognize that my analogy creates an artificial binary when organizational behavior exists on a spectrum. In nature, an organism is either predator or prey in a specific relationship, but humans in organizations can simultaneously participate in both ethical and unethical behaviors. The model doesn't capture how the same individual might act ethically in one context while enabling unethical behavior in another, or how "predatory" and "prey" behaviors might gradually blur together rather than existing as distinct categories.

Despite these limitations, I still find the model useful as a tool for tracking how an organization's emergent behavior can lead to unintended or calculated outcomes. These shortcomings present opportunities for further refinement of the conceptual framework.

The ethical paradox in organizations - where good people collectively enable bad systems - isn't inevitable. By understanding the dynamics of how ethical culture deteriorates as organizations scale, we can design structural interventions that maintain ethical equilibrium even in large institutions.

The predator-prey framework provides a powerful lens for understanding these dynamics and identifying leverage points for intervention. Rather than focusing exclusively on individual ethics training or compliance policies, organizations must address the systemic properties that allow ethical deterioration to occur.

By recognizing the inherent limitations of human social capacity and deliberately designing organizations to work with rather than against these limitations, we can create institutions where ethical behavior thrives rather than becomes endangered. This approach moves beyond individual blame to address the true causes of organizational ethical failures.

The most important insight from this analysis is that ethical failures in organizations aren't primarily about bad people making bad choices. They're about predictable system dynamics that naturally emerge as organizations grow beyond certain thresholds. By understanding these dynamics through the predator-prey model and implementing appropriate countermeasures, we can build organizations that scale without sacrificing their ethical foundations.

What I hope to accomplish with my research is developing tools that can track when organizations are trending towards network architectures that can create the conditions for an unbalanced environment, and make recommendations for restructuring that is not based or led by the very actors that created conditions that can lead to an extinction event within an organization's culture.

Authors Note: My expressed views are personal and are not the opinions of my employer, AWS