MIT study warns of major AI risk. Is governance keeping up?

What the MIT study reveals about AI risk

The study, which was published by MIT FutureTech and the University of Queensland School of Psychology, asked participants to evaluate 24 categories of AI risk over a five-year horizon. Rather than judging whether AI is "good" or "bad," respondents assessed the likelihood, severity and potential for catastrophic outcomes under different development scenarios.

In this context, catastrophic refers to outcomes that involve mass casualties, major global economic disruption or comparable societal harm. The threshold is deliberately high, pushing respondents beyond familiar concerns, such as AI bias, hallucinations and isolated security incidents, instead asking whether AI could plausibly contribute to large-scale, systemic crises.

The research used the Delphi method, a structured forecasting approach in which experts provide responses in multiple rounds with anonymized feedback, designed to reduce bias and move toward a more considered consensus. It's commonly used in nuclear safety and pandemic forecasting to assess low-probability, high-impact risks.

The following were among the most concerning categories the study identified:

• AI-enabled cyberattacks and offensive security capabilities.
• Development of AI-assisted biological or chemical weapons.
• Large-scale misinformation and manipulation.
• Concentration of power in a small number of organizations or governments.
• Competitive pressure leading to unsafe deployment practices.

One of the biggest takeaways was the consensus that the main groups most vulnerable to various AI risks were also the ones least responsible for them.

The study defined catastrophic outcomes as events involving more than one million deaths, more than $100 billion in economic losses or comparable societal harm. Even when improved mitigation scenarios were applied to the examples, five risks still exceeded the 10% threshold: dangerous capabilities, weapons and cyberattacks, environmental harm, inequality and unemployment, and power centralization. All 24 were considered to carry at least a 5% risk.

A consistent theme in the responses was that the most significant risks weren't just technical failures, but rather they reflected systemic misuse and governance breakdowns; that is, how AI is deployed, scaled and shaped by real-world incentives.

Another key finding of the study was an accountability gap. Experts said the primary responsibility for reducing AI risks falls on developers, regulators and standards bodies, yet they identified the general public as the group most likely to experience the consequences if those risks aren't addressed.

Andrew Gamino-Cheong, a participant in the MIT study and the CTO and co-founder of Trustible, a purpose-built AI governance platform, said this imbalance was one of the most striking conclusions of the study. "One of the biggest takeaways was the consensus that the main groups most vulnerable to various AI risks were also the ones least responsible for them," he said.

The finding reinforces a central challenge in AI governance: Those most exposed to AI risks often have the least influence over how systems are developed and deployed.

Taken together, the findings suggest that experts are increasingly concerned with the broader forces shaping AI risk, such as competitive pressure, gaps in governance and the incentives driving AI development. This is happening even though much of today's attention is focused on security-related threats.

Why security is shaping AI governance.

Sectors such as aviation, nuclear energy and pharmaceuticals are built around the idea that even low-probability, critical risks need strict oversight, with slow rollout cycles, mandatory safety testing and centralized regulation. AI doesn't operate under a comparable system. Even as concern among researchers rises, development continues at a rapid pace, competition is intense and governance frameworks are fragmented and inconsistent across jurisdictions.

If it was aviation or nuclear or medication, regulators would respond immediately and be insanely aggressive. For some reason, AI just isn't looked at as catastrophic.

Regulators would likely react differently if another industry faced similar risk assessments, said Russell Twilligear, head of AI research and development at BlogBuster, an automated SEO content production service. "If it were aviation or nuclear or medication, regulators would respond immediately and be insanely aggressive," he said. "For some reason, AI just isn't looked at as catastrophic -- or that it can be."

That difference helps explain why AI governance has largely evolved through a security lens. Security-related threats, such as cyberattacks, disinformation campaigns and the misuse of advanced models for biological, chemical or military purposes, tend to fit existing regulatory and national security frameworks. They're easier to identify and assign ownership to, so they often receive attention before broader societal concerns, such as accountability, labor disruption or concentration of power.

The White House's executive order requiring voluntary early government access to frontier AI models illustrates how policymakers are approaching the issue. The action was primarily driven by concerns about cybersecurity, misuse and national security, rather than broader responsible AI concerns.

While security is an important component of governance, it shouldn't be mistaken for the entire governance challenge, said Trustible's Gamino-Cheong. "You can have a perfectly secure AI model that's resilient to jailbreaking, prompt injections and data leakage, but that doesn't mean it's okay to upload a ton of resumes and ask the AI who you should hire," he said. "That action may be safe technically speaking but has massive ethical and legal implications that need to be addressed."

Tina Paikeday, general manager and senior advisor for responsible AI at Findem, a talent acquisition platform, noted that the distinction between security and responsible AI is becoming increasingly blurred. "They're converging fast, and keeping them siloed is a liability," she said. "A breach and a bias failure are both failures of responsible AI."

According to Paikeday, governance frameworks can no longer treat security and responsible AI as separate tracks because risks in either area ultimately converge on organizational trust, compliance exposure and real-world harm.

Trevor Horwitz, co-founder and chief information security officer (CISO) at TrustNet, a cybersecurity and compliance organization, said many of the risks the MIT study highlights intersect with security concerns, which helps explain why governance efforts often gravitate toward them.

"What stands out in the study is that many of the highest-ranked risks are not purely technical problems," he said. "AI-enabled cyberattacks, misinformation, power concentration and misuse of advanced capabilities are all influenced by human decisions, organizational incentives and governance structures."

Why broader AI risks are difficult to govern

While security risks often fit within existing governance frameworks, many of the broader risks experts highlighted are considerably harder to address.

Organizations move quickly to capture value, while policy, regulation, audit and accountability move much more slowly.

Issues such as bias, transparency, accountability, labor impacts and concentration of power rarely have a single point of failure or a clearly defined owner. Instead, they emerge through organizational decisions, deployment practices and economic incentives that evolve over time.

That complexity makes these issues more difficult to measure, regulate and enforce than traditional cybersecurity threats.

This is because AI risk is increasingly operational rather than theoretical, said Marcelo Lorenzetti, founder and chief AI officer at SavvyLex, an AI-powered platform for legal professionals. "The important point is that AI risk is no longer just about whether a chatbot gives a wrong answer. In real organizations, the risk is becoming operational. It touches security, data access, workflow design, decision-making, accountability and concentration of power," he said.

That shift means governance can't be limited to high-level principles, but must instead be embedded in the systems, controls and processes that determine how AI is actually used.

Several structural factors reinforce this dynamic. AI risk is uncertain and forward-looking, making it difficult to establish clear regulatory thresholds. Competition pushes companies and countries to move fast. Governance responsibilities are distributed across multiple actors. At the same time, the benefits of AI are immediate and visible, while many risks are longer term or uncertain.

These incentives create a predictable governance lag, Lorenzetti said. "The benefits of fast AI deployment are immediate and visible, while the risks are often delayed, distributed and harder to assign to one person or one company," he said. "Organizations move quickly to capture value, while policy, regulation, audit and accountability move much more slowly."

What MIT's findings mean for AI governance

There's a misconception that governance slows you down on the way to getting more out of AI; in reality, governance is how you get more out of AI.

For policymakers and enterprise leaders, the MIT findings aren't merely a warning about future AI risks. They're a test of whether governance systems are equipped to manage the types of risks experts said are becoming increasingly possible.

In many organizations, security teams and CISOs are taking on AI governance responsibilities, while broader responsible AI functions often lack clear ownership or have been deprioritized altogether.

TrustNet's Horwitz said that gap is becoming increasingly visible as organizations deploy AI faster than they can govern it. "Organizations are rapidly deploying AI capabilities across their business operations, but many are still working to understand where AI is being used, what decisions it influences, who owns the associated risks and what governance mechanisms should be in place," he said.

SavvyLex's Lorenzetti said many governance efforts are stuck at the policy level, with insufficient focus on the operational controls needed to manage AI risk. Leaders need clearer visibility into how AI systems are being used, what authority they have, how risky applications are reviewed and who owns responsibility when something goes wrong, he said.

Findem's Paikeday challenged the notion that governance slows innovation. "There's a misconception that governance slows you down on the way to getting more out of AI; in reality, governance is how you get more out of AI."

As AI systems become more capable and widely deployed, pressure is likely to grow for clearer standards, more formal accountability structures, and potentially industry-wide baselines for safety and testing. But the challenge isn't only technical or regulatory; it's institutional. The question is whether governance systems are evolving fast enough to match the systems they're designed to oversee, or they remain shaped by the risks that fit existing security frameworks.

The imbalance between innovation and governance continues to widen, according to BlogBuster's Twilligear. "The concern is moving way faster than the governance response, and everyone knows there are big risks, but no one wants to slow down and lose the race," he said.

That tension might be one of the central issues emerging from the MIT study. Expert concerns are broad, spanning not only security threats but also governance failures, incentive structures and accountability gaps. Yet many visible policy responses continue to focus primarily on security.

Whether governance can expand beyond a security-first framework to address the wider set of risks experts view as consequential might define the next phase of AI oversight.

Kinza Yasar is a technical writer for Informa TechTarget's AI and Emerging Tech group and has a background in computer networking.