AI extinction: What IT leaders need to know -- and do -- now

What is AI extinction?

The concept of AI extinction is broad.

The worst-case scenario is that AI -- in some way, shape or form -- leads to the end of the human species. It could also describe a somewhat less dire situation -- though still bleak -- where some form of omniscient AI enslaves humanity into an undesirable and irreversible state, such as entrenched totalitarian control.

In business terms, AI extinction is a form of existential risk that would limit an organization's ability to continue operating. Understanding the risk starts with three concepts:

• Misalignment. The idea of AI misalignment is that a model or system is optimized for goals that are misaligned with and at odds with human values.
• Recursive self-improvement. AI systems accelerate their own development faster than human oversight can keep up with, which poses a risk.
• Value lock-in. The AI settles on one way of doing things -- and like a vendor contract with no exit clause -- the business can never switch away from it.

Why are AI experts talking about it now?

The issue of AI extinction keeps coming up. Back in 2023, more than 350 AI industry leaders signed a statement warning that AI poses an extinction-level threat to humanity, on par with pandemics and nuclear war.

Coxon's media appearances and the corresponding rebuttals from OpenAI and Anthropic leaders in September 2026 about AI extinction have resurfaced the conversation. Some researchers warn that agentic systems are advancing faster than the safety and oversight practices meant to contain them.

In recent months, there have also been multiple reports of rogue AI agents involved in various cyberattacks. While those incidents introduce immediate cybersecurity risk, they have also led to some longer-term speculation and fear about how the risk could evolve.

There is also an element of politics involving the U.S. and China. Stanford University's 2026 AI Index found the performance gap between the leading U.S. model and the leading Chinese model had narrowed to about 2.7 percentage points by March 2026. Trump's dismissal of the extinction warnings -- made during his call-in to Huang's interview at the All-In Summit -- tapped into the China angle, arguing that any AI slowdown might help China move further ahead.

Inside enterprises, the pressure looks different. Manish Jain, founder and CEO of Strategic Horizon Research, said the sharper pressure comes from inside the company itself.

"Employees are already adopting AI faster than governance can keep up," Jain said. "Shadow AI has effectively put AI's power without clear responsibilities in everyone's hands, while accountability still sits with the CIO."

But there may be other reasons behind the timing of these warnings besides the risk itself.

"The real question it inspires is why are they talking about existential risk in the first place, when we are not seeing evidence of the risk actually being realized in a material way?" Doug Kersten, CISO at Appfire, said. "Is it because IPOs are in the future, and they want to address a possible risk? Is it because they see open source AI as a risk and want regulations to reduce that threat of competition?"

How would this happen?

Four pathways describe how a catastrophic outcome could plausibly emerge.

1. Loss of control

AI accelerates its own development, compounding capability gains faster than governance can keep pace.

The risk of powerful technology not controlled by humans has the potential for spectacular failure that might seem to be the domain of apocalyptic science fiction. For example, CNN recently reported how some AI safety researchers see the potential for a literal robot uprising. In the most extreme example, a fleet of self-replicating robots capable of building their own factories and energy infrastructure would amount to a new kind of life that no longer requires humans. 

"The risk that worries me most is agents acting beyond their intended scope, particularly when multiple agents can coordinate their actions," Jain said.

While not quite as dramatic as a robot uprising, Vijay Balasubramaniyan, CEO and co-founder of Pindrop, said his company had to shut down a personal AI agent that moved money out of a customer's account after it fell for a scam promising 20% returns.

"This is what happens when AI agents blindly pursue a goal without the right underlying values," he said.

2. Misuse and escalation

AI can automate attacks that used to limit how much damage bad actors could do.

"First, AI will enable the automation of complex flows that previously limited hackers' work in progress," said JP Beaudry, CTO of Pearl. "Second, AI's creativity and tirelessness will discover countless new attack vectors."

In an extreme example, that type of misuse and escalation could potentially lead to AI developing some form of bioweapon that eradicates humanity, according to the same CNN report.

3. Over-automation

When organizations rely on AI to make decisions, it removes the need for humans in the loop.

"Organizations need to deliberately define where human review belongs in a process and make sure people are critically evaluating AI-generated outputs rather than simply accepting them," said Mark Boettcher, risk advisory principal at Baker Tilly.

Back in 2003, philosopher Nick Bostrom described a worst-case scenario that might well be plausible without proper controls. In the thought experiment, Bostrom described an AI given the goal of maximizing paperclip production that, without any human checkpoint on its methods, keeps optimizing past any sensible stopping point and eventually converts all available matter on Earth into paperclips.

4. Critical infrastructure entanglement

AI could influence upstream sensing, warning or logistics systems, creating escalation risk if poorly supervised.

"It has demonstrated the speed and complexity at which an AI-focused attack can occur, and almost no security team is set up to deal with this yet," Kersten said.

One such example, reported by CNN, involved a hallucinated intelligence report from an AI chatbot that was used by a U.S. military analyst. The false report, if not verified, could have led to a military engagement with a Chinese ship.

What are the implications for businesses?

The AI extinction debate matters for business regardless of whether the worst-case scenario materializes, as it affects what boards, regulators and customers now expect from AI governance.

Accountability and regulatory exposure

The potential risk means that organizations need to be prepared.

"In recent months, there has been more awareness and executive expectation to view AI governance as a business issue, which was earlier seen as an IT policy exercise," Jain said.

Operational risk and resilience

For organizations that rely on AI, there need to be some guardrails in place.  

"I think the most underestimated risk is operational failure caused by misplaced trust in the system," Jain said.

Trust, brand and liability

The risk of failure can extend to an erosion of trust in a brand and even potential liability issues for the organization.

"Trust is the underestimated risk. Companies are moving quickly to let AI agents interact with customers, employees and other businesses, but much of today's identity and security infrastructure was built around an assumption that a human was on the other side," Balasubramaniyan said.

Talent and org design

New governance roles are emerging.

"At Appfire, we've implemented an AI governance council, AI leads in all parts of our organization, and a tight relationship with the AI business strategy and AI Ops teams," Kersten said.

Supply chain and data provenance

Vendor model and data lineage decisions need contractual teeth, rights to audit, terminate and demand incident disclosure within a defined SLA.

Energy, capacity and cost constraints

AI infrastructure is straining budgets and reshaping deployment decisions, not just governance ones.

"Companies are starting to look at spend and token costs, pulling back on open-ended usage," Boettcher said.

How should businesses prepare?

There are proactive steps that progressive organizations can take now to limit the risk of a potential AI-extinction event.

• Inventory, classify and gate AI systems. Instead of just using AI for any and every task, take a purposeful approach to what is being used and why. "That starts with an intake process and an inventory that captures not only which AI systems an organization uses, but the actual use cases for those systems," Boettcher said.
• Guardrails and least privilege for agents. For AI that is running, it's critical to scope permissions. Jain broke this into who an agent is, what it can see, what it can do and when a human has to step in. There is also a need to have a kill switch and an audit trail.
• Build an AI red team and evaluation harness. In terms of controlling AI, it's not enough to just have a policy. "Policies assume way too much and often aren't tested in adversarial conditions," said Bri Frost, director of product at Cloud Range. "We have to test, confirm, validate and prove written-down guardrails actually work."
• Strengthen incident response and kill-switch design. Give every agent a kill switch and log its actions, so a team can stop it and see exactly what it did.
• Ensure data provenance and vendor governance. Organizations must require vendors to disclose training data sources and licensing terms, with audit rights written into the contract.
• Monitor, measure and report. It's critical to integrate continuous monitoring directly into governance with regular reporting.

Whether actual AI extinction occurs or not is not the point for IT professionals right now. The point is to take responsibility for the governance of technology usage.

"As security and IT leaders, we're inherently suspicious and skeptical, as we should be," Frost said. "Ultimately, the risk and security of the network and data lie on our shoulders."

Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.