AI existential risk: Is AI a threat to humanity?
What should enterprises make of the recent warnings about AI's threat to humanity? AI experts and ethicists offer opinions and practical advice for managing AI risk.
Rapid innovation in AI has fueled debate among industry experts about the existential threat posed by so-called intelligent machines that can perform tasks previously done by humans.
Artificial general intelligence (AGI) refers to machines that are able to think and experience the world like a human. Doomsayers argue that AGI will be upon us sooner than expected and have the capability to outwit people. Shorter term, they warn our overreliance on AI systems could spell disaster: Disinformation will flood the internet, terrorists will craft dangerous and cheap weapons, superintelligent AI models will lead to mass unemployment, automated AI could start a nuclear war and killer drones could run rampant.
Even Geoffrey Hinton, who's referred to as the "godfather of AI" for his seminal work on neural networks, has expressed growing concerns over AI's threat to humanity, He issued a warning in 2023 about the rapidly advancing abilities of generative AI chatbots, like ChatGPT. "Right now, they're not more intelligent than us, as far as I can tell," he told the BBC. "But I think they soon may be." Hinton predicted it would take 5 to 10 years, rather than his previous timeline of 30 to 50 years.
Rising concerns about AI's existential risks have led to calls for moratoriums on AI R&D -- an AI pause -- from industry and academic experts, including executives at many companies fueling AI innovation. Yet even staunch American AI doomerists like Sam Altman and Elon Musk have accelerated their AGI efforts.
This article is part of
What is GenAI? Generative AI explained
Davi Ottenheimer
Others argue that this AI doom narrative distracts from more likely AI dangers enterprises need to heed: AI bias, inequity, inequality, hallucinations, new failure modes, privacy risks and security breaches. A big concern among people in this group is that a pause might create a protective moat for major AI companies, like OpenAI, maker of ChatGPT and an AI pause advocate.
"Releasing ChatGPT to the public while calling it dangerous seems little more than a cynical ploy by those planning to capitalize on fears without solving them," said Davi Ottenheimer, vice president of trust and digital ethics at Inrupt, a secure data-sharing platform provider. A bigger risk might lie in enabling AI doomsayers to profit by abusing our trust, he said.
The existential threat discourse "turned out to be a new form of mysticism, to rope in true believers who wouldn't question the premise," Ottenheimer argued. Every major AI company gave what he called "saccharin safety pledges," then abandoned them. For enterprise leaders evaluating vendors, Ottenheimer offered a diagnostic: "The tells are architectural. When a vendor says 'safety,' ask where the controls are enforced." If they can't show evidence within a trend line, the commitment is decorative.
Responsible AI or virtue signaling?
Signed letters calling for an AI pause get a lot of media play, but they beg the question of what happens next.
The late Abhishek Gupta, who founded the Montreal AI Ethics Institute, was one of many prominent AI ethicists who viewed the calls for an AI pause ineffective, if not disingenuous. "I find it difficult to sign letters that primarily serve as virtue signaling without any tangible action or the necessary clarity to back them up," Gupta said in an interview in 2023. Such letters are often counterproductive as they consume attention cycles without leading to any real change, he noted.
Media-fueled doomsday narratives consume valuable time and resources and, in the process, they confuse the discourse and potentially put a lid on the levelheaded conversation required to make sound policy decisions, according to Gupta and like-minded ethicists. People seeking to manage AI risks are better off educating themselves on the actual risks instead of focusing on muddled musings about existential threats. They need to collaborate with technical experts who have practical experience in developing production-grade AI and machine learning systems, as well as with academic professionals who work on the theoretical foundations of AI.
Building consensus on what needs to be tackled to ensure responsible AI programs shouldn't be that hard.
What are realistic AI risks?
Brian Green
If AI doomerism isn't likely to prove useful in controlling AI risks, how should enterprises be thinking about the problem?
It's helpful to frame AI risks as those that come from AI itself and risks that come from the use of AI by humans, said Brian Green, director of technology ethics at the Markkula Center for Applied Ethics at Santa Clara University.
Risks from the AI technology itself range from simple errors in computation that lead to bad outcomes, to the existential threat of AI gaining a will of its own and deciding to attack humankind, Green said. At present, there's no clear way for the latter to happen, he said. Green is the co-author of Ethics in the Age of Disruptive Technologies: An Operational Roadmap, a handbook that lays out what he considers practical steps businesses can take to make ethical decisions
Risks stemming from the human use of AI cover every action people can imagine. They could be automated and made more efficiently evil with AI, such as more centralized nuclear weapons with hair-trigger controls, more powerful disinformation campaigns, deadlier biological weapons and more effective planning for social control.
"Everything horrible that human intelligence can do, artificial intelligence might be programmed to do as well as or better than humans," Green said. "There are vastly more chances that humans might use AI for existentially risky purposes than there are chances that AI would just pursue these goals on its own."
Green said he believes the realistic AI existential risks we face are more mundane: AI-generated content trained to catch our attention that inadvertently blinds us to important issues, or AI-based marketing apps trained to lure us into buying products and services detrimental to our well-being.
"I would argue that both of these things are already happening, so this possible existential AI risk is already upon us and is, therefore, 100% real," Green said.
Nell Watson, an AI ethics researcher and author, identified a risk that most frameworks miss: AI-automated psychological decomposition at scale. AI could use tactics similar to the Zersetzung approach of East Germany's Stasi secret police, she said, in which gaslighting, reputation destruction and social isolation were used to break individuals covertly.
Agentic AI makes this scalable and personalized in ways that were previously impossible, Watson added. The infrastructure is already built; recommender systems optimized for engagement can profile individual vulnerabilities and craft personalized content to exploit them, she explained. This approach becomes effective when the optimization target shifts from attention to influence. A new field of psychosecurity is needed, according to Watson, dedicated to protecting cognitive integrity, in the same way cybersecurity protects digital infrastructure.
It's important to keep on top of known problems, such as AI bias and misalignment with organizational objectives, Green said: "Immediate problems that are ignored can turn into big problems later, and conversely, it's easier to solve big problems later if you first get some practice solving problems now."
Could AI stir up social unrest by displacing workers?
Andrew Pery
How AI technology is changing the nature of work is an issue companies should focus on now, said Andrew Pery, AI ethics evangelist at Abbyy, an intelligent automation company. "With the commercialization of generative AI, the magnitude of labor disruption could be unprecedented," he said. The International Monetary Fund predicted that nearly 40% of global employment is exposed to AI.
"Such a dramatic displacement of labor is a recipe for growing social tensions by shifting millions of people to the margins of society with unsustainable unemployment levels and without the dignity of work that gives us meaning," Pery said. Labor displacement could give rise to more nefarious and dangerous uses of GenAI technology that subvert the foundations of a rule-based order, he added.
Fostering digital upskilling for new jobs and rethinking social safety-net programs will play a pivotal role in safely transitioning into an age of AI, Pery said. However, he added that there's an even deeper structural concern: GenAI is replacing apprenticeships, eliminating the roles through which people learn how to become experts. The 2026 Anthropic labor market study found evidence that hiring of younger workers has slowed in highly exposed occupations, even though aggregate unemployment hasn't yet significantly increased.
Reskilling, Pery argued, has become the moral placebo of the AI age: It sounds responsible but often means little. Workers need wage insurance, transition stipends, portable benefits and mechanisms for sharing automation dividends. California's SB 53: Transparency in Frontier Artificial Intelligence Act and New York's Responsible AI and Safety Education Act impose obligations on frontier developers to publish safety frameworks and disclose catastrophic risk assessments, Pery said. They function as a de facto national benchmark in the absence of comprehensive federal legislation.
How enterprises can manage AI risks
A key component of responsible AI is identifying and mitigating risks arising from AI systems. These risks can manifest in various forms, including data privacy breaches, biased outputs, AI hallucinations, deliberate attacks on AI systems, and concentration of power in compute and data.
AI experts recommended enterprises and stakeholders take a holistic and proactive approach that considers the potential effect of each AI risk across different domains and stakeholders to prioritize these risk scenarios effectively. This approach requires a deep understanding of AI systems and their algorithmic biases, the data inputs used to train and test the models, and the potential vulnerabilities and attack vectors that hackers or malicious actors can exploit.
AI risk heat map
A practical approach applies the methods used in cybersecurity that evaluate risks according to their probability and severity. Many risks haven't been identified yet, so businesses would need to distinguish between areas of uncertainty and known potential risks to build their heat maps. Uncertainty considers the unknown unknowns, while risk refers to assessment based on known unknowns.
Trustworthy AI pledge
Pery suggested that enterprises make a top-down organizational commitment to trustworthy AI principles and guidelines. Trustworthy AI includes human-centered values of fairness of AI outcomes, accuracy, integrity, confidentiality, security, accountability and transparency associated with the use of AI.
Organizations that offer frameworks for trustworthy AI include the following:
- Organization for Economic Cooperation and Development.
- Berkman Klein Center at Harvard University.
- Stanford Center for Human-Centered Artificial Intelligence.
- AI Now Institute.
In addition, the NIST AI Risk Management Framework provides a comprehensive roadmap for implementing responsible AI best practices and a model for mitigating potential AI harms. Other standards that businesses might consider include the ISO/IEC 23894:2023 framework and the EU-sponsored AI governance framework by the European Committee for Standardization and the European Electrotechnical Committee for Standardization.
Checklist of questions for monitoring AI
Enterprises should implement measures to ensure continuous human monitoring of AI system performance. These steps can include identifying potential deviations from expected outcomes, taking remediation steps to correct adverse results and including processes for overriding automated decisions by AI systems.
'Kimberly Nevala, strategic advisor at SAS, recommended companies consider the following questions:
Kimberly Nevala
- How will and could this solution go astray or make errors?
- Is intentional misuse probable and in what circumstances?
- How might the system be inadvertently misunderstood or misapplied?
- What are the impacts, and how do they scale?
- Does the system's design exacerbate or attenuate the potential for misuse and misunderstanding?
- How might this system be integrated into or influence others within and beyond our scope of control?
- What might be the second- or third-order effects thereof?
Will AI regulation help or harm?
Governments worldwide are starting to draft AI regulations that might prevent some of the worst AI risks. But poorly drafted regulations could slow the adoption of AI applications that solve some of our more pressing problems in healthcare and sustainable development -- or they could create new problems. Some industry observers have argued that regulation could stifle innovation and create a regulatory moat around incumbent companies, limiting competition and disruption from startups.
Effective AI regulation requires being specific about the processes and requirements to make AI transparent, understandable and safe. Drafting broad laws that focus on whether a process is ultimately harmful won't take us nearly far enough.
Discussions about regulating AI could take a cue from the regulations that helped us transition through the Industrial Revolution, such as minimum AI safety standards for work conditions, minimum pay requirements, child labor restrictions and environmental standards. The AI equivalent would address how processes should be regulated, including how they use data, whether and how they drive decisions, and how we can ensure AI's processes remain transparent, understandable and accountable.
Tackling existential AI risks will require identifying and addressing the present dangers of the AI systems we're deploying today, Nevala reasoned. "This is an issue that will only be addressed through a combination of public literacy and pressure, regulation and law, and -- history sadly suggests -- after a yet-to-be-determined critical threshold of actual harm has occurred," she said.
George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.
