AI slowdown: Do we really need it?

Focus on AI control, not slowdown

However, the debate should not be summarized as whether AI development should stop entirely. In reality, an AI slowdown is difficult to achieve because AI development is spread across different companies, research institutes and countries. Instead, the main focus should be on whether our ability to understand, control and secure AI usage is keeping pace with its development.

If the gap between development and control continues to grow, the risk will not simply come from developing a superintelligent machine powered by AI. It could come from exploiting traditional problems: giving excessive permissions to AI agents, insecure integration of AI systems with external systems, poorly tested autonomous actions. This also includes the malicious use of AI or humans relying on AI systems in situations where they should not. These problems are not science-fiction failures, they are the same issues that IT enterprises have suffered from for decades. However, AI amplifies them by speed, scale and autonomy. As AI becomes more capable of acting alone in the real world, the need to ensure our ability to control these actions should develop at the same pace. 

Even if some companies such as OpenAI or Anthropic agreed to slow down AI development, other companies may not respond the same way. On the other hand, an AI slowdown could prevent legitimate AI research that can provide significant benefits to humanity. In healthcare, for example, AI systems are already achieving excellent results in drug discovery and improving diagnostic accuracy, which can benefit millions of patients.

Still, this does not mean there is no reason to slow down. Experts' concerns become more reasonable when we look at how quickly AI systems are moving from research environments to deployment. Organizations are accelerating the adoption of AI agents that can perform multiple tasks without human oversight. These autonomous agents can access sensitive data, such as customer and patient records, connect to internal applications, external APIs and cloud infrastructure and perform tasks and make decisions with limited human involvement. An OpenAI agent evaluation demonstrated how an autonomous agent, when given broad permissions and insufficient safeguards, could take unintended actions against external systems.

This is where the slowdown debate should be focused. The main risk is not in continuing to develop AI, but in giving autonomous systems more permissions to act without human approval. For instance, there is a significant difference between an AI agent that summarizes or generates a report and an AI agent that can modify a production database, execute code, send emails, access confidential customer information or make changes to cloud infrastructure configurations.

This is why the discussion should move from the simple binary question of whether AI development should be stopped to a more precise question: Where do additional cautions need to be concentrated? AI systems that interact with critical systems, sensitive information, financial systems or security controls deserve a different level of testing and human oversight than an AI system used to do basic tasks like summarizing documents or drafting marketing copy. 

AI development will continue and trying to stop it is not a realistic scenario. But deployment should not move at the same speed as development. For the benefit of humanity, continuing to develop more capable AI systems is a good thing. However, we should be very careful in giving these systems autonomy, access to sensitive systems and the ability to execute critical actions without human approval.

The real AI threat is not a robot uprising

When people discuss the potential dangers of AI, one of the common scenarios that comes to mind is turning an AI system into a self-aware machine that decides to turn against humanity. Actually, this is the Hollywood version of the risk: a conscious machine that acts against its creators and aims to destroy them. While this scenario looks interesting, it's unlikely that it will unfold. The direct risk will come from how humans use AI and the amount of control they are willing to give to these systems.

When used in the wrong direction, AI can facilitate executing different types of attacks. In cybersecurity, AI can already make cyberattacks more efficient by helping threat actors identify targets, generate phishing messages that seem similar to those created by humans and automate reconnaissance activity by scanning massive volumes of public data and using it to facilitate different types of social engineering attacks. As AI systems become more advanced and capable, attackers can use them to boost their malicious activities by accelerating the process of vulnerability discovery, modifying malware and developing techniques intended to evade existing security controls. They can also be used to facilitate fraud through deepfake technology, where AI generates voice and video content that resembles real people with impressive accuracy.

Another question that comes to mind is how humans are going to use advanced AI systems. AI itself might not be aware that it wants to harm people if a human uses it to execute cyberattacks, manipulate information through deepfake or automate activities that would previously require a considerable amount of time and expertise from the attackers. So the risk does not come from an AI system that turns against humans, but from humans using AI to accelerate malicious activities in a scalable and cost-effective way. This is why concentrating merely on the possibility of an AI uprising distracts from some of the more realistic security concerns. An AI system does not need to act with malicious intent to cause considerable damage. If it has enough access to the real world, misunderstanding an objective, being given excessive permission, carrying out a malicious instruction or taking unintended actions can cause the AI system to do serious damage. 

As AI systems become more capable, it is not enough to monitor what these systems can do, but also what we allow them to do. The greatest risk comes less from an AI deciding to attack humanity, and more from humans giving wide permissions to AI systems -- and consequently more capability to cause damage -- before the necessary controls are established around their usage.

Could AI contribute to human extinction?

We should take seriously the possibility that AI could lead to human extinction. Although AI researchers do not agree how likely such a situation is, or how quickly it could develop, the fact that they disagree does not mean that caution is unnecessary. A possible scenario is that humans lose control of autonomous AI systems. If future AI systems become more capable of operating alone, using external tools and services and finding ways to achieve an objective without human involvement, the consequences could be serious. This does not mean the AI system will suddenly become conscious and decide it should eliminate humanity, but a system pursuing an objective using unintended or harmful ways could cause significant damage if humans are unable to stop it.

Another concern is the wide capability of AI systems to accelerate existing threats. For example, AI is already used to support cyber operations. AI can help threat actors analyze large volumes of information, identify possible vulnerabilities, prioritize their targets and automate certain aspects of reconnaissance and exploitation. It can also generate convincing phishing emails and automate the early stages of different cyberattacks at a scale and speed that would be difficult to achieve manually.

On the other hand, AI systems are already being developed for medical areas such as drug discovery and other related scientific research. However, the same capabilities can be misused by threat actors to support the development of biological or chemical weapons. The same principle applies to military systems. AI systems can process large volumes of information and generate recommendations or decisions much faster than humans in some tasks. Although this can boost the efficiency of military operations by automating a large number of tasks, it leads to ethical concerns regarding who should be held responsible for decisions to use lethal force and whether humans can intervene at the correct time in order to prevent a situation from escalating.

None of this means that human extinction is the most likely outcome of AI development. However, the important thing to consider is that AI does not have to turn evil or become a conscious machine to make the consequences catastrophic for humans. The combination of increasing AI capabilities, autonomy and access to critical systems -- in addition to human misuse of AI -- could create risks that are very difficult to reverse once they reach a certain point.

AI regulation is necessary, but will not solve the main problem

Governments have the primary role in establishing boundaries for AI development and deployment. As AI systems become more advanced and integrated in different areas that can impact people's lives, relying entirely on individual companies to decide how AI should be used is not enough. Government regulations can provide the baseline requirements for safety, transparency and accountability when using AI systems in critical environments such as public infrastructure, healthcare and medical decision-making, financial systems, autonomous vehicles, defense, employment and hiring decisions.

Still, regulation has a major limitation: AI capabilities can develop faster than governments can create or update regulations addressing those capabilities. By the time governments create rules for a specific AI capability, the technology could have changed significantly by adding more capabilities. While this does not mean government regulations are unnecessary, we cannot expect regulations alone to solve every AI-related risk.

Because government regulations are not enough, organizations need to implement technical controls for developing or using AI in their work. For instance, organizations developing or deploying AI systems should carefully monitor how these systems behave, track their actions and understand the type of data and other systems that they have access to. This is particularly important when AI systems are given autonomy or granted access to critical infrastructure. A system should not receive full, broad permission because it is capable of performing a task, AI agents should follow the principle of least privilege just as human users and applications do.

AI development should not move faster than our ability to control it

We should not stop AI development. This technology continues to provide numerous benefits to humanity. However, this does not mean we should continue deploying capable AI technology without carefully considering its risks.

As AI systems become more capable, it is important to have the appropriate security controls, testing, monitoring and human oversight over their critical actions. This is especially relevant when AI systems are given autonomy or access to sensitive data, critical infrastructure and systems that can impact people's lives.

This moves our debate away from choosing between stopping AI development and allowing unrestricted deployment. It would be better to continue with the development of AI but to exercise great care when it comes to the areas in which it is deployed. As the AI system becomes more capable and more autonomous, it will require more safety measures and more human supervision.

Nihad A. Hassan is an independent cybersecurity consultant, digital forensics and cyber OSINT expert, online blogger and author with more than 15 years of experience in information security research. He has authored six books and numerous articles on information security. Nihad is highly involved in security training, education and motivation.