Building cyber-resilient AI in the enterprise

AI breaches are different

The introduction of AI into enterprise production environments creates an entirely different and potentially more expansive attack surface. This fact is not lost on adversaries, who have been quick to capitalize on exposed AI infrastructure.

AI-driven applications differ from traditional software in numerous ways, starting with how they handle user input. In conventional applications, user input security controls run on predictability -- identical input equals identical output. Large language model (LLM) outputs, however, can change based on factors ranging from temperature, settings and context length to model updates and tool availability. This makes it challenging to verify when vulnerabilities are patched.

Another significant difference with LLMs is that adversaries don't have to exploit software vulnerabilities. Instead, threat actors can work in a manner resembling social engineering, manipulating an ambiguity or shifting context to penetrate the model. Plus, attackers don't have to take over infrastructure to exfiltrate sensitive information. They can manipulate an AI model to trigger malicious actions. Threat actors can also poison outputs by manipulating the data pipeline.

By its nature, AI is susceptible to tactics such as prompt injections and instruction hacking that adversaries use to trick the engine into ignoring rules and following nefarious instructions. Data exfiltration using retrieval-augmented generation (RAG) and connectors is another common attack method in which threat actors bypass access controls during retrieval. Bad actors also use AI to launch machine-speed attacks that can identify and exploit supply chain vulnerabilities.

Adversaries can use language to bypass policies and controls maintained by conventional security tools. LLMs are often connected to multiple environments, including code, HR, tickets and CRM systems. Infiltrating an LLM workflow can therefore compromise multiple domains simultaneously. Data can be leaked through generated texts, summaries, tool outputs, logs and other unauthorized actions.

AI breaches are difficult to detect, too, with leaks occurring over multiple seemingly harmless inquiries. This forces investigators to determine whether the leaked data was from training, memory or a connector.

Building a cyber-resilient AI environment

The impact of an AI breach can be significant, ranging from exposed sensitive data and regulatory fines to integrated AI systems working improperly. Enterprises need to approach AI with security as an integral part of its use. Security practitioners must set governance and threat modeling from the outset. Security teams should model LLM-specific threats such as prompt injection, indirect injection and data leakage via RAG.

Authorization requirements at retrieval time, not just in the UI, are critical. Security practitioners need to ensure identity permissions extend to the database and search layers. While certainly not unique to AI, it is important to use data classification and tagging to keep potentially confidential and high-value documents from being indexed.

It is critical to safeguard all connectors and credentials. This means applying least-privilege access controls for connectors. Build security into tool and agent execution through policies that incorporate controls such as allowlists and constraints. It is also important to mandate human intervention for permanent actions, including payments and customer-facing emails.

To deflect prompt injections, security practitioners should use strong system prompts. Additionally, implement zero-trust controls that assume external content is potentially malicious until proven otherwise. Data loss prevention protocols are critical to block users from pasting sensitive content into AI that could be leaked.

The supply chain also needs security, which requires vetting all checkpoints and consistent maintenance of the model registry. Harden all infrastructure with isolation applied to tenants and indexes. Put strong identity and access management in place through single sign-on and MFA, maintaining zero-trust principles.

SecOps teams must be vigilant about logging and monitoring, looking for indicators such as abnormal query patterns and escalations in retrievals of sensitive labels. All organizations need to have an AI incident response guide that outlines elements such as taking tools and connectors offline, rotating tokens, purging indexes and verifying data leakage sources.

As AI continues its rapid integration into enterprise operations, organizations must recognize that speed without security is a recipe for disaster. The transformative potential of AI can only be realized when built on a foundation of cybersecurity and proactive risk management. Organizations that prioritize cyber-resilience today will be the ones that thrive in the AI-driven future, while those that neglect it could face breaches that could have otherwise been prevented.

Amy Larsen DeCarlo has covered the IT industry for more than 30 years, as a journalist, editor and analyst. As a principal analyst at GlobalData, she covers managed security and cloud services.