metamorworks - stock.adobe.com
Why conversational AI is redefining your security perimeter
As employees become more accustomed to using AI, they might be getting a little too comfortable. Learn how strong AI governance helps manage this new human risk.
As enterprises race to deploy AI across their operations, a perfect storm is brewing: New AI-generated attack vectors are colliding with employees' growing emotional trust in chatbots and AI assistants, creating security blind spots that traditional defenses weren't designed to handle.
Security teams are knee-deep in mitigating the threats that accompany AI adoption, from prompt injections and data poisoning to bias exploitation, deepfakes, and models acting in unexpected ways. Though a constant struggle, this more technical concern accompanies the psychological issue of employees oversharing sensitive information with conversational AI systems they've come to trust as helpful and even friendly digital assistants -- a challenge that is harder to address.
The problem of oversharing
The personal use of generative AI and chatbots has broad social implications that bleed into the workplace. Psychologists understand that human beings tend to connect with anything that talks to them, even if it's a machine. And although most users know that AI isn't sentient, it can still elicit emotions -- specifically, misplaced trust.
The line between workplace and personal AI is blurry, and some employees are bringing their bad habits to work. Many organizations have yet to establish firm policies for the use of AI assistants, and many employees use AI without awareness of their organization's AI strategy, suggesting widespread use of personal tools outside official channels. According to a Microsoft study, 78% of users bring their own AI tools to work, with the practice being more common at small and midsize companies. Further, a National Cybersecurity Alliance and CybSafe survey found that 43% of employees who use AI for work tasks send sensitive data to AI applications without their employer's knowledge.
This reality is creating a new problem for security teams. Employees, already conditioned to trust their personal AI assistants -- everything from ChatGPT to AI friend apps -- are more likely to let their guard down and share personally identifiable information or sensitive company data with systems that lack inherent privacy safeguards. In fact, many publicly available GenAI platforms clearly state in their T&Cs that they use inputs as training data.
There are real-world implications. For example, Samsung suffered several security incidents related to AI assistants. In 2023, an engineer pasted proprietary source code for semiconductor equipment into ChatGPT to help correct errors, exposing confidential code used in the company's chip manufacturing process. Another employee exposed sensitive business intelligence and internal discussions after feeding the content of a high-level meeting into ChatGPT.
According to Naynesh Patel, managing director of cybersecurity at Accenture, the ease of information sharing with AI assistants is problematic, and traditional enterprise security was not designed for it. "The concept of a text box -- where you are able to put information in with little to no friction -- and the fact that it's helpful creates risk," he said.
Governance for AI trust
The convergence of technical vulnerabilities and human psychology requires CISOs and their teams to adopt controls to defend against data loss via AI.
According to Patel, the solution isn't fixing AI; it's rethinking how the organization itself governs AI use among its employees. He said that most data security failures aren't model failures, but identity and governance failures running at machine speed.
He recommended that security teams deploy the following basic protections alongside enterprise GenAI and chatbot instances:
- Restrict the ability to post information that's shared anywhere else, such as with the AI parent company and related technology providers.
- Keep all data inputs within the boundaries of the organization.
- Grant just-in-time, least-privileged access for all employees.
- Set telemetry that enables SecOps teams to see prompts and intervene at the moment of risk.
Data: The new perimeter
Data is the new perimeter, and GenAI and conversational AI represent both productivity tools and data exfiltration points. Security teams must treat them as they would any other approved digital tool: secure them, put controls around them, audit their use and educate employees on how to use them safely.
As far as human threats are concerned, companies must enforce strict policies. In the wake of its AI security woes, Samsung pursued disciplinary action against the employees, developed its own internal AI system with data controls and eventually enhanced its security protocols.
At its best, conversational AI provides efficiency to many at work and comfort to some at home. At their worst, AI assistants can make an already daunting threat landscape worse. What's certain, however, is that human nature is difficult to change, and people will continue to share more information than they should, which requires a fundamental evolution in how security leaders think about risk.
Richard Livingston is an editor with Informa TechTarget's SearchSecurity site, covering cybersecurity news, trends and analysis.
