AI can be a helpful tool when users respect its limitations and verify what it claims to be fact. If not, the impact on the organization can be catastrophic.
When used correctly, AI is an excellent force multiplier. It can automate routine tasks, assess massive data sets and improve efficiency across an organization.
The key, of course, is that organizations are using it correctly. AI technology is constantly evolving, so it can be difficult for businesses to keep up with what it is capable of and what data it should have access to.
Machine-speed work can result in machine-speed mistakes, which might require some serious cleanup efforts. As more organizations emphasize strengthening resilience, AI-related mishaps can become a roadblock.
Here are four real AI-related incidents that put organizational data at risk and damaged resilience efforts.
1. Bye bye, production database
In April of this year, there was a high-profile AI incident at PocketOS, a software vendor. Despite being instructed otherwise, an AI coding agent had deleted a production database, while still providing the user with what it thought should be the correct outcome.
This is not an uncommon incident; many users lose data to faulty agents because the agents are drawing incorrect conclusions from the data provided.
The primary lesson here: don’t blindly say yes to the AI. It’s critical to understand what the AI is doing and not giving it carte blanche over the environment. Ensure production is gated as much as possible and keep a human in the loop wherever possible as a sanity stop switch.
2. Build an app, show it to the world (company data included)
Users who are empowered with AI might create usable products, but many don’t understand that system security and data privacy can create an AI incident.
Unfortunately, when companies fail to provide an environment to host these ad-hoc applications they're placed on third-party tools that have live, confidential company data. This use of unsanctioned AI tools for company data is referred to as shadow AI, and it's a major concern as AI tools proliferate.
Earlier this year, Community Bank, a regional bank in Pennsylvania, dealt with this very issue. Rather than an outside attacker, an employee of the bank exposed sensitive customer information by uploading that data into an unauthorized AI application. Not only was this a reputational hit, but it also resulted in the bank filing a Form 8-K to the SEC.
Leadership can mitigate a crisis like this by ensuring users have internal infrastructure to host and demo their products, safe in the knowledge the data won’t be scooped up by some bad actor scanning the internet for just this type of information. The costs of developing such a system are trivial when compared with cleanup costs and potential liabilities.
3. Trusted but not verified
Everyone knows AI systems can hallucinate, despite best efforts. Users can tell agents to verify everything and "do not hallucinate," but there is no way to guarantee it will work. Professional services network KPMG found that out only too well when they released a report stating factually incorrect AI usage tied to actual real-life entities.
The fallout was significant, resulting in a serious reputational loss for KPMG. Had those named entities been so inclined, this could have ended up in court.
The best way to mitigate or prevent a disaster like this is to make sure that someone is actually reading the documents AI produces, verifying the data sources and recognizing the algorithms that manipulate the data. Often, subtle bugs can result in numbers that are slightly off or just completely wrong.
4. Legal chatbots, illegal suggestions
The NYC MyCity chatbot became a legal minefield when it gave NYC business owners incorrect legal information, including misrepresenting the minimum wage and fabricating claims about stores being cashless. Not only was this AI incident bad PR, but it also eroded trust in the accuracy an official city website.
Lawyers carry liability insurance. A chatbot doesn’t.
On the user end, mitigation for this is simple: Don’t trust the chatbot as a source of truth, and pay a proper legal professional when it involves significant risk. Chatbots frequently make up answers to please the user, so any responses should be taken with a grain of salt.
For organizations, it's critical that any official company chatbots are monitored not only for preventing private data leaks, but making sure the answers and advice provided are accurate and don't put the organization at risk. If a company chatbot gives incorrect, falsified or irresponsible answers, that creates a liability for the organization. Lawyers carry liability insurance. A chatbot doesn’t.
Stuart Burns is an enterprise Linux administrator at a leading company that specializes in catastrophe and disaster modeling.
Dig Deeper on Disaster recovery planning and management
