Who owns your AI data? Navigate security and proprietary risks

Data ownership in AI systems

Data is not a uniform construct -- there are different types of data. And to make matters more confusing, vendor terminology on data varies as well. There are three primary types of data that are related to enterprise AI, including the following:

• Input data. This includes the content that is put into a prompt, such as text, documents and corrections.
• Training data. Before an organization ever touches a model, the vendor has already used data to train the model.
• Output data. The generated content that a model provides as a response.

Misunderstandings about AI data

The default assumption is that if you typed it, you own it. Terms of service usually say otherwise. While there are three primary data types, there are numerous areas where organizations have misunderstandings about the data types and how they collect data.

Eric Johnson, CIO at PagerDuty, said one of the biggest misunderstandings he sees is that if a vendor says, "We don't train on your data," the buyer usually assumes that it covers all use cases.

"Input, training and output are three separate questions with three separate answers, and vendors don't always volunteer them all," Johnson said. "The confusion starts because the contract language is often vague on retention and reuse, so IT leaders assume the safest interpretation instead of confirming it."

Sebastian Arriada, CIO at Globant, argued that companies make the same mistake internally that vendors make in their contracts. They govern one of the three data categories, he said, and treat the job as done.

"Most governance conversations negotiate only the first one and declare victory," Arriada said. "Leaders treat this as a legal question when it's first an observability question. Most organizations can't answer who sent what to which model yesterday."

According to Keyur Ajmera, CIO at Boomi, there's actually another specific category in play.

"The lifecycle nobody has a good name for is the one that matters most now: retrieval context," Ajmera said. "Data that isn't in the prompt and isn't in the training set, but gets pulled in at runtime to ground the answer. That's where most enterprise data actually meets a model in 2026, and almost nobody governs it, because it doesn't fit either of the two categories people have words for."

Understanding the differences of the various kinds of data is critical to enterprise AI governance, according to Kristie Grinnell, CIO and executive vice president at TD Synnex.

"The confusion starts when we treat all AI-related data as if it's the same thing," Grinnell said. "The organizations that are getting this right are taking the time upfront to define categories such as customer data, prompts, outputs, logs, training data and derived data, then ensuring those definitions carry through into their contracts, controls and governance practices."

Public vs. proprietary tools: Do different rules apply?

Public AI tools generally operate with broad consumer terms of service that apply to data input and usage. In contrast, proprietary tools are usually sold under negotiated enterprise contracts that will have specific terms and exclusions for data to help ensure privacy.

The risk isn't theoretical. Adnan Masood, chief AI architect at UST, pointed to a case from 2023, when Samsung engineers pasted proprietary source code into a consumer chatbot.

"The data left through the front door under terms nobody had read," he said.

Is 'no training' the whole story?

"The enterprise commitments are real, and I want to be clear about that because the cynical take gets repeated a lot," Masood said. "We have gone through the DPAs, the SOC 2 reports, and the audit provisions with clients in banking, healthcare and insurance, and the zero data retention terms are contractual obligations."

Arriada has had the same experience, noting that the contractual commitments on enterprise tiers are generally real. That said, he noted that doesn't mean providers don't still learn.

"Providers still gain from aggregate usage patterns, telemetry, and service improvement carve-outs, and retention for abuse monitoring means your data can sit on their infrastructure for days even under a no-training clause," Arriada said. "The more important question is what exactly happens to every byte you send, for how long and who can see it."

Open source vs. open weight models

Another important distinction beyond just public and private models is understanding the impact of open source and open weight models. While some marketing folks might use the terms interchangeably, there is a difference between the two.

An open weight model refers to the training parameter -- the weights -- being made publicly available for anyone to see. To be clear, a model weight is not the training data; rather, it is the information that the model learned by analyzing the data.

In contrast, an open source model will also include code and training methodology alongside the weights, though there is also broad ongoing debate on what is and what isn't actually an open source AI license. The practical reality in many cases, though, is that the terms are blurred and organizations consider the terms as mostly the same thing.

The case for open weight

"We evaluated open weight models and moved straight into running them in production," Arriada said. "For several months now, we've been running GLM on our own Nvidia infrastructure. And it has grown to roughly 35% of our total token consumption -- at near-zero marginal cost per request -- because the economics live in hardware we already own. For a large share of our workloads, the capability gap versus frontier models simply doesn't matter, and the data never leaves our environment."

Masood said that at his organization, there is a governed model gateway that routes work by data classification as well as workload complexity. Frontier reasoning goes to the enterprise APIs under zero retention terms. Regulated or sovereignty-bound workloads run open weights inside the client's own perimeter, where nothing leaves.

"Open weights buy you architectural certainty about where your data lives," he said.

What it costs you

Going the open route does have other costs.

"You inherit the security burden -- like patching, guardrails, prompt-injection defenses, red-teaming -- plus the MLOps overhead of running inference infrastructure," Arriada said.

Where most companies will land

"I think the future is tiered, not either/or," Ajmera said, "splitting that slice between a frontier model for planning and self-hosted open weights for execution."

Service level agreements: What CIOs must demand from AI vendors

Service level agreements (SLAs) and contracts written for traditional software never anticipated the AI vendor management challenges related to a customer's data.

Non-negotiable terms

CIOs describe a similar starting point: testing a vendor's promises against its paperwork -- not its sales pitch.

Orla Daly, CIO at Skillsoft, noted that her company uses a formal AI security assessment questionnaire for vendors.

The most important thing is to ensure that enterprise data remains in a private, secure tenant and is not accessible to the AI vendor or third parties for selling, mining or training purposes.

"With respect to data, we look for details on how a vendor is handling data governance and privacy, model performance, bias, accuracy, reliability, transparency and explainability, compliance to regulations," she said. "It's important to push on understanding how a written policy is embedded into operations: clear data usage rights so our inputs and outputs are not used to train or improve any model without explicit opt-in, defined retention windows, transparency on subprocessors with notification before that list changes, and the ability to evaluate model updates before they hit our environment."

Data isolation is also critical.

"The most important thing is to ensure that enterprise data remains in a private, secure tenant and is not accessible to the AI vendor or third parties for selling, mining or training purposes," Ajmera said.

Masood has a list of five specific must-have items in an SLA:

1. First, no training on inputs or outputs, extended explicitly to derivatives, including embeddings, fine-tuned checkpoints, evaluation sets and feedback signals.
2. Second, retention measured in days, with deletion SLAs and certification.
3. Third, no human review of the company's content without written consent.
4. Fourth, output ownership assigned to the customer, with intellectual property indemnification behind it.
5. Fifth, exit rights: subprocessor transparency, residency commitments, audit access, and the ability to take your fine-tunes and embeddings with you when you leave.

Enforcement and liability

Contracts are all well and good until terms are broken. That's why it's critical to have enforcement and liability terms in place. It's important that those terms include clear breach notification timelines and incident response commitments.

"Ensure that any future contract language changes do not supersede our initial opt-out choices and require formal, legally binding approval from us for any data protection changes," Ajmera said. "Indemnification for data misuse or unauthorized training use. Meaningful liability cap for data breach or security incidents."

Red flags that should trigger deeper diligence

Two CIOs pointed to the same kind of warning sign, one contractual and one operational.

"The contractual red flag is vague, 'We may use data to improve our services,' language, and that phrase opens the door to swallow every other protection in the document," Arriada said.

"I push hardest on explicit language around training use, data retention and deletion -- not general assurances," Johnson said. "One potential challenge we watch for is any AI agent or tool that starts producing information based on data it shouldn't have access to. That is usually the first sign the data boundaries aren't what they should be."

Security concerns

There are a series of legitimate security concerns related to data and AI.

The AI security risks

Memorization. Models can potentially expose training data that was intended to remain private.

"There are documented cases of large language models regurgitating training data under specific prompting conditions, including data that was never supposed to be in the training set," Ajmera said.

Prompt injection. Attackers use the technique known as prompt injection to insert instructions into a prompt that force AI systems to do unexpected and potentially malicious things.

"That's a real attack vector, especially as we move toward agentic AI where models take autonomous actions," Ajmera said. "If your agent can be manipulated into exfiltrating data, making unauthorized API calls, or escalating privileges, you have a serious problem."

Oversharing and third-party integrations. Sometimes the risk comes from exposure to connected systems.

"Deploy an AI assistant with broad access to enterprise systems, and it can surface data from connected systems the user was never permitted to see," Ajmera said.

Real-world examples, and the guardrails not keeping pace

There is no shortage of publicly reported real-world vulnerabilities and incidents related to AI and data.

"EchoLeak, CVE-2025-32711, disclosed in June 2025, was a zero-click indirect prompt injection in Microsoft 365 Copilot, where a single crafted email could pull data out of a user's context with no click at all," Masood said. "The New York Times litigation against OpenAI taught two lessons in one docket: The exhibits showed near-verbatim memorization, and the 2025 preservation order forced retention of chats users believed they had deleted, while zero retention API customers sat outside the hold because there was nothing to preserve."

The basic approach to defense is to use guardrails, but they are not universally deployed. Identity and role-based controls are the most common safeguard, deployed by 60% of organizations running generative AI in their security stack, according to Kiteworks' State of AI Cybersecurity in 2026 report.  Data loss prevention tools trail at 54%, and prompt filtering with input and output controls sits at just 34%.

Insider threat and training

AI has already changed how insiders access and move company data, but governance hasn't caught up.

The Ponemon Institute and DTEX Systems' Cost of Insider Risks Global Report found that 92% of organizations say generative AI has changed how employees access and share data, yet only 13% have a formally integrated AI into business strategies.

There is also a governance risk related to training. Forrester's State of AI Survey, 2025 found that most existing AI policies cover only basics such as data use or copyright compliance, and few mandate responsible AI training.

As part of a CIO AI strategy, none of these risks get solved by technology alone. They get managed -- or missed -- in how honest the relationship with a vendor actually is.

"The most productive vendor relationships are built on openness and accountability," Grinnell said. "If a provider can clearly explain how your data moves through their environment and can demonstrate the controls behind those explanations, that's a very positive sign."

Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.