Tokenmaxxing: How CIOs can extract maximum value from AI tokens

The economics of tokens

Every LLM transaction has two sides. Input tokens cover everything sent into the model -- system prompts, conversation history, retrieved documents and the user query. Output tokens cover the response, and they typically cost two to four times more than input tokens, depending on the model and provider.

Pricing also varies significantly by model tier. Frontier models carry substantially higher per-thousand-token rates than smaller models.

Chris Thomas, U.S. hybrid cloud infrastructure leader at Deloitte, said that the same model has different economics when deployed using on-premises, using a public cloud API, or in co-location.

"One of the greatest areas of waste is organizations not understanding the dynamics of how and where tokens are generated across their cost structure," Thomas said.

Beyond base rates, four hidden cost drivers consistently inflate enterprise token bills.

• Long prompts. Loading entire documents or unfiltered retrieval results into every model call increases input token counts per request.
• Repeated context injection. Resending the same system prompts or retrieval context on every call means paying the model provider for identical tokens repeatedly.
• Verbose outputs. Without explicit response-length constraints, models return far more text than most tasks require.
• Agent loops. Agentic systems that re-plan or call models recursively without clear stop conditions multiply costs in ways that rarely surface until the invoice arrives.

Inefficient token use

Knowing the factors that influence token costs is important. However, understanding how enterprises trigger those factors is a separate and equally critical challenge.

"The biggest waste is structural, not conversational," explained Brian Fending, managing director of Ordovera Advisory and a former CIO and CTO across healthcare, private and nonprofit sectors. "A system that loads a 50-page document for every query when the model only needs three paragraphs is burning tokens on an initial architecture decision that was never revisited."

Monika Malik, lead data AI engineer at AT&T, identifies multiple patterns that compound across typical deployments:

• Sending too much context every time. Teams dump entire documents, long chat histories or full retrieval results into prompts when only a small portion is needed.
• Using the most expensive model by default. Not every workflow needs a top-tier reasoning model. Classification, extraction, summarization and routing work can be done much more cheaply.
• Poor RAG hygiene. Systems retrieve too many chunks, duplicate similar passages or pass raw retrieved text without filtering or ranking it first.
• Prompt bloat. Enterprises keep appending system prompts, repeated instructions and formatting rules to every call. Over time, prompt templates become cost-heavy and harder to maintain.
• No caching. Repeated instructions, summaries and retrieval results are regenerated on every call rather than reused.

"Teams optimize first for speed of rollout, not for cost-aware architecture," Malik said. "That is understandable early on, but once usage scales, those shortcuts become expensive."

The consequences show up in the data. Nicholas Arcolano, Ph.D., head of research at Jellyfish, points to a pattern that often goes unaddressed. "Extreme token use often isn't a sign of good engineering," he said. "It suggests poorly specced out tasks, lots of unnecessary rework or outsized bootstrapping costs."

Tokenmaxxing techniques CIOs should know

There are proven techniques at several levels that consistently reduce token spend without sacrificing output quality.

• Model selection. Route tasks to the right model tier, not the most powerful one available. Fending runs agentic systems where research tasks use Claude Opus, file retrieval routes to Claude Haiku, and formatting work bypasses frontier models entirely. "I've found that alone can cut token spend by 60% on mixed workloads without any loss in output quality," Fending said.
• Context management. Give the model what it needs for the task, not everything available. Tighter RAG pipelines, summarization layers before inference and prompt caching for stable content all reduce input size without sacrificing output quality. Prompt engineering costs compound quickly when these disciplines are skipped. "Most production systems load too much context too eagerly because the architecture was set up before anyone understood what the model actually needed," Fending said.
• Agent design. Set explicit limits on retries, tool calls and loop depth. Malik recommends response-length constraints since output tokens are billable. Matias Madou, co-founder and chief technology officer at Secure Code Warrior, added that enforcing structured JSON outputs eliminates conversational filler and keeps every output token purposeful.

Governance and FinOps for AI

The practice of FinOps, that is, applying financial discipline to IT ops, needs to include AI token spend.

"Without chargeback and visibility, token usage becomes the new shadow IT, quietly expanding without accountability," Dion Hinchliffe, vice president and practice lead at Futurum Group, said.

Set budgets, alerts and quotas. Hinchliffe describes a layered governance model emerging at mature organizations:

• At the platform level, CIOs set quotas, rate limits and model access policies, including who can use high-cost models and when.
• At the application level, teams track cost per transaction or per business outcome.
• At the executive level, those figures roll up into AI unit economics: cost per customer interaction, cost per support resolution.

Track cost per user, workflow and business unit. The most useful tracking connects token spend to business output, not just API calls. "Once you factor in the quality and security of the AI-generated code versus token spend, you're getting a really interesting view based on data, not on surveys or what people think is happening on the ground with AI," Madou said.

Build chargeback models carefully. Chargeback works best once usage patterns have stabilized, and teams understand their baseline. Rushing it can backfire. "Being overly aggressive with cost controls can backfire and slow adoption in the critical middle of the curve," Arcolano said.

Tooling and metrics

Most enterprises running AI at scale still cannot answer basic questions about the cost per workflow. The four metrics that matter most:

• Cost per 1K tokens. Broken down by model, so frontier and budget model usage can be compared across the same workflow.
• Tokens per request. Tracks input and output volume per call, surfacing which workflows are consuming disproportionately.
• Cost per business outcome. Raw token counts don't tell you whether the spend is productive."I strongly advocate for tracking what Salesforce calls an 'agentic work unit,'" Hinchliffe said. "This is a generic way to measure the cost to complete a business outcome, not just raw token consumption."
• Latency versus cost tradeoffs. Surfaces where speed is being traded for spend, enabling routing decisions based on outcome value rather than default settings.

"The real value is linking token spend to metrics such as merged pull requests or rolling up to business outcomes like shipped features," Arcolano said. "That's what allows you to diagnose whether and how tokens are being used effectively to produce business value."

Risk and security implications

Poor token hygiene creates security and compliance risks that are distinct from cost. Exposure areas CIOs should address include:

• Data leakage. Every prompt sent to an external model carries whatever is in the context window. "Every prompt sent to an external model is data leaving your perimeter via sensitive data in prompts, RAG context pulling from systems that contain PII, or agents that paste credentials into tool calls," Fending warned.
• Audit gaps. Most production AI deployments have no record of what the model was asked or what it returned. "If an agent generates a contract clause, a customer message, or a code change, you minimally need a logged record of the prompt, the model version, and the output," Fending said. "Most production AI deployments have nothing close to this."
• Prompt injection. Hinchliffe flags this as a distinct attack surface. In poorly designed RAG systems, untrusted retrieved content can manipulate model behavior if inputs are not sandboxed, making token volume both a security and a cost variable.
• Compliance exposure. Regulatory accountability is now a token governance issue. "If multiple AI agents share a single 'master token,' your logs will show the AI performed an action, but you won't know which agent or which user initiated it," Madou said. "This makes it impossible to comply with 2026 regulations (like the EU AI Act) that require clear accountability for automated decisions."
• Data sovereignty. Poor API credential management creates a related exposure.
  "A compromised token can allow an attacker to query your sensitive EU-based data from a non-compliant region, triggering massive 'Failure to Protect' fines under GDPR and the newer AI-specific frameworks," Madou said.

Call to action: What CIOs should do now

Token usage is now a measurable, controllable cost driver, just as compute and storage became manageable once enterprises started treating them as resources to be monitored and governed.

The starting point is a token usage audit. Without baseline visibility into what is being spent, where and why, none of the governance or optimization work lands. Malik identifies the priority actions:

• Classify which data is allowed in prompts and which is not.
• Apply redaction and minimization before model calls.
• Put access controls around retrieval and memory layers.
• Log model usage at the workflow level for auditability.
• Require guardrails for agentic systems before scaling them.

The stakes extend beyond cost. Hinchliffe sees the three disciplines coming together as one.

"Token governance, cost governance and data governance are converging," he said. "The CIOs who recognize that early and quickly build integrated frameworks are the ones who will scale AI safely and economically."

Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.