Based on McKinsey July 2026 study.
Despite dramatic drops in raw inference costs—with GPT-3.5-level token prices plummeting from $20 per million to under $0.07—enterprise AI spending is skyrocketing. Over 90% of organizations report exceeding their AI budgets, and one-fifth have been forced to cap AI operations due to runaway costs.
The issue isn’t token pricing; it’s agentic complexity. As models move from simple chatbots to multi-step reasoning agents, operational overhead expands exponentially. To prevent AI from becoming a cost sink, enterprise leaders must shift their focus from token discount conversations to governing machine work for business value.
1. The 6 Cost Drivers of Agentic Work
Traditional single-turn API calls are predictable. Agentic workflows, by contrast, introduce compounding execution overhead driven by six structural factors:
Long-Lived Context Repetition: Because LLMs are stateless, agents must resubmit complete historical context at each step. Multi-turn agentic workflows can consume up to 1,000 times more tokens than standard coding or chat tasks.
The Refinement Sink: The initial model output is rarely the primary expense; approximately 60% of an agentic task’s cost goes into self-checking, error correction, tool re-execution, and output verification.
Autonomy-Driven Cost Variance: Agents dynamically determine execution paths. The same programming prompt can exhibit a 30x cost variation depending on the reasoning branches, retries, and tool calls the agent chooses to run.
Unrouted Reasoning Overhead: Using expensive, high-reasoning frontier models for simple data extraction or basic formatting creates massive, unnecessary spending.
Suboptimal Tool Orchestration: Inefficient task decomposition, redundant agent handoffs, and poor coordination protocols compound token usage without improving the final business outcome.
Information Structure Inefficiencies: Poor prompt formatting, non-standardized JSON schemas, and non-English text (which splits into significantly more tokens per word) inflate baseline API consumption.
2. Four Strategic Shifts in the Agent Era
Process Advantages Shrink: Traditional execution advantages (e.g., faster claims processing or underwriting) are easily replicated when platform agents scale across competitors. Advantage moves from process design to execution speed.
Context Beats Models: As foundation models converge in baseline capability, proprietary context—operational data, decision history, and process telemetry—becomes the primary source of differentiation.
Governance as a Competitive Advantage: Organizations that master intelligence routing, cost-per-outcome measurement, and agent autonomy limits will extract far higher ROI from identical AI budgets.
Sourcing Boundary Recalibration: Sourcing is no longer driven strictly by labor costs. Proprietary context and core business logic must be kept strictly in-house, while non-core administrative workflows are offloaded to third-party agent services.
3. The CEO Operating Blueprint
To bring discipline to machine work, enterprise leadership must implement five structural imperatives:
Allocate Intelligence Like Capital: Reserve top-tier frontier models exclusively for high-complexity, high-value tasks. Route routine workloads to smaller, open-weight, or deterministic models.
Anchor Technology to Core Business Moats: Focus AI investments on the 2–3 core operational domains where automation fundamentally transforms unit economics (e.g., R&D in pharma, claims in insurance).
Treat Agentic Operations as a Core Discipline: Establish formal governance over tool architecture, model routing, and autonomy thresholds. Enforce strict stopping rules and budgets for all autonomous systems.
Establish Explicit Capability Ownership: Assign clear executive accountability (e.g., a Chief AI Operations Officer or dedicated steering committee) for machine-work KPIs like cost-per-completed-outcome and vendor performance.
Measure Outcomes Over Tokens: Track completion of business goals rather than low-level technical metrics like token counts or API calls.
Workload Placement Matrix: Balancing Cost, Sensitivity, and Quality
Where an enterprise workload should run depends on its operational characteristics. Most organizations should maintain 3–4 deployment tiers managed dynamically by an intelligent routing layer:
