Per-Token Costs Dropped 1,000x. Your AI Bill Didn't.

Per-token costs for running frontier-class AI dropped roughly 1,000x in three years. GPT-4 class inference was around $30 per million tokens in early 2023. You can run equivalent-quality models for under $0.07 per million today. NVIDIA's Blackwell B200 cuts it further, to as low as $0.02 per million on large models with a properly optimized stack. The trajectory is obvious. So why are enterprise AI bills climbing?

This is the Jevons Paradox applied to inference. In 1865, William Stanley Jevons observed that more efficient steam engines didn't reduce total coal consumption, they increased it, because efficiency made coal-burning economical for applications that were previously too expensive to run. Efficiency didn't shrink demand. It expanded it faster than the efficiency gain shrank per-unit cost. The same dynamic is playing out in AI right now.

What Actually Happened When Tokens Got Cheap

When a chatbot turn cost $0.30, you kept interactions short. When that same turn costs $0.003, you build an agent. And agents are not chatbots. A single agentic task typically fans out into an orchestrator prompt, several tool calls, one or more retrieval steps, a sub-agent invocation, output validation, and often a retry. Research from Microsoft and Stanford puts the token multiplier at roughly 1,000x compared to a standard chat exchange. Gartner's 2026 analysis lands at a more conservative 5 to 30x, but either way the direction is clear.

Add reasoning models into this. Thinking-mode prompts on Claude Opus 5 or GPT-5.6 Sol can generate thousands of internal reasoning tokens before the visible response, all billed at the same per-token rate. A task that costs $0.01 in standard mode can run $0.50 in deep-reasoning mode. The quality often justifies it. But that math scales differently than people expect when it's embedded in a pipeline running thousands of times a day.

Inference spending now accounts for 70 to 85 percent of corporate AI budgets, and that share is growing. Every new agentic use case you ship adds tokens, not reduces them.

AT&T Is the Clearest Illustration Right Now

AT&T published their numbers this year and they're worth paying attention to. The company went from 8 billion tokens per day to 45 billion over the course of a few months, driven by internal AI tools scaling across the business. That's not 5x more users. That's the same users doing more with AI, each interaction generating far more tokens than the previous generation of tools.

Their response was to build an AI Gateway: a routing layer that classifies each prompt by complexity and sends it to the cheapest model that can handle it reliably. Simple classification, structured extraction, tool-call parsing: small open-weight models. Complex multi-step reasoning: frontier models. The result was a 90% reduction in inference costs at the same scale. Not by spending less on AI, but by spending smarter on which model handles which task.

I haven't built at AT&T's scale, but the pattern matches what I've seen in smaller production systems. Teams spending $40k/month on API costs often have the same issue: every prompt hits the frontier model by default, even when 70% of those prompts are things a 14B parameter model handles correctly at 1/20th the price.

Three Levers That Actually Work

Model routing. This is the highest-leverage move. Map your pipeline's subtasks to tiers. Tool schema parsing and simple extraction: a distilled open-weight model (Qwen3.8-Max or Llama 3.1 70B at $0.10 to $0.20/M). Structured reasoning with clear success criteria: mid-tier models like Kimi K3 or Claude Sonnet 5. Ambiguous, high-stakes, or creative tasks: frontier. One team cut their monthly bill from $40k to $24k by doing this audit without changing any functionality.

Prompt caching. If you're sending the same system prompt or document context on every call in an agentic loop, you're paying full price to re-read it each time. Most major providers now offer prompt caching at 80 to 90% discount on cached tokens. A 10,000-token system prompt sent 1,000 times a day costs roughly $1 with caching versus $100 without. This requires structuring your prompts to put stable content at the top, but it's not complex to implement and the ROI is immediate.

Context discipline. Every token in the prompt is billed. Production agent systems that haven't been audited often have context bloat: tool schemas for tools that aren't used in this step, full conversation history when only the last three turns are relevant, verbose retrieval chunks when a single paragraph answers the question. I've found 30 to 40% context bloat in most agent systems I've reviewed. It's the easiest thing to cut and the least glamorous, which is probably why it gets skipped.

The Right Mental Model Going Forward

The price of tokens will keep falling. Blackwell hardware, open-weight model competition, and better software stacks are all pointing in the same direction. But you can't cost-optimize your way out of a volume problem. The Jevons dynamic doesn't reverse: every time tokens get cheaper, you find more use cases to build, and each new use case has its own token multiplier.

The teams that stay in control of their AI spend treat token cost as a first-class metric alongside latency and accuracy. They set token budgets per pipeline, alert when those budgets drift, and run cost regression as part of CI when they ship new agent logic.

That discipline matters less at proof-of-concept scale. At production scale, the bill always arrives.