Token Economics: LLM Cost, Infrastructure Cost or Business Outcome Cost?

One of the first things we learned to measure with Generative AI was tokens. How many are we sending? How many are coming back? Which model is cheaper? Can we shrink the context, cache repeated requests, route simple work to a smaller model?

All fair questions. Token consumption translates directly into cost, and knowing how to optimize it matters. But there’s a bigger question underneath that most cost conversations skip: what are we actually trying to achieve with those tokens?

If we optimize token usage without understanding the problem we’re solving, we can end up reducing the LLM bill while making the overall business process more expensive.

Part 1 — What actually makes up AI cost

The layered cost model

It’s tempting to define AI cost as the cost of the LLM call. Input tokens in, output tokens out, and we pay for both. Clean, visible, easy to reason about. And almost never the whole story.

An agent rarely lives alone. It retrieves from a knowledge base, queries a database, calls APIs, executes code, sometimes invokes multiple models, validates its own output, retries failed operations, and often pulls in a human. All of that architecture is also part of the cost. Vector databases, storage, orchestration, compute, networking, observability, tool execution: some directly attributable to an AI request, some shared platform costs allocated across workloads.

Then there’s the human. Say an agent costs a few cents to run. But if it hands a partial or wrong result back to a person who has to review, fix, or restart the workflow, the LLM bill is small and the real cost isn’t. Consider a task a person used to do in ten minutes. The AI workflow now spends two minutes of LLM and infrastructure cost but still needs eight minutes of human review. We didn’t eliminate eight minutes of work. We moved it. From the LLM bill onto the payroll.

And failures compound. When an AI run fails, the model call wasn’t free. You already paid for the tokens, the retrieval, the tool calls, and the infrastructure before finding out. Then a human unwinds it. That’s part of the cost of producing the outcome.

Put it together:

LLM Cost + Infrastructure Cost + Tool Cost + Human Cost + Failure/Rework Cost = Cost of Producing the Outcome

Cost grows with autonomy

A single LLM chat might involve one or two model calls. An agentic workflow can involve dozens: retrieving context, reasoning about the next step, calling a tool, interpreting the result, deciding again, validating output, retrying when something fails.

Every extra step adds potential cost. Which is why optimizing the cost of an individual model call can mislead you. A cheaper model saves tokens per call but earns retries. A smaller context saves tokens but drops the piece of information the model needed. A more aggressive caching strategy saves calls but returns a stale answer. The cheapest individual operation doesn’t necessarily produce the cheapest successful outcome.

Cost per Successful Outcome

The most meaningful metric for an agentic system isn’t cost per token, or even cost per task. It’s closer to Cost per Successful Business Outcome.

That denominator matters. An agent that burns twice the tokens but pushes success from 70% to 98% may be economically superior, despite the bigger LLM bill. An agent that sips tokens but constantly needs a human standing behind it may cost far more than the invoice suggests.

The tricky part is defining what successful outcome means for a given task. For a support agent, it might be a ticket resolved without escalation. For a code agent, a merged PR that passes review. For a document extraction agent, fields a human doesn’t have to correct. Success is domain-specific, and getting the definition right is a prerequisite to any real cost analysis.

Then you can start measuring. Four numbers tell you more than the LLM invoice ever will: success rate, human intervention rate, retry rate, and end-to-end task cost including human minutes. Once those are wired in, cost per successful outcome stops being a slogan and becomes a number you can compare across models, prompts, and architectures.

Which brings us to the practical question.

Part 2 — How to actually optimize AI cost

Now that the target is clear (cheapest cost per successful outcome, not cheapest token bill), here are the levers that actually move the needle. Roughly ordered from fastest wins to biggest structural bets.

Tighten the prompt

Long, meandering prompts cost more, take longer, and often produce weaker output. Every extra instruction is tokens in, and every extra “please explain your reasoning” is tokens out. Get concrete: state the task, the format, and the constraints.

A few patterns that work well:

• Cap response length explicitly. Set max_tokens in the API call and add a plain-language limit in the prompt (“answer in under 100 words”, “return only the JSON, no preamble”). Left unchecked, models add throat-clearing before the actual answer.
• Ask for a specific output shape. For a format like JSON, use the provider’s structured output or function calling mode. The model is forced to produce something that matches, saving reformatting tokens and post-processing time.
• Front-load the important stuff. Models pay the most attention to the beginning and end of a prompt. Critical instructions go there, not buried in the middle of a wall of context.
• Keep system prompts stable. Anything that doesn’t change per request (role, task definition, format rules) belongs in the system prompt. Keep it stable and most providers will cache it, so you pay a fraction of the normal rate on every call.
• Skip “think step by step” unless you need reasoning. Chain-of-thought prompting helps on hard problems, but can double or triple output length. Wasteful for classification or short factual answers.
• Cut politeness padding and repeated instructions. “Please”, “kindly”, and repeating the same instruction two different ways all cost tokens. Tiny wins each, meaningful at 100k calls a day.
• A/B test small wording changes. Run two versions of the same prompt on real examples and compare the results. The leaner one often cuts token count 30–50% with no quality loss. Measure, don’t guess.

The goal isn’t the shortest possible prompt. It’s the prompt that reliably gets the outcome for the least token spend. And the payoff isn’t just cheaper calls. Tight prompts save your time, cut iteration cycles, and make you a sharper thinker about what you actually want from the model.

Send only the context that matters

The single biggest cost driver for most teams is sending too much context. Stuffing the whole document, conversation, or codebase into every call is the AI equivalent of paying overnight shipping on a paperclip. Retrieve only what’s relevant for the specific request in front of you.

• Break big documents into small, meaningful pieces. Split on natural boundaries (paragraphs, sections, function definitions) at 256–1024 tokens per chunk. Bad chunking wrecks retrieval quality no matter how good your Vector DB is.
• Use a search that understands meaning, not just keywords. These are called vector databases (Pinecone, Weaviate, Chroma, pgvector, Qdrant). Store your chunks in one, and the AI pulls back just the few most relevant to the question instead of you stuffing everything in.
• For relationship-heavy information, use a graph database instead. Some knowledge is about connections: who reports to whom, what depends on what, how concepts link. A graph database like Neo4j captures those relationships, which meaning-based search tends to miss.
• Combine two search techniques and rerank. The strongest retrieval mixes meaning-based search with keyword search (BM25), then uses a small reranker to re-order results by fit. Small upfront cost, big savings on unnecessary context.
• Filter before you search. Metadata filters (date, source, permission, category) narrow the pool before similarity kicks in. Same reason you filter emails by date before searching inside them.
• Summarize long conversations. For multi-turn agents, don’t resend the full history on every call. Roll up older turns into a running summary and keep recent ones verbatim. Cuts context 3–5x on any long conversation.

Done well, this one area doesn’t just cut token spend by 5–10x. It multiplies the quality of the output. The model isn’t drowning in irrelevant text, so it can focus on what actually matters. You get lower cost and sharper, better-grounded answers, all at once.

Right-size the context window

Related trap: using a long-context model (one that accepts huge amounts of input in a single call) when you don’t need one. Long-context variants cost several times more per token than standard-size siblings. Renting a moving truck to carry a suitcase. If most of your requests fit in 32,000 tokens, running them all through a 200,000-token model is paying for space you’re not using.

Match the model’s context size to your typical workload. And if you have one occasional monster request, route that one to the long-context model, not all of them.

The same principle applies whether you’re building an agent or using one:

• Building your own agent? Set explicit memory limits (many frameworks default to sending the full conversation history on every step), summarize older turns instead of resending them verbatim, and cap how much prior tool output gets re-included.
• Using an Agentic IDE like Cursor, Claude Code, Windsurf, or Copilot Agent Mode? You don’t control the context directly, but you can influence it. Pick the right model in settings. Add relevant files explicitly instead of letting the tool grep the whole repo. Use .cursorignore / .aiignore to exclude noise (node_modules, dist, generated code, lockfiles). Start a fresh chat when you switch topics. Watch the token indicator most IDEs now show.

Cache what you can

Caching is one of the fastest ways to cut cost. Two kinds worth knowing about:

• Prompt caching reuses the model’s compute for parts of the prompt that don’t change from call to call. Long system prompts, task definitions, and reference material get stored on the provider’s side and billed at a fraction of the normal rate. Like a barista remembering your usual order. Native in Anthropic and OpenAI APIs. Turn it on wherever your prompt structure is stable.
• Semantic caching stores answers and matches new questions by meaning, not by exact string. “How do I reset my password?” today and “I forgot my password, what do I do?” tomorrow can share the same cached answer. Great for FAQs, support flows, and anything with repeated question shape.

Track your cache hit rate as a first-class metric. A poor hit rate is usually a routing or key-generation problem, and it’s fixable. At scale, a 50% hit rate at 100k requests a day means 50k requests you didn’t pay full price for. That’s the difference between a manageable AI bill and a runaway one.

Route by task complexity

Not every task needs the biggest model. A quick classification (“is this email spam?”) needs a fraction of the reasoning of an architecture recommendation. Sending both through the same top-tier model is like taking a helicopter to the corner store.

Model routing means sending simple work to a cheap model and reserving the expensive one for hard reasoning. A few ways to do it:

• Set up a lightweight router. A small first-pass model looks at each request and decides which model handles it. Classification and short extractions to the cheap tier; complex reasoning, code generation, and multi-step analysis to the expensive tier.
• Or use the escalation pattern. Send every request to the cheap model first, escalate to the expensive one only if confidence is low or validation fails. Captures most of the savings with far less design work.
• In an Agentic IDE, pick your model per task manually. Trivial edits and refactors to the smaller model; architecture questions and tricky debugging to the bigger one. Most IDEs let you switch mid-session.

Done well, routing can cut costs by 60–80% on high-volume workflows without touching quality on the tasks that actually need reasoning power.

Fine-tune small models for narrow, high-volume tasks

For narrow, high-volume tasks your product does over and over (extracting fields from documents, classifying support tickets, tagging products, routing requests), a small fine-tuned model often beats a large general model on both cost and quality.

Fine-tuning means taking a smaller model and training it further on examples of your specific task, so it becomes a specialist. Like hiring a general contractor versus a specialist who does one thing all day. On unfamiliar work the generalist wins. On repetitive work the specialist is faster, cheaper, and often more reliable.

Trade-off: upfront investment in collecting examples, training, and evaluation, plus ongoing retraining when the task drifts. In exchange, per-call inference costs 10–50x less and accuracy often improves, because the model has seen many examples of exactly your task.

Rule of thumb: the math starts favoring fine-tuning around a few hundred thousand monthly calls in that specific task category. Below that, routing plus good prompting usually wins.

Replace reasoning with code where you can

The fastest cost cut most teams overlook. If a task can be handled with a few lines of traditional code, there’s no good reason to burn tokens on it. LLMs are for the ambiguous parts (natural language, fuzzy classification, reasoning across incomplete information). Everything else, if/else is free, instant, and predictable.

Things that should almost never be an LLM call:

• Math and calculations. Adding numbers, converting units, computing dates, running formulas. LLMs are famously unreliable at arithmetic. Use code.
• Validation and format checks. Is this a valid email? Does this JSON match the schema? Is this date in the past? All deterministic. Write the check once and run it forever.
• Data transformations. Parsing CSVs, reshaping JSON, applying business rules that can be expressed as if this, then that. This is what code was invented for.
• Simple lookups. “What’s the price of product X?” is a database query, not an LLM query.

A pattern that works well: use the LLM to interpret a request and pull out structured fields, then run the deterministic pipeline on those fields. You get the flexibility of natural language on the front, the reliability of code on the back, and a much smaller bill at the end.

Plan the workflow before you build

Before touching an LLM, get concrete about the workflow, the data, and what success looks like. The most expensive AI systems are the ones designed without this step, because they end up handling edge cases by burning tokens (and human time) instead of by design.

A short list of questions worth answering before you write any prompt:

• What is the agent trying to accomplish? State the outcome in one sentence. If you can’t, the agent won’t either.
• What information does it need to make each decision? For every step, list the inputs the agent depends on.
• Is that information available, structured, and current? If it lives in a PDF nobody’s touched in three years, you have a data problem before an AI problem.
• Where does human judgment matter more than autonomous execution? Some decisions should never be delegated. Name them upfront.
• Where do exceptions happen? How does the agent recognize an unusual situation, and what should it do (escalate, ask a human, try harder, give up cleanly)?

Skip this step and you’ll build something that works on the happy path and burns tokens on everything else. Do it well and you’ll spend a lot less on inference and have a much better shot at working in production.

Treat prompting as a team skill

Prompting isn’t a solo art. When one person figures out a better prompt, the whole team should benefit. When one person makes a prompt worse, everyone should catch it. The teams that get the most out of AI treat prompting like any other engineering discipline: shared, measured, and iterated on.

A few habits that make this real:

• Build a prompt library. A shared repo or wiki where prompts live alongside notes on what they’re for and what works. Prompts scattered across Slack DMs are prompts nobody can improve.
• Review prompts like code. A second set of eyes catches ambiguity, missing constraints, and format issues before they hit production.
• Version your prompts. Track changes so you can roll back if a “better” prompt makes quality worse in the wild.
• Run evaluations, not just eyeball tests. Frameworks like RAGAS, TruLens, and DeepEval score prompt outputs on accuracy, faithfulness, and relevance. A prompt change either measurably improves things or it doesn’t.
• Bring non-engineers into prompt design. Support leads, domain experts, and product managers often know how to phrase things better than engineers do.

Individual prompting is a lucky guess. Team-level prompting is a discipline. The teams that treat it as the second get far more from every dollar they spend on AI.

Token economics is really outcome economics

Token economics isn’t really a cost-reduction exercise. Tokens are just a measurable unit of AI consumption. What matters is the chain they sit at the front of.

The technology will keep changing. Models will get cheaper and more capable. Context windows will grow. Agents will get more autonomous. But the underlying question stays the same:

An organization doesn’t ultimately create value by consuming fewer tokens. It creates value by producing better outcomes at an acceptable cost. The goal of Agentic AI isn’t the agent that uses the fewest tokens. It’s the right agent, for the right process, with the right level of autonomy, at the right cost, tied to a clearly understood business outcome.

Token economics is really outcome economics. Anything less is just watching the meter run.