## Why cost per token misleads
When finance asks how expensive an AI feature is, the easiest answer is cost per token or cost per call. Both are useful inputs, neither is a useful unit. A model that costs half as much per token but needs three retries to get the right answer is more expensive in practice. A small model that fails forty percent of the time is not cheap, it is a customer experience risk.
The metric that actually matters is **cost per successful task**: the total spend required, on average, to get a result that satisfies the user.
## Defining a successful task
The definition is product-specific and worth real effort. For a support copilot it might be a session that ends without escalation and earns a positive rating. For a code agent it might be a task whose generated change passes tests and is merged. For a sales research agent it might be a brief that the rep actually uses in the meeting.
Write the definition down. Make it measurable. Get product, engineering and finance to agree on it. Without that agreement, every cost discussion ends in a different room.
## What goes into the cost
A full cost per successful task includes more than the model bill:
- **LLM tokens** across all attempts, retries and tool-calling steps.
- **Embedding and vector search** costs for retrieval.
- **Reranking and other supporting model calls.**
- **Compute and storage** for caching, logs and traces.
- **Vendor fees** for guardrails, evaluation and observability platforms.
- **Human review** for tasks that need a person in the loop, costed at fully loaded rates.
Divide the total by the number of successful tasks, not the number of attempts. The first time you do this, the number is usually two to five times higher than people expect.
## The levers that move the number
Once you can measure it, the optimisation playbook is clear.
### Better routing
Not every request needs your most expensive model. Build a router that classifies the request and sends easy ones to a cheap, fast model and hard ones to a frontier model. Train the router on historical traces. A well-tuned router routinely cuts cost by thirty to sixty percent without hurting quality.
### Caching
Many requests are near-duplicates. Cache at the response level for exact matches and at the embedding level for semantic matches. Set a sensible TTL. For agents, cache intermediate tool results too: the same lookup in three places should not cost three times.
### Smaller context
Every extra token in the prompt costs money on every call. Aggressively trim system prompts, summarise long histories and only include the retrieval chunks the model actually needs. Measure quality before and after.
### Better retrieval
If retrieval brings the right information on the first try, the model rarely needs to reason its way around gaps. Investing in retrieval quality pays back in lower model costs as well as higher answer quality.
### Model upgrades
New models often offer better quality at lower cost. Evaluate them on your own task definitions, not on public benchmarks. Sometimes the right move is a smaller model with stronger fine-tuning rather than the latest flagship.
### Failure cost
Every failed task is wasted spend. Add automatic retries with backoff for transient errors, but cap them. Surface persistent failure modes to the team so they fix the root cause rather than retry forever.
## Reporting that drives action
Report cost per successful task alongside two siblings: success rate and median latency. Together they tell the real story. A drop in cost that comes with a drop in success rate is not a win. A rise in cost that comes with a much higher success rate often is.
Break the metric down by feature, by tenant and by user segment. The aggregate hides the segments that lose money on every interaction.
## Pricing implications
Unit economics changes pricing strategy. Flat per-seat pricing for an AI feature can be ruinous if a small subset of users drives the majority of cost. Consider tiered usage, soft caps with overage or feature gating for power users. Be transparent so customers can self-regulate.
Measuring cost per successful task forces every team to share one definition of value and one number that captures it. That alignment is worth as much as any single optimisation it produces.