3 — The cheapest token is the one you don't send
Not every token-saving move is about using AI less. Some of it is just buying the same tokens cheaper, sending fewer of them for the same result, or routing the easy 90% away from the model entirely. All of it requires knowing the levers exist.
The first lever is the dumbest one, and it's the one most people leave on the table: if the response doesn't need to come back in real time, you can reduce the cost. Anthropic's Message Batches API and OpenAI's Batch API both cut the token price in half — flat 50% off standard rates, for anything you can queue instead of wait on. Nightly reports, bulk classification, eval runs, backfilling a dataset — none of that needs a synchronous response, and a lot of workloads are running at double the price they need to be for a lot of teams.
Same tokens, same model, half the price — the diagram illustrates how much you can save by being patient, or realising your requirements aren't immediate.
The discount is possible due to how the hardware works and is also why output tokens cost roughly 5x more than input tokens, on every provider. Inference has two phases with very different economics. Prefill — reading your entire prompt — happens in one parallel pass across all the input tokens at once: compute-bound, the GPU's tensor cores are busy the whole time, cheap per token. Decode — generating the output — happens one token at a time, and each single token requires a full pass through the model's weights: memory-bandwidth-bound, the GPU's compute sits mostly idle waiting on data to arrive. That asymmetry is why output is the expensive half of every API call.
Input is a parallel read. Output is a sequential, one-token-at-a-time reload of the whole model.
It's also why batching pays for itself rather than just being a promotion: serving one request at a time wastes most of the GPU waiting on memory, but running many requests together lets that same weight-load be shared across far more tokens generated per pass — better hardware utilization, more tokens per GPU-hour, and idle capacity a provider can slot batch jobs into instead of holding a GPU in reserve for your one request right now. Neither Anthropic nor OpenAI has published the exact cost math behind the 50% figure itself, so treat that specific number as a business decision — but the mechanism that makes a discount like it affordable at all is real, documented, and the reason I'd bet on batch pricing sticking around rather than disappearing as a promo would.
The second lever is shrinking the tokens you send and receive. You can use a tool like LLMLingua to compress your prompts — up to 20x — with close to no loss in quality. In fact, you get improved performance on RAG use cases, because a shorter, denser context fights the "lost in the middle" problem that long contexts create. On the output side, a tool like Caveman aims to do the same thing in reverse: its own published number is 65% fewer output tokens, byte-for-byte identical for things like code and error output, though that's measured across a small sample of prompts so far, not a large production workload.
The third lever is the one with the most room to be misused, so it's worth being precise about it: don't call a model for a decision a deterministic system can make correctly and near-instantly. A regex, a keyword match, or a rules engine can resolve a large share of requests before a model ever needs to see them — open-source rules engines like GoRules' ZEN evaluate decisions in microseconds, not the hundreds of milliseconds a model call costs even before you count tokens.
The gate doesn't replace the model. It narrows what reaches the model down to the part that actually needs judgment.
This isn't a case to replace all your AI flows, but instead to route only the hardest parts that the deterministic layer can't handle to a satisfactory level for whatever metric is important to you (e.g. precision, coverage).
None of these three levers touch how good the model is, how you prompt it, or how much AI your team uses. They touch the plumbing around it — when you call it, how much you hand it, and whether you needed to call it at all. That's the whole point of tokenmaxxing done right: the savings that don't cost you anything to take. That's a better way to tokenmaxx, imo.