Most teams find out their AI feature is expensive the same way: the invoice arrives.
By then it’s too late to design around it. The system prompt is already live, the query volume is already what it is, and the only remaining move is a retroactive optimization pass under budget pressure. The better version of this story starts earlier — before the feature ships, with a number.
Tokens are not words, and the gap is not fixed
Every LLM API bills you in tokens, not words or characters. A token is a sub-word fragment — sometimes a whole word, sometimes a piece of one. That distinction sounds pedantic until you notice it isn’t a fixed conversion rate. Plain conversational English tends to sit close to one token per word. Technical writing — compound terms, jargon, hyphenated phrases — routinely costs 50% more tokens per word than plain prose covering the same ground, because the tokenizer has to break unfamiliar compound terms into multiple pieces. Code-adjacent text (type annotations, UUIDs, punctuation-heavy syntax) does the same thing for a different reason.
The practical consequence: two prompts of identical word count can bill meaningfully differently depending on register. If your system prompt reads like an engineering spec instead of a memo, you’re likely paying a jargon tax you never budgeted for.
The multiplication problem
A single system prompt sent once is a rounding error. A single system prompt sent on every request, at production volume, is a line item — and the line item scales linearly with something engineers rarely think about when they’re writing prompt copy: query volume.
The formula is unglamorous: tokens per call, multiplied by calls per day, multiplied by the provider’s per-token price, multiplied by 365. A modest system prompt at a few dozen tokens, sent a thousand times a day, lands somewhere in the range of tens to low hundreds of dollars a year on a mid-tier model. That number alone doesn’t sound alarming. It stops being a rounding error once you notice most production AI features run more than one prompt — a router prompt, a classifier prompt, a response-formatting prompt — and volume assumptions from a pilot rarely survive contact with real usage.
This is the part most cost conversations skip: the unit economics are decided the day someone writes the system prompt, not the day someone reviews the AWS bill.
The fix nobody markets, because it’s not a product
The single highest-leverage cost intervention available to most teams isn’t a cheaper model, a caching layer, or a routing strategy — it’s rewriting the prompt itself. Redundant phrasing, restated constraints, and verbose formatting instructions are all token cost with no functional payoff. A prompt that says the same thing in fewer tokens produces identical behavior at a lower bill, every single call, forever.
This is unglamorous advice compared to “add a caching layer” or “route to a cheaper model,” which is probably why it’s underused. It also requires zero new infrastructure — just a pass over the prompt with the same editorial instinct you’d apply to cutting a bloated PR description.
Temperature is a separate lever, and it’s not a quality dial
Cost and quality tend to get bundled into one mental category — “make it better” — but temperature belongs to neither. Temperature controls how sharply the model commits to its highest-probability next token versus how willing it is to sample further down the distribution. It doesn’t change what the model knows, and it doesn’t cost more or less. It changes consistency.
A classifier, an extraction task, anything that needs to produce the same output for the same input every time, wants temperature near zero — that’s a reliability requirement, not a taste preference. A brainstorming feature, a creative-copy generator, anything where variety is the actual product, wants it higher. Picking the wrong one doesn’t blow up your bill; it produces a support ticket when a “deterministic” feature quietly isn’t.
Measure before you ship, not after the invoice
None of this requires a production account or an API key to start. Any tokenizer playground will show you real token counts for a candidate prompt in seconds, and the cost formula above is arithmetic, not infrastructure. The habit worth building is small: before a prompt goes into a PR, know its token count, know the volume it’ll run at, and know the number that produces. That fifteen-minute check is the difference between a design decision and a postmortem.
Token cost is the first constraint you hit. The second shows up once a conversation keeps growing turn over turn — that’s covered in how LLM context and memory management actually works.
This is session one of a 12-part AI/ML engineering course covering the fundamentals through production system design: How LLMs Actually Work →