Controlling AI Costs in E-Commerce: Tokens and Caching

Per-token prices are falling faster than almost any other IT cost item: across capability thresholds, Epoch AI measures a decline of 50x per year at the median (Epoch AI). Even so, a third of German companies running AI report costs significantly above expectations: 33% (Bitkom). The contradiction dissolves once you break the bill apart. Cost is not driven by the price per token but by context length, output length and call frequency. Anyone moving AI features for online shops from prototype to production therefore needs a cost model that exists before go-live.

Understanding the cost model: input, output, context

Language models are billed per token, separately for input and output. One token corresponds to roughly three to four characters of running text. Two properties of this billing model matter for your calculation: output tokens cost a multiple of input tokens across the common model classes, typically around five times as much (project experience). And input covers not just the current question but the entire context you send along: system instruction, product data, conversation history, attached documents.

From this follows the single most important sentence in this article: context length is the main cost driver, not answer quality. An assistant that ships 8,000 tokens of catalogue excerpt with every message costs ten times that context across ten messages, even though the catalogue has not changed in between. The API is stateless: every call carries the full history again, and every repetition is billed again.

This structure dictates the right control metric. A monthly invoice tells you something was expensive but not why. The robust figure is cost per interaction: per generated product text, per chat answer, per classified ticket. Only that figure can be set against a benefit, and only it makes forecasting possible as volume grows.

Cost per use case instead of interpreting a monthly invoice

A shop rarely runs one AI feature; usually it runs four to six. They differ by orders of magnitude in cost and benefit, yet disappear behind a single number on the combined invoice. The first step towards cost control is therefore mundane and still rarely taken: calculate every feature separately, with tokens per operation and expected volume.

Benefit belongs next to cost, expressed as contribution margin rather than gut feeling. For product text generation the calculation is usually clear: editorial time saved per item can be valued at an internal hourly rate. For advisory dialogue it gets fuzzier but not impossible: comparing the share of sessions with chat contact and their order rate against a control group gives a workable approximation. The same reasoning carries profitability and margin work in e-commerce.

That this calculation is still rarely done properly shows on the impact side. According to the enterprise survey analysed by Stanford HAI, organisations report the largest cost savings in software engineering and manufacturing at 56% each (Stanford HAI, AI Index 2026), while revenue effects are most often cited for marketing and sales at 67% (Stanford HAI, AI Index 2026) and for product development at 62% (Stanford HAI, AI Index 2026). At enterprise level, however, only 39% (McKinsey) of respondents report a measurable EBIT effect, and roughly 6% (McKinsey) attribute more than five percent of EBIT to AI use.

An AI feature without a cost per operation is an experiment. That number is what turns it into a product you can scale or switch off.

XICTRON development team

Seven levers that actually reduce the bill

The following seven measures differ in impact, but in this order they take effect fastest. The order matters because the first three are purely technical and need no business sign-off, whereas the last four require decisions about behaviour and quality.

1. Prompt caching for recurring context. System instruction, legal texts, shipping rules and catalogue excerpts typically do not change between two calls. Cached as a prefix, they are billed at a fraction of the input price on the next call.
2. Result caching for identical requests. The same question against the same data produces the same answer. A cache key built from a normalised question plus data version intercepts a substantial share of requests before any model is called.
3. Batch processing for bulk runs. Alt texts, category texts or data enrichment are not time-critical. Asynchronous bulk processing is billed considerably lower than real-time individual calls, in the order of half the price at the common providers.
4. Model routing per task. Classification, extraction and short decisions do not need a large model. A small model for structured tasks and a large one only for free text reduces the bill without a noticeable quality drop at the customer-facing surface.
5. Trim context instead of attaching the catalogue. Rather than shipping 200 products, a semantic product search supplies the five relevant ones. This is the lever with the best effort-to-effect ratio.
6. Pre-filters against pointless calls. Empty inputs, duplicates within seconds, obvious bot patterns and requests outside the intended scope should be rejected before the model call, not after it.
7. Cap output length. A hard token limit plus a validated output schema prevents sprawling answers. Because output tokens are the most expensive, this measure has a disproportionate effect.

Prompt caching deserves a closer look, because the mechanics are often misunderstood. What gets cached is a prefix, meaning the beginning of the request in exactly the same byte sequence. If anything ahead of it changes, everything after it drops out of the cache. A timestamp injected into the system instruction, an unsorted JSON serialisation or a per-user tool set is enough to push the hit rate to zero. Stable content first, volatile content last is the rule.

Batch processing is the most frequently overlooked lever because it requires a change in the workflow. Bulk runs belong in a queue, not in the shop's request cycle. The discount matters less than the side effect: bulk runs are predictable, interruptible and can be capped with a quota. Robust asynchronous processing is the same discipline that carries Shopware performance work in general.

Budget governance: caps, cost centres, alerts

Technical levers lower the price per operation. They do not protect against the number of operations rising unexpectedly. That is what governance is for, and at its core it is unspectacular: every AI feature gets its own cap, its own cost centre and its own alert.

• Cap per feature, not per account. A daily and a monthly cap per feature prevents a runaway bulk run from consuming the customer dialogue's budget.
• Carry a cost centre per feature. Every model call is logged with feature identifier, model, input, output and cache tokens. Without that attribution, later optimisation is guesswork.
• Alert on outliers instead of a monthly report. Two thresholds work well: an absolute one (daily budget 80 percent consumed) and a relative one (cost per operation deviates by more than a third from the mean).
• A kill path without a deployment. Every AI feature needs a configuration switch that disables it within minutes or reroutes it to a cheaper model.
• Define the degradation step. What happens when the cap is reached? An error message, classic search, a contact form or a queue are four defensible answers; a silent failure is not.
• Cost in the definition of done. An AI feature counts as finished only once cost per operation, cap, cost centre and degradation step are documented.

The most common mistake is a cap without a defined reaction. When the budget bites and the feature then breaks off without comment, support cases arise whose handling costs more than the inference saved. The degradation step therefore belongs in the business requirement, not in a technical footnote. For recurring workflows the same principle applies as in any other process automation: the exception belongs in the design.

Abuse protection for public AI endpoints

An AI endpoint any visitor can call without signing in is, from an attacker's perspective, free compute. Running it unprotected means paying for someone else's inference. The attack needs no sophistication: a script calling the chat endpoint ten thousand times overnight produces a bill that blows the monthly plan without triggering anything in classic security monitoring.

• Rate limit per IP and per session, with different thresholds: a short one (calls per minute) and a long one (calls per hour and day).
• Session quota instead of a pure frequency brake. An advisory dialogue rarely needs more than around twenty answers. A hard quota per session limits the damage even when frequency stays inconspicuous.
• Cap input length. A field without a length limit invites flooding the context with foreign text and multiplying the cost per call.
• Verify origin. Same-origin checks, short-lived tokens and a server-side proxy prevent the endpoint from being used out of third-party applications.
• Log cost per session. Being able to see the most expensive sessions of a day surfaces abuse patterns typically within hours rather than at month end.
• Limit anonymous use, treat signed-in use more generously. This gradation protects the budget without slowing genuine customers down.

Make or buy: when self-hosting becomes financially interesting

As soon as the bill becomes visible, the self-hosting question arrives. It is legitimate but usually asked too early and argued with the wrong numbers. The comparison is not API price against server price, but API invoice against the fully loaded cost of a permanently operated model.

Self-hosting becomes financially interesting only when two conditions hold at once: the monthly interface invoice clearly exceeds the fully loaded cost of permanently reserved accelerator capacity, and utilisation sits stably above roughly 40 percent (project experience). Below that threshold you are paying for idle time. For most mid-sized shops this argues clearly against self-hosting the language model, while smaller specialist models for classification or embeddings can run sensibly in your own hosting.

The same fixed costs are regularly overlooked: on-call outside business hours, model and driver maintenance, headroom for load peaks, a second site for resilience, and the staff effort for quality comparisons at every model change. Adding these positions to the calculation usually pushes the break-even considerably higher. Sober architecture consulting saves more here than any price negotiation.

Worked example: a mid-sized shop

To close, a worked example. The starting point is a shop with around 45,000 sessions a month, roughly 18,000 items and four AI features in operation. All values are example values based on typical volume structures; they are meant to show the calculation grid, not to replace your own figures.

The saving amounts to roughly 62 percent, without switching off a feature or changing answer quality at the customer-facing surface. It comes from the sum of the levers: prompt caching of the shared preamble in dialogue, a result cache for recurring questions, batch runs for alt texts and product texts, a small model for ticket classification, and a leaner context through semantic pre-selection instead of a catalogue excerpt.

The second figure is more interesting than the saving. Without a rate limit and session quota, a single automated access to the public chat endpoint can trigger 40,000 additional answers overnight. At the baseline price that would be EUR 320 in a single day, more than the entire planned monthly bill. In this calculation, abuse protection is therefore not the last item on the list but the one with the largest single effect on risk.

Build AI features with cost control from day one

Adoption has made this a mainstream question. 41% (Bitkom) of German companies with 20 or more employees already use AI, up from 17% (Bitkom) a year earlier, and a further 48% (Bitkom) are planning or discussing it. Internationally, the share of organisations using AI in at least one business function stands at 88% (Stanford HAI, AI Index 2026) after 78% (Stanford HAI, AI Index 2026) the year before, and at 91% (Stanford HAI, AI Index 2026) in Europe. Anyone building an AI feature today builds it into an environment where competitors use the same technology. The difference does not come from access but from economics.

The good news: cost control is not a retrofit project but a design decision. A caching layer, a model router, cost logging per feature and a cap with a defined degradation step together form a manageable building block when they are planned from the start. Retrofitted into a grown integration, the same components cost a multiple, because prompt assembly, data access and error handling have to be re-cut.

XICTRON develops AI features for online shops with exactly this set-up: prompt and result caching as a dedicated layer, model routing per task, batch paths for bulk runs, budget caps with a degradation step per feature, and reporting that makes cost per operation and per feature visible. That way the economics of an AI feature are settled before go-live rather than with the first invoice.

Frequently asked questions