Most teams find out what their long-context requests cost when the invoice arrives, and the gap is usually wider than the extra tokens explain. Some current models bill the whole request at a higher rate once the prompt crosses a size threshold, and a spend tool that stores one rate per token class has no way to represent that.
At the end of August I wrote about Tollgate, an open-source gateway that prices every LLM request by token, reserves the worst-case cost before forwarding, and refuses the request if that reservation would break a budget. Once the provider reports usage, the reservation is settled to the actual figure. That post described v0.1.1. Four releases later, v0.2.3 is out (Stevens, 2026), and most of the work in between went into the pricing rather than into the reserve and settle path itself.
Where a rate per token stops working
This is the part worth checking against your own provider’s pricing page, because it is easy to miss and it costs money in one direction only.
Google prices Gemini 2.5 Pro at $1.25 per million input tokens for prompts up to 200,000 tokens and $2.50 above it, with output going from $10.00 to $15.00 at the same threshold. Gemini 3.1 Pro Preview uses the same 200,000 token threshold and the same 2x input and 1.5x output multiples at different absolute rates (Google, 2026a). The same tiering appears on Vertex AI, which is the Google surface Tollgate proxies (Google, 2026b). What re-rates is the whole request, rather than the excess above the threshold, so a prompt one token over the line costs twice as much on its input leg as the same prompt one token under it.
This is a property of individual models. Anthropic states that Claude 4.6 and later include the full one million token context window at standard pricing, and that a 900,000 token request is billed at the same per-token rate as a 9,000 token one (Anthropic, 2026). A provider that tiered last year may stop, and a new model family may arrive tiered.
For a proxy sitting in front of the model, that creates a specific problem. A price, as most systems store one, is one rate per token class. It cannot hold a rate that changes with request size. A proxy that stores “input costs $1.25 per million” and multiplies will under-charge every long request by the tier multiple, without any error to show for it. The ledger then disagrees with the invoice by an amount that looks like a rounding difference right up until somebody runs a long-context workload, at which point the same defect shows up as a several-thousand-dollar variance nobody can account for.
Migration 0010, in v0.2.2, puts a threshold and separate input and output multiples on the price row, resolved field by field against a deployment default. The tier applies to the reservation as well as to the settlement, so the two cannot drift apart on exactly the largest requests, which are the ones where drift is expensive.
Tiering stays off until you configure it, per model.
tollgate admin price set \
--provider vertex --model gemini-2.5-pro \
--long-context-threshold 200000 \
--long-context-input-permille 2000 \
--long-context-output-permille 1500Listing 1. Long-context tiering is per model and off until set. Multiples are given in permille, so 2000 is the 2x input multiple and 1500 the 1.5x output one.
Defaulting to an uplift would invent a charge and break the meaning of the unit, since a million tokens would stop costing the per-million rate. Until the tier is set, a prompt above the threshold is logged as a possible under-charge rather than re-rated. I would rather leave a gap somebody can see than fill it with a number I made up.
What else changed since v0.1.1
It speaks the protocols the clients already speak. There is a native Anthropic route at POST /v1/messages, which takes the key as x-api-key and returns the upstream body unwrapped, and an OpenAI-compatible route at POST /v1/chat/completions that takes a bearer token. The Anthropic SDK, OpenAI clients, LiteLLM and Google ADK agents route through it by changing a base URL and a key, with no Tollgate SDK to adopt. Where a request does cross protocols, as an OpenAI-shaped request to a Google model does, the cost is metered from the usage the provider reports rather than from the shape of the request, because token accounting is usually the first thing a translation layer loses. Tollgate builds on proxy and cost-control patterns that LiteLLM established (BerriAI, 2026).
Streaming is metered from the stream. Usage is read from the provider’s own end-of-stream reporting rather than estimated afterwards. A stream that does not finish cleanly is charged its full reservation rather than going unbilled, and the row is recorded as estimated, with /console/usage reporting measured_cost alongside estimated_cost. Keeping the measured figure separate from the assumed one has been more useful to me than the total, because it tells you how much of a month’s spend number rests on an estimate.
Prompt-cache classes are priced per class, per model. A class left unpriced falls back to a conservative multiple of the input rate, 1x on reads and 2x on writes, and logs a warning naming the model. Several under-charges that had been present from the first release were fixed across v0.1.2 and v0.2.0: Gemini thinking tokens and tool-use prompt tokens were not billed at all, reasoning tokens through Vertex’s OpenAI shim were not billed, and Anthropic cached requests were under-counted by up to 99%.
Budgets recover from losing their cache. Budget counters live in Valkey for speed and are rebuilt from the Postgres ledger. The reserve script used INCRBY, which recreates a missing key at the reserve amount, so a flush, a failover to a cold replica or an eviction restarted every budget’s period with nothing logged, and left it restarted until somebody bounced the gateway. An absent counter carries no information about the spend that preceded it, and INCRBY treats it as a counter at zero. Reserve now refuses on an absent counter, rebuilds it from the ledger and retries once, and refuses the request outright if the ledger cannot be reached.
A refusal says whose fault it is. x-tollgate-reason now uses one vocabulary on every route. The native Anthropic path used to emit the SDK’s error kind, so a budget refusal on /v1/messages reported permission_error while the same refusal on /v1/chat/completions reported budget_exceeded, and an alert keyed on the latter would have missed every Claude refusal. v0.2.3 extends that to failures. A pre-flight count that fails is now reported as an upstream error with a 502 rather than a backend error with a 503, which sends the operator to the provider instead of to Valkey and Postgres. A count endpoint that rejects a malformed body surfaces as a 400 instead of a 502, which SDKs had been retrying three times before anyone saw the real fault.
Privileged changes leave a trail. The audit_log table has existed with append-only triggers since migration 0002, and nothing wrote to it. Key issuance, revocation, budget changes and price changes now each append a row. The principal recorded is the OS user and host the command ran as, which is enough to line a change up against a shell history. It is not evidence that anyone was authenticated, and SECURITY.md says so.
If you are already running v0.1.x
Three of the fixes above change numbers you may have relied on, so they are worth acting on rather than reading past.
If you ran cache-heavy Anthropic traffic, or Gemini traffic using thinking or tools, on v0.1.1, the recorded cost for that period is low against the invoice, in the Anthropic cache case by as much as 99%. Reconcile that window against the bill rather than trusting the ledger for it.
If your Valkey instance was flushed, evicted or failed over to a cold replica while a budget period was open, that period restarted at zero and stayed restarted until the gateway was next bounced. Any cap that looked comfortable during such a window was not being applied as configured.
If you use exact admission on Vertex, it was refusing requests rather than counting them, because the body it sent carried fields the count endpoint rejects.
The upgrade path is tollgate admin migrate before starting the new binary, migration 0010 included, and a review of budgets before you start, because cache-heavy workloads will show higher spend once the classes are priced properly. Finish the rollout before relying on the counter repair. A v0.2.0 instance still recreates a missing counter at the reserve amount, so during a mixed rollout the old behaviour is live on the old instances.
Three bugs found by measuring the provider
Three of the fixes in v0.2.3 came out of a script that calls the providers’ token-counting endpoints and prints what comes back. None were found by reading the code, and two contradicted a reading of the documentation that looked obvious at the time.
Vertex :countTokens rejects safetySettings, labels, toolConfig and cachedContent with a 400. An SDK-built body routinely carries safetySettings, so for most callers exact admission on Vertex refused the request instead of counting it, and the code that built the body looked reasonable. The endpoint accepts and counts contents, systemInstruction, tools and generationConfig, which is what is now sent.
generationConfig being counted matters more than it sounds. A responseSchema lives inside it and is billed as prompt material. A one-word message counted 1 token bare and 51 tokens with a small four-property schema attached. The schema had been dropped from the count on the reasoning that it describes the output rather than the input. That reasoning is tidy and it is wrong, and it under-reserved every structured-output request on Vertex.
The third was not a counting bug at all. reqwest has been built without its default features since the first release, to make the TLS backend an explicit choice, and that took http2 out with it. Every outbound provider request in every release up to now went over HTTP/1.1, with no connection multiplexing, on the busiest path in the product. A dependency’s default feature list is not something most of us re-read after the first time we set it.
Where a provider’s behaviour is the ground truth, measure it. Reasoning about which fields cannot possibly affect a token count is how two of those three bugs were written.
What it still does not do
The README carries a long limitations list on purpose, and it is the part I would read before the feature list.
The cap is a reservation, and a settlement can exceed it. Server-injected tool-use tokens, prompts that name a stored context cache, and long-context re-rating can all settle above what was reserved, and under the default fast admission the input estimate is deliberately rough. What the design aims for is direction rather than precision. It errs upward, because under-charging is the failure that makes a spend control worth nothing.
Provider-side context cache creation and storage are not visible from the request path. They bill against the cache resource, which no proxy in this position sees.
/metrics reports something as of v0.2.3, where it previously exposed a liveness gauge alone: request totals by decision, a cost total, and the configured limit for each budget. A scrape reads process memory only, and does not query the ledger, because an observability path that depends on Postgres and Valkey stops answering during the incident you would be using it to diagnose.
Streaming under exact admission is refused on the OpenAI route, and supported on the native Anthropic one. External media by reference is refused on both. The ledger drops partitions older than 90 days by default, which is a retention setting worth looking at before it becomes a gap in your own reporting.
It is beta and pre-1.0. Interfaces and the schema may still change.
Try it without infrastructure
cargo run -- demoListing 2. The demo boots an in-memory gateway against a mock provider, so nothing needs provisioning.
No Postgres, no Valkey, no cloud credentials. It prints an API key and two budgets, serves a mock provider and the console, and you can watch the per-key meter climb and hard-stop.
The open-source core remains single-operator and single-tenant, with keys and budgets managed through the CLI and an observe-only console. A Tollgate Enterprise edition will follow for teams running across several tenants and operators, adding web and API management, role-based access control and SSO, budget-change approval workflows, multi-org scoping, and audit export. If that edition is relevant to your environment, a GitHub Discussion or a note to Sakura Sky is the place to say so.
The repository is at github.com/sakura-sky/tollgate and the release is v0.2.3. If you put a real budget in front of real traffic and the ledger disagrees with your invoice, open an issue with the shape of the workload. That is the report I want most, because the failure this release was built around is one you only see on the bill. Please keep the SECURITY.md inbox for vulnerability reports.
Disclosure: I created Tollgate. It is published as a Sakura Sky open-source project under MIT, with copyright held by me and its contributors, and there is no contributor licence agreement. Sakura Sky intends to offer a commercial Enterprise edition alongside the free core, which is a commercial interest worth stating. Sakura Sky also builds and runs managed LLM gateway environments for customers, including LiteLLM deployments, and continues to do so; Tollgate is a narrower tool for a specific job and is not a migration recommendation for anyone. The provider prices quoted here were checked on the dates given in the references and change often.
References
Anthropic (2026) Pricing. Available at: https://platform.claude.com/docs/en/about-claude/pricing (Accessed: 11 September 2026).
BerriAI (2026) LiteLLM. Available at: https://github.com/BerriAI/litellm (Accessed: 11 September 2026).
Google (2026a) Gemini Developer API pricing. Available at: https://ai.google.dev/gemini-api/docs/pricing (Accessed: 11 September 2026).
Google (2026b) Vertex AI pricing for generative AI. Available at: https://cloud.google.com/vertex-ai/generative-ai/pricing (Accessed: 11 September 2026).
Stevens, A. (2026) Tollgate v0.2.3. Available at: https://github.com/sakura-sky/tollgate/releases/tag/v0.2.3 (Accessed: 11 September 2026).