Tollgate v0.3.0: Stream Metering, Per-Key Budgets, and Vertex Endpoints

When your teams route LLM traffic through a spend-control gateway, two things have to be true before its figures are worth much. The spend it records has to match what the provider bills, and a cap that appears on a dashboard has to apply to the traffic it appears to cover. If either one slips, the gateway still produces numbers, but finance will struggle to reconcile them against the invoice, and the caps lose their value as controls.

Tollgate is the open-source gateway I created for this job. It prices each request by token, reserves the worst-case cost against every budget that applies, and refuses the request with a 402 before it reaches the model if a hard cap would be exceeded. The previous post covered long-context pricing and the other changes up to v0.2.3. v0.3.0 is out now (Stevens, 2026a). It fixes metering errors in both directions, mostly streams charged too much and one case that could have been charged too little, and it enforces per-key budgets that were stored but never applied. The release notes sum it up as “streams charged what they measured, Vertex locations that resolve, and budgets that are enforced as set or reported when they cannot be.”

Streamed requests are charged what they used

On the OpenAI-compatible route, /v1/chat/completions, a stream that finished normally but whose client then stopped reading was charged its full worst-case reservation instead of its measured usage. The common case is a client that stops reading as soon as it sees finish_reason, before the data: [DONE] end marker. That is ordinary client behaviour.

Once the provider has sent its finish, Tollgate now keeps reading the upstream for up to 10 seconds after it notices the client has stopped, forwarding nothing, to collect the final usage report and the end marker. The stream is then charged what it measured. An upstream error or timeout after the finish still charges the reservation.

The same change tightens what counts as finished. A stream is measured only when the provider sent a non-empty finish_reason with no output after it, a usage report in or after that chunk, and [DONE]. A connection that closed cleanly without those used to be treated as finished, and on an upstream that reports usage on every chunk, a stream cut mid-generation could have been under-charged. That is the one metering case in this release that ran low. For a spend control, under-charging is the more serious direction of error, because it lets traffic past a cap.

Two related changes can also move your figures. A stream that reports usage more than once is now charged the most expensive of its reports, each priced on its own, rather than the last one. A stream whose prompt or output count falls from one report to the next is treated as self-contradictory and charged its reservation. Tollgate assumes usage is cumulative across a stream, so a custom upstream that reports per-chunk deltas will be measured low, or charged its reservation when a delta is smaller than the one before.

Every stream the relay settles now logs how it ended, at info for any stream not settled cleanly at the end marker, so an operator can see why a given stream was charged its reservation without inferring it from the ledger. There is one consequence to plan for. A custom upstream that never sends a finish reason or the end marker is now charged its reservation on every stream, and some streams previously recorded as measured may now be recorded as estimated. If you run such an upstream, the estimated share of your spend will rise after the upgrade. That rise reflects streams the gateway could not measure, which earlier releases counted as measured.

Per-key budgets that were not being enforced

This is the fix I would read first if you run Tollgate today. A per-key budget whose key id was entered in any form a UUID parser accepts other than lowercase with hyphens (upper case, no hyphens, braces, or a urn:uuid: prefix) was stored as typed. Requests carry the lowercase hyphenated form, so the budget never matched its key, and that key ran under the wider provider, model, and global budgets only. No log line or metric reported the mismatch, so the budget appeared in listings while the key ran without it.

Listing 1 shows the shape of the bug. Every form a UUID parser accepts describes the same key, but only one of them matches the string a request carries.

import uuid

def canonical_key_id(typed: str) -> str:
    """Return the one form a stored per-key budget id should take."""
    return str(uuid.UUID(typed))  # lowercase, hyphenated

request_key_id = "3f2b8c1e-9d4a-4c6b-8e2f-1a7d5c9b0e44"
typed_forms = [
    "3F2B8C1E-9D4A-4C6B-8E2F-1A7D5C9B0E44",
    "3f2b8c1e9d4a4c6b8e2f1a7d5c9b0e44",
    "{3f2b8c1e-9d4a-4c6b-8e2f-1a7d5c9b0e44}",
    "urn:uuid:3f2b8c1e-9d4a-4c6b-8e2f-1a7d5c9b0e44",
]
for typed in typed_forms:
    stored_as_typed = typed == request_key_id
    stored_canonical = canonical_key_id(typed) == request_key_id
    print(f"{typed:46} as typed: {stored_as_typed!s:5}  canonical: {stored_canonical}")

use uuid::Uuid;

/// Return the one form a stored per-key budget id should take.
fn canonical_key_id(typed: &str) -> Result<String, uuid::Error> {
    Ok(Uuid::parse_str(typed)?.hyphenated().to_string()) // lowercase, hyphenated
}

fn main() -> Result<(), uuid::Error> {
    let request_key_id = "3f2b8c1e-9d4a-4c6b-8e2f-1a7d5c9b0e44";
    let typed_forms = [
        "3F2B8C1E-9D4A-4C6B-8E2F-1A7D5C9B0E44",
        "3f2b8c1e9d4a4c6b8e2f1a7d5c9b0e44",
        "{3f2b8c1e-9d4a-4c6b-8e2f-1a7d5c9b0e44}",
        "urn:uuid:3f2b8c1e-9d4a-4c6b-8e2f-1a7d5c9b0e44",
    ];
    for typed in typed_forms {
        let stored_as_typed = typed == request_key_id;
        let stored_canonical = canonical_key_id(typed)? == request_key_id;
        println!("{typed:46} as typed: {stored_as_typed:5}  canonical: {stored_canonical}");
    }
    Ok(())
}

Listing 1. Four accepted spellings of one illustrative key id: none matches the request's id as typed, and all four match once canonicalised.

Key ids are now stored and loaded in a single canonical form. Migration 0011 converts existing rows and adds a database constraint so no other form can be stored. The rule is now also enforced in the schema, so a row written outside the CLI is refused as well.

The release also deals with the wider class of budget that can never apply. tollgate admin budget set now refuses scopes that could never match: an unknown provider, or a model with no current price. If you want to set a model’s cap before its price exists, the new --allow-unpriced flag does that deliberately.

Budgets that can’t match some or all of their requests are logged at startup and on reload, and counted by a new Prometheus gauge on /metrics, tollgate_unmatchable_budgets. An alert on that gauge above zero is a sensible default, unless you have staged a cap on purpose. It isn’t exposed by tollgate demo, so look for it on a real deployment. One case is logged but not counted: a Vertex budget while /v1/chat/completions also reaches Vertex, because those requests are scoped as openai. Check the log for it as well as the gauge.

Vertex AI locations that resolve

Requests for Vertex’s global, us, and eu locations now go to the hosts Google documents for them. Google’s endpoint documentation gives aiplatform.googleapis.com for the global endpoint and aiplatform.us.rep.googleapis.com and aiplatform.eu.rep.googleapis.com for the two multi-region endpoints (Google, 2026). Earlier releases built a {location}-aiplatform.googleapis.com hostname for these locations, which Google does not list. The native adapter, token counting, and the OpenAI-compatible endpoint now share the same host. Thanks to Florian Valéry for the global fix.

Vertex project and location values are also validated at startup. Both are built into the URL of requests that carry the gateway’s Google access token, and the location goes into the hostname, so a bad value is refused before any request carries that token anywhere. The validation accepts either a Google Cloud project number or a project ID.

Hardening and supply chain

OpenTelemetry moves to 0.33. An OTLP endpoint the exporter can’t use is now refused at startup with a clear message, and that message names only the scheme of the refused value, never the value itself, because an endpoint string can carry credentials.

Configuration is checked before Tollgate connects to anything. Billing, the API-key pepper, the upstream, admission mode, Vertex, and Redis settings are validated before the gateway connects to Postgres or Valkey, so a configuration mistake is reported as one instead of surfacing behind a connection error.

On the supply chain, dependency updates in this release clear published RustSec advisories, including in rustls (0.23.46) and tokio (1.53.3), and GitHub Actions are pinned by commit SHA (Stevens, 2026b). Dependabot now waits before proposing version updates and no longer proposes tokio releases past the 1.53 long-term-support line, while security updates still come through immediately.

Developer experience

The compose file can now run Tollgate itself with --profile app, alongside Postgres and Valkey. The app profile applies migrations at boot, so don’t point it at an existing deployment’s database without previewing 0011 first. The demo can run in Docker when cargo isn’t installed, so trying the gateway no longer needs a Rust toolchain.

Before you upgrade

Upgrading to v0.3.0 requires migration 0011, and applying it is the point at which enforcement changes. Instances already running pick up the corrected budgets at their next config reload after 0011 is applied, or at restart if config reload is disabled, so keys can start receiving 402s before you deploy v0.3.0. v0.3.0 itself won’t start until 0011 is applied. Work through these steps in order.

1. Preview the affected budgets. Run the query in Listing 2, from the release notes. For each row, compare the key’s spend this period with its limit (limit_micros is in millionths of the billing currency). A key already past a hard limit will be refused straight away once its budget loads.
2. Fix what would stop the migration. It refuses, listing the rows, if a per-key budget’s id is not a UUID at all, or if two budgets name the same key and period in different forms. For each row the query returns, check whether another budget exists for the same key and period; if one does, keep the row with the limit you intend, because the migration won’t choose for you.
3. Tell the owners of affected keys before their budgets start enforcing against spend already recorded this period.
4. Check configuration that earlier releases accepted. An OTLP endpoint must now be http://host:port or unix:///path. Both https:// endpoints (this build has no TLS) and values with no scheme, such as collector:4317, are refused at startup. An empty endpoint now disables export; if you relied on the OTEL_EXPORTER_OTLP_* variables, set TOLLGATE_TELEMETRY__OTLP_ENDPOINT explicitly. If Vertex is in use, natively or as the OpenAI-compatible upstream, the project must be a project number or ID, and the location must be lowercase letters, digits, and inner hyphens.
5. Apply 0011 with tollgate admin migrate, then deploy v0.3.0.

SELECT id, scope_id, period, limit_micros
FROM budgets
WHERE scope_type = 'api_key'
  AND scope_id !~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$';

Listing 2. Preview the per-key budgets whose key id is not in canonical form, before running migration 0011.

Recorded spend can move in both directions after the upgrade: down for streams that were over-charged, and up in the estimated share on upstreams that don’t send finish_reason and [DONE], and on keys whose budgets now enforce. Marking the upgrade date on spend dashboards makes that easier to explain later.

Where to get it

Tollgate is beta and pre-1.0, so interfaces and the schema may still change. The open-source core is single-operator and single-tenant by design, and a Tollgate Enterprise edition is planned.

The release is v0.3.0 and the repository is at github.com/sakura-sky/tollgate. If your ledger and your provider invoice disagree after this release, an issue describing the shape of the workload (without keys, prompts, or customer data) helps most. Please keep the SECURITY.md inbox for vulnerability reports.

Disclosure: I created Tollgate. It is a Sakura Sky open-source project published under MIT, copyright is held by me and its contributors, and there is no contributor licence agreement. Sakura Sky plans to offer a commercial Enterprise edition alongside the free core, which is a commercial interest. Sakura Sky also builds and runs managed LLM gateway environments for customers, including LiteLLM deployments, and continues to do so. Nothing in this post is a recommendation to change gateways.

References

Google, 2026. Deployments and endpoints. Google Cloud Documentation. Available at: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/learn/locations [Accessed 30 October 2026].

Stevens, A., 2026a. Tollgate v0.3.0. GitHub release. Available at: https://github.com/sakura-sky/tollgate/releases/tag/v0.3.0 [Accessed 30 October 2026].

Stevens, A., 2026b. Tollgate. GitHub repository. Available at: https://github.com/sakura-sky/tollgate [Accessed 30 October 2026].