The four core golden signals for a multi-model AI gateway are latency, traffic, errors, and saturation. Monitor them together, then segment them by model, tenant, provider, route, workload, region, and deployment version. Extend this framework with AI-specific measurements such as token throughput, cost per request or token, cache behavior, routing distribution, fallback frequency, retry amplification, and capacity headroom.
The short answer: monitor latency, traffic, errors, and saturation together
A multi-model gateway sits between applications and a changing mix of models, serving infrastructure, and external providers. A fleet-wide success rate or average response time can therefore look healthy while one tenant, model, or route is experiencing severe degradation.
The traditional golden signals provide a durable operating framework:
| Signal | Recommended measurements | Useful dimensions | User-visible symptom | Operational decision |
|---|---|---|---|---|
| Latency | Time to first token, generation rate, end-to-end response time, queue time | Model, route, workload, streaming mode, deployment version | Slow initial response, uneven streaming, delayed completion | Investigate routing, queues, batching, or capacity |
| Traffic | Requests, input tokens, output tokens, token throughput | Tenant, model, provider, route, region, workload | Demand spikes, uneven load, unexpected usage | Adjust capacity plans or routing policy |
| Errors | Gateway failures, upstream failures, timeouts, rate limits, policy denials | Failure source, tenant, model, provider, route, version | Failed, blocked, incomplete, or degraded responses | Isolate the failure domain and choose an appropriate response |
| Saturation | Accelerator utilization, memory pressure, concurrency, queue depth, batch occupancy | Cluster, node, model, route, region, deployment | Rising latency, timeouts, rejected work | Rebalance demand, revise serving policy, or add headroom |
No single signal is sufficient. For example, rising latency with stable traffic may point to saturation, an upstream dependency, or a deployment regression. Rising traffic without latency or error impact may simply represent healthy demand. High GPU utilization may be acceptable when queue time and tail latency remain within workload objectives; low utilization may still coexist with inefficient batching or another bottleneck.
Interpretation must also reflect the workload. Latency-sensitive chat, batch enrichment, and agentic workflows create different serving-policy problems. An interactive assistant may prioritize fast response initiation and stable streaming, while a batch process may prioritize completion throughput and predictable cost.
Measure latency across the complete request and generation path
AI gateway latency is not one number. Measure the stages that explain what the user experiences and where time is being spent:
- Time to first token (TTFT): Time from request submission until the first generated token reaches the client. This is particularly important for interactive streaming experiences.
- Inter-token latency or generation rate: The pace of generation after the response begins. A fast first token followed by inconsistent generation can still produce a poor experience.
- End-to-end response time: Total duration from request receipt to completion, cancellation, or failure.
- Queue time: Time spent waiting before execution. This can reveal capacity pressure even when model execution remains stable.
- Upstream model latency: Time attributable to an external provider or model-serving dependency, where it can be measured separately.
Use percentile distributions rather than relying on averages. An average can remain stable while a smaller but commercially important group of requests experiences a major slowdown. Tail percentiles help reveal intermittent queueing, overloaded routes, provider degradation, unusually long generations, and deployment-specific regressions.
Latency should be segmented by model and route because different models can have different generation characteristics. It should also be segmented by workload, tenant, prompt class, streaming mode, region, and deployment version where those dimensions support a real diagnostic decision.
Avoid prescribing one universal latency threshold. The relevant objective depends on user expectations and workload behavior. A chat interface, background extraction job, and multi-step agent should not necessarily share the same target. Establish workload-specific service objectives and evaluate both user-visible latency and its underlying components.
Latency is a service-health signal—not evidence of answer quality. A fast response can still be inaccurate, ungrounded, unsafe, or unsuitable for the task.
Segment traffic by requests, tokens, routes, and workload context
Request rate alone does not describe AI serving demand. Two routes can process the same number of requests while consuming very different capacity because their input lengths, output lengths, models, and generation patterns differ.
A useful traffic view combines:
- Request count and request rate
- Input and output token volume
- Token throughput over time
- Streaming versus non-streaming requests
- Completed, cancelled, retried, and redirected requests
Segment these measurements by model, tenant, provider, route, workload, region, and deployment version. This makes traffic data useful for diagnosis, attribution, and capacity planning. For example, a traffic increase isolated to one tenant requires a different response from a fleet-wide increase. A shift toward longer outputs may increase serving demand even if request volume remains flat.
Route-level visibility is especially important in a multi-model environment. Operators should be able to distinguish intended routing changes from unexpected concentration on one model or provider. Deployment-version correlation helps determine whether a traffic shift followed an application release, gateway policy change, or model update.
Segmentation requires cardinality discipline. Use bounded identifiers and controlled categories for operational metrics. Raw prompts, arbitrary user strings, request IDs, and other unbounded values should not become metric labels. Detailed request-level context is generally better handled through appropriately governed logs or traces, while metrics retain stable dimensions for aggregation and alerting.
Token volume also needs context. It can help explain demand and economics, but it does not independently predict compute requirements or cost. Model architecture, quantization, batching, hardware, cache behavior, prompt structure, and output generation all affect the relationship between tokens and serving resources.
Classify errors by failure source and user-visible impact
A single aggregate error rate hides where failures originate and what users experience. Build an error taxonomy that distinguishes at least the following conditions:
- Gateway failures: Internal gateway processing or connectivity failures
- Provider or model failures: Errors returned by an upstream model endpoint or serving system
- Timeouts: Requests that exceed a gateway, client, or upstream time limit
- Rate limits: Work rejected or delayed because a request, token, or concurrency limit was reached
- Policy denials: Requests intentionally blocked by an applicable access or usage policy
- Malformed requests: Invalid schemas, unsupported parameters, or incompatible inputs
- Fallback exhaustion: Cases in which the available fallback path cannot complete the request
- Degraded responses: Requests that technically complete but use an unintended fallback, return partial output, or lose an expected capability
Track both the technical source and the user-visible result. A timeout caused by an upstream dependency should not be attributed to the same operating domain as invalid client input. Likewise, a fallback response may produce a successful transport status while delivering a materially different model, latency profile, cost, or capability.
Retry behavior deserves separate attention. Retries can increase traffic and upstream load, creating retry amplification during an incident. Correlate retry volume with timeouts, rate limits, queue depth, and fallback activity rather than viewing retries only as successful recovery attempts.
Segment errors by model, provider, route, tenant, workload, prompt class, and deployment version when useful. This allows operators to distinguish a localized model problem from a gateway-wide event without creating uncontrolled metric cardinality.
A successful HTTP response does not establish task success. Model-output failures—including poor groundedness, unsafe output, incorrect answers, and low human acceptance—require a separate evaluation system.
Monitor saturation across compute, queues, batches, and upstream limits
Saturation indicates how close a constrained resource or dependency is to limiting useful work. In an AI serving path, the constraint may appear in compute, memory, queues, network connections, or an upstream quota.
Relevant measurements can include:
- GPU or accelerator utilization and memory pressure
- Active concurrency and available execution slots
- Queue depth and time spent waiting
- Batch occupancy, formation time, and execution behavior
- Connection-pool usage and pending connections
- Upstream rate-limit or quota headroom
Interpret these measurements together. Accelerator utilization alone does not prove that capacity is healthy or efficient. High utilization may be productive if queues, errors, and user-facing latency remain controlled. Low utilization may still accompany memory constraints, poor batch formation, connection bottlenecks, uneven routing, or idle capacity that cannot serve the waiting workload.
Queue depth is similarly contextual. A batch workload may intentionally accumulate requests briefly to improve batch formation, while the same queue behavior may be unacceptable for interactive chat. Queue time, rather than depth alone, often provides a clearer connection to user impact.
Upstream limits belong in the saturation view even when compute is privately deployed elsewhere. A route can have ample local capacity but insufficient provider quota, connection headroom, or concurrency allowance. Monitor dependency constraints alongside internal resources so operators do not respond to the wrong bottleneck.
Avoid universal utilization or queue thresholds. Baselines should account for workload type, model behavior, hardware, batching policy, and service objectives. Alerting should focus on sustained symptoms and diminishing headroom—not every short-lived utilization spike.
Add AI-specific signals for economics, routing, and capacity
The traditional golden signals remain the foundation, but AI gateways need additional measurements to explain serving behavior and inference economics. Cost is an extension to the framework, not one of the original four golden signals.
Useful AI-specific extensions include:
- Cost per request or token: Attribute modeled or billed consumption by tenant, model, route, and workload where feasible.
- Cache hit and bypass rates: Identify how often eligible work uses a cache and why requests bypass it.
- Routing distribution: Show how demand is allocated across models, providers, and serving pools.
- Fallback frequency: Reveal when secondary paths are being used and whether fallback has become normal rather than exceptional.
- Retry amplification: Compare original demand with the additional traffic generated by retries.
- Token throughput: Track input processing and output generation in a way that complements request rate.
- Capacity headroom: Estimate how much additional workload a route or serving pool can absorb under its current operating assumptions.
These signals turn observability into decision support. A change in routing distribution can be compared with latency, error, cost, and output-evaluation data before a routing policy is retained. Cache behavior can be examined alongside workload eligibility and response requirements. Queue and batch measurements can inform batching decisions. Memory pressure and workload tests can inform whether a quantization option merits evaluation. Demand, headroom, and queue behavior can guide GPU scheduling decisions.
None of these measurements identifies a universally optimal policy. Routing, batching, caching, quantization, and GPU scheduling involve trade-offs among latency, throughput, model behavior, infrastructure control, and cost. Changes should be evaluated against workload-specific objectives and separate model-quality tests.
Token Forge Cloud Private LLM Inference focuses on private deployment and serving-layer optimization using capabilities including caching, model routing, batching, quantization, and GPU scheduling. Private deployment paths can keep models, prompts, and telemetry in the customer’s controlled environment. The appropriate telemetry design and operating controls should be defined for the actual deployment and governance model.
For teams still validating demand, Token Forge Cloud Managed Model APIs offer an API-first path to model access before committing to private serving capacity. Broad usage data can help establish workload patterns, while a private serving plan becomes easier to size once model demand is more predictable.
Turn telemetry into alerts, serving decisions, and evaluation criteria
Production alerts should represent sustained user-facing symptoms or a credible threat to a service objective. An isolated GPU spike is usually less actionable than rising tail latency accompanied by queue growth and declining capacity headroom.
A practical alerting model combines:
- A workload-specific objective: Define what acceptable service means for chat, batch enrichment, agents, or another workload.
- A sustained symptom: Avoid paging on transient events that resolve without user impact.
- Correlated signals: Combine latency, traffic, errors, and saturation to identify likely causes.
- A clear operator action: Every alert should lead to a diagnostic or intervention path.
For example, rising TTFT, increasing queue time, and stable upstream latency may justify investigating local scheduling or capacity. More timeouts accompanied by upstream rate limits may point toward quota pressure or route concentration. Higher request volume combined with falling cache-hit rates may warrant checking workload eligibility or a deployment change. These are diagnostic patterns, not universal rules.
Keep operational telemetry separate from model-output evaluation. Service-health metrics answer whether the system received, routed, and served a request reliably. Evaluation signals answer whether the output accomplished its task. Depending on the application, those signals may include task success, groundedness, safety evaluation, human acceptance, or downstream business outcomes. Both layers matter, but one cannot substitute for the other.
Checklist for actionable gateway observability
When planning a multi-model AI gateway or private inference control plane, determine whether the operating model provides:
- Latency decomposition across queueing, first token, generation, and completion
- Percentile views by model, route, tenant, workload, and deployment version
- Request and token traffic measurements rather than request counts alone
- Failure attribution across the gateway, model, provider, policy, and client
- Traceability across routing, retries, fallbacks, and deployment changes
- Cost allocation at a level appropriate for finance and FinOps decisions
- Capacity views spanning compute, memory, concurrency, queues, batches, and quotas
- Controlled metric dimensions and a plan for high-cardinality diagnostic data
- A separate evaluation process for quality, groundedness, safety, and human acceptance
- Operator controls with defined permissions, review paths, and rollback procedures
The central question is not whether a gateway collects many metrics. It is whether teams can connect a user-visible symptom to a model, route, dependency, deployment change, and safe operational decision. That traceability is what makes telemetry useful for production serving and inference economics.
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.