Record latency as a trace-level sequence of named boundary events with timestamps, derived durations, request and trace identifiers, attempt numbers, parent-child relationships, outcomes, and timing sources. Preserve the raw events so an auditor can reproduce each calculation, identify overlapping intervals, and distinguish gateway-observed upstream time from processing reported by the provider.
Exact boundaries vary by architecture and provider. The important principle is to define every interval through observable start and end events rather than relying on ambiguous labels such as “provider latency” or “processing time.”
| Interval | Recommended start boundary | Recommended end boundary | Important qualification |
|---|---|---|---|
| Queue time | Enqueue or admission-wait start | Dequeue or execution admission | Separate batch formation or scheduler wait when those are distinct stages |
| Gateway overhead | Gateway ingress or resumption after an external stage | Queue entry, upstream dispatch, or final response flush | The complete gateway span includes other stages and is not itself pure gateway processing |
| Provider pre-stream time | Upstream provider dispatch | First response byte or first token | Label it gateway-observed unless correlated provider-native timing is available |
| Streaming time | First byte or first token | Final byte or final token | Record which convention is used; byte and token boundaries are not interchangeable |
| End-to-end time | Declared request boundary, such as gateway ingress | Declared completion boundary, such as final flush or client acknowledgment | Keep separate from the sum of component durations |
The Direct Recommendation: Preserve a Trace-Level Event Timeline
A single aggregate latency value is not sufficient audit evidence. It does not show whether a request waited for capacity, spent time in gateway logic, waited for an upstream response, streamed slowly, retried, changed route, or ended before completion.
Instead, retain a trace-level timeline for every request attempt. Each timeline should contain:
- Explicit stage and event names
- Start and end timestamps where an interval is represented directly
- Raw point-in-time boundary events
- Calculated duration and unit
- Trace ID and request ID
- Attempt number
- Parent-child or other causal relationships
- Service or component that observed the event
- Timing source and clock information
- Route, model, region, or deployment identifiers where appropriate
- Outcome, status, and error or cancellation reason
- Schema version
This structure lets an auditor determine what was directly observed, what was calculated, and which system supplied each observation. It also makes it possible to detect missing events and avoid presenting assumptions as measured facts.
Record raw boundary events alongside calculated durations
Store the original event timestamps even when the operational dashboard uses precomputed duration fields. For example, retain provider_dispatch, first_response_byte, and stream_complete events in addition to fields such as provider_pre_stream_ms and streaming_ms.
Raw events provide three benefits:
- Reproducibility: An auditor can recalculate the duration using the documented formula.
- Schema migration: A revised definition can be applied to historical events without depending entirely on an old derived field.
- Anomaly detection: Negative durations, impossible ordering, missing boundaries, and inconsistent units become easier to identify.
Every derived field should specify its boundary convention. A field named time_to_first_token_ms, for example, should be derived from a token event—not silently calculated from the first network byte.
Link stages with trace IDs, request IDs, attempts, and parent-child relationships
A trace ID connects the end-to-end operation, while a request ID identifies the logical request according to the platform’s documented convention. These identifiers may be the same in a simple architecture, but they should not be assumed to be interchangeable.
Attempt numbers are essential when retries or fallbacks occur. A logical request might generate one failed provider attempt followed by a successful attempt through another route. If those attempts are merged into one provider interval, the resulting latency record hides the retry and makes route-level analysis unreliable.
Parent-child relationships should show how work was initiated. A gateway request might contain a queue span, which leads to an upstream attempt containing pre-stream and streaming intervals. If some stages run concurrently, the record should preserve those relationships rather than forcing the events into a misleading additive sequence.
Define Each Latency Interval by Observable Start and End Events
Latency names are useful only when their boundaries are documented. Define the event that starts each interval, the event that ends it, the system that observed both events, and the clock used for the calculation.
Queue time: admission wait or enqueue to dequeue or execution admission
Queue time should run from the point at which work begins waiting for admission to the point at which it is admitted for execution or dispatched to the next execution stage.
Common boundaries include:
- Start:
admission_wait_startedorrequest_enqueued - End:
execution_admitted,request_dequeued, or another precisely defined admission event
Do not use the request’s gateway ingress time as the queue start unless ingress itself means immediate queue admission in the documented architecture. Gateway validation, authentication, routing, or cache lookup may occur before queue entry.
Batching also requires explicit treatment. If a request waits while a batch is assembled and then waits again for execution capacity, record those as separate intervals when possible. Otherwise, document that the queue interval includes batch formation. This distinction matters when evaluating interactive chat separately from batch enrichment or other delay-tolerant workloads.
Gateway time: isolate ingress, dispatch, and completion overhead
The full gateway span usually contains queue time and upstream-provider time. It should not be labeled entirely as “gateway processing.” Instead, record enough boundaries to isolate gateway-controlled segments:
- Gateway ingress
- Validation or policy-processing completion, where relevant
- Queue entry and queue exit
- Cache lookup and outcome, where relevant
- Route selection
- Upstream dispatch
- First upstream response event
- Upstream completion
- Final response flush
Gateway overhead may consist of multiple non-contiguous segments, such as ingress-to-dispatch processing plus upstream-completion-to-final-flush processing. Report those segments separately or calculate a documented gateway-exclusive duration. Do not subtract unrelated spans unless their clocks and nesting relationships make the calculation valid.
A complete gateway span remains useful as an envelope, but it should be clearly distinguished from gateway-exclusive work.
Provider pre-stream time: dispatch to first byte or first token
Provider pre-stream time should begin when the gateway dispatches the upstream request and end when the gateway receives the selected first-response event.
Two common conventions are:
- Time to first byte: Provider dispatch to receipt of the first response byte
- Time to first token: Provider dispatch to receipt of the first output token
These measurements answer different questions. A response byte may contain headers or framing rather than generated output. The evidence record must therefore identify whether the ending boundary is first_response_byte or first_output_token.
Unless provider-native telemetry is available and correlated to the same attempt, describe this interval as gateway-observed provider pre-stream time. It includes network transit and any intermediary behavior visible between dispatch and receipt; it does not, by itself, prove the provider’s internal compute duration.
Record Streaming and End-to-End Time Separately
Streaming time begins with the declared first-response boundary and ends with the corresponding final-response boundary. A token-based interval should use first token to final token. A byte-based interval should use first byte to final byte.
For interrupted responses, record the stream as partial rather than treating the last observed token as successful completion. Capture the termination status, such as client cancellation, provider error, timeout, connection loss, or gateway cancellation.
End-to-end time should be a separate measurement with explicitly declared boundaries. For example:
- Start: Gateway ingress
- End: Final response flush from the gateway
Another implementation may use client submission and client acknowledgment. That is valid if both timestamps are available from suitably correlated clocks and the convention is documented. It is a different measurement from gateway-ingress-to-final-flush and should have a different field name.
Do not add overlapping provider and streaming intervals
A “provider total” span may run from upstream dispatch through final response completion. If streaming time is nested inside that span, adding provider total and streaming time double-counts the stream.
Use one of these approaches instead:
- Report provider pre-stream and streaming as separate, non-overlapping intervals.
- Preserve a provider-total parent span with pre-stream and streaming child intervals.
- Report end-to-end time independently, without claiming it equals the sum of every recorded span.
Durations should only be added when their boundaries are non-overlapping, their units match, and their timestamps or local duration measurements are clock-compatible.
Minimum Evidence Schema
A practical latency record should include enough context to reconstruct the request without storing sensitive payload content.
| Field | Purpose |
|---|---|
trace_id, request_id | Connect the logical request and its distributed trace |
attempt | Separate retries, fallbacks, and repeated dispatches |
parent_span_id, span_id | Preserve nesting and causal relationships |
stage, event_name | State exactly what occurred and where |
start_timestamp_utc, end_timestamp_utc | Support correlation using declared UTC timestamps |
duration, duration_unit | Provide the locally calculated interval |
clock_source, sync_status | Explain how timing was measured and whether clocks were synchronized |
service, component | Identify the observer that emitted the record |
region, deployment_id | Distinguish relevant execution environments |
model_id, route_id | Associate latency with the selected model or serving route |
provider_request_id | Correlate provider evidence when available |
status, error_reason, cancellation_reason | Explain normal and abnormal termination |
schema_version | Preserve the meaning of fields as definitions evolve |
Identifiers should be synthetic, pseudonymous, or otherwise handled according to the organization’s data policies. Prompts, generated content, credentials, and sensitive payloads are generally unnecessary for stage-level latency reconstruction.
Vendor-Neutral Sample Timeline and JSON Record
The following is a recommended illustration, not a representation of Token Forge Cloud product output.
| Sequence | Event | Observer | Interpretation |
|---|---|---|---|
| 1 | gateway_ingress | Gateway | End-to-end start boundary |
| 2 | admission_wait_started | Scheduler | Queue start |
| 3 | execution_admitted | Scheduler | Queue end |
| 4 | provider_dispatch | Gateway | Provider pre-stream start |
| 5 | first_response_byte | Gateway | Byte-based pre-stream end |
| 6 | first_output_token | Stream processor | Token-based pre-stream end |
| 7 | final_output_token | Stream processor | Token-based streaming end |
| 8 | final_response_flush | Gateway | Gateway-observed end-to-end completion |
``json { "schema_version": "latency-evidence.v1", "trace_id": "tr_synthetic_8f31", "request_id": "req_synthetic_2047", "attempt": 2, "parent_span_id": "span_gateway_01", "span_id": "span_upstream_02", "route_id": "route_example_b", "model_id": "model_alias_example", "provider_request_id": "provider_ref_when_available", "deployment_id": "deployment_example", "region": "region_example", "events": [ { "stage": "upstream", "event_name": "provider_dispatch", "timestamp_utc": "2026-09-08T10:15:30.120Z", "monotonic_ns": 842001000000, "service": "gateway" }, { "stage": "upstream", "event_name": "first_output_token", "timestamp_utc": "2026-09-08T10:15:30.410Z", "monotonic_ns": 842291000000, "service": "stream_processor" }, { "stage": "stream", "event_name": "final_output_token", "timestamp_utc": "2026-09-08T10:15:31.260Z", "monotonic_ns": 843141000000, "service": "stream_processor" } ], "derived_intervals": [ { "name": "gateway_observed_time_to_first_token", "start_event": "provider_dispatch", "end_event": "first_output_token", "duration": 290, "duration_unit": "ms" }, { "name": "token_streaming_time", "start_event": "first_output_token", "end_event": "final_output_token", "duration": 850, "duration_unit": "ms" } ], "clock_source": "local_monotonic_with_utc_correlation", "sync_status": "documented_by_deployment", "status": "completed" } ``
In a real implementation, a monotonic value emitted by one process should not be subtracted from a monotonic value emitted by another process. Each duration should identify the observer or clock domain responsible for both boundaries.
Handle Retries, Fallbacks, Batching, Caching, and Incomplete Requests
Operational behavior can distort latency evidence unless it is represented explicitly.
- Retries: Create a separate attempt record for every upstream dispatch. Link all attempts to the logical request and retain failed-attempt durations.
- Fallbacks and routing changes: Record the route selected for each attempt and the event that triggered a fallback. Do not overwrite the original route.
- Batching: Separate batch-formation wait from capacity or execution wait when the architecture exposes both boundaries.
- Cache outcomes: Record whether the request was a hit, miss, bypass, stale result, or error. A cache-served response should not be represented as a zero-duration provider call.
- Cancellations: Capture who initiated cancellation and the last completed boundary event.
- Timeouts: Identify which timeout fired and whether the upstream operation may have continued after the gateway stopped waiting.
- Partial streams: Retain the first-response event, last observed event, emitted unit count where appropriate, and incomplete status.
These distinctions prevent successful requests from being compared directly with retried, cached, timed-out, or partially streamed requests without suitable segmentation.
Protect Clock Integrity and Timing Provenance
Use a monotonic clock for local duration measurement where possible because wall-clock adjustments can create incorrect intervals. Retain UTC timestamps as well so events can be correlated across services and operational records.
For each duration, document:
- The clock source
- The service or process that observed both boundaries
- The measurement unit and precision
- Synchronization status for cross-system timestamps
- Any known clock uncertainty or collection delay
Avoid subtracting timestamps from unsynchronized systems without qualification. If provider-native timestamps are available, retain them in their original clock domain and correlate them through request identifiers or documented synchronization methods. Do not silently combine them with gateway timestamps as though they came from one clock.
Auditor Checklist for Reproducible Latency Evidence
An auditor or technical reviewer should be able to answer the following questions:
- Are queue, gateway, provider pre-stream, streaming, and end-to-end boundaries explicitly defined?
- Does each derived duration point to reproducible raw events?
- Is trace continuity preserved across gateway, scheduler, router, and upstream attempts?
- Are missing, duplicated, or out-of-order events detectable?
- Are nested and overlapping intervals represented without double-counting?
- Is each provider interval labeled as gateway-observed or provider-native?
- Does the schema distinguish first byte from first token?
- Are clock source, clock domain, synchronization status, units, and precision recorded?
- Are retries and fallback routes assigned to distinct attempts?
- Are batching delay, cache outcome, cancellation, timeout, failure, and partial-stream states visible?
- Is the end-to-end completion boundary final token, final byte, gateway flush, or client acknowledgment?
- Is the schema version retained with the record?
- Is retention context documented so the reviewer knows which records remain available for reconstruction?
- Are identifiers and metadata handled without unnecessarily retaining prompts, generated content, credentials, or sensitive payloads?
- Can an independent reviewer reproduce the reported durations from retained events?
A trace can establish timing sequence and correlation, but it does not automatically prove why a delay occurred. Causal conclusions may require scheduler metrics, capacity data, network observations, provider-native telemetry, or controlled testing.
Using Latency Evidence to Evaluate an Inference Control Plane
Stage-level evidence is particularly useful when evaluating a private inference control plane because serving decisions can affect different parts of the timeline. Caching may change whether an upstream attempt occurs. Routing can create distinct provider attempts. Batching may introduce formation wait while improving infrastructure utilization. GPU scheduling can affect admission timing, while quantization choices may need to be evaluated alongside workload quality and operating objectives.
Token Forge Cloud Private LLM Inference focuses on private deployment and serving-layer optimization for enterprise AI workloads, including caching, routing, batching, quantization, and GPU scheduling. Token Forge Cloud treats latency-sensitive chat, batch enrichment, and agentic workflows as different serving-policy problems.
During evaluation, ask for a demonstration of how the proposed deployment represents the boundaries relevant to your architecture. Review event definitions, attempt handling, clock provenance, export format, retention context, and the treatment of incomplete requests. Do not assume that generic tracing terminology proves the availability of a particular field, integration, or retention mechanism.
Teams beginning with API-first model access can apply the same evidence model to Token Forge Cloud Managed Model APIs: preserve gateway-observed boundaries, identify each upstream attempt, and avoid treating externally observed time as provider-internal compute time.
Next Step
A reproducible latency timeline helps technical, operations, product, and finance teams evaluate serving behavior using the same definitions. Establish the timing schema before comparing routes or deployment models so that changes in queueing, upstream response, and streaming behavior remain interpretable.
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.