When a client cancels after a request’s latency deadline has already passed, the platform should treat deadline expiry and cancellation as separate events. A practical policy is to stop avoidable work, attempt best-effort termination of running work, suppress delivery after cancellation, record any consumption that could not be stopped, and determine charges under a clearly disclosed billing contract. Work performed solely because the platform failed to enforce a supported deadline should generally not be charged to the client, but the applicable agreement must define the actual treatment.
The recommended policy in brief
Request handling should separate four decisions that are often incorrectly combined:
- Admission: Should the request enter the queue or begin execution if its deadline cannot be met?
- Execution: Can queued or running work still be stopped?
- Delivery: Should a result be returned after expiry or acknowledged cancellation?
- Billing: Which accepted work, generated tokens, or consumed compute is chargeable under the contract?
A deadline is not the same as a cancellation signal. Likewise, an HTTP disconnect, client-side timeout, explicit cancellation, latency objective, and contractual SLA event are not interchangeable. Each needs documented semantics.
Treat deadline expiry and client cancellation as separate events
The declared deadline states when the result will cease to be useful under the request contract. It should inform admission and scheduling before the deadline passes, not merely trigger reporting afterward.
If work has not started and the system determines that the deadline is no longer achievable, the recommended behavior is to reject or drop that work—provided the API contract supports this policy. Continuing to queue predictably stale work consumes capacity that could serve requests with achievable deadlines.
A later cancellation communicates that the client no longer wants the operation to continue. It may arrive because the client timed out, switched to a fallback, ended a user session, or decided that the result no longer has value. The cancellation should be handled idempotently, but it should not retroactively change the event history. The request may already have consumed resources before cancellation reached the server.
Deadline expiry alone also does not establish an SLA breach or financial liability. A request-level deadline, a service objective, and an SLA with defined remedies can use different measurements and exclusions. The commercial agreement should explain their relationship.
Stop avoidable work, suppress stale delivery, and meter unavoidable consumption
For work that remains queued, cancellation or confirmed deadline expiry should normally remove it from consideration before execution starts. For work already running, the platform should attempt termination at a safe interruption point.
Immediate interruption is not always possible. A request may already be:
- incorporated into a shared inference batch;
- resident on a GPU and partway through generation;
- streaming output through an active processing pipeline;
- forwarded to an upstream model provider;
- executing a tool or agent step with separate cancellation behavior.
In those cases, the cancellation acknowledgment and execution stop may occur at different times. The platform should avoid returning a stale result after cancellation is acknowledged, while still recording tokens or compute consumed before the work could be terminated. Suppressing the response does not erase backend consumption.
The following decision table is a recommended model rather than a universal billing rule:
| Request state when expiry or cancellation is processed | Response handling | Execution action | Metering | Billing-policy consideration |
|---|---|---|---|---|
| Queued and not started | Return the documented expiry or cancellation outcome | Remove from the queue when possible | Record queue activity; no generation usage | Avoid charging for execution that never began |
| Running | Suppress delivery after acknowledged cancellation | Attempt best-effort termination at a safe point | Record actual tokens or compute consumed | Contract should state whether partial consumption is billable and who caused the missed deadline |
| Streaming or batched | Stop further client delivery when the contract requires it | Propagate cancellation where technically possible without corrupting shared work | Separate produced, delivered, and discarded output where measurable | Do not assume undelivered output is automatically free or chargeable; disclose the rule |
| Completed before cancellation was received | Resolve using authoritative completion and cancellation timestamps | No work remains to terminate | Record completed usage and delivery status | Contract should define whether completion, delivery, or client receipt controls the charge |
| Expired but not yet cancelled | Do not deliver a stale result unless the API explicitly permits late results | Drop unstarted work or evaluate whether running work should stop | Record work performed after expiry | Consider not charging for work caused solely by failure to enforce a supported deadline |
Billing should therefore distinguish among request acceptance, queueing, execution start, generated tokens, consumed compute, completed output, delivered output, and responsibility for a deadline miss. Charging only for completed responses is simple but may ignore substantial unavoidable compute. Charging every accepted request is also simple but can make clients pay for work the platform should have rejected. A state-based policy gives engineering, finance, and customers a clearer basis for reconciliation.
Establish the authoritative order of deadline, detection, and cancellation events
Race conditions are unavoidable in distributed inference systems. The solution is not to infer event order from what the client happened to observe. The platform should define an authoritative clock, server-side timestamps, state transitions, and tie-breaking rules.
Consider this example:
- 12:00:00.000: The server accepts a request with a declared deadline of 12:00:02.000.
- 12:00:01.700: Execution begins.
- 12:00:02.000: The declared deadline passes.
- 12:00:02.040: The server detects that the deadline has passed.
- 12:00:02.100: The client sends an explicit cancellation.
- 12:00:02.130: The server receives and acknowledges that cancellation.
- 12:00:02.180: Execution reaches a safe interruption point and stops.
This timeline contains three primary events: the declared deadline, server detection of expiry, and receipt of the later cancellation. The 180 milliseconds of post-deadline execution cannot be analyzed as a single undifferentiated “cancelled request.” Some work occurred before detection, some after detection, and some after cancellation receipt while termination was in progress.
The declared latency deadline
The contract should define what the deadline means and which clock controls it. Common questions include:
- Is the deadline an absolute server timestamp or a relative duration measured from acceptance?
- Does it apply to the first streamed token, the final token, or completion of the full operation?
- Does queue time count?
- Can the platform reject a request at admission when meeting the deadline is already improbable?
- Are late responses suppressed, returned with a late status, or allowed for designated workloads?
These choices affect scheduling. Interactive chat may prioritize time to first token, while batch enrichment may care more about completion within a broader processing window. Agentic workflows may require separate deadlines for model calls, tools, and the overall task. Token Forge Cloud treats latency-sensitive chat, batch enrichment, and agentic workflows as different serving-policy problems, reflecting the need to match policy to workload rather than apply one timeout rule everywhere.
Server-side detection that the deadline has passed
The deadline timestamp and the moment the platform detects expiry are not necessarily identical. Queue polling intervals, scheduler behavior, in-flight kernels, batch boundaries, network propagation, and upstream APIs can introduce a delay.
The request contract should state what happens during that interval. A strong design uses deadline-aware admission and scheduling to prevent avoidable work before expiry, then applies an explicit transition when expiry is detected. That transition might mark the request as expired, remove queued work, signal running components to stop, and suppress subsequent delivery. Actual implementation details will vary by serving architecture.
For billing purposes, it is useful to distinguish work that was unavoidable before detection from work that continued because a supported deadline was not enforced. As a policy recommendation, clients should generally not be charged for work performed solely because the platform failed to apply its own supported deadline semantics. This is not a universal refund or credit rule; pricing terms should specify the disposition and any dispute process.
Receipt and acknowledgment of the later cancellation
Cancellation should be idempotent. If a client sends the same cancellation repeatedly because an acknowledgment was lost, the request should not move through conflicting states or create multiple billing adjustments.
The server also needs a documented answer for completion and cancellation arriving nearly simultaneously. For example:
- If completion is committed before cancellation receipt, the request may remain completed even if the client never sees the response.
- If cancellation is recorded first, later delivery should normally be suppressed even if execution finishes during propagation.
- If the client receives an ambiguous timeout, a retry should use an idempotency mechanism so it does not unintentionally create duplicate work or charges.
A network disconnect should not automatically be interpreted as explicit cancellation unless the API contract says so. The client may reconnect, the transport may fail while the operation remains useful, or backend execution may already have been delegated. Explicit cancellation gives the platform a clearer lifecycle signal, but it still cannot guarantee immediate reversal of every queued, batched, GPU-resident, streamed, or upstream operation.
Build an auditable request and billing record
Operational telemetry should make the final decision explainable to engineering, FinOps, support, and the client. A useful request record can include:
- request acceptance and queue-entry timestamps;
- the declared deadline and its interpretation;
- execution start and relevant scheduling transitions;
- deadline-expiry detection;
- cancellation receipt and acknowledgment;
- termination request and actual termination status;
- generated, delivered, and discarded tokens where measurable;
- compute or upstream usage that could not be stopped;
- completion status and response-delivery status;
- final billing disposition and the reason for it.
Not every platform will expose every field directly to customers. Internally, however, the billing result should be traceable to authoritative lifecycle events rather than inferred only from a client error code.
This separation is especially important for streamed requests. A client may receive some tokens before cancelling, while additional tokens are generated during propagation. Product and finance teams must decide whether charges are based on generated output, delivered output, consumed compute, another unit, or a combination. The answer should be visible in the pricing and API terms before production use.
Questions to address before deployment
Before deploying managed model API access, self-deployed serving, or a private inference control plane, confirm how cancellation and deadline policy behaves across the workload’s actual execution path:
- Does the system use deadlines during admission and scheduling, or only report that a completed request was late?
- What happens to requests that remain queued after their deadline becomes unachievable?
- Where are safe interruption points for generation, streaming, batching, and agent steps?
- Does cancellation propagate to upstream providers, and how is uninterruptible upstream consumption treated?
- Can one request be removed from a shared batch without disrupting other requests?
- Which event controls billing: acceptance, execution, generated tokens, delivered tokens, completion, or consumed compute?
- How are provider-caused deadline misses, credits, refunds, and billing disputes handled under the contract?
- What records are available to explain a charge for a cancelled or undelivered response?
- How are repeated cancellations and ambiguous retries deduplicated?
- Are disconnects and client-side timeouts treated differently from explicit cancellation?
These questions should be tested against interactive, batch, streaming, and agentic scenarios. A policy that works for single-request generation may behave differently when routing, shared batching, GPU scheduling, or upstream dependencies are involved.
Serving-layer control and Token Forge Cloud
Token Forge Cloud Private LLM Inference focuses on private deployment and serving-layer optimization using caching, routing, batching, quantization, and GPU scheduling. Those controls are relevant to how enterprises reason about admission, capacity use, request lifecycle policy, and inference economics.
Cancellation propagation, deadline enforcement, metering, and billing still require deployment-specific definitions. For Token Forge Cloud Private LLM Inference, teams should confirm the applicable behavior for queued, running, batched, streaming, and upstream work, along with the commercial treatment of partial or undelivered execution.
For teams validating model demand before private deployment, Token Forge Cloud Managed Model APIs provides an API-first path for model access and usage validation. In either deployment model, cancellation semantics should be evaluated alongside routing and cost controls rather than treated as a transport-level afterthought.
Next step
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.