An AI platform should preserve a versioned decision record containing the request context, eligible candidates, active routing policy, selected provider and model, reason codes, operational and price inputs, fallback history, and any override. Ordinary request logs are not enough because they rarely show which alternatives existed or why the router chose one.
The short answer: preserve a versioned selection decision record, not only request and response logs
A useful audit record must capture the routing decision as it was made. It should connect the original request to the options available at that moment, the policy and configuration applied, the resulting selection, and the events that changed the selection through retries, fallbacks, or overrides.
A conventional API or access log may show a timestamp, endpoint, response status, and latency. Those fields can establish that a call occurred, but they usually cannot answer questions such as:
- Which providers, models, versions, and deployment endpoints were eligible?
- Which candidates were excluded before ranking, and for what reason?
- Which routing-policy and configuration versions were active?
- Which cost, latency, availability, capacity, or data-handling constraints influenced the decision?
- Did the platform choose the initial candidate, use a fallback, or apply an override?
- What operational information was available when the choice was made?
For auditability, the unit of evidence should therefore be a request-level selection decision record, linked to—but distinct from—general infrastructure logs, provider billing records, and prompt or response content.
What an auditor must be able to reconstruct
A well-designed record should let an authorized reviewer reconstruct the decision sequence:
- A particular workload submitted a request at a recorded time.
- The platform interpreted that request as requiring specific capabilities and constraints.
- A defined set of provider, model, endpoint, and deployment options was eligible.
- A particular routing-policy and configuration version evaluated those options.
- Candidates passed, failed, or ranked differently for recorded reasons.
- The platform selected a provider, model version, endpoint, and location.
- Any execution failure, retry, fallback, exception, or override changed the path.
- The request ultimately reached the recorded deployment—or failed without a final successful selection.
This does not require retaining every internal computation. Stable identifiers, version references, bounded input values, standardized reason codes, and selected operational snapshots are usually more useful than an unstructured diagnostic dump.
Why routing evidence is different from explaining a model-generated answer
Provider-and-model selection evidence explains why the serving layer routed a request to a particular option. It does not fully explain why the selected model produced a particular answer.
Output analysis may require separate evidence, such as the effective prompt, retrieval context, tool results, generation parameters, model version, content-filter events, or evaluation results. Even that information may not make a nondeterministic output exactly reproducible.
Hidden chain-of-thought is not necessary for reconstructing a routing decision. An audit design should focus on observable policy inputs, candidate evaluations, reason codes, and outcomes rather than attempting to retain private model reasoning.
Record the request context and every option eligible at decision time
Routing can only be understood relative to the request the system received and the choices available at that moment. The platform should preserve enough context to identify the request without automatically retaining all sensitive payload content.
Request identity, timing, workload, and correlation fields
Start with fields that connect the decision record to surrounding systems and events:
- A unique request, event, or trace ID
- Decision and execution timestamps, including an appropriate time-zone reference
- Tenant, application, environment, or workload identifiers
- Correlation IDs for upstream requests, agents, tools, or downstream provider calls
- The requesting service or actor identity, where appropriate
- The audit-event schema version
Identifiers should be stable across the routing, provider-call, retry, billing, and observability layers. If each system generates unrelated IDs, reconstruction becomes a manual matching exercise and may remain ambiguous.
For multi-step agentic workflows, one parent trace may contain several model-selection events. Each call should have its own decision ID while retaining a link to the parent workflow and, where relevant, the preceding step.
Capability, modality, context, cost, latency, data-handling, residency, and availability constraints
The decision record should capture the requirements supplied to the router or derived by an authorized policy. Depending on the workload, these may include:
- Requested capability and task type
- Input and output modality
- Required context-length category or estimated token demand
- Latency objective, timeout budget, or batch-processing window
- Cost ceiling or pricing preference
- Tool-use, structured-output, or other functional needs
- Data-handling restrictions
- Permitted or excluded deployment locations
- Required availability characteristics
- Whether private or externally managed execution is allowed
It is important to distinguish a request’s original requirements from values derived later by the routing system. If a classifier labels a task as latency-sensitive or batch-suitable, preserve the classifier or rule version and its bounded output—not merely the final label.
Workload context also matters because latency-sensitive chat, batch enrichment, and agentic workflows present different serving-policy problems. A route that is reasonable for asynchronous enrichment may not be reasonable for an interactive assistant, even when both can use the same underlying model.
Candidate providers, models, versions, endpoints, locations, and exclusions
Recording only the winner creates selection bias in the audit trail. The event should identify the complete eligible choice set at decision time, including, where applicable:
- Provider and model identifiers
- Model or deployment versions
- Managed endpoint or private deployment identifiers
- Deployment location or region
- Candidate order or fallback position
- Eligibility status
- Exclusion and rejection reason codes
The record should distinguish between an option that was never configured, one excluded by policy, one temporarily unavailable, and one evaluated but ranked below the winner. These cases lead to different conclusions about how the routing system behaved.
A candidate-set snapshot or content-addressed reference can reduce record size. However, the referenced configuration must remain retrievable for the relevant lifecycle period; retaining only a hash without the underlying version does not allow meaningful reconstruction.
Preserve routing-policy provenance and contemporaneous evaluation inputs
The candidate set shows what could have been selected. Policy provenance shows how the platform evaluated those choices.
At minimum, the decision event should reference the active policy name and version, rule or workflow version, configuration snapshot, and deployment or change identifier. If a policy depends on another service—such as a capability registry, pricing catalog, health monitor, or capacity scheduler—the record should also identify the version or timestamp of the inputs used.
Reason codes should be specific enough to support analysis without exposing sensitive implementation details. Useful categories might indicate that a candidate:
- Did not meet a capability or modality requirement
- Violated a data-handling or location constraint
- Exceeded a configured price or latency threshold
- Was unhealthy, unavailable, or capacity-constrained
- Fell below a routing score or confidence threshold
- Was reserved for a later fallback position
If the router calculates scores, preserve the score definition or version, the candidate-level scores, and the threshold applied. A score such as 0.78 has little audit value unless reviewers can determine what it represented and which scoring logic produced it.
Price evidence should reflect the input used by the decision—not merely a later invoice or current public price. The same principle applies to health, capacity, and availability signals. Store the observed value or a durable reference to the time-bounded snapshot that influenced the route.
Capture the selection result, execution path, and human context
The selection record should explicitly state the chosen provider, model, model version, endpoint or deployment, deployment location, and fallback position. It should also separate selection from successful execution: choosing an endpoint does not prove that the request completed there.
Fallbacks, retries, and circuit-breaker events
Preserve the ordered execution history when the initial route changes. For each attempt, record the candidate, attempt number, start and end time, outcome, bounded error category, and the policy action that followed.
Relevant events may include:
- Connection or provider errors
- Timeouts
- Rate or capacity limits
- Health-check failures
- Circuit-breaker state changes
- Policy-directed retries
- Movement to the next fallback candidate
- Final success or terminal failure
Avoid retaining credentials or unrestricted provider error bodies. Standardized error categories and carefully redacted diagnostic metadata are generally safer and easier to aggregate.
Approvals, overrides, and exceptions
Where a person or organizational process can alter routing, the record should show whether the normal policy was followed. Capture the approval, override, or exception identifier; the actor or service identity; the relevant role; the time; and a bounded reason code.
The objective is not to store informal commentary indefinitely. It is to establish that an authorized action changed the normal route and to link that change to its governing record. Automated overrides should receive the same treatment as human ones: identify the service, rule, or workflow that applied the exception.
Illustrative provider-and-model audit-event design
The following table is an illustrative design, not a universal schema. Teams should adapt it to their architecture, risk profile, contractual commitments, and applicable obligations.
| Evidence category | Example fields | Reconstruction purpose | Privacy consideration |
|---|---|---|---|
| Event identity | decision_id, request_id, trace_id, timestamp, schema_version | Connect the selection to related application and infrastructure events | Prefer opaque identifiers over user-identifying values |
| Workload context | tenant_id, application_id, environment, task_type, modality | Establish which workload made the request and how it was classified | Use pseudonymous or bounded labels where possible |
| Decision constraints | capability_requirements, latency_class, cost_limit, data_policy, location_policy | Show the requirements against which candidates were evaluated | Do not copy sensitive prompt text into constraint fields |
| Candidate snapshot | candidate_set_ref, provider, model, version, endpoint, location, eligibility | Recreate the options available at decision time | Avoid embedding endpoint credentials or proprietary payloads |
| Policy provenance | policy_id, policy_version, config_ref, deployment_change_id | Identify the exact routing logic and configuration in force | Restrict access to operational configuration details |
| Evaluation evidence | reason codes, rules matched, thresholds, scores, health, capacity, price input | Explain candidate exclusion, ranking, and selection | Store bounded values or references rather than unrestricted diagnostic dumps |
| Selection result | provider, model, model version, endpoint, location, fallback position | Identify the option selected by the router | Apply the same protection used for infrastructure metadata |
| Attempt history | attempt order, candidate, timestamps, outcome, error category, retry action | Reconstruct retries, failures, and fallback behavior | Redact provider responses that may contain sensitive content |
| Override context | override ID, actor or service ID, role, reason code, approval reference | Show when and why normal policy was changed | Minimize personal data and limit access to authorized reviewers |
| Lifecycle metadata | retention class, redaction status, access classification, integrity reference, export version | Support controlled preservation and later interpretation | Match retention to purpose rather than retaining all data indefinitely |
A corresponding event can be stored as one document or as linked records. The architectural choice matters less than preserving stable relationships, versions, and ordering across the decision lifecycle.
Balance reconstructability with privacy and security
Auditability does not require indiscriminate storage of raw prompts and responses. Those payloads may contain personal information, credentials, confidential business data, source code, or proprietary context. Retaining everything can create a larger security and governance burden without improving routing reconstruction.
A privacy-conscious design can favor:
- Opaque identifiers and correlation IDs
- Prompt or configuration hashes where they serve a defined comparison purpose
- Bounded task, modality, and sensitivity classifications
- Reason codes instead of unrestricted free text
- References to separately controlled records
- Redacted provider errors
- Explicit indicators showing whether sensitive fields were removed
Hashes must be used carefully. Predictable or low-entropy content may still be vulnerable to guessing, and a hash cannot replace the underlying record when an auditor needs to inspect its meaning.
Retention periods should be selected according to the organization’s operational, contractual, security, and legal needs rather than treated as universal. Access controls, encryption, redaction, exportability, schema migration, and tamper-evident mechanisms are important design considerations. Teams should determine how those controls apply to both the decision event and every referenced configuration snapshot.
Plan for reconstruction even when exact replay is impossible
A complete audit trail does not guarantee that a historical request can be reproduced exactly. External provider availability, deployment state, capacity, pricing, model updates, and endpoint behavior can change. Generation may also be nondeterministic even when the same apparent inputs are reused.
The practical goal is to preserve what the routing system knew and did at the time. That means retaining contemporaneous versions and observations rather than querying current systems during an audit and assuming they reflect past conditions.
For example, a current pricing catalog cannot establish the price input used several months earlier. A model name without a version cannot show whether the provider changed the underlying deployment. A current health status says nothing about whether an endpoint was unavailable during the original decision.
Auditors should therefore be able to distinguish among:
- Decision reconstruction: determining why the router chose a provider and model
- Execution reconstruction: determining which attempts occurred and where the request completed
- Replay: attempting to run equivalent inputs again
- Output explanation: investigating why the model produced a particular result
These are related but separate objectives. A platform can support strong decision reconstruction even when exact output replay is not possible.
Buyer questions for evaluating an inference control plane
When assessing routing or inference infrastructure, ask for a demonstration using one request that encounters both a rejected candidate and a fallback. This is more revealing than reviewing a generic access-log screenshot.
Useful questions include:
- Can the platform produce a request-level selection record without requiring unrestricted prompt retention?
- Does the record identify all eligible and excluded candidates, including versions, endpoints, and deployment locations?
- Can reviewers determine which routing-policy, rule, configuration, and deployment versions were active?
- Are candidate-level reason codes, scores, thresholds, and relevant operational inputs available?
- Can the record distinguish policy exclusion from health, capacity, timeout, and provider errors?
- Are retries and fallbacks ordered and linked to the final execution result?
- How are human and automated overrides identified?
- Can decision records be exported in a documented, versioned format?
- How are redaction, access, retention, encryption, and integrity handled?
- What remains reconstructable if a provider changes a model, endpoint, price, or service state?
Buyers should also test whether the evidence remains understandable outside the operations team. A record that depends on undocumented internal abbreviations or a dashboard’s current state may be difficult for security, risk, finance, or procurement stakeholders to interpret later.
Evaluating auditability with Token Forge Cloud
Token Forge Cloud Private LLM Inference focuses on private deployment and serving-layer control for enterprise AI workloads. Token Forge Cloud supports private deployment paths where models, prompts, and telemetry remain in the customer’s controlled environment, while capabilities such as model routing, semantic caching, batching, quantization, and GPU scheduling address how inference workloads are served.
For an auditability-focused deployment, routing and serving controls should be evaluated together with the decision-record requirements above. Teams should confirm which request identifiers, policy versions, candidate details, reason codes, operational signals, fallback events, and lifecycle controls are available for their proposed architecture. The required design will vary depending on whether workloads use private routing, external providers, or a combination of deployment paths.
Token Forge Cloud Managed Model APIs offers an API-first option for teams validating model demand before committing to private serving capacity. When moving from managed access toward private inference, preserve consistent correlation and decision fields so routing evidence remains comparable as the serving architecture evolves.
Audit telemetry is most useful when it is connected to versioned policy and execution context—not treated as a substitute for them. Cost optimization should likewise remain subordinate to workload capability, data-handling, availability, and governance constraints recorded for each decision.
Next step
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.