A familiar prompt starts producing different answers. The application still sends the same model name, so the first suspicion is that the provider changed something. That is possible, but the output alone cannot establish what changed. Application defaults, conversation history, retrieved material, tool responses, routing configuration, and generation variability can all affect the result.
Useful incident records preserve what the application actually sent and what it could observe. They help an engineer reconstruct the conditions around a request. They do not promise to reproduce generated text word for word or reveal infrastructure the application cannot inspect.
Separate the layers you can observe
Keep application, routing, and provider information distinct. At the application layer, record the release, client library version, prompt template revision, and effective request settings. At the routing layer, record the configuration you control or information the service explicitly provides. At the provider layer, retain the submitted model identifier and any model information returned with the response.
For teams evaluating GPTZZZ (https://gptzzz.ai/), whose website describes an independent AI API relay and multi-model gateway, the same boundary applies: save application-side evidence first, then add service-provided metadata where available. A public service description does not establish the internal route taken by a particular request.
If a third-party service does not expose its routing configuration, mark it as unavailable. Do not infer a hidden route from writing style, a hostname, or a familiar-looking response field.
Distinguish requested names from reported identities
An application may maintain an alias such as primary-text. That is an illustrative application-defined name, not a model offered by a particular service. A configuration change can make the same alias select a different target while leaving the caller's input unchanged.
A request record can distinguish the following fields:
- requested_model: the exact model value sent in that attempt.
- reported_model: a model identifier supplied in the response, if present.
- alias_mapping_revision: the version of an alias mapping the application controls.
- config_revision: the effective application configuration for that attempt.
- client_revision: the application and client-library versions used.
These are suggested fields in your own records, not promised API response properties. Store the source of reported metadata alongside its value. A reported identifier is evidence of what the service reported; it is not independent verification of the underlying model. Preserve that distinction when writing an incident report.
Bind each attempt to its actual configuration
Capture the configuration reference when preparing the request. Reading the latest configuration after the response arrives can attach the wrong revision if an update happened while the request was running.
Retain immutable snapshots or versioned configuration references for settings you control. Record when each revision became effective, what changed, and which application environments used it. The reference is only useful while the corresponding configuration remains available.
Give a business operation one correlation ID, and give each attempt its own ID. A retry or fallback may use a different target or configuration. Keep those attempts separate so the final successful response does not erase the preceding failures. Link any service-provided request ID to the relevant attempt rather than treating it as a complete account of the operation.
Preserve enough context without copying secrets
Alongside configuration references, retain timestamps with time zones, the execution environment, completion status, and observed error category. A timeout should remain a timeout or an unknown completion state; it should not automatically become evidence that no processing occurred.
Record an approved input reference, prompt revision, effective parameters, and the versions of retrieved material or tool results when relevant. If reproducing the context requires protected content, store that material under appropriate access controls and retention rules. A reference or checksum can help identify a retained artifact, but cannot reconstruct content that has been deleted.
Do not copy credentials, cookies, session tokens, or complete authorization headers into ordinary logs. Endpoint URLs may also contain sensitive query parameters or private paths. Prefer a non-secret target label and a sanitized endpoint reference. Keep diagnostic access narrower than general application-log access when the retained material requires it.
Walk through a hypothetical alias change
Suppose an application maps primary-text to target A under configuration revision 12. Two days later, revision 13 maps it to target B. These are illustrative labels, not real provider model names or measured results.
An older result should lead back to revision 12 and the configuration that selected A. A later result should lead to revision 13 and B. That establishes a configuration difference. Without further request evidence, it does not independently establish how either service processed the requests internally.
Before attributing an output difference to that change, compare the input, prompt revision, conversation context, generation settings, and tool or retrieval results. Note other differences and missing evidence. If an approved replay is useful, label it as a new run under recorded conditions, not a reconstruction guaranteed to match the original response.
Test the records from both directions
Choose an old result and try to locate its attempts, configuration, and retained input references. Then choose a configuration revision and identify the requests associated with it. This checks whether the records support both investigating one failure and assessing the scope of a change.
Document anything the records cannot answer. Adjust collection or retention where that gap matters, then repeat the exercise on a later sample. A small record with reliable references is more useful during an incident than a large log whose fields cannot be connected to the configuration actually used.
