Traces, Logs & Replay

Applies to: ChengOS v0.1.0+ ｜ Last verified: 2026-08-12 ｜ Source: crates/cheng-core/src/domain/execution_archive.rs, crates/cheng-core/src/domain/archive_policy.rs, crates/cheng-nodes/src/nodes/builtin/tools/get_trace.rs, crates/cheng-api/src/rest/handlers/execution/mod.rs

There are three separate views of one execution, and the most important thing on this page is that they are never conflated:

Treating them as one thing is how systems end up either leaking payloads into a model's context or losing the audit record when a UI cache expires.

The UI trace projection

While an execution runs, the editor and CLI render it from the WebSocket event stream — EXECUTION_START, EXECUTION_PROGRESS, NODE_COMPLETE, NODE_FAILED, streaming and tool events, and the terminal EXECUTION_COMPLETE / EXECUTION_FAILED / EXECUTION_CANCELLED. See the WebSocket API.

This projection is transient. For reconnect recovery there is a bounded snapshot:

GET /executions/:id/trace-snapshot

The ownership check runs before the store lookup, deliberately, so the response cannot leak whether another user's snapshot exists. A client that reconnects mid-run fetches this and merges it rather than showing a gap — see Embedded Chat.

Execution logs

GET /executions/:id/logs?offset=&limit=

Paginated structured logs at DEBUG / INFO / WARN / ERROR. This is the "what happened, in order" view for a specific run.

The durable execution archive

An archive is the versioned, structured record of one completed agent turn: appendable while the loop runs, immutable once committed.

Three rules govern it:

1. The persisted assistant message stays the canonical public answer. An archive never substitutes for it.
2. ArchiveRef is opaque. It never encodes a path, tenant name, filename, or predictable turn number.
3. Native tool-call reconstruction reads structured steps only. Rendered trace text is never parsed back into protocol messages.

The archive is the authoritative detail source; the compact history summary is a projection of it, never the other way around.

The reference

ArchiveRef is a 32-character lowercase hex UUIDv4 with no hyphens. Parsing from untrusted input rejects anything else, which is what makes path traversal structurally impossible in the storage adapter rather than something the adapter has to defend against with string checks. Directory sharding uses a two-level prefix derived from the ID itself (ab/cd), so directories stay bounded without any caller-controlled data reaching the filesystem.

Steps

Each archived step carries a sequence number, the originating agent-loop iteration, the assistant action, the tool results, and a timestamp.

replay_blocked is the subtle field: when capture itself was ambiguous — a duplicate provider call ID, or a second result for one call — the step is still archived, because it is real history an auditor must see, but it is never replayed, because "which result belongs to which call" has no truthful answer. Replay validation checks the one-to-one pairing rules in full rather than sampling them.

Query modes

An archive is read at one of three widths:

An unrecognised mode string falls back to summary rather than silently widening exposure.

The caller chooses a mode, never a response size. The rendered projection is capped server-side at 24 000 characters, so a model cannot ask for an unbounded dump.

Policy: retention and redaction

The archive policy is stamped onto every archive, so an old object stays interpretable after the policy changes.

Retention classes:

ephemeral deliberately caps below the base: it exists to back the transient UI recovery window, not to be a second durable store. The class is durable metadata read by the cleanup job from the catalog, not re-derived from current configuration — so lowering the default TTL never retroactively shortens, nor raising it retroactively extends, an already-written archive's advertised lifetime.

Redaction modes:

Redaction happens at the producer boundary: the archive on disk is already redacted. A later policy change can therefore never turn previously written data into more-exposed data. The policy version is bumped when the meaning of a redaction decision changes, so old archives keep their original interpretation.

Per-workspace trace retention is configurable separately, with distinct day counts for completed and failed executions — failures usually deserve to outlive successes.

Reading a trace from inside a workflow

tools/get_trace lets a ReAct agent read a previous turn's record on demand. The model first sees a compact summary in turn metadata; when it needs detail it calls the tool.

[agent/react].tools ──▶ [agent/get_execution_trace]

Two independent lookup paths:

Old conversations have only trace_key; new ones write archive_ref; both may coexist during migration, and no legacy key is removed or reinterpreted. Supply one of the two.

Ask "which tools did you use" with mode: tool_calls — summary will not have the call detail, and detail costs context you probably do not need.

Replay and rerun

GET  /executions/:id/containers/:cid/iterations           list loop iteration traces
POST /executions/:id/containers/:cid/iterations/:idx/rerun rerun one iteration

Loop iterations are individually traceable and individually re-runnable, so debugging a 100-item batch does not mean re-running all 100. See Subflows & Batch Subflows.

Retrying a failed execution produces a new execution id rather than mutating the old record: the failed run stays intact as evidence.

The rest of the surface

GET  /executions                      list
GET  /executions/active               currently running
GET  /executions/history              historical
GET  /executions/:id                  one execution
GET  /executions/archives/:ref        read an archive
POST /executions/:id/cancel|pause|resume|approve

Next steps

• Execution Model — the states these records describe.
• Approvals & Human Review — the approve endpoint in context.
• Execution Failures — using all of this to diagnose a bad run.