codex/app_server_tracing_design.md

App-server v2 tracing design

This document proposes a simple, staged tracing design for codex-rs/app-server with these goals:

• support distributed tracing from client-initiated app-server work into app-server and codex-core
• keep tracing consistent across the app-server v2 surface area
• minimize tracing boilerplate in request handlers
• avoid introducing tracing-owned lifecycle state that duplicates existing app-server runtime state

This design explicitly avoids a RequestKind taxonomy and avoids app-server-owned long-lived lifecycle span registries.

Summary

The design has four major pieces:

1. A transport-level W3C trace carrier on inbound JSON-RPC request envelopes.
2. A centralized app-server request tracing layer that wraps every inbound request in the same request span.
3. An internal trace-context handoff through codex_protocol::Submission so work that continues in codex-core inherits the inbound app-server request ancestry.
4. A core-owned long-lived turn span for turn-producing operations such as turn/start and review/start.

Every inbound JSON-RPC request gets a standardized request span.

When an app-server request submits work into core, the current span context is captured into Submission.trace. Core then creates a short-lived dispatch span parented from that carrier and, for turn-producing operations, creates a long-lived turn span beneath it before continuing into its existing task and model request tracing.

Important:

• request spans stay short-lived
• long-lived turn spans are owned by core, not app-server
• the design does not add app-server-owned long-lived thread or realtime spans

Design goals

• Distributed tracing first

  • Clients should be able to send trace context to app-server.
  • App-server should preserve that trace ancestry across the async handoff into core.
  • Existing core model request tracing should continue to inherit from the active core span once the handoff occurs.

• Consistent request instrumentation

  • Every inbound request should produce the same request span with the same base attributes.
  • Request tracing should be wired at the transport boundary, not repeated in individual handlers.

• Minimal boilerplate

  • Request handlers should not manually parse carriers or build request spans.
  • Existing calls to thread.submit(...) and similar APIs should pick up trace propagation automatically.

• Minimal business logic pollution

  • W3C parsing, OTEL conversion, and span-parenting rules should live in tracing-specific modules.
  • App-server business logic should stay focused on request handling, not span management.

• Incremental rollout

  • The first rollout should prove inbound request tracing and app-server -> core propagation.
  • Once propagation is in place, core should add a long-lived turn span so a single span covers the actual duration of a turn.
  • Thread and realtime lifecycle tracing should wait until there is a concrete need.

Non-goals

• This design does not attempt to make every loaded thread or realtime session correspond to a long-lived tracing span.
• This design does not add tracing-owned thread or realtime state stores in the initial design.
• This design does not require every app-server v2 *Params type to carry trace metadata.
• This design does not require outbound JSON-RPC trace propagation in the initial rollout.

Why not RequestKind

An earlier direction considered a central RequestKind taxonomy such as Unary, TurnLifecycle, or RealtimeLifecycle.

That is workable, but it makes tracing depend on a classification that can drift from runtime behavior. The simpler design instead treats tracing as two generic mechanics:

• every inbound request gets the same request span
• any async work that crosses from app-server into core gets the current span context attached to Submission

This keeps the initial implementation small and avoids turning tracing into a taxonomy maintenance problem.

Terminology

• Request span

  • A short-lived span for one inbound JSON-RPC request to app-server.

• W3C trace context

  • A serializable representation of distributed trace context based on traceparent and tracestate.

• Submission trace handoff

  • The optional serialized trace context attached to codex_protocol::Submission so core can restore parentage after the app-server request handler returns.

• Dispatch span

  • A short-lived core span created when the submission loop receives a Submission with trace context.

• Turn span

  • A long-lived core-owned span representing the actual runtime of a turn from turn start until completion, interruption, or failure.

High-level tracing model

1. Inbound request

For every inbound JSON-RPC request:

1. parse an optional W3C trace carrier from the JSON-RPC envelope
2. create a standardized request span
3. parent that span from the incoming carrier when present
4. process the request inside that span

This is true for every request, regardless of whether the API is unary or starts work that continues later.

2. Async handoff into core

Some app-server requests submit work that continues in core after the original request returns. The critical example is turn/start, but the mechanism should be generic.

To preserve trace ancestry:

• add an optional W3cTraceContext to codex_protocol::Submission
• have CodexThread::submit() capture the current span context into that field automatically
• have codex-core create a per-submission dispatch span parented from that carrier

This gives a clean causal chain:

• client span
• app-server request span
• core dispatch span
• core turn span for turn-producing operations
• existing core spans such as run_turn, sampling, and model request spans

3. Core-owned turn spans

For turn-producing operations such as turn/start and review/start:

• app-server creates the inbound request span
• app-server propagates that request context through Submission.trace
• core creates a dispatch span when it receives the submission
• core then creates a long-lived turn span beneath that dispatch span
• existing core work such as run_turn and model request tracing runs beneath the turn span

This keeps long-lived span ownership with the layer that actually owns turn execution and completion.

4. Defer thread and realtime lifecycle-heavy tracing

The design should not add:

• app-server-owned thread residency stores
• app-server-owned realtime session stores

App-server already maintains thread subscription and runtime state in existing structures. If later tracing work needs thread loaded-duration or realtime duration metrics, that data should extend those existing structures rather than introducing a parallel tracing-only state machine.

Span model by API shape

The initial implementation keeps the app-server side uniform.

Unary request/response APIs

Examples:

• thread/list
• thread/read
• model/list
• config/read
• skills/list
• app/list

Behavior:

• create request span
• return response
• no additional app-server span state

Turn-producing APIs

Examples:

• turn/start
• review/start
• thread/compact/start when it executes as a normal turn lifecycle

Behavior:

• create request span
• submit work under that request span
• capture the current span context into Submission.trace
• let core create a dispatch span and then a long-lived turn span
• let the turn span remain open until the real core turn lifecycle ends

Important: request spans should not stay open until eventual streamed completion. The request span ends quickly; the core-owned turn span carries the long-running work.

Other APIs that submit work into core

Examples:

• thread/realtime/start
• thread/realtime/appendAudio
• thread/realtime/appendText
• thread/realtime/stop

Behavior:

• create request span
• submit work under that request span
• capture the current span context into Submission.trace
• let core continue tracing from there

These APIs do not automatically imply a long-lived app-server or core lifecycle span in the initial design.

Thread lifecycle APIs

Examples:

• thread/start
• thread/resume
• thread/fork
• thread/unsubscribe

Behavior in the initial design:

• create request span
• annotate with thread.id when known
• do not introduce separate app-server lifecycle spans or tracing-only state

If later work needs thread loaded/unloaded metrics, it should reuse the existing thread runtime state already maintained by app-server.

Where the code should live

codex-rs/protocol

Add a small shared W3cTraceContext type to codex-rs/protocol/src/protocol.rs.

Responsibilities:

• define a serializable W3C trace context type
• avoid direct dependence on OTEL runtime types
• be usable from both protocol crates and runtime crates

Suggested contents:

• W3cTraceContext
  • traceparent: Option<String>
  • tracestate: Option<String>

Suggested Submission change:

• Submission { id, op, trace: Option<W3cTraceContext> }

This is the only new internal async handoff needed for the initial rollout.

codex-rs/otel

Add a small helper module or extend existing tracing helpers so OTEL-specific logic stays centralized.

Responsibilities:

• convert W3cTraceContext -> OTEL Context
• convert the current tracing span context -> W3cTraceContext
• parent a tracing span from an explicit carrier when present
• apply precedence rules:
  • explicit carrier from app-server transport or Submission.trace
  • fallback to env TRACEPARENT / TRACESTATE
  • otherwise root span

Important:

• keep this focused on carrier parsing and span parenting
• do not move app-server runtime state into codex-otel
• do not overload OtelManager with app-server lifecycle ownership in the initial design

codex-rs/app-server-protocol

Extend inbound JSON-RPC request envelopes in codex-rs/app-server-protocol/src/jsonrpc_lite.rs with a dedicated optional trace carrier field.

Suggested shape:

• JSONRPCRequest { id, method, params, trace }

Where:

• trace: Option<W3cTraceContext>

Important:

• use a dedicated tracing field, not a generic meta bag
• keep tracing transport-level and method-agnostic
• do not add trace fields to individual *Params business payloads

codex-rs/core

Make small changes in the submission path in codex-rs/core/src/codex.rs.

Responsibilities:

• read Submission.trace
• create a per-submission dispatch span parented from that carrier
• run existing submission handling under that span

This is enough for existing core tracing to inherit the correct ancestry, and it is the right place to add the long-lived turn span required for turn lifecycles.

For turn-producing operations, core responsibilities should include:

• read Submission.trace
• create a per-submission dispatch span parented from that carrier
• create a long-lived turn span beneath the dispatch span when the operation actually starts a turn
• finish that turn span when the real core turn lifecycle completes, interrupts, or fails

codex-rs/app-server

Add a small dedicated tracing module rather than spreading request tracing logic across handlers. A likely shape is:

• app_server_tracing/mod.rs
• app_server_tracing/request_spans.rs
• app_server_tracing/incoming.rs

Responsibilities:

• extract incoming W3C trace carriers from JSON-RPC requests
• build standardized request spans
• provide a small API that wraps request handling in the correct span

Non-responsibilities in the initial design:

• no thread residency registry
• no realtime session registry

Standardized request spans

Every inbound request should use the same request-span builder.

Suggested name:

• app_server.request

Suggested attributes:

• rpc.system = "jsonrpc"
• rpc.service = "codex-app-server"
• rpc.method
• rpc.transport
  • stdio
  • websocket
• rpc.request_id
• app_server.connection_id
• app_server.api_version = "v2" when applicable
• app_server.client_name when known from initialize
• app_server.client_version when known

Optional useful attributes:

• thread.id when already known from params
• turn.id when already known from params

Important:

• the span factory should be the only place that assembles these fields
• handlers should not manually construct request-span attributes
• for the initialize request itself, read clientInfo.name and clientInfo.version directly from the request params when present
• for later requests on the same connection, read client metadata from per-connection session state populated during initialize

No app-server tracing registries

The design should not introduce app-server-owned tracing registries for turns, threads, or realtime sessions.

Why:

• app-server already has thread subscription and runtime state
• core already owns the real task and turn lifecycle
• a second tracing-specific state machine adds more code and more ways for lifecycle tracking to drift

Future guidance:

• if thread loaded/unloaded metrics become important, extend existing app-server thread state
• keep long-lived turn spans in core
• if realtime lifecycle metrics become important, extend the existing realtime runtime path rather than creating a parallel tracing store

No direct span construction in handlers

Request handlers should not call info_span!, trace_span!, set_parent, or OTEL APIs directly for app-server request tracing.

Instead:

• message_processor should wrap inbound request handling through the centralized request-span helper
• CodexThread::submit() should capture the current span context into Submission.trace

That keeps request tracing transport-level and largely invisible to business handlers.

Layering

The intended call graph is:

• message_processor -> app_server_tracing
  • create and enter the standardized inbound request span
• CodexThread::submit() -> codex-otel trace-context helper
  • snapshot the current span context into Submission.trace
• codex-core submission loop -> codex-otel trace-context helper
  • create a dispatch span parented from Submission.trace
  • create a long-lived turn span for turn-producing operations

Important:

• app-server owns inbound request tracing
• core owns execution after the async handoff
• core owns long-lived turn spans
• the design does not add app-server-owned long-lived thread or realtime spans

Inbound flow in app-server

The inbound request path should work like this:

1. Parse the JSON-RPC request envelope, including trace.
2. Use the tracing module to create a request span.
3. Process the request inside that span.
4. If the request submits work into core, let CodexThread::submit() capture the active span context into Submission.trace.

Integration point:

• codex-rs/app-server/src/message_processor.rs

Core handoff flow

The turn/start and similar flows cross an async boundary:

• app-server handler submits work
• core submission loop receives Submission
• actual work continues later on different tasks

To preserve parentage:

1. app-server request handling runs inside app_server.request
2. CodexThread::submit() captures that active context into Submission.trace
3. core submission loop creates a dispatch span parented from Submission.trace
4. if the submission starts a turn, core creates a long-lived turn span beneath that dispatch span
5. existing core spans naturally nest under the turn span

This lets:

• submission handling
• a single long-lived turn span for turn-producing APIs
• run_turn
• model client request tracing

inherit the app-server request trace without broad tracing changes across core.

Behavior for key v2 APIs

thread/start

• create request span
• annotate with thread.id once known
• send response and thread/started
• no separate thread lifecycle span in the initial design

thread/resume

• create request span
• annotate with thread.id when known
• no separate lifecycle span

thread/fork

• create request span
• annotate with the new thread.id
• no separate lifecycle span

thread/unsubscribe

• create request span
• no separate unload span
• if later thread unload metrics are needed, reuse existing thread state rather than adding a tracing-only registry

turn/start

• create request span
• submit work into core under that request span
• propagate the active span context through Submission.trace
• let core create a dispatch span and then a long-lived turn span
• let that turn span cover the full duration until completion, interruption, or failure

turn/steer

• create request span
• if the request submits core work, propagate via Submission.trace
• otherwise request span only

turn/interrupt

• create request span
• request span only unless core submission is involved

review/start

• treat like turn/start
• let core create the same kind of long-lived turn span

thread/realtime/start, appendAudio, appendText, stop

• create request span
• if the API submits work into core, propagate via Submission.trace
• do not introduce separate realtime lifecycle spans in the initial design

Unary methods such as thread/list

• create request span only

Runtime checks

Keep runtime checks narrowly scoped in the initial rollout:

• warn when an inbound trace carrier is present but invalid
• test that Submission.trace is set when work is submitted from a traced request

Do not add lifecycle consistency checks for tracing registries that do not exist yet.

Tests

Add tests for the initial mechanics:

• inbound request tracing accepts a valid W3C carrier
• invalid carriers are ignored cleanly
• unary methods create request spans without needing any extra handler changes
• turn/start propagates request ancestry through Submission.trace into core
• turn/start creates a long-lived core-owned turn span
• the turn span closes on completion, interruption, or failure
• existing core spans inherit from the propagated parent

The goal is to verify the centralized propagation behavior, not to exhaustively test OTEL internals.

Suggested PR sequence

PR 1: Foundation plus inbound request spans

Scope:

1. Introduce a shared W3cTraceContext type in codex-protocol.
2. Add trace to inbound JSON-RPC request envelopes in app-server protocol.
3. Add focused trace-context helpers in codex-rs/otel.
4. Add the centralized app-server request tracing module.
5. Wrap inbound request handling in message_processor.rs.

Why this PR:

• proves the transport and request-span shape with minimal scope
• gives all inbound app-server APIs consistent request tracing immediately
• avoids mixing lifecycle questions into the initial plumbing review

PR 2: Async handoff into core via Submission

Scope:

1. Add trace to Submission.
2. Have CodexThread::submit() capture the current span context automatically.
3. Have the core submission loop restore parentage with a dispatch span.
4. Validate the flow with turn/start.

Why this PR:

• validates the critical async handoff from app-server into core
• proves that existing core tracing can inherit the app-server request ancestry
• keeps the behavior change focused on one boundary

PR 3: Core-owned long-lived turn spans

Scope:

1. Add a long-lived turn span in core for turn/start.
2. Reuse the same turn-span pattern for review/start.
3. Ensure the span closes on completion, interruption, or failure.

Why this PR:

• completes the minimum useful tracing story for turn lifecycles
• keeps long-lived span ownership in the layer that actually owns the turn
• still builds on the simpler propagation model from PR 2 instead of mixing everything into one change

PR 4: Optional follow-ups

Possible follow-ups:

1. Reuse existing app-server thread state to add thread loaded/unloaded duration metrics if needed.
2. Reuse existing realtime runtime state to add realtime duration metrics if needed.
3. Add outbound JSON-RPC trace propagation only if there is a concrete client-side tracing use case.

Rollout guidance

Start with:

• inbound request spans for all app-server requests
• turn/start request -> core propagation
• a core-owned long-lived turn span for turn/start

Those pieces exercise the important mechanics:

• inbound carrier extraction
• request span creation
• async handoff into core
• inherited core tracing beneath the propagated parent
• a single span covering the full duration of a turn

After that, only add more lifecycle-specific tracing if a real debugging or observability gap remains.

Bottom line

The recommended initial design is:

• trace context on inbound JSON-RPC request envelopes
• one standardized request span for every inbound request
• automatic propagation through Submission into core
• core-owned long-lived turn spans for turn-producing APIs
• OTEL conversion and carrier logic centralized in codex-otel
• no app-server-owned tracing registries for turns, threads, or realtime sessions in the initial implementation

This gives app-server distributed tracing that is:

• consistent
• low-boilerplate
• modular
• aligned with the existing ownership boundaries in app-server and core