e15ecc9c35
## Why While investigating `codex exec hi` startup latency, the useful questions were not "is startup slow?" but "which durable bucket is slow in production?" The path we observed has a few distinct stages: 1. `thread/start` creates the session 2. startup prewarm builds the turn context, tools, and prompt 3. startup prewarm warms the websocket 4. the first real turn resolves the prewarm 5. the model produces the first token Before this PR, production telemetry had some of the raw measurements already: - aggregate startup-prewarm duration / age-at-first-turn metrics - TTFT as a metric - websocket request telemetry But there was no coherent production event stream for the startup breakdown itself, and TTFT was metric-only. That made it hard to answer the same latency questions from OpenTelemetry-backed logs without adding one-off local instrumentation. ## What changed Add durable production telemetry on the existing `SessionTelemetry` path: - new `codex.startup_phase` OTel log/trace events plus `codex.startup.phase.duration_ms` - new `codex.turn_ttft` OTel log/trace events while preserving the existing TTFT metric The startup phase event is emitted for the coarse buckets we actually observed while running `exec hi`: - `thread_start_create_thread` - `startup_prewarm_total` - `startup_prewarm_create_turn_context` - `startup_prewarm_build_tools` - `startup_prewarm_build_prompt` - `startup_prewarm_websocket_warmup` - `startup_prewarm_resolve` These phases are intentionally low-cardinality so they remain safe as production telemetry tags. ## Why this shape This keeps the instrumentation on the same production path as the rest of the session telemetry instead of adding a local debug-only trace mode. It also avoids changing startup behavior: - prewarm still runs - no control flow changes - no extra remote calls - no user-visible behavior changes One boundary is intentional: very early process bootstrap that happens before a session exists is not included here, because this PR uses session-scoped production telemetry. The expensive buckets we were trying to understand after `thread/start` are now covered durably. ## Verification - `cargo test -p codex-otel` - `cargo test -p codex-core turn_timing` - `cargo test -p codex-core regular_turn_emits_turn_started_without_waiting_for_startup_prewarm` - `cargo test -p codex-core interrupting_regular_turn_waiting_on_startup_prewarm_emits_turn_aborted` - `cargo test -p codex-app-server thread_start` - `just fix -p codex-otel -p codex-core -p codex-app-server` I also ran `cargo test -p codex-core`; it built successfully and then hit an existing unrelated stack overflow in `tools::handlers::multi_agents::tests::tool_handlers_cascade_close_and_resume_and_keep_explicitly_closed_subtrees_closed`.
codex-core
This crate implements the business logic for Codex. It is designed to be used by the various Codex UIs written in Rust.
Dependencies
Note that codex-core makes some assumptions about certain helper utilities being available in the environment. Currently, this support matrix is:
macOS
Expects /usr/bin/sandbox-exec to be present.
When using the workspace-write sandbox policy, the Seatbelt profile allows
writes under the configured writable roots while keeping .git (directory or
pointer file), the resolved gitdir: target, and .codex read-only.
Network access and filesystem read/write roots are controlled by
SandboxPolicy. Seatbelt consumes the resolved policy and enforces it.
Seatbelt also keeps the legacy default preferences read access
(user-preference-read) needed for cfprefs-backed macOS behavior.
Linux
Expects the binary containing codex-core to run the equivalent of codex sandbox linux (legacy alias: codex debug landlock) when arg0 is codex-linux-sandbox. See the codex-arg0 crate for details.
Legacy SandboxPolicy / sandbox_mode configs are still supported on Linux.
They can continue to use the legacy Landlock path when the split filesystem
policy is sandbox-equivalent to the legacy model after cwd resolution.
Split filesystem policies that need direct FileSystemSandboxPolicy
enforcement, such as read-only or denied carveouts under a broader writable
root, automatically route through bubblewrap. The legacy Landlock path is used
only when the split filesystem policy round-trips through the legacy
SandboxPolicy model without changing semantics. That includes overlapping
cases like /repo = write, /repo/a = none, /repo/a/b = write, where the
more specific writable child must reopen under a denied parent.
The Linux sandbox helper prefers the first bwrap found on PATH outside the
current working directory whenever it is available. If bwrap is present but
too old to support --argv0, the helper keeps using system bubblewrap and
switches to a no---argv0 compatibility path for the inner re-exec. If
bwrap is missing, it falls back to the bundled codex-resources/bwrap
binary shipped with Codex and Codex surfaces a startup warning through its
normal notification path instead of printing directly from the sandbox helper.
Codex also surfaces a startup warning when bubblewrap cannot create user
namespaces. WSL2 uses the normal Linux bubblewrap path. WSL1 is not supported
for bubblewrap sandboxing because it cannot create the required user
namespaces, so Codex rejects sandboxed shell commands that would enter the
bubblewrap path before invoking bwrap.
Windows
Legacy SandboxPolicy / sandbox_mode configs are still supported on
Windows. Legacy read-only and workspace-write policies imply full
filesystem read access; exact readable roots are represented by split
filesystem policies instead.
The elevated Windows sandbox also supports:
- legacy
ReadOnlyandWorkspaceWritebehavior - split filesystem policies that need exact readable roots, exact writable roots, or extra read-only carveouts under writable roots
- backend-managed system read roots required for basic execution, such as
C:\Windows,C:\Program Files,C:\Program Files (x86), andC:\ProgramData, when a split filesystem policy requests platform defaults
The unelevated restricted-token backend still supports the legacy full-read
Windows model for legacy ReadOnly and WorkspaceWrite behavior. It also
supports a narrow split-filesystem subset: full-read split policies whose
writable roots still match the legacy WorkspaceWrite root set, but add extra
read-only carveouts under those writable roots.
New [permissions] / split filesystem policies remain supported on Windows
only when they can be enforced directly by the selected Windows backend or
round-trip through the legacy SandboxPolicy model without changing semantics.
Policies that would require direct explicit unreadable carveouts (none) or
reopened writable descendants under read-only carveouts still fail closed
instead of running with weaker enforcement.
All Platforms
Expects the binary containing codex-core to simulate the virtual
apply_patch CLI when arg1 is --codex-run-as-apply-patch. See the
codex-arg0 crate for details.