Files
codex/codex-rs/otel
Eric Traut 0d344aca9b goal: pause continuation loops on usage limits and blockers (#23094)
Addresses #22833, #22245, #23067

## Why
`/goal` can keep synthesizing turns even when the next turn cannot make
meaningful progress. Hard usage exhaustion can replay failing turns, and
repeated permission or external-resource blockers can keep burning
tokens while waiting for user or system intervention.

## What changed
- Add resumable `blocked` and `usageLimited` goal states. As with
`paused`, goal continuation stops with these states.
- Move to `usageLimited` after usage-limit failures.
- Allow the built-in `update_goal` tool to set `blocked` only under
explicit repeated-impasse guidance. Updated goal continuation prompt to
specify that agent should use `blocked` only when it has made at least
three attempts to get past an impasse.

Most of the files touched by this PR are because of the small app server
protocol update.

## Validation

I manually reproduced a number of situations where an agent can run into
a true impasse and verified that it properly enters `blocked` state. I
then resumed and verified that it once again entered `blocked` state
several turns later if the impasse still exists.

I also manually reproduced the usage-limit condition by creating a
simulated responses API endpoint that returns 429 errors with the
appropriate error message. Verified that the goal runtime properly moves
the goal into `usageLimited` state and TUI UI updates appropriately.
Verified that `/goal resume` resumes (and immediately goes back into
`ussageLImited` state if appropriate).


## Follow-up PRs

Small changes will be needed to the GUI clients to properly handle the
two new states.
2026-05-18 11:28:53 -07:00
..

codex-otel

codex-otel is the OpenTelemetry integration crate for Codex. It provides:

  • Provider wiring for log/trace/metric exporters (codex_otel::OtelProvider and codex_otel::provider).
  • Session-scoped business event emission via codex_otel::SessionTelemetry.
  • Low-level metrics APIs via codex_otel::metrics.
  • Trace-context helpers via codex_otel::trace_context and crate-root re-exports.

Tracing and logs

Create an OTEL provider from OtelSettings. The provider also configures metrics (when enabled), then attach its layers to your tracing_subscriber registry:

use codex_otel::config::OtelExporter;
use codex_otel::config::OtelHttpProtocol;
use codex_otel::config::OtelSettings;
use codex_otel::OtelProvider;
use tracing_subscriber::prelude::*;

let settings = OtelSettings {
    environment: "dev".to_string(),
    service_name: "codex-cli".to_string(),
    service_version: env!("CARGO_PKG_VERSION").to_string(),
    codex_home: std::path::PathBuf::from("/tmp"),
    exporter: OtelExporter::OtlpHttp {
        endpoint: "https://otlp.example.com".to_string(),
        headers: std::collections::HashMap::new(),
        protocol: OtelHttpProtocol::Binary,
        tls: None,
    },
    trace_exporter: OtelExporter::OtlpHttp {
        endpoint: "https://otlp.example.com".to_string(),
        headers: std::collections::HashMap::new(),
        protocol: OtelHttpProtocol::Binary,
        tls: None,
    },
    metrics_exporter: OtelExporter::None,
    span_attributes: std::collections::BTreeMap::new(),
    tracestate: std::collections::BTreeMap::new(),
};

if let Some(provider) = OtelProvider::from(&settings)? {
    let registry = tracing_subscriber::registry()
        .with(provider.logger_layer())
        .with(provider.tracing_layer());
    registry.init();
}

Configured span attributes and W3C tracestate member fields are applied to exported trace spans and propagated trace context:

[otel.span_attributes]
"example.trace_attr" = "enabled"

[otel.tracestate.example]
alpha = "one"
beta = "two"

Configured tracestate members and encoded values must be valid W3C tracestate. Each nested table is encoded as semicolon-separated key:value fields inside that member. If propagated trace context already has the named member, Codex upserts configured fields and preserves other fields in that member. This config shape does not support setting opaque tracestate member values. Invalid trace metadata entries are ignored during config load and reported as startup warnings.

SessionTelemetry (events)

SessionTelemetry adds consistent metadata to tracing events and helps record Codex-specific session events. Rich session/business events should go through SessionTelemetry; subsystem-owned audit events can stay with the owning subsystem.

use codex_otel::SessionTelemetry;

let manager = SessionTelemetry::new(
    conversation_id,
    model,
    slug,
    account_id,
    account_email,
    auth_mode,
    originator,
    log_user_prompts,
    terminal_type,
    session_source,
);

manager.user_prompt(&prompt_items);

Metrics (OTLP or in-memory)

Modes:

  • OTLP: exports metrics via the OpenTelemetry OTLP exporter (HTTP or gRPC).
  • In-memory: records via opentelemetry_sdk::metrics::InMemoryMetricExporter for tests/assertions; call shutdown() to flush.

codex-otel also provides OtelExporter::Statsig, a shorthand for exporting OTLP/HTTP JSON metrics to Statsig using Codex-internal defaults.

Statsig ingestion (OTLP/HTTP JSON) example:

use codex_otel::config::{OtelExporter, OtelHttpProtocol};

let metrics = MetricsClient::new(MetricsConfig::otlp(
    "dev",
    "codex-cli",
    env!("CARGO_PKG_VERSION"),
    OtelExporter::OtlpHttp {
        endpoint: "https://api.statsig.com/otlp".to_string(),
        headers: std::collections::HashMap::from([(
            "statsig-api-key".to_string(),
            std::env::var("STATSIG_SERVER_SDK_SECRET")?,
        )]),
        protocol: OtelHttpProtocol::Json,
        tls: None,
    },
))?;

metrics.counter("codex.session_started", 1, &[("source", "tui")])?;
metrics.histogram("codex.request_latency", 83, &[("route", "chat")])?;

In-memory (tests):

let exporter = InMemoryMetricExporter::default();
let metrics = MetricsClient::new(MetricsConfig::in_memory(
    "test",
    "codex-cli",
    env!("CARGO_PKG_VERSION"),
    exporter.clone(),
))?;
metrics.counter("codex.turns", 1, &[("model", "gpt-5.1")])?;
metrics.shutdown()?; // flushes in-memory exporter

Trace context

Trace propagation helpers remain separate from the session event emitter:

use codex_otel::current_span_w3c_trace_context;
use codex_otel::set_parent_from_w3c_trace_context;

Shutdown

  • OtelProvider::shutdown() stops the OTEL exporter.
  • SessionTelemetry::shutdown_metrics() flushes and shuts down the metrics provider.

Both are optional because drop performs best-effort shutdown, but calling them explicitly gives deterministic flushing (or a shutdown error if flushing does not complete in time).