## Summary
`cargo test` has entails both running standard Rust tests and doctests.
It turns out that the doctest discovery is fairly slow, and it's a cost
you pay even for crates that don't include any doctests.
This PR disables doctests with `doctest = false` for crates that lack
any doctests.
For the collection of crates below, this speeds up test execution by
>4x.
E.g., before this PR:
```
Benchmark 1: cargo test -p codex-utils-absolute-path -p codex-utils-cache -p codex-utils-cli -p codex-utils-home-dir -p codex-utils-output-truncation -p codex-utils-path -p codex-utils-string -p codex-utils-template -p codex-utils-elapsed -p codex-utils-json-to-toml
Time (mean ± σ): 1.849 s ± 4.455 s [User: 0.752 s, System: 1.367 s]
Range (min … max): 0.418 s … 14.529 s 10 runs
```
And after:
```
Benchmark 1: cargo test -p codex-utils-absolute-path -p codex-utils-cache -p codex-utils-cli -p codex-utils-home-dir -p codex-utils-output-truncation -p codex-utils-path -p codex-utils-string -p codex-utils-template -p codex-utils-elapsed -p codex-utils-json-to-toml
Time (mean ± σ): 428.6 ms ± 6.9 ms [User: 187.7 ms, System: 219.7 ms]
Range (min … max): 418.0 ms … 436.8 ms 10 runs
```
For a single crate, with >2x speedup, before:
```
Benchmark 1: cargo test -p codex-utils-string
Time (mean ± σ): 491.1 ms ± 9.0 ms [User: 229.8 ms, System: 234.9 ms]
Range (min … max): 480.9 ms … 512.0 ms 10 runs
```
And after:
```
Benchmark 1: cargo test -p codex-utils-string
Time (mean ± σ): 213.9 ms ± 4.3 ms [User: 112.8 ms, System: 84.0 ms]
Range (min … max): 206.8 ms … 221.0 ms 13 runs
```
Co-authored-by: Codex <noreply@openai.com>
## DISCLAIMER
This is experimental and no production service must rely on this
## Why
Built-in MCPs are product-owned runtime capabilities, but they were
previously flattened into the same config-backed stdio path as
user-configured servers. That made them depend on a hidden `codex
builtin-mcp` re-exec path, exposed them through config-oriented CLI
flows, and erased distinctions the runtime needs to preserve—most
notably whether an MCP call should count as external context for
memory-mode pollution.
## What changed
- Model product-owned built-ins separately from config-backed MCP
servers via `BuiltinMcpServer` and `EffectiveMcpServer`.
- Launch built-ins in process through a reusable async transport instead
of the hidden `builtin-mcp` stdio subcommand.
- Keep config-oriented CLI operations such as `codex mcp
list/get/login/logout` scoped to configured servers, while merging
built-ins only into the effective runtime server set.
- Retain server metadata after launch so parallel-tool support and
context classification come from the live server set; built-in
`memories` is now classified as local Codex state rather than external
context.
## Test plan
- `cargo test -p codex-mcp`
- `cargo test -p codex-core --test suite
builtin_memories_mcp_call_does_not_mark_thread_memory_mode_polluted_when_configured`
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
Related to
https://openai.slack.com/archives/C095U48JNL9/p1777537279707449
TLDR:
We update the meaning of session ids and thread ids:
* thread_id stays as now
* session_id become a shared id between every thread under a /root
thread (i.e. every sub-agent share the same session id)
This PR introduces an explicit `SessionId` and threads it through the
protocol/client boundary so `session_id` and `thread_id` can diverge
when they need to, while preserving compatibility for older serialized
`session_configured` events.
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- make `thread_source` an explicit optional thread-level field on
`thread/start`, `thread/fork`, and returned thread payloads
- persist `thread_source` in rollout/session metadata so resumed live
threads retain the original value
- replace the old best-effort `session_source` -> `thread_source`
mapping with an explicit caller-supplied analytics classification
## Why
Before this change, analytics `thread_source` was populated by a
best-effort mapping from `session_source`. `session_source` describes
the runtime/client surface, not the actual thread-level origin, so that
projection was not accurate enough to distinguish cases such as `user`,
`subagent`, `memory_consolidation`, and future thread origins reliably.
Making `thread_source` explicit keeps one thread-level analytics field
while letting callers provide the real classification directly instead
of recovering it indirectly from `session_source`.
## Impact
For new analytics events, `thread_source` now reflects the explicit
thread-level classification supplied by the caller rather than an
inferred value derived from `session_source`. Existing protocol fields
remain optional; callers that omit `threadSource` now produce `null`
instead of a best-effort inferred value.
## Validation
- `just write-app-server-schema`
- `cargo test -p codex-analytics -p codex-core -p
codex-app-server-protocol --no-run`
- `cargo test -p codex-app-server-protocol
generated_ts_optional_nullable_fields_only_in_params`
- `cargo test -p codex-analytics
thread_initialized_event_serializes_expected_shape`
- `cargo test -p codex-core
resume_stopped_thread_from_rollout_preserves_thread_source`
## Why
Memory search currently treats separators literally, so callers need to
know whether a stored term uses spaces, hyphens, or no separators at
all. That makes recall brittle for terms such as `MultiAgentV2` vs.
`multi agent v2` and `cold-resume` vs. `cold resume`.
## What changed
- Add an opt-in `normalized` mode to memory search that removes
non-alphanumeric separators after any requested case folding.
- Thread the new flag through the MCP `search` tool into the local
backend while keeping existing literal matching as the default.
- Reject queries that normalize to an empty string, and add regression
coverage for both normalized matching and that validation path.
## Testing
- `cargo test -p codex-memories-mcp`
## Why
Memory search currently supports either independent substring matches or
requiring every query to appear on the same line. That is too
restrictive for memory files where related terms often land on nearby
lines in the same note or bullet block.
## What changed
- Replace the old `all` match mode with explicit tagged modes:
`all_on_same_line` and `all_within_lines { line_count }`.
- Add windowed matching in `codex-rs/memories/mcp/src/local.rs` so
callers can require every query to appear within a bounded line range
while returning only the minimal qualifying windows.
- Reject invalid zero-width windows and update the MCP tool description
plus argument parsing to expose the new mode.
- Add coverage for same-line matching, windowed matching, and invalid
`line_count` input.
## Verification
- Added targeted coverage in `codex-rs/memories/mcp/src/local_tests.rs`
for `search_supports_all_within_lines_match_mode` and
`search_rejects_zero_line_window`.
- Added server-side parsing coverage in
`codex-rs/memories/mcp/src/server.rs` for
`search_args_accept_windowed_all_match_mode`.
## Why
The local memories root can contain implementation details such as
`.git` plus incidental OS metadata like `.DS_Store`. Those entries are
not authored memory content, so the memories MCP should keep them
invisible instead of exposing them through normal discovery or direct
lookup.
Only for local implementation ofc
## What changed
- Return `NotFound` for scoped `list`, `read`, and `search` requests
that include a hidden path component.
- Skip hidden files and directories while listing a directory or
recursively searching the memories tree.
- Add regression coverage for hidden files, hidden directories, and
hidden scoped requests across `list`, `read`, and `search`.
## Testing
- Added focused regression tests in `memories/mcp/src/local_tests.rs`
covering hidden-path behavior across the affected APIs.
## Why
The local memories MCP backend only rejected symlinks after resolving
the final path. That left room for scoped requests like
`skills/secret.md` to walk through a symlinked ancestor directory and
escape the configured memories root.
This change also makes missing scoped paths fail explicitly instead of
looking like an empty `list` / `search` result or a `NotFile` read
error.
## What Changed
- walk each scoped path component in
`LocalMemoriesBackend::resolve_scoped_path` and reject symlinked
ancestors before accessing the target
- reject scoped paths that traverse through a non-directory intermediate
component
- add a `NotFound` backend error for missing `read`, `list`, and
`search` paths and map it through the MCP server error conversion
- add coverage for missing paths and symlinked ancestor directories in
`codex-rs/memories/mcp/src/local_tests.rs`
## Testing
- added unit coverage in `codex-rs/memories/mcp/src/local_tests.rs` for
missing paths and symlinked ancestor directories across `read`, `list`,
and `search`
## Why
The memories MCP server currently keeps handwritten JSON Schema beside
the Rust types that actually serialize and deserialize the tool
payloads:
[`schema.rs`](2f5c06a29c/codex-rs/memories/mcp/src/schema.rs (L4-L133)),
[`server.rs`](2f5c06a29c/codex-rs/memories/mcp/src/server.rs (L44-L75)),
and
[`backend.rs`](2f5c06a29c/codex-rs/memories/mcp/src/backend.rs (L41-L117)).
That duplicates the tool contract and makes schema drift easier as the
API evolves.
## What changed
- derive `JsonSchema` for the memories tool arguments, responses, and
nested response types
- replace the handwritten schema builders with shared `schemars`
generation
- preserve the existing wire shape while generating schemas, including
nullable output `Option` fields and non-nullable optional input fields
- wire the `list`, `read`, and `search` tools to the generated schemas
## Verification
- CI pending
## Why
The memories MCP `search` tool only accepts a single substring today,
which makes it hard for clients to express combined queries or explain
why a line matched. This change adds the richer search shape needed for
the next client iteration while keeping the legacy single-`query` call
working.
## What changed
- accept either the legacy `query` field or a new `queries` array, plus
`match_mode: any|all`
- teach the local memories backend to evaluate multi-query line matches
and return `matched_queries` on each hit
- update the MCP input/output schema and add coverage for parser
behavior, ordering, pagination, case sensitivity, and match modes
## Testing
- added unit coverage in `memories/mcp/src/local_tests.rs` and
`memories/mcp/src/server.rs`
## Why
The paginated memories MCP `search` tool still returned only the
matching line text, which made it harder for clients to present useful
search results or decide whether they needed to follow up with a
separate `read` call. Adding a small amount of surrounding context makes
individual hits much more usable while keeping the search response
deterministic and line-addressable.
## What changed
- add an optional `context_lines` search argument and thread it through
the MCP server into the local memories backend
- change search matches to return the matched `line_number` plus a
`start_line_number` and multi-line `content` block for the requested
context window
- update the search tool schema and description to document the new
request/response shape
- extend the local backend tests to cover zero-context matches,
contextual results, pagination, and invalid cursors that point past the
end of the result set
## Testing
- Added targeted unit coverage in `memories/mcp/src/local_tests.rs`
- GitHub Actions are running for the branch
---------
Co-authored-by: Codex <noreply@openai.com>
## Why
The memories MCP `search` tool previously stopped once it hit
`max_results`, so callers could tell there were more matches via
`truncated` but had no way to fetch the rest of the result set. That
made large searches awkward for clients that need to keep paging through
a stable, deterministic view of the matches.
## What changed
- add an optional `cursor` field to `SearchMemoriesRequest` / tool input
and return `next_cursor` in `SearchMemoriesResponse`
- update the MCP schemas and tool wiring so clients can request
subsequent pages explicitly
- change the local memories backend to collect and sort the full scoped
match list, then slice the requested page and reject invalid cursors
- add unit coverage for paginated search results and invalid cursor
handling in `memories/mcp/src/local_tests.rs`
## Testing
- Added targeted unit coverage in `memories/mcp/src/local_tests.rs`
- GitHub Actions are running for the branch
## Why
The memories MCP `list` tool should behave like a directory listing, not
a recursive tree walk. Recursive results make pagination harder to
reason about, return unexpectedly deep paths for scoped requests, and no
longer match the intended tool contract.
## What Changed
- Changed the local memories backend so `list` returns only the
immediate children of the requested path.
- Preserved file-scoped requests by returning the file itself, and
missing paths by returning an empty result.
- Updated cursor handling to paginate over the shallow sibling set and
reject cursors past the available results.
- Updated the MCP tool description to say it lists immediate files and
directories under a path.
- Reworked the local backend tests to cover shallow top-level listing,
shallow scoped listing, sibling ordering, and pagination.
## Testing
- `cargo test -p codex-memories-mcp`
## Why
Large memories trees do not fit well into a single MCP `list` response.
This change makes the memories MCP server page `list` results so callers
can continue walking the tree without overfetching or relying on
ambiguous truncation.
## What changed
- add an optional `cursor` input to the memories MCP `list` API and
return `next_cursor` alongside `truncated` in the response
- paginate recursive local-memory traversal while preserving
lexicographic path order across directories
- reject malformed and out-of-range cursors as invalid MCP requests
- update the server/schema wiring and add coverage for pagination,
ordering, and cursor validation in `memories/mcp/src/local_tests.rs`
## Testing
- `cargo test -p codex-memories-mcp`
## Why
The memories MCP `read` tool already supports `line_offset`, but it
cannot return a bounded line range. That makes it awkward to page
through large memory files or request a small slice without relying on
token truncation.
## What changed
- add an optional `max_lines` parameter to the memories MCP `read` tool
schema and request parsing
- cap local backend reads to the requested number of lines before token
truncation
- treat `max_lines = 0` as an invalid request and surface it as
`invalid_params`
- add backend tests for bounded reads and invalid line request
validation
## Testing
- added coverage in `memories/mcp/src/local_tests.rs` for `max_lines`
reads and invalid `max_lines` / `line_offset` requests
## Why
Memory clients sometimes need to continue reading a file from a known
line instead of starting over from the top. Adding a line offset to the
`read` MCP keeps that resume logic simple and avoids re-reading
already-consumed content.
## What changed
- Added an optional `line_offset` argument to the memory `read` tool,
defaulting to `1`.
- Read content starting at the requested 1-indexed line before token
truncation, and return `start_line_number` in the response.
- Treat invalid offsets as invalid params errors and cover the new
behavior in `codex-rs/memories/mcp/src/local_tests.rs`.
## Testing
- Added unit tests for reading from a non-default starting line.
- Added unit tests for rejecting `0` and past-end line offsets.
## Summary
Ad-hoc memory notes are written under `memories/extensions/ad_hoc/`, but
the consolidation agent only knows how to interpret an extension when
the extension folder has an `instructions.md`. Seed those instructions
from the memories write pipeline so an enabled memories startup creates
the expected ad-hoc extension layout automatically.
This also moves extension-specific write behavior behind a dedicated
`memories/write/src/extensions/` module. `ad_hoc` owns the seeded
instructions template, while the existing resource-retention cleanup
lives in its own `prune` module so future memory extensions can add
their own write-side setup without growing a flat helper file.
## Changes
- Seed `memories/extensions/ad_hoc/instructions.md` during eligible
memory startup without overwriting an existing file.
- Store the ad-hoc instructions template under
`memories/write/templates/extensions/ad_hoc/`, keeping ownership in
`codex-memories-write`.
- Split memory extension support into `extensions::ad_hoc` and
`extensions::prune`.
- Keep the existing old-resource pruning behavior unchanged.
## Verification
- `cargo test -p codex-memories-write`
- `bazel build //codex-rs/memories/write:write`
---------
Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
- Build one app-server process ThreadStore from startup config and share
it with ThreadManager and CodexMessageProcessor.
- Remove per-thread/fork store reconstruction so effective thread config
cannot switch the persistence backend.
- Add params to ThreadStore create/resume for specifying thread
metadata, since otherwise the metadata from store creation would be used
(incorrectly).
## Why
This bug is exposed by Guardian/auto-review approvals. With the managed
network proxy enabled, a blocked network request can be reported back
through the network approval service as an approval denial after the
command has already started. Before this change, the shell and unified
exec runtimes registered those network approval calls, but did not have
a way to observe an async proxy denial as a cancellation/failure signal
for the running process.
The result was confusing: Guardian/auto-review could correctly deny
network access, but the command path could keep running or unregister
the approval without surfacing the denial as the command failure.
## What Changed
- `NetworkApprovalService` now attaches a cancellation token to active
and deferred network approvals.
- Proxy-denial outcomes are recorded only for active registrations,
cancel the owning token, and are consumed when the approval is
finalized.
- The shell runtime combines the normal command timeout with the
network-denial cancellation token.
- Unified exec stores the deferred network approval object, terminates
tracked processes when the proxy denial arrives, and returns the denial
as a process failure while polling or completing the process.
- Tool orchestration passes the active network approval cancellation
token into the sandbox attempt and preserves deferred approval errors
instead of silently unregistering them.
- App-server `command/exec` now handles the combined
timeout-or-cancellation expiration variant used by the runtime.
## Verification
- `cargo test -p codex-core network_approval --lib`
- `cargo clippy -p codex-app-server --all-targets -- -D warnings`
- `cargo clippy -p codex-core --all-targets -- -D warnings`
---------
Co-authored-by: Codex <noreply@openai.com>
Summary:
- Add codex-thread-manager-sample, a one-shot binary that starts a
ThreadManager thread, submits a prompt, and prints the final assistant
output.
- Pass ThreadStore into ThreadManager::new and expose
thread_store_from_config for existing callsites.
- Build the sample Config directly with only --model and prompt inputs.
Verification:
- just fmt
- cargo check -p codex-thread-manager-sample -p codex-app-server -p
codex-mcp-server
- git diff --check
Tests: Not run per request.
## Why
Memory startup runs in the background after an eligible turn, but it can
consume Codex backend quota at exactly the wrong time: when the user is
already near a rate-limit boundary. This PR adds a guard so the memory
pipeline backs off when the Codex rate-limit snapshot says the remaining
budget is too low.
## What Changed
- Added `memories.min_rate_limit_remaining_percent` with a default of
`25`, clamped to `0..=100`, and regenerated `core/config.schema.json`.
- Added `codex-rs/memories/write/src/guard.rs`, which fetches Codex
backend rate limits before memory startup and skips phase 1 / phase 2
when the Codex limit is reached or either tracked window is above the
configured usage ceiling.
- Keeps startup best-effort: non-Codex auth or rate-limit fetch/client
failures preserve the existing memory startup behavior.
- Records a `codex.memory.startup` counter with
`status=skipped_rate_limit` when startup is skipped.
- Added config parsing/clamping coverage and guard unit tests.
## Verification
- Added `codex-rs/memories/write/src/guard_tests.rs` for threshold,
primary/secondary window, and reached-limit behavior.
- Added config tests for TOML parsing and clamping.
## Why
Memory startup was tied to thread lifecycle events such as create, load,
and fork. That can run memory work before a thread receives real user
input, and it makes startup cost scale with thread management instead of
actual turns. Moving the trigger to `thread/sendInput` keeps memory
startup aligned with the first real user turn and lets it use the
current thread config at turn time.
The idea is to prevent ghost cost due to pre-warm triggered by the app
Turn-based startup can also make global phase-2 consolidation easier to
request repeatedly, so this adds a success cooldown and tightens the
default startup scan window.
## What Changed
- Start `codex_memories_write::start_memories_startup_task` after a
non-empty `thread/sendInput` turn is submitted, instead of from thread
create/load/fork paths:
d4a6885b78/codex-rs/app-server/src/codex_message_processor.rs (L6477-L6487)
- Expose `CodexThread::config()` so app-server can pass the live config
into memory startup at turn time.
- Add a six-hour successful-run cooldown for global phase-2
consolidation via `SkippedCooldown`:
d4a6885b78/codex-rs/state/src/runtime/memories.rs (L963-L966)
- Reduce memory startup defaults to at most 2 rollouts over 10 days:
d4a6885b78/codex-rs/config/src/types.rs (L31-L34)
## Verification
Updated the memory runtime coverage around phase-2 reclaim behavior,
including `phase2_global_lock_respects_success_cooldown`.
---------
Co-authored-by: Codex <noreply@openai.com>
## Why
Phase 2 still needs to choose the most relevant stage-1 memory outputs
by usage and recency, but exposing that ranking as the rendered
`raw_memories.md` order creates unnecessary large diff. Usage-count or
timestamp changes can reshuffle otherwise unchanged memories, making the
workspace diff noisy and giving the consolidation prompt a misleading
recency signal from file position.
This fix will reduce token consumption
## What Changed
- Keep the existing top-N Phase 2 selection ranking by `usage_count`,
`last_usage`, `source_updated_at`, and `thread_id`.
- Return the selected rows in stable ascending `thread_id` order before
syncing Phase 2 filesystem inputs.
- Update the memory README, raw memories header, and consolidation
prompt so they describe the stable order and tell the prompt to use
metadata and workspace diffs instead of file order as the recency
signal.
- Adjust the memory runtime tests to use deterministic thread IDs and
assert the stable return order separately from the ranked selection
semantics.
## Test Coverage
- Existing memory runtime tests in
`codex-rs/state/src/runtime/memories.rs` now cover the stable returned
ordering for Phase 2 inputs.
---------
Co-authored-by: Codex <noreply@openai.com>
Keep extracting memories out of core and moving the write trigger in the
app-server
This is temporary and it should move at the client level as a follow-up
This makes core fully independant from `codex-memories-write`
---------
Co-authored-by: Codex <noreply@openai.com>
