Compare commits

...

15 Commits

Author SHA1 Message Date
Ahmed Ibrahim
8aa5e7770c fmt + clippy: codex-core deterministic shell tool tests, conflict cleanup 2025-09-10 23:42:57 -07:00
Ahmed Ibrahim
162e1235a8 Change forking to read the rollout from file (#3440)
This PR changes get history op to get path. Then, forking will use a
path. This will help us have one unified codepath for resuming/forking
conversations. Will also help in having rollout history in order. It
also fixes a bug where you won't see the UI when resuming after forking.
2025-09-10 17:42:54 -07:00
jif-oai
c09ed74a16 Unified execution (#3288)
## Unified PTY-Based Exec Tool

Note: this requires to have this flag in the config:
`use_experimental_unified_exec_tool=true`

- Adds a PTY-backed interactive exec feature (“unified_exec”) with
session reuse via
  session_id, bounded output (128 KiB), and timeout clamping (≤ 60 s).
- Protocol: introduces ResponseItem::UnifiedExec { session_id,
arguments, timeout_ms }.
- Tools: exposes unified_exec as a function tool (Responses API);
excluded from Chat
  Completions payload while still supported in tool lists.
- Path handling: resolves commands via PATH (or explicit paths), with
UTF‑8/newline‑aware
  truncation (truncate_middle).
- Tests: cover command parsing, path resolution, session
persistence/cleanup, multi‑session
  isolation, timeouts, and truncation behavior.
2025-09-10 17:38:11 -07:00
Michael Bolin
65f3528cad feat: add UserInfo request to JSON-RPC server (#3428)
This adds a simple endpoint that provides the email address encoded in
`$CODEX_HOME/auth.json`.

As noted, for now, we do not hit the server to verify this is the user's
true email address.
2025-09-10 17:03:35 -07:00
Michael Bolin
44262d8fd8 fix: ensure output of codex-rs/mcp-types/generate_mcp_types.py matches codex-rs/mcp-types/src/lib.rs (#3439)
https://github.com/openai/codex/pull/3395 updated `mcp-types/src/lib.rs`
by hand, but that file is generated code that is produced by
`mcp-types/generate_mcp_types.py`. Unfortunately, we do not have
anything in CI to verify this right now, but I will address that in a
subsequent PR.

#3395 ended up introducing a change that added a required field when
deserializing `InitializeResult`, breaking Codex when used as an MCP
client, so the quick fix in #3436 was to make the new field `Optional`
with `skip_serializing_if = "Option::is_none"`, but that did not address
the problem that `mcp-types/generate_mcp_types.py` and
`mcp-types/src/lib.rs` are out of sync.

This PR gets things back to where they are in sync. It removes the
custom `mcp_types::McpClientInfo` type that was added to
`mcp-types/src/lib.rs` and forces us to use the generated
`mcp_types::Implementation` type. Though this PR also updates
`generate_mcp_types.py` to generate the additional `user_agent:
Optional<String>` field on `Implementation` so that we can continue to
specify it when Codex operates as an MCP server.

However, this also requires us to specify `user_agent: None` when Codex
operates as an MCP client.

We may want to introduce our own `InitializeResult` type that is
specific to when we run as a server to avoid this in the future, but my
immediate goal is just to get things back in sync.
2025-09-10 16:14:41 -07:00
Jeremy Rose
95a9938d3a fix trampling projects table when accepting trusted dirs (#3434)
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
2025-09-10 23:01:31 +00:00
Jeremy Rose
f69f07b028 put workspace roots in the environment context (#3375)
to keep the tool description constant when the writable roots change.
2025-09-10 15:10:52 -07:00
Gabriel Peal
8d766088e6 Make user_agent optional (#3436)
# External (non-OpenAI) Pull Request Requirements

Currently, mcp server fail to start with:
```
🖐  MCP client for `<CLIENT>` failed to start: missing field `user_agent`
````

It isn't clear to me yet why this is happening. My understanding is that
this struct is simply added as a new field to the response but this
should fix it until I figure out the full story here.

<img width="714" height="262" alt="CleanShot 2025-09-10 at 13 58 59"
src="https://github.com/user-attachments/assets/946b1313-5c1c-43d3-8ae8-ecc3de3406fc"
/>
2025-09-10 14:15:02 -07:00
dedrisian-oai
87654ec0b7 Persist model & reasoning changes (#2799)
Persists `/model` changes across both general and profile-specific
sessions.
2025-09-10 20:53:46 +00:00
Michael Bolin
51d9e05de7 Back out "feat: POSIX unification and snapshot sessions (#3179)" (#3430)
This reverts https://github.com/openai/codex/pull/3179.

#3179 appears to introduce a regression where sourcing dotfiles causes a
bunch of activity in the title bar (and potentially slows things down?)


https://github.com/user-attachments/assets/a68f7fb3-0749-4e0e-a321-2aa6993e01da

Verified this no longer happens after backing out #3179.

Original commit changeset: 62bd0e3d9d
2025-09-10 12:40:24 -07:00
Jeremy Rose
8068cc75f8 replace tui_markdown with a custom markdown renderer (#3396)
Also, simplify the streaming behavior.

This fixes a number of display issues with streaming markdown, and paves
the way for better markdown features (e.g. customizable styles, syntax
highlighting, markdown-aware wrapping).

Not currently supported:
- footnotes
- tables
- reference-style links
2025-09-10 12:13:53 -07:00
Eric Traut
acb28bf914 Improved resiliency of two auth-related tests (#3427)
This PR improves two existing auth-related tests. They were failing when
run in an environment where an `OPENAI_API_KEY` env variable was
defined. The change makes them more resilient.
2025-09-10 11:46:02 -07:00
Kazuhiro Sera
97338de578 Remove a broken link to prompting_guide.md in docs/getting-started.md (#2858)
The file no longer exists. We've been receiving this feedback several
times.
- https://github.com/openai/codex/issues/2374
- https://github.com/openai/codex/issues/2810
- https://github.com/openai/codex/issues/2826

My previous PR https://github.com/openai/codex/pull/2413 for this issue
restored the file but now it's compatible with the current file
structure. Thus, let's simply delete the link.
2025-09-10 10:52:50 -07:00
katyhshi
5200b7a95d docs: fix codex exec heading typo (#2703)
# External (non-OpenAI) Pull Request Requirements

Before opening this Pull Request, please read the "Contributing" section
of the README or your PR may be closed:
https://github.com/openai/codex#contributing

If your PR conforms to our contribution guidelines, replace this text
with a detailed and high quality description of your changes.
2025-09-10 10:39:53 -07:00
Michael Bolin
64e6c4afbb fix: remove empty file: chatwidget_stream_tests.rs (#3356)
Originally added in https://github.com/openai/codex/pull/2029.
2025-09-10 10:35:24 -07:00
61 changed files with 4866 additions and 1927 deletions

190
codex-rs/Cargo.lock generated
View File

@@ -311,15 +311,6 @@ version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a8241f3ebb85c056b509d4327ad0358fbbba6ffb340bf388f26350aeda225b1"
[[package]]
name = "bincode"
version = "1.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1f45e9417d87227c7a56d22e471c6206462cba514c7590c09aff4cf6d1ddcad"
dependencies = [
"serde",
]
[[package]]
name = "bit-set"
version = "0.5.3"
@@ -655,7 +646,7 @@ dependencies = [
"tokio-test",
"tokio-util",
"toml",
"toml_edit 0.23.4",
"toml_edit",
"tracing",
"tree-sitter",
"tree-sitter-bash",
@@ -778,6 +769,7 @@ version = "0.0.0"
dependencies = [
"anyhow",
"assert_cmd",
"base64",
"codex-arg0",
"codex-common",
"codex-core",
@@ -879,6 +871,7 @@ dependencies = [
"path-clean",
"pathdiff",
"pretty_assertions",
"pulldown-cmark",
"rand 0.9.2",
"ratatui",
"regex-lite",
@@ -895,7 +888,6 @@ dependencies = [
"tracing",
"tracing-appender",
"tracing-subscriber",
"tui-markdown",
"unicode-segmentation",
"unicode-width 0.1.14",
"url",
@@ -1763,12 +1755,6 @@ version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988"
[[package]]
name = "futures-timer"
version = "3.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24"
[[package]]
name = "futures-util"
version = "0.3.31"
@@ -1854,12 +1840,6 @@ version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f"
[[package]]
name = "glob"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8d1add55171497b4705a648c6b583acafb01d58050a51727785f0b2c8e0a2b2"
[[package]]
name = "globset"
version = "0.4.16"
@@ -2567,12 +2547,6 @@ dependencies = [
"libc",
]
[[package]]
name = "linked-hash-map"
version = "0.5.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0717cef1bc8b636c6e1c1bbdefc09e6322da8a9321966e8928ef80d20f7f770f"
[[package]]
name = "linux-raw-sys"
version = "0.4.15"
@@ -3014,28 +2988,6 @@ version = "1.70.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4895175b425cb1f87721b59f0f286c2092bd4af812243672510e1ac53e2e0ad"
[[package]]
name = "onig"
version = "6.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "336b9c63443aceef14bea841b899035ae3abe89b7c486aaf4c5bd8aafedac3f0"
dependencies = [
"bitflags 2.9.1",
"libc",
"once_cell",
"onig_sys",
]
[[package]]
name = "onig_sys"
version = "69.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c7f86c6eef3d6df15f23bcfb6af487cbd2fed4e5581d58d5bf1f5f8b7f6727dc"
dependencies = [
"cc",
"pkg-config",
]
[[package]]
name = "openssl"
version = "0.10.73"
@@ -3361,15 +3313,6 @@ dependencies = [
"yansi",
]
[[package]]
name = "proc-macro-crate"
version = "3.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "edce586971a4dfaa28950c6f18ed55e0406c1ab88bbce2c6f6293a7aaba73d35"
dependencies = [
"toml_edit 0.22.27",
]
[[package]]
name = "proc-macro2"
version = "1.0.95"
@@ -3381,9 +3324,9 @@ dependencies = [
[[package]]
name = "pulldown-cmark"
version = "0.13.0"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e8bbe1a966bd2f362681a44f6edce3c2310ac21e4d5067a6e7ec396297a6ea0"
checksum = "76979bea66e7875e7509c4ec5300112b316af87fa7a252ca91c448b32dfe3993"
dependencies = [
"bitflags 2.9.1",
"getopts",
@@ -3394,9 +3337,9 @@ dependencies = [
[[package]]
name = "pulldown-cmark-escape"
version = "0.11.0"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "007d8adb5ddab6f8e3f491ac63566a7d5002cc7ed73901f72057943fa71ae1ae"
checksum = "bd348ff538bc9caeda7ee8cad2d1d48236a1f443c1fa3913c6a02fe0043b1dd3"
[[package]]
name = "pxfm"
@@ -3627,12 +3570,6 @@ version = "0.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c"
[[package]]
name = "relative-path"
version = "1.9.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2"
[[package]]
name = "reqwest"
version = "0.12.23"
@@ -3691,51 +3628,12 @@ dependencies = [
"windows-sys 0.52.0",
]
[[package]]
name = "rstest"
version = "0.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6fc39292f8613e913f7df8fa892b8944ceb47c247b78e1b1ae2f09e019be789d"
dependencies = [
"futures-timer",
"futures-util",
"rstest_macros",
"rustc_version",
]
[[package]]
name = "rstest_macros"
version = "0.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f168d99749d307be9de54d23fd226628d99768225ef08f6ffb52e0182a27746"
dependencies = [
"cfg-if",
"glob",
"proc-macro-crate",
"proc-macro2",
"quote",
"regex",
"relative-path",
"rustc_version",
"syn 2.0.104",
"unicode-ident",
]
[[package]]
name = "rustc-demangle"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "989e6739f80c4ad5b13e0fd7fe89531180375b18520cc8c82080e4dc4035b84f"
[[package]]
name = "rustc_version"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
dependencies = [
"semver",
]
[[package]]
name = "rustix"
version = "0.38.44"
@@ -3975,12 +3873,6 @@ dependencies = [
"libc",
]
[[package]]
name = "semver"
version = "1.0.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56e6fa9c48d24d85fb3de5ad847117517440f6beceb7798af16b4a87d616b8d0"
[[package]]
name = "serde"
version = "1.0.219"
@@ -4464,28 +4356,6 @@ dependencies = [
"syn 2.0.104",
]
[[package]]
name = "syntect"
version = "5.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "874dcfa363995604333cf947ae9f751ca3af4522c60886774c4963943b4746b1"
dependencies = [
"bincode",
"bitflags 1.3.2",
"flate2",
"fnv",
"once_cell",
"onig",
"plist",
"regex-syntax 0.8.5",
"serde",
"serde_derive",
"serde_json",
"thiserror 1.0.69",
"walkdir",
"yaml-rust",
]
[[package]]
name = "sys-locale"
version = "0.3.2"
@@ -4809,18 +4679,12 @@ dependencies = [
"indexmap 2.10.0",
"serde",
"serde_spanned",
"toml_datetime 0.7.0",
"toml_datetime",
"toml_parser",
"toml_writer",
"winnow",
]
[[package]]
name = "toml_datetime"
version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c"
[[package]]
name = "toml_datetime"
version = "0.7.0"
@@ -4830,17 +4694,6 @@ dependencies = [
"serde",
]
[[package]]
name = "toml_edit"
version = "0.22.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
dependencies = [
"indexmap 2.10.0",
"toml_datetime 0.6.11",
"winnow",
]
[[package]]
name = "toml_edit"
version = "0.23.4"
@@ -4848,7 +4701,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7211ff1b8f0d3adae1663b7da9ffe396eabe1ca25f0b0bee42b0da29a9ddce93"
dependencies = [
"indexmap 2.10.0",
"toml_datetime 0.7.0",
"toml_datetime",
"toml_parser",
"toml_writer",
"winnow",
@@ -5058,22 +4911,6 @@ dependencies = [
"termcolor",
]
[[package]]
name = "tui-markdown"
version = "0.3.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d10648c25931bfaaf5334ff4e7dc5f3d830e0c50d7b0119b1d5cfe771f540536"
dependencies = [
"ansi-to-tui",
"itertools 0.14.0",
"pretty_assertions",
"pulldown-cmark",
"ratatui",
"rstest",
"syntect",
"tracing",
]
[[package]]
name = "typenum"
version = "1.18.0"
@@ -5855,15 +5692,6 @@ version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec107c4503ea0b4a98ef47356329af139c0a4f7750e621cf2973cd3385ebcb3d"
[[package]]
name = "yaml-rust"
version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56c1936c4cc7a1c9ab21a1ebb602eb942ba868cbd44a99cb7cdc5892335e1c85"
dependencies = [
"linked-hash-map",
]
[[package]]
name = "yansi"
version = "1.0.1"

View File

@@ -35,7 +35,7 @@ npx @modelcontextprotocol/inspector codex mcp
You can enable notifications by configuring a script that is run whenever the agent finishes a turn. The [notify documentation](../docs/config.md#notify) includes a detailed example that explains how to get desktop notifications via [terminal-notifier](https://github.com/julienXX/terminal-notifier) on macOS.
### `codex exec` to run Codex programmatially/non-interactively
### `codex exec` to run Codex programmatically/non-interactively
To run Codex non-interactively, run `codex exec PROMPT` (you can also pass the prompt via `stdin`) and Codex will work on your task until it decides that it is done and exits. Output is printed to the terminal directly. You can set the `RUST_LOG` environment variable to see more about what's going on.

View File

@@ -54,6 +54,7 @@ tracing = { version = "0.1.41", features = ["log"] }
tree-sitter = "0.25.9"
tree-sitter-bash = "0.25.0"
uuid = { version = "1", features = ["serde", "v4"] }
which = "6"
wildmatch = "2.4.0"
@@ -69,9 +70,6 @@ openssl-sys = { version = "*", features = ["vendored"] }
[target.aarch64-unknown-linux-musl.dependencies]
openssl-sys = { version = "*", features = ["vendored"] }
[target.'cfg(target_os = "windows")'.dependencies]
which = "6"
[dev-dependencies]
assert_cmd = "2"
core_test_support = { path = "tests/common" }

View File

@@ -9,6 +9,9 @@ use std::sync::atomic::AtomicU64;
use std::time::Duration;
use crate::AuthManager;
use crate::config_edit::CONFIG_KEY_EFFORT;
use crate::config_edit::CONFIG_KEY_MODEL;
use crate::config_edit::persist_non_null_overrides;
use crate::event_mapping::map_response_item_to_event_messages;
use async_channel::Receiver;
use async_channel::Sender;
@@ -16,13 +19,14 @@ use codex_apply_patch::ApplyPatchAction;
use codex_apply_patch::MaybeApplyPatchVerified;
use codex_apply_patch::maybe_parse_apply_patch_verified;
use codex_protocol::mcp_protocol::ConversationId;
use codex_protocol::protocol::ConversationHistoryResponseEvent;
use codex_protocol::protocol::ConversationPathResponseEvent;
use codex_protocol::protocol::RolloutItem;
use codex_protocol::protocol::TaskStartedEvent;
use codex_protocol::protocol::TurnAbortReason;
use codex_protocol::protocol::TurnAbortedEvent;
use futures::prelude::*;
use mcp_types::CallToolResult;
use serde::Deserialize;
use serde::Serialize;
use serde_json;
use tokio::sync::oneshot;
@@ -109,6 +113,7 @@ use crate::safety::assess_command_safety;
use crate::safety::assess_safety_for_untrusted_command;
use crate::shell;
use crate::turn_diff_tracker::TurnDiffTracker;
use crate::unified_exec::UnifiedExecSessionManager;
use crate::user_instructions::UserInstructions;
use crate::user_notification::UserNotification;
use crate::util::backoff;
@@ -277,6 +282,7 @@ pub(crate) struct Session {
/// Manager for external MCP servers/tools.
mcp_connection_manager: McpConnectionManager,
session_manager: ExecSessionManager,
unified_exec_manager: UnifiedExecSessionManager,
/// External notifier command (will be passed as args to exec()). When
/// `None` this feature is disabled.
@@ -403,7 +409,7 @@ impl Session {
let rollout_fut = RolloutRecorder::new(&config, rollout_params);
let mcp_fut = McpConnectionManager::new(config.mcp_servers.clone());
let default_shell_fut = shell::default_user_shell(conversation_id.0, &config.codex_home);
let default_shell_fut = shell::default_user_shell();
let history_meta_fut = crate::message_history::history_metadata(&config);
// Join all independent futures.
@@ -462,12 +468,12 @@ impl Session {
tools_config: ToolsConfig::new(&ToolsConfigParams {
model_family: &config.model_family,
approval_policy,
sandbox_policy: sandbox_policy.clone(),
include_plan_tool: config.include_plan_tool,
include_apply_patch_tool: config.include_apply_patch_tool,
include_web_search_request: config.tools_web_search_request,
use_streamable_shell_tool: config.use_experimental_streamable_shell_tool,
include_view_image_tool: config.include_view_image_tool,
experimental_unified_exec_tool: config.use_experimental_unified_exec_tool,
}),
user_instructions,
base_instructions,
@@ -476,12 +482,12 @@ impl Session {
shell_environment_policy: config.shell_environment_policy.clone(),
cwd,
};
let sess = Arc::new(Session {
conversation_id,
tx_event: tx_event.clone(),
mcp_connection_manager,
session_manager: ExecSessionManager::default(),
unified_exec_manager: UnifiedExecSessionManager::default(),
notify,
state: Mutex::new(state),
rollout: Mutex::new(Some(rollout_recorder)),
@@ -1101,10 +1107,10 @@ async fn submission_loop(
let provider = prev.client.get_provider();
// Effective model + family
let (effective_model, effective_family) = if let Some(m) = model {
let (effective_model, effective_family) = if let Some(ref m) = model {
let fam =
find_family_for_model(&m).unwrap_or_else(|| config.model_family.clone());
(m, fam)
find_family_for_model(m).unwrap_or_else(|| config.model_family.clone());
(m.clone(), fam)
} else {
(prev.client.get_model(), prev.client.get_model_family())
};
@@ -1141,12 +1147,12 @@ async fn submission_loop(
let tools_config = ToolsConfig::new(&ToolsConfigParams {
model_family: &effective_family,
approval_policy: new_approval_policy,
sandbox_policy: new_sandbox_policy.clone(),
include_plan_tool: config.include_plan_tool,
include_apply_patch_tool: config.include_apply_patch_tool,
include_web_search_request: config.tools_web_search_request,
use_streamable_shell_tool: config.use_experimental_streamable_shell_tool,
include_view_image_tool: config.include_view_image_tool,
experimental_unified_exec_tool: config.use_experimental_unified_exec_tool,
});
let new_turn_context = TurnContext {
@@ -1162,25 +1168,34 @@ async fn submission_loop(
// Install the new persistent context for subsequent tasks/turns.
turn_context = Arc::new(new_turn_context);
if cwd.is_some() || approval_policy.is_some() || sandbox_policy.is_some() {
sess.record_conversation_items(&[ResponseItem::from(EnvironmentContext::new(
cwd,
approval_policy,
sandbox_policy,
// Shell is not configurable from turn to turn
None,
))])
.await;
// Optionally persist changes to model / effort
let effort_str = effort.map(|_| effective_effort.to_string());
if let Err(e) = persist_non_null_overrides(
&config.codex_home,
config.active_profile.as_deref(),
&[
(&[CONFIG_KEY_MODEL], model.as_deref()),
(&[CONFIG_KEY_EFFORT], effort_str.as_deref()),
],
)
.await
{
warn!("failed to persist overrides: {e:#}");
}
}
Op::UserInput { items } => {
// attempt to inject input into current task
if let Err(items) = sess.inject_input(items) {
// no current task, spawn a new one
let task =
AgentTask::spawn(sess.clone(), Arc::clone(&turn_context), sub.id, items);
sess.set_task(task);
}
submit_user_input(
turn_context.cwd.clone(),
turn_context.approval_policy,
turn_context.sandbox_policy.clone(),
&sess,
&turn_context,
sub.id.clone(),
items,
)
.await;
}
Op::UserTurn {
items,
@@ -1225,13 +1240,14 @@ async fn submission_loop(
tools_config: ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy,
sandbox_policy: sandbox_policy.clone(),
include_plan_tool: config.include_plan_tool,
include_apply_patch_tool: config.include_apply_patch_tool,
include_web_search_request: config.tools_web_search_request,
use_streamable_shell_tool: config
.use_experimental_streamable_shell_tool,
include_view_image_tool: config.include_view_image_tool,
experimental_unified_exec_tool: config
.use_experimental_unified_exec_tool,
}),
user_instructions: turn_context.user_instructions.clone(),
base_instructions: turn_context.base_instructions.clone(),
@@ -1240,11 +1256,16 @@ async fn submission_loop(
shell_environment_policy: turn_context.shell_environment_policy.clone(),
cwd,
};
// TODO: record the new environment context in the conversation history
// no current task, spawn a new one with the perturn context
let task =
AgentTask::spawn(sess.clone(), Arc::new(fresh_turn_context), sub.id, items);
sess.set_task(task);
submit_user_input(
fresh_turn_context.cwd.clone(),
fresh_turn_context.approval_policy,
fresh_turn_context.sandbox_policy.clone(),
&sess,
&Arc::new(fresh_turn_context),
sub.id.clone(),
items,
)
.await;
}
}
Op::ExecApproval { id, decision } => match decision {
@@ -1378,14 +1399,29 @@ async fn submission_loop(
sess.send_event(event).await;
break;
}
Op::GetHistory => {
Op::GetPath => {
let sub_id = sub.id.clone();
// Flush rollout writes before returning the path so readers observe a consistent file.
let (path, rec_opt) = {
let guard = sess.rollout.lock_unchecked();
match guard.as_ref() {
Some(rec) => (rec.get_rollout_path(), Some(rec.clone())),
None => {
error!("rollout recorder not found");
continue;
}
}
};
if let Some(rec) = rec_opt
&& let Err(e) = rec.flush().await
{
warn!("failed to flush rollout recorder before GetHistory: {e}");
}
let event = Event {
id: sub_id.clone(),
msg: EventMsg::ConversationHistory(ConversationHistoryResponseEvent {
msg: EventMsg::ConversationPath(ConversationPathResponseEvent {
conversation_id: sess.conversation_id,
entries: sess.state.lock_unchecked().history.contents(),
path,
}),
};
sess.send_event(event).await;
@@ -2063,6 +2099,72 @@ async fn handle_response_item(
Ok(output)
}
async fn handle_unified_exec_tool_call(
sess: &Session,
call_id: String,
session_id: Option<String>,
arguments: Vec<String>,
timeout_ms: Option<u64>,
) -> ResponseInputItem {
let parsed_session_id = if let Some(session_id) = session_id {
match session_id.parse::<i32>() {
Ok(parsed) => Some(parsed),
Err(output) => {
return ResponseInputItem::FunctionCallOutput {
call_id: call_id.to_string(),
output: FunctionCallOutputPayload {
content: format!("invalid session_id: {session_id} due to error {output}"),
success: Some(false),
},
};
}
}
} else {
None
};
let request = crate::unified_exec::UnifiedExecRequest {
session_id: parsed_session_id,
input_chunks: &arguments,
timeout_ms,
};
let result = sess.unified_exec_manager.handle_request(request).await;
let output_payload = match result {
Ok(value) => {
#[derive(Serialize)]
struct SerializedUnifiedExecResult<'a> {
session_id: Option<String>,
output: &'a str,
}
match serde_json::to_string(&SerializedUnifiedExecResult {
session_id: value.session_id.map(|id| id.to_string()),
output: &value.output,
}) {
Ok(serialized) => FunctionCallOutputPayload {
content: serialized,
success: Some(true),
},
Err(err) => FunctionCallOutputPayload {
content: format!("failed to serialize unified exec output: {err}"),
success: Some(false),
},
}
}
Err(err) => FunctionCallOutputPayload {
content: format!("unified exec failed: {err}"),
success: Some(false),
},
};
ResponseInputItem::FunctionCallOutput {
call_id,
output: output_payload,
}
}
async fn handle_function_call(
sess: &Session,
turn_context: &TurnContext,
@@ -2090,6 +2192,38 @@ async fn handle_function_call(
)
.await
}
"unified_exec" => {
#[derive(Deserialize)]
struct UnifiedExecArgs {
input: Vec<String>,
#[serde(default)]
session_id: Option<String>,
#[serde(default)]
timeout_ms: Option<u64>,
}
let args = match serde_json::from_str::<UnifiedExecArgs>(&arguments) {
Ok(args) => args,
Err(err) => {
return ResponseInputItem::FunctionCallOutput {
call_id,
output: FunctionCallOutputPayload {
content: format!("failed to parse function arguments: {err}"),
success: Some(false),
},
};
}
};
handle_unified_exec_tool_call(
sess,
call_id,
args.session_id,
args.input,
args.timeout_ms,
)
.await
}
"view_image" => {
#[derive(serde::Deserialize)]
struct SeeImageArgs {
@@ -2318,25 +2452,13 @@ pub struct ExecInvokeArgs<'a> {
pub stdout_stream: Option<StdoutStream>,
}
fn should_translate_shell_command(
shell: &crate::shell::Shell,
shell_policy: &ShellEnvironmentPolicy,
) -> bool {
matches!(shell, crate::shell::Shell::PowerShell(_))
|| shell_policy.use_profile
|| matches!(
shell,
crate::shell::Shell::Posix(shell) if shell.shell_snapshot.is_some()
)
}
fn maybe_translate_shell_command(
params: ExecParams,
sess: &Session,
turn_context: &TurnContext,
) -> ExecParams {
let should_translate =
should_translate_shell_command(&sess.user_shell, &turn_context.shell_environment_policy);
let should_translate = matches!(sess.user_shell, crate::shell::Shell::PowerShell(_))
|| turn_context.shell_environment_policy.use_profile;
if should_translate
&& let Some(command) = sess
@@ -2705,6 +2827,30 @@ async fn handle_sandbox_error(
}
}
async fn submit_user_input(
cwd: PathBuf,
approval_policy: AskForApproval,
sandbox_policy: SandboxPolicy,
sess: &Arc<Session>,
turn_context: &Arc<TurnContext>,
sub_id: String,
items: Vec<InputItem>,
) {
sess.record_conversation_items(&[ResponseItem::from(EnvironmentContext::new(
Some(cwd),
Some(approval_policy),
Some(sandbox_policy),
// Shell is not configurable from turn to turn
None,
))])
.await;
if let Err(items) = sess.inject_input(items) {
// no current task, spawn a new one
let task = AgentTask::spawn(Arc::clone(sess), Arc::clone(turn_context), sub_id, items);
sess.set_task(task);
}
}
fn format_exec_output_str(exec_output: &ExecToolCallOutput) -> String {
let ExecToolCallOutput {
aggregated_output, ..
@@ -2953,15 +3099,10 @@ fn convert_call_tool_result_to_function_call_output_payload(
#[cfg(test)]
mod tests {
use super::*;
use crate::config_types::ShellEnvironmentPolicyInherit;
use mcp_types::ContentBlock;
use mcp_types::TextContent;
use pretty_assertions::assert_eq;
use serde_json::json;
use shell::ShellSnapshot;
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration as StdDuration;
fn text_block(s: &str) -> ContentBlock {
@@ -2972,48 +3113,6 @@ mod tests {
})
}
fn shell_policy_with_profile(use_profile: bool) -> ShellEnvironmentPolicy {
ShellEnvironmentPolicy {
inherit: ShellEnvironmentPolicyInherit::All,
ignore_default_excludes: false,
exclude: Vec::new(),
r#set: HashMap::new(),
include_only: Vec::new(),
use_profile,
}
}
fn zsh_shell(shell_snapshot: Option<Arc<ShellSnapshot>>) -> shell::Shell {
shell::Shell::Posix(shell::PosixShell {
shell_path: "/bin/zsh".to_string(),
rc_path: "/Users/example/.zshrc".to_string(),
shell_snapshot,
})
}
#[test]
fn translates_commands_when_shell_policy_requests_profile() {
let policy = shell_policy_with_profile(true);
let shell = zsh_shell(None);
assert!(should_translate_shell_command(&shell, &policy));
}
#[test]
fn translates_commands_for_zsh_with_snapshot() {
let policy = shell_policy_with_profile(false);
let shell = zsh_shell(Some(Arc::new(ShellSnapshot::new(PathBuf::from(
"/tmp/snapshot",
)))));
assert!(should_translate_shell_command(&shell, &policy));
}
#[test]
fn bypasses_translation_for_zsh_without_snapshot_or_profile() {
let policy = shell_policy_with_profile(false);
let shell = zsh_shell(None);
assert!(!should_translate_shell_command(&shell, &policy));
}
#[test]
fn prefers_structured_content_when_present() {
let ctr = CallToolResult {

View File

@@ -38,7 +38,7 @@ const OPENAI_DEFAULT_MODEL: &str = "gpt-5";
/// the context window.
pub(crate) const PROJECT_DOC_MAX_BYTES: usize = 32 * 1024; // 32 KiB
const CONFIG_TOML_FILE: &str = "config.toml";
pub(crate) const CONFIG_TOML_FILE: &str = "config.toml";
/// Application configuration loaded from disk and merged with overrides.
#[derive(Debug, Clone, PartialEq)]
@@ -172,8 +172,15 @@ pub struct Config {
pub use_experimental_streamable_shell_tool: bool,
/// If set to `true`, used only the experimental unified exec tool.
pub use_experimental_unified_exec_tool: bool,
/// Include the `view_image` tool that lets the agent attach a local image path to context.
pub include_view_image_tool: bool,
/// The active profile name used to derive this `Config` (if any).
pub active_profile: Option<String>,
/// When true, disables burst-paste detection for typed input entirely.
/// All characters are inserted as they are received, and no buffering
/// or placeholder replacement will occur for fast keypress bursts.
@@ -257,17 +264,7 @@ pub fn load_config_as_toml(codex_home: &Path) -> std::io::Result<TomlValue> {
}
}
/// Patch `CODEX_HOME/config.toml` project state.
/// Use with caution.
pub fn set_project_trusted(codex_home: &Path, project_path: &Path) -> anyhow::Result<()> {
let config_path = codex_home.join(CONFIG_TOML_FILE);
// Parse existing config if present; otherwise start a new document.
let mut doc = match std::fs::read_to_string(config_path.clone()) {
Ok(s) => s.parse::<DocumentMut>()?,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => DocumentMut::new(),
Err(e) => return Err(e.into()),
};
fn set_project_trusted_inner(doc: &mut DocumentMut, project_path: &Path) -> anyhow::Result<()> {
// Ensure we render a human-friendly structure:
//
// [projects]
@@ -283,14 +280,26 @@ pub fn set_project_trusted(codex_home: &Path, project_path: &Path) -> anyhow::Re
// Ensure top-level `projects` exists as a non-inline, explicit table. If it
// exists but was previously represented as a non-table (e.g., inline),
// replace it with an explicit table.
let mut created_projects_table = false;
{
let root = doc.as_table_mut();
let needs_table = !root.contains_key("projects")
|| root.get("projects").and_then(|i| i.as_table()).is_none();
if needs_table {
root.insert("projects", toml_edit::table());
created_projects_table = true;
// If `projects` exists but isn't a standard table (e.g., it's an inline table),
// convert it to an explicit table while preserving existing entries.
let existing_projects = root.get("projects").cloned();
if existing_projects.as_ref().is_none_or(|i| !i.is_table()) {
let mut projects_tbl = toml_edit::Table::new();
projects_tbl.set_implicit(true);
// If there was an existing inline table, migrate its entries to explicit tables.
if let Some(inline_tbl) = existing_projects.as_ref().and_then(|i| i.as_inline_table()) {
for (k, v) in inline_tbl.iter() {
if let Some(inner_tbl) = v.as_inline_table() {
let new_tbl = inner_tbl.clone().into_table();
projects_tbl.insert(k, toml_edit::Item::Table(new_tbl));
}
}
}
root.insert("projects", toml_edit::Item::Table(projects_tbl));
}
}
let Some(projects_tbl) = doc["projects"].as_table_mut() else {
@@ -299,12 +308,6 @@ pub fn set_project_trusted(codex_home: &Path, project_path: &Path) -> anyhow::Re
));
};
// If we created the `projects` table ourselves, keep it implicit so we
// don't render a standalone `[projects]` header.
if created_projects_table {
projects_tbl.set_implicit(true);
}
// Ensure the per-project entry is its own explicit table. If it exists but
// is not a table (e.g., an inline table), replace it with an explicit table.
let needs_proj_table = !projects_tbl.contains_key(project_key.as_str())
@@ -323,6 +326,21 @@ pub fn set_project_trusted(codex_home: &Path, project_path: &Path) -> anyhow::Re
};
proj_tbl.set_implicit(false);
proj_tbl["trust_level"] = toml_edit::value("trusted");
Ok(())
}
/// Patch `CODEX_HOME/config.toml` project state.
/// Use with caution.
pub fn set_project_trusted(codex_home: &Path, project_path: &Path) -> anyhow::Result<()> {
let config_path = codex_home.join(CONFIG_TOML_FILE);
// Parse existing config if present; otherwise start a new document.
let mut doc = match std::fs::read_to_string(config_path.clone()) {
Ok(s) => s.parse::<DocumentMut>()?,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => DocumentMut::new(),
Err(e) => return Err(e.into()),
};
set_project_trusted_inner(&mut doc, project_path)?;
// ensure codex_home exists
std::fs::create_dir_all(codex_home)?;
@@ -472,6 +490,7 @@ pub struct ConfigToml {
pub experimental_instructions_file: Option<PathBuf>,
pub experimental_use_exec_command_tool: Option<bool>,
pub experimental_use_unified_exec_tool: Option<bool>,
pub projects: Option<HashMap<String, ProjectConfig>>,
@@ -653,7 +672,11 @@ impl Config {
tools_web_search_request: override_tools_web_search_request,
} = overrides;
let config_profile = match config_profile_key.as_ref().or(cfg.profile.as_ref()) {
let active_profile_name = config_profile_key
.as_ref()
.or(cfg.profile.as_ref())
.cloned();
let config_profile = match active_profile_name.as_ref() {
Some(key) => cfg
.profiles
.get(key)
@@ -818,7 +841,11 @@ impl Config {
use_experimental_streamable_shell_tool: cfg
.experimental_use_exec_command_tool
.unwrap_or(false),
use_experimental_unified_exec_tool: cfg
.experimental_use_unified_exec_tool
.unwrap_or(true),
include_view_image_tool,
active_profile: active_profile_name,
disable_paste_burst: cfg.disable_paste_burst.unwrap_or(false),
};
Ok(config)
@@ -1192,7 +1219,9 @@ model_verbosity = "high"
tools_web_search_request: false,
preferred_auth_method: AuthMode::ChatGPT,
use_experimental_streamable_shell_tool: false,
use_experimental_unified_exec_tool: true,
include_view_image_tool: true,
active_profile: Some("o3".to_string()),
disable_paste_burst: false,
},
o3_profile_config
@@ -1248,7 +1277,9 @@ model_verbosity = "high"
tools_web_search_request: false,
preferred_auth_method: AuthMode::ChatGPT,
use_experimental_streamable_shell_tool: false,
use_experimental_unified_exec_tool: true,
include_view_image_tool: true,
active_profile: Some("gpt3".to_string()),
disable_paste_burst: false,
};
@@ -1319,7 +1350,9 @@ model_verbosity = "high"
tools_web_search_request: false,
preferred_auth_method: AuthMode::ChatGPT,
use_experimental_streamable_shell_tool: false,
use_experimental_unified_exec_tool: true,
include_view_image_tool: true,
active_profile: Some("zdr".to_string()),
disable_paste_burst: false,
};
@@ -1376,7 +1409,9 @@ model_verbosity = "high"
tools_web_search_request: false,
preferred_auth_method: AuthMode::ChatGPT,
use_experimental_streamable_shell_tool: false,
use_experimental_unified_exec_tool: true,
include_view_image_tool: true,
active_profile: Some("gpt5".to_string()),
disable_paste_burst: false,
};
@@ -1387,17 +1422,14 @@ model_verbosity = "high"
#[test]
fn test_set_project_trusted_writes_explicit_tables() -> anyhow::Result<()> {
let codex_home = TempDir::new().unwrap();
let project_dir = TempDir::new().unwrap();
let project_dir = Path::new("/some/path");
let mut doc = DocumentMut::new();
// Call the function under test
set_project_trusted(codex_home.path(), project_dir.path())?;
set_project_trusted_inner(&mut doc, project_dir)?;
// Read back the generated config.toml and assert exact contents
let config_path = codex_home.path().join(CONFIG_TOML_FILE);
let contents = std::fs::read_to_string(&config_path)?;
let contents = doc.to_string();
let raw_path = project_dir.path().to_string_lossy();
let raw_path = project_dir.to_string_lossy();
let path_str = if raw_path.contains('\\') {
format!("'{raw_path}'")
} else {
@@ -1415,12 +1447,10 @@ trust_level = "trusted"
#[test]
fn test_set_project_trusted_converts_inline_to_explicit() -> anyhow::Result<()> {
let codex_home = TempDir::new().unwrap();
let project_dir = TempDir::new().unwrap();
let project_dir = Path::new("/some/path");
// Seed config.toml with an inline project entry under [projects]
let config_path = codex_home.path().join(CONFIG_TOML_FILE);
let raw_path = project_dir.path().to_string_lossy();
let raw_path = project_dir.to_string_lossy();
let path_str = if raw_path.contains('\\') {
format!("'{raw_path}'")
} else {
@@ -1432,13 +1462,12 @@ trust_level = "trusted"
{path_str} = {{ trust_level = "untrusted" }}
"#
);
std::fs::create_dir_all(codex_home.path())?;
std::fs::write(&config_path, initial)?;
let mut doc = initial.parse::<DocumentMut>()?;
// Run the function; it should convert to explicit tables and set trusted
set_project_trusted(codex_home.path(), project_dir.path())?;
set_project_trusted_inner(&mut doc, project_dir)?;
let contents = std::fs::read_to_string(&config_path)?;
let contents = doc.to_string();
// Assert exact output after conversion to explicit table
let expected = format!(
@@ -1453,5 +1482,37 @@ trust_level = "trusted"
Ok(())
}
// No test enforcing the presence of a standalone [projects] header.
#[test]
fn test_set_project_trusted_migrates_top_level_inline_projects_preserving_entries()
-> anyhow::Result<()> {
let initial = r#"toplevel = "baz"
projects = { "/Users/mbolin/code/codex4" = { trust_level = "trusted", foo = "bar" } , "/Users/mbolin/code/codex3" = { trust_level = "trusted" } }
model = "foo""#;
let mut doc = initial.parse::<DocumentMut>()?;
// Approve a new directory
let new_project = Path::new("/Users/mbolin/code/codex2");
set_project_trusted_inner(&mut doc, new_project)?;
let contents = doc.to_string();
// Since we created the [projects] table as part of migration, it is kept implicit.
// Expect explicit per-project tables, preserving prior entries and appending the new one.
let expected = r#"toplevel = "baz"
model = "foo"
[projects."/Users/mbolin/code/codex4"]
trust_level = "trusted"
foo = "bar"
[projects."/Users/mbolin/code/codex3"]
trust_level = "trusted"
[projects."/Users/mbolin/code/codex2"]
trust_level = "trusted"
"#;
assert_eq!(contents, expected);
Ok(())
}
}

View File

@@ -0,0 +1,582 @@
use crate::config::CONFIG_TOML_FILE;
use anyhow::Result;
use std::path::Path;
use tempfile::NamedTempFile;
use toml_edit::DocumentMut;
pub const CONFIG_KEY_MODEL: &str = "model";
pub const CONFIG_KEY_EFFORT: &str = "model_reasoning_effort";
/// Persist overrides into `config.toml` using explicit key segments per
/// override. This avoids ambiguity with keys that contain dots or spaces.
pub async fn persist_overrides(
codex_home: &Path,
profile: Option<&str>,
overrides: &[(&[&str], &str)],
) -> Result<()> {
let config_path = codex_home.join(CONFIG_TOML_FILE);
let mut doc = match tokio::fs::read_to_string(&config_path).await {
Ok(s) => s.parse::<DocumentMut>()?,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
tokio::fs::create_dir_all(codex_home).await?;
DocumentMut::new()
}
Err(e) => return Err(e.into()),
};
let effective_profile = if let Some(p) = profile {
Some(p.to_owned())
} else {
doc.get("profile")
.and_then(|i| i.as_str())
.map(|s| s.to_string())
};
for (segments, val) in overrides.iter().copied() {
let value = toml_edit::value(val);
if let Some(ref name) = effective_profile {
if segments.first().copied() == Some("profiles") {
apply_toml_edit_override_segments(&mut doc, segments, value);
} else {
let mut seg_buf: Vec<&str> = Vec::with_capacity(2 + segments.len());
seg_buf.push("profiles");
seg_buf.push(name.as_str());
seg_buf.extend_from_slice(segments);
apply_toml_edit_override_segments(&mut doc, &seg_buf, value);
}
} else {
apply_toml_edit_override_segments(&mut doc, segments, value);
}
}
let tmp_file = NamedTempFile::new_in(codex_home)?;
tokio::fs::write(tmp_file.path(), doc.to_string()).await?;
tmp_file.persist(config_path)?;
Ok(())
}
/// Persist overrides where values may be optional. Any entries with `None`
/// values are skipped. If all values are `None`, this becomes a no-op and
/// returns `Ok(())` without touching the file.
pub async fn persist_non_null_overrides(
codex_home: &Path,
profile: Option<&str>,
overrides: &[(&[&str], Option<&str>)],
) -> Result<()> {
let filtered: Vec<(&[&str], &str)> = overrides
.iter()
.filter_map(|(k, v)| v.map(|vv| (*k, vv)))
.collect();
if filtered.is_empty() {
return Ok(());
}
persist_overrides(codex_home, profile, &filtered).await
}
/// Apply a single override onto a `toml_edit` document while preserving
/// existing formatting/comments.
/// The key is expressed as explicit segments to correctly handle keys that
/// contain dots or spaces.
fn apply_toml_edit_override_segments(
doc: &mut DocumentMut,
segments: &[&str],
value: toml_edit::Item,
) {
use toml_edit::Item;
if segments.is_empty() {
return;
}
let mut current = doc.as_table_mut();
for seg in &segments[..segments.len() - 1] {
if !current.contains_key(seg) {
current[*seg] = Item::Table(toml_edit::Table::new());
if let Some(t) = current[*seg].as_table_mut() {
t.set_implicit(true);
}
}
let maybe_item = current.get_mut(seg);
let Some(item) = maybe_item else { return };
if !item.is_table() {
*item = Item::Table(toml_edit::Table::new());
if let Some(t) = item.as_table_mut() {
t.set_implicit(true);
}
}
let Some(tbl) = item.as_table_mut() else {
return;
};
current = tbl;
}
let last = segments[segments.len() - 1];
current[last] = value;
}
#[cfg(test)]
mod tests {
use super::*;
use pretty_assertions::assert_eq;
use tempfile::tempdir;
/// Verifies model and effort are written at top-level when no profile is set.
#[tokio::test]
async fn set_default_model_and_effort_top_level_when_no_profile() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
persist_overrides(
codex_home,
None,
&[
(&[CONFIG_KEY_MODEL], "gpt-5"),
(&[CONFIG_KEY_EFFORT], "high"),
],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"model = "gpt-5"
model_reasoning_effort = "high"
"#;
assert_eq!(contents, expected);
}
/// Verifies values are written under the active profile when `profile` is set.
#[tokio::test]
async fn set_defaults_update_profile_when_profile_set() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// Seed config with a profile selection but without profiles table
let seed = "profile = \"o3\"\n";
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
persist_overrides(
codex_home,
None,
&[
(&[CONFIG_KEY_MODEL], "o3"),
(&[CONFIG_KEY_EFFORT], "minimal"),
],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"profile = "o3"
[profiles.o3]
model = "o3"
model_reasoning_effort = "minimal"
"#;
assert_eq!(contents, expected);
}
/// Verifies profile names with dots/spaces are preserved via explicit segments.
#[tokio::test]
async fn set_defaults_update_profile_with_dot_and_space() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// Seed config with a profile name that contains a dot and a space
let seed = "profile = \"my.team name\"\n";
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
persist_overrides(
codex_home,
None,
&[
(&[CONFIG_KEY_MODEL], "o3"),
(&[CONFIG_KEY_EFFORT], "minimal"),
],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"profile = "my.team name"
[profiles."my.team name"]
model = "o3"
model_reasoning_effort = "minimal"
"#;
assert_eq!(contents, expected);
}
/// Verifies explicit profile override writes under that profile even without active profile.
#[tokio::test]
async fn set_defaults_update_when_profile_override_supplied() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// No profile key in config.toml
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), "")
.await
.expect("seed write");
// Persist with an explicit profile override
persist_overrides(
codex_home,
Some("o3"),
&[(&[CONFIG_KEY_MODEL], "o3"), (&[CONFIG_KEY_EFFORT], "high")],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"[profiles.o3]
model = "o3"
model_reasoning_effort = "high"
"#;
assert_eq!(contents, expected);
}
/// Verifies nested tables are created as needed when applying overrides.
#[tokio::test]
async fn persist_overrides_creates_nested_tables() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
persist_overrides(
codex_home,
None,
&[
(&["a", "b", "c"], "v"),
(&["x"], "y"),
(&["profiles", "p1", CONFIG_KEY_MODEL], "gpt-5"),
],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"x = "y"
[a.b]
c = "v"
[profiles.p1]
model = "gpt-5"
"#;
assert_eq!(contents, expected);
}
/// Verifies a scalar key becomes a table when nested keys are written.
#[tokio::test]
async fn persist_overrides_replaces_scalar_with_table() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
let seed = "foo = \"bar\"\n";
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
persist_overrides(codex_home, None, &[(&["foo", "bar", "baz"], "ok")])
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"[foo.bar]
baz = "ok"
"#;
assert_eq!(contents, expected);
}
/// Verifies comments and spacing are preserved when writing under active profile.
#[tokio::test]
async fn set_defaults_preserve_comments() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// Seed a config with comments and spacing we expect to preserve
let seed = r#"# Global comment
# Another line
profile = "o3"
# Profile settings
[profiles.o3]
# keep me
existing = "keep"
"#;
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
// Apply defaults; since profile is set, it should write under [profiles.o3]
persist_overrides(
codex_home,
None,
&[(&[CONFIG_KEY_MODEL], "o3"), (&[CONFIG_KEY_EFFORT], "high")],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"# Global comment
# Another line
profile = "o3"
# Profile settings
[profiles.o3]
# keep me
existing = "keep"
model = "o3"
model_reasoning_effort = "high"
"#;
assert_eq!(contents, expected);
}
/// Verifies comments and spacing are preserved when writing at top level.
#[tokio::test]
async fn set_defaults_preserve_global_comments() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// Seed a config WITHOUT a profile, containing comments and spacing
let seed = r#"# Top-level comments
# should be preserved
existing = "keep"
"#;
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
// Since there is no profile, the defaults should be written at top-level
persist_overrides(
codex_home,
None,
&[
(&[CONFIG_KEY_MODEL], "gpt-5"),
(&[CONFIG_KEY_EFFORT], "minimal"),
],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"# Top-level comments
# should be preserved
existing = "keep"
model = "gpt-5"
model_reasoning_effort = "minimal"
"#;
assert_eq!(contents, expected);
}
/// Verifies errors on invalid TOML propagate and file is not clobbered.
#[tokio::test]
async fn persist_overrides_errors_on_parse_failure() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// Write an intentionally invalid TOML file
let invalid = "invalid = [unclosed";
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), invalid)
.await
.expect("seed write");
// Attempting to persist should return an error and must not clobber the file.
let res = persist_overrides(codex_home, None, &[(&["x"], "y")]).await;
assert!(res.is_err(), "expected parse error to propagate");
// File should be unchanged
let contents = read_config(codex_home).await;
assert_eq!(contents, invalid);
}
/// Verifies changing model only preserves existing effort at top-level.
#[tokio::test]
async fn changing_only_model_preserves_existing_effort_top_level() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// Seed with an effort value only
let seed = "model_reasoning_effort = \"minimal\"\n";
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
// Change only the model
persist_overrides(codex_home, None, &[(&[CONFIG_KEY_MODEL], "o3")])
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"model_reasoning_effort = "minimal"
model = "o3"
"#;
assert_eq!(contents, expected);
}
/// Verifies changing effort only preserves existing model at top-level.
#[tokio::test]
async fn changing_only_effort_preserves_existing_model_top_level() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// Seed with a model value only
let seed = "model = \"gpt-5\"\n";
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
// Change only the effort
persist_overrides(codex_home, None, &[(&[CONFIG_KEY_EFFORT], "high")])
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"model = "gpt-5"
model_reasoning_effort = "high"
"#;
assert_eq!(contents, expected);
}
/// Verifies changing model only preserves existing effort in active profile.
#[tokio::test]
async fn changing_only_model_preserves_effort_in_active_profile() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// Seed with an active profile and an existing effort under that profile
let seed = r#"profile = "p1"
[profiles.p1]
model_reasoning_effort = "low"
"#;
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
persist_overrides(codex_home, None, &[(&[CONFIG_KEY_MODEL], "o4-mini")])
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"profile = "p1"
[profiles.p1]
model_reasoning_effort = "low"
model = "o4-mini"
"#;
assert_eq!(contents, expected);
}
/// Verifies changing effort only preserves existing model in a profile override.
#[tokio::test]
async fn changing_only_effort_preserves_model_in_profile_override() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
// No active profile key; we'll target an explicit override
let seed = r#"[profiles.team]
model = "gpt-5"
"#;
tokio::fs::write(codex_home.join(CONFIG_TOML_FILE), seed)
.await
.expect("seed write");
persist_overrides(
codex_home,
Some("team"),
&[(&[CONFIG_KEY_EFFORT], "minimal")],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"[profiles.team]
model = "gpt-5"
model_reasoning_effort = "minimal"
"#;
assert_eq!(contents, expected);
}
/// Verifies `persist_non_null_overrides` skips `None` entries and writes only present values at top-level.
#[tokio::test]
async fn persist_non_null_skips_none_top_level() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
persist_non_null_overrides(
codex_home,
None,
&[
(&[CONFIG_KEY_MODEL], Some("gpt-5")),
(&[CONFIG_KEY_EFFORT], None),
],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = "model = \"gpt-5\"\n";
assert_eq!(contents, expected);
}
/// Verifies no-op behavior when all provided overrides are `None` (no file created/modified).
#[tokio::test]
async fn persist_non_null_noop_when_all_none() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
persist_non_null_overrides(
codex_home,
None,
&[(&["a"], None), (&["profiles", "p", "x"], None)],
)
.await
.expect("persist");
// Should not create config.toml on a pure no-op
assert!(!codex_home.join(CONFIG_TOML_FILE).exists());
}
/// Verifies entries are written under the specified profile and `None` entries are skipped.
#[tokio::test]
async fn persist_non_null_respects_profile_override() {
let tmpdir = tempdir().expect("tmp");
let codex_home = tmpdir.path();
persist_non_null_overrides(
codex_home,
Some("team"),
&[
(&[CONFIG_KEY_MODEL], Some("o3")),
(&[CONFIG_KEY_EFFORT], None),
],
)
.await
.expect("persist");
let contents = read_config(codex_home).await;
let expected = r#"[profiles.team]
model = "o3"
"#;
assert_eq!(contents, expected);
}
// Test helper moved to bottom per review guidance.
async fn read_config(codex_home: &Path) -> String {
let p = codex_home.join(CONFIG_TOML_FILE);
tokio::fs::read_to_string(p).await.unwrap_or_default()
}
}

View File

@@ -150,13 +150,13 @@ impl ConversationManager {
/// caller's `config`). The new conversation will have a fresh id.
pub async fn fork_conversation(
&self,
conversation_history: Vec<ResponseItem>,
num_messages_to_drop: usize,
config: Config,
path: PathBuf,
) -> CodexResult<NewConversation> {
// Compute the prefix up to the cut point.
let history =
truncate_after_dropping_last_messages(conversation_history, num_messages_to_drop);
let history = RolloutRecorder::get_rollout_history(&path).await?;
let history = truncate_after_dropping_last_messages(history, num_messages_to_drop);
// Spawn a new conversation with the computed initial history.
let auth_manager = self.auth_manager.clone();
@@ -171,36 +171,36 @@ impl ConversationManager {
/// Return a prefix of `items` obtained by dropping the last `n` user messages
/// and all items that follow them.
fn truncate_after_dropping_last_messages(items: Vec<ResponseItem>, n: usize) -> InitialHistory {
fn truncate_after_dropping_last_messages(history: InitialHistory, n: usize) -> InitialHistory {
if n == 0 {
let rolled: Vec<RolloutItem> = items.into_iter().map(RolloutItem::ResponseItem).collect();
return InitialHistory::Forked(rolled);
return InitialHistory::Forked(history.get_rollout_items());
}
// Walk backwards counting only `user` Message items, find cut index.
let mut count = 0usize;
let mut cut_index = 0usize;
for (idx, item) in items.iter().enumerate().rev() {
if let ResponseItem::Message { role, .. } = item
// Work directly on rollout items, and cut the vector at the nth-from-last user message input.
let items: Vec<RolloutItem> = history.get_rollout_items();
// Find indices of user message inputs in rollout order.
let mut user_positions: Vec<usize> = Vec::new();
for (idx, item) in items.iter().enumerate() {
if let RolloutItem::ResponseItem(ResponseItem::Message { role, .. }) = item
&& role == "user"
{
count += 1;
if count == n {
// Cut everything from this user message to the end.
cut_index = idx;
break;
}
user_positions.push(idx);
}
}
if cut_index == 0 {
// No prefix remains after dropping; start a new conversation.
// If fewer than n user messages exist, treat as empty.
if user_positions.len() < n {
return InitialHistory::New;
}
// Cut strictly before the nth-from-last user message (do not keep the nth itself).
let cut_idx = user_positions[user_positions.len() - n];
let rolled: Vec<RolloutItem> = items.into_iter().take(cut_idx).collect();
if rolled.is_empty() {
InitialHistory::New
} else {
let rolled: Vec<RolloutItem> = items
.into_iter()
.take(cut_index)
.map(RolloutItem::ResponseItem)
.collect();
InitialHistory::Forked(rolled)
}
}
@@ -256,7 +256,13 @@ mod tests {
assistant_msg("a4"),
];
let truncated = truncate_after_dropping_last_messages(items.clone(), 1);
// Wrap as InitialHistory::Forked with response items only.
let initial: Vec<RolloutItem> = items
.iter()
.cloned()
.map(RolloutItem::ResponseItem)
.collect();
let truncated = truncate_after_dropping_last_messages(InitialHistory::Forked(initial), 1);
let got_items = truncated.get_rollout_items();
let expected_items = vec![
RolloutItem::ResponseItem(items[0].clone()),
@@ -268,7 +274,12 @@ mod tests {
serde_json::to_value(&expected_items).unwrap()
);
let truncated2 = truncate_after_dropping_last_messages(items, 2);
let initial2: Vec<RolloutItem> = items
.iter()
.cloned()
.map(RolloutItem::ResponseItem)
.collect();
let truncated2 = truncate_after_dropping_last_messages(InitialHistory::Forked(initial2), 2);
assert!(matches!(truncated2, InitialHistory::New));
}
}

View File

@@ -26,6 +26,7 @@ pub(crate) struct EnvironmentContext {
pub approval_policy: Option<AskForApproval>,
pub sandbox_mode: Option<SandboxMode>,
pub network_access: Option<NetworkAccess>,
pub writable_roots: Option<Vec<PathBuf>>,
pub shell: Option<Shell>,
}
@@ -57,6 +58,16 @@ impl EnvironmentContext {
}
None => None,
},
writable_roots: match sandbox_policy {
Some(SandboxPolicy::WorkspaceWrite { writable_roots, .. }) => {
if writable_roots.is_empty() {
None
} else {
Some(writable_roots.clone())
}
}
_ => None,
},
shell,
}
}
@@ -72,6 +83,7 @@ impl EnvironmentContext {
/// <cwd>...</cwd>
/// <approval_policy>...</approval_policy>
/// <sandbox_mode>...</sandbox_mode>
/// <writable_roots>...</writable_roots>
/// <network_access>...</network_access>
/// <shell>...</shell>
/// </environment_context>
@@ -94,6 +106,16 @@ impl EnvironmentContext {
" <network_access>{network_access}</network_access>"
));
}
if let Some(writable_roots) = self.writable_roots {
lines.push(" <writable_roots>".to_string());
for writable_root in writable_roots {
lines.push(format!(
" <root>{}</root>",
writable_root.to_string_lossy()
));
}
lines.push(" </writable_roots>".to_string());
}
if let Some(shell) = self.shell
&& let Some(shell_name) = shell.name()
{
@@ -115,3 +137,77 @@ impl From<EnvironmentContext> for ResponseItem {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use pretty_assertions::assert_eq;
fn workspace_write_policy(writable_roots: Vec<&str>, network_access: bool) -> SandboxPolicy {
SandboxPolicy::WorkspaceWrite {
writable_roots: writable_roots.into_iter().map(PathBuf::from).collect(),
network_access,
exclude_tmpdir_env_var: false,
exclude_slash_tmp: false,
}
}
#[test]
fn serialize_workspace_write_environment_context() {
let context = EnvironmentContext::new(
Some(PathBuf::from("/repo")),
Some(AskForApproval::OnRequest),
Some(workspace_write_policy(vec!["/repo", "/tmp"], false)),
None,
);
let expected = r#"<environment_context>
<cwd>/repo</cwd>
<approval_policy>on-request</approval_policy>
<sandbox_mode>workspace-write</sandbox_mode>
<network_access>restricted</network_access>
<writable_roots>
<root>/repo</root>
<root>/tmp</root>
</writable_roots>
</environment_context>"#;
assert_eq!(context.serialize_to_xml(), expected);
}
#[test]
fn serialize_read_only_environment_context() {
let context = EnvironmentContext::new(
None,
Some(AskForApproval::Never),
Some(SandboxPolicy::ReadOnly),
None,
);
let expected = r#"<environment_context>
<approval_policy>never</approval_policy>
<sandbox_mode>read-only</sandbox_mode>
<network_access>restricted</network_access>
</environment_context>"#;
assert_eq!(context.serialize_to_xml(), expected);
}
#[test]
fn serialize_full_access_environment_context() {
let context = EnvironmentContext::new(
None,
Some(AskForApproval::OnFailure),
Some(SandboxPolicy::DangerFullAccess),
None,
);
let expected = r#"<environment_context>
<approval_policy>on-failure</approval_policy>
<sandbox_mode>danger-full-access</sandbox_mode>
<network_access>enabled</network_access>
</environment_context>"#;
assert_eq!(context.serialize_to_xml(), expected);
}
}

View File

@@ -24,6 +24,9 @@ pub(crate) struct ExecCommandSession {
/// JoinHandle for the child wait task.
wait_handle: StdMutex<Option<JoinHandle<()>>>,
/// Tracks whether the underlying process has exited.
exit_status: std::sync::Arc<std::sync::atomic::AtomicBool>,
}
impl ExecCommandSession {
@@ -34,6 +37,7 @@ impl ExecCommandSession {
reader_handle: JoinHandle<()>,
writer_handle: JoinHandle<()>,
wait_handle: JoinHandle<()>,
exit_status: std::sync::Arc<std::sync::atomic::AtomicBool>,
) -> Self {
Self {
writer_tx,
@@ -42,6 +46,7 @@ impl ExecCommandSession {
reader_handle: StdMutex::new(Some(reader_handle)),
writer_handle: StdMutex::new(Some(writer_handle)),
wait_handle: StdMutex::new(Some(wait_handle)),
exit_status,
}
}
@@ -52,6 +57,10 @@ impl ExecCommandSession {
pub(crate) fn output_receiver(&self) -> broadcast::Receiver<Vec<u8>> {
self.output_tx.subscribe()
}
pub(crate) fn has_exited(&self) -> bool {
self.exit_status.load(std::sync::atomic::Ordering::SeqCst)
}
}
impl Drop for ExecCommandSession {

View File

@@ -6,6 +6,7 @@ mod session_manager;
pub use exec_command_params::ExecCommandParams;
pub use exec_command_params::WriteStdinParams;
pub(crate) use exec_command_session::ExecCommandSession;
pub use responses_api::EXEC_COMMAND_TOOL_NAME;
pub use responses_api::WRITE_STDIN_TOOL_NAME;
pub use responses_api::create_exec_command_tool_for_responses_api;

View File

@@ -3,6 +3,7 @@ use std::io::ErrorKind;
use std::io::Read;
use std::sync::Arc;
use std::sync::Mutex as StdMutex;
use std::sync::atomic::AtomicBool;
use std::sync::atomic::AtomicU32;
use portable_pty::CommandBuilder;
@@ -19,6 +20,7 @@ use crate::exec_command::exec_command_params::ExecCommandParams;
use crate::exec_command::exec_command_params::WriteStdinParams;
use crate::exec_command::exec_command_session::ExecCommandSession;
use crate::exec_command::session_id::SessionId;
use crate::truncate::truncate_middle;
use codex_protocol::models::FunctionCallOutputPayload;
#[derive(Debug, Default)]
@@ -327,11 +329,14 @@ async fn create_exec_command_session(
// Keep the child alive until it exits, then signal exit code.
let (exit_tx, exit_rx) = oneshot::channel::<i32>();
let exit_status = Arc::new(AtomicBool::new(false));
let wait_exit_status = exit_status.clone();
let wait_handle = tokio::task::spawn_blocking(move || {
let code = match child.wait() {
Ok(status) => status.exit_code() as i32,
Err(_) => -1,
};
wait_exit_status.store(true, std::sync::atomic::Ordering::SeqCst);
let _ = exit_tx.send(code);
});
@@ -343,116 +348,11 @@ async fn create_exec_command_session(
reader_handle,
writer_handle,
wait_handle,
exit_status,
);
Ok((session, exit_rx))
}
/// Truncate the middle of a UTF-8 string to at most `max_bytes` bytes,
/// preserving the beginning and the end. Returns the possibly truncated
/// string and `Some(original_token_count)` (estimated at 4 bytes/token)
/// if truncation occurred; otherwise returns the original string and `None`.
fn truncate_middle(s: &str, max_bytes: usize) -> (String, Option<u64>) {
// No truncation needed
if s.len() <= max_bytes {
return (s.to_string(), None);
}
let est_tokens = (s.len() as u64).div_ceil(4);
if max_bytes == 0 {
// Cannot keep any content; still return a full marker (never truncated).
return (format!("{est_tokens} tokens truncated…"), Some(est_tokens));
}
// Helper to truncate a string to a given byte length on a char boundary.
fn truncate_on_boundary(input: &str, max_len: usize) -> &str {
if input.len() <= max_len {
return input;
}
let mut end = max_len;
while end > 0 && !input.is_char_boundary(end) {
end -= 1;
}
&input[..end]
}
// Given a left/right budget, prefer newline boundaries; otherwise fall back
// to UTF-8 char boundaries.
fn pick_prefix_end(s: &str, left_budget: usize) -> usize {
if let Some(head) = s.get(..left_budget)
&& let Some(i) = head.rfind('\n')
{
return i + 1; // keep the newline so suffix starts on a fresh line
}
truncate_on_boundary(s, left_budget).len()
}
fn pick_suffix_start(s: &str, right_budget: usize) -> usize {
let start_tail = s.len().saturating_sub(right_budget);
if let Some(tail) = s.get(start_tail..)
&& let Some(i) = tail.find('\n')
{
return start_tail + i + 1; // start after newline
}
// Fall back to a char boundary at or after start_tail.
let mut idx = start_tail.min(s.len());
while idx < s.len() && !s.is_char_boundary(idx) {
idx += 1;
}
idx
}
// Refine marker length and budgets until stable. Marker is never truncated.
let mut guess_tokens = est_tokens; // worst-case: everything truncated
for _ in 0..4 {
let marker = format!("{guess_tokens} tokens truncated…");
let marker_len = marker.len();
let keep_budget = max_bytes.saturating_sub(marker_len);
if keep_budget == 0 {
// No room for any content within the cap; return a full, untruncated marker
// that reflects the entire truncated content.
return (format!("{est_tokens} tokens truncated…"), Some(est_tokens));
}
let left_budget = keep_budget / 2;
let right_budget = keep_budget - left_budget;
let prefix_end = pick_prefix_end(s, left_budget);
let mut suffix_start = pick_suffix_start(s, right_budget);
if suffix_start < prefix_end {
suffix_start = prefix_end;
}
let kept_content_bytes = prefix_end + (s.len() - suffix_start);
let truncated_content_bytes = s.len().saturating_sub(kept_content_bytes);
let new_tokens = (truncated_content_bytes as u64).div_ceil(4);
if new_tokens == guess_tokens {
let mut out = String::with_capacity(marker_len + kept_content_bytes + 1);
out.push_str(&s[..prefix_end]);
out.push_str(&marker);
// Place marker on its own line for symmetry when we keep line boundaries.
out.push('\n');
out.push_str(&s[suffix_start..]);
return (out, Some(est_tokens));
}
guess_tokens = new_tokens;
}
// Fallback: use last guess to build output.
let marker = format!("{guess_tokens} tokens truncated…");
let marker_len = marker.len();
let keep_budget = max_bytes.saturating_sub(marker_len);
if keep_budget == 0 {
return (format!("{est_tokens} tokens truncated…"), Some(est_tokens));
}
let left_budget = keep_budget / 2;
let right_budget = keep_budget - left_budget;
let prefix_end = pick_prefix_end(s, left_budget);
let suffix_start = pick_suffix_start(s, right_budget);
let mut out = String::with_capacity(marker_len + prefix_end + (s.len() - suffix_start) + 1);
out.push_str(&s[..prefix_end]);
out.push_str(&marker);
out.push('\n');
out.push_str(&s[suffix_start..]);
(out, Some(est_tokens))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -616,50 +516,4 @@ Output:
abc"#;
assert_eq!(expected, text);
}
#[test]
fn truncate_middle_no_newlines_fallback() {
// A long string with no newlines that exceeds the cap.
let s = "abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ";
let max_bytes = 16; // force truncation
let (out, original) = truncate_middle(s, max_bytes);
// For very small caps, we return the full, untruncated marker,
// even if it exceeds the cap.
assert_eq!(out, "…16 tokens truncated…");
// Original string length is 62 bytes => ceil(62/4) = 16 tokens.
assert_eq!(original, Some(16));
}
#[test]
fn truncate_middle_prefers_newline_boundaries() {
// Build a multi-line string of 20 numbered lines (each "NNN\n").
let mut s = String::new();
for i in 1..=20 {
s.push_str(&format!("{i:03}\n"));
}
// Total length: 20 lines * 4 bytes per line = 80 bytes.
assert_eq!(s.len(), 80);
// Choose a cap that forces truncation while leaving room for
// a few lines on each side after accounting for the marker.
let max_bytes = 64;
// Expect exact output: first 4 lines, marker, last 4 lines, and correct token estimate (80/4 = 20).
assert_eq!(
truncate_middle(&s, max_bytes),
(
r#"001
002
003
004
…12 tokens truncated…
017
018
019
020
"#
.to_string(),
Some(20)
)
);
}
}

View File

@@ -16,6 +16,7 @@ mod codex_conversation;
pub mod token_data;
pub use codex_conversation::CodexConversation;
pub mod config;
pub mod config_edit;
pub mod config_profile;
pub mod config_types;
mod conversation_history;
@@ -34,6 +35,8 @@ mod mcp_tool_call;
mod message_history;
mod model_provider_info;
pub mod parse_command;
mod truncate;
mod unified_exec;
mod user_instructions;
pub use model_provider_info::BUILT_IN_OSS_MODEL_PROVIDER_ID;
pub use model_provider_info::ModelProviderInfo;

View File

@@ -17,7 +17,7 @@ use anyhow::Result;
use anyhow::anyhow;
use codex_mcp_client::McpClient;
use mcp_types::ClientCapabilities;
use mcp_types::McpClientInfo;
use mcp_types::Implementation;
use mcp_types::Tool;
use serde_json::json;
@@ -159,10 +159,14 @@ impl McpConnectionManager {
// indicates this should be an empty object.
elicitation: Some(json!({})),
},
client_info: McpClientInfo {
client_info: Implementation {
name: "codex-mcp-client".to_owned(),
version: env!("CARGO_PKG_VERSION").to_owned(),
title: Some("Codex".into()),
// This field is used by Codex when it is an MCP
// server: it should not be used when Codex is
// an MCP client.
user_agent: None,
},
protocol_version: mcp_types::MCP_SCHEMA_VERSION.to_owned(),
};

View File

@@ -8,7 +8,6 @@ use std::collections::HashMap;
use crate::model_family::ModelFamily;
use crate::plan_tool::PLAN_TOOL;
use crate::protocol::AskForApproval;
use crate::protocol::SandboxPolicy;
use crate::tool_apply_patch::ApplyPatchToolType;
use crate::tool_apply_patch::create_apply_patch_freeform_tool;
use crate::tool_apply_patch::create_apply_patch_json_tool;
@@ -58,7 +57,7 @@ pub(crate) enum OpenAiTool {
#[derive(Debug, Clone)]
pub enum ConfigShellToolType {
DefaultShell,
ShellWithRequest { sandbox_policy: SandboxPolicy },
ShellWithRequest,
LocalShell,
StreamableShell,
}
@@ -70,17 +69,18 @@ pub(crate) struct ToolsConfig {
pub apply_patch_tool_type: Option<ApplyPatchToolType>,
pub web_search_request: bool,
pub include_view_image_tool: bool,
pub experimental_unified_exec_tool: bool,
}
pub(crate) struct ToolsConfigParams<'a> {
pub(crate) model_family: &'a ModelFamily,
pub(crate) approval_policy: AskForApproval,
pub(crate) sandbox_policy: SandboxPolicy,
pub(crate) include_plan_tool: bool,
pub(crate) include_apply_patch_tool: bool,
pub(crate) include_web_search_request: bool,
pub(crate) use_streamable_shell_tool: bool,
pub(crate) include_view_image_tool: bool,
pub(crate) experimental_unified_exec_tool: bool,
}
impl ToolsConfig {
@@ -88,12 +88,12 @@ impl ToolsConfig {
let ToolsConfigParams {
model_family,
approval_policy,
sandbox_policy,
include_plan_tool,
include_apply_patch_tool,
include_web_search_request,
use_streamable_shell_tool,
include_view_image_tool,
experimental_unified_exec_tool,
} = params;
let mut shell_type = if *use_streamable_shell_tool {
ConfigShellToolType::StreamableShell
@@ -103,9 +103,7 @@ impl ToolsConfig {
ConfigShellToolType::DefaultShell
};
if matches!(approval_policy, AskForApproval::OnRequest) && !use_streamable_shell_tool {
shell_type = ConfigShellToolType::ShellWithRequest {
sandbox_policy: sandbox_policy.clone(),
}
shell_type = ConfigShellToolType::ShellWithRequest;
}
let apply_patch_tool_type = match model_family.apply_patch_tool_type {
@@ -126,6 +124,7 @@ impl ToolsConfig {
apply_patch_tool_type,
web_search_request: *include_web_search_request,
include_view_image_tool: *include_view_image_tool,
experimental_unified_exec_tool: *experimental_unified_exec_tool,
}
}
}
@@ -200,7 +199,56 @@ fn create_shell_tool() -> OpenAiTool {
})
}
fn create_shell_tool_for_sandbox(sandbox_policy: &SandboxPolicy) -> OpenAiTool {
fn create_unified_exec_tool() -> OpenAiTool {
let mut properties = BTreeMap::new();
properties.insert(
"input".to_string(),
JsonSchema::Array {
items: Box::new(JsonSchema::String { description: None }),
description: Some(
"When no session_id is provided, treat the array as the command and arguments \
to launch. When session_id is set, concatenate the strings (in order) and write \
them to the session's stdin."
.to_string(),
),
},
);
properties.insert(
"session_id".to_string(),
JsonSchema::String {
description: Some(
"Identifier for an existing interactive session. If omitted, a new command \
is spawned."
.to_string(),
),
},
);
properties.insert(
"timeout_ms".to_string(),
JsonSchema::Number {
description: Some(
"Maximum time in milliseconds to wait for output after writing the input."
.to_string(),
),
},
);
OpenAiTool::Function(ResponsesApiTool {
name: "unified_exec".to_string(),
description:
"Runs a command in a PTY. Provide a session_id to reuse an existing interactive session.".to_string(),
strict: false,
parameters: JsonSchema::Object {
properties,
required: Some(vec!["input".to_string()]),
additional_properties: Some(false),
},
})
}
const SHELL_TOOL_DESCRIPTION: &str = r#"Runs a shell command and returns its output"#;
fn create_shell_tool_for_sandbox() -> OpenAiTool {
let mut properties = BTreeMap::new();
properties.insert(
"command".to_string(),
@@ -212,82 +260,29 @@ fn create_shell_tool_for_sandbox(sandbox_policy: &SandboxPolicy) -> OpenAiTool {
properties.insert(
"workdir".to_string(),
JsonSchema::String {
description: Some("The working directory to execute the command in".to_string()),
description: Some("Working directory to execute the command in.".to_string()),
},
);
properties.insert(
"timeout_ms".to_string(),
JsonSchema::Number {
description: Some("The timeout for the command in milliseconds".to_string()),
description: Some("Timeout for the command in milliseconds.".to_string()),
},
);
if matches!(sandbox_policy, SandboxPolicy::WorkspaceWrite { .. }) {
properties.insert(
properties.insert(
"with_escalated_permissions".to_string(),
JsonSchema::Boolean {
description: Some("Whether to request escalated permissions. Set to true if command needs to be run without sandbox restrictions".to_string()),
description: Some("Request escalated permissions, only for when a command would otherwise be blocked by the sandbox.".to_string()),
},
);
properties.insert(
properties.insert(
"justification".to_string(),
JsonSchema::String {
description: Some("Only set if with_escalated_permissions is true. 1-sentence explanation of why we want to run this command.".to_string()),
description: Some("Required if and only if with_escalated_permissions == true. One sentence explaining why escalation is needed (e.g., write outside CWD, network fetch, git commit).".to_string()),
},
);
}
let description = match sandbox_policy {
SandboxPolicy::WorkspaceWrite {
network_access,
writable_roots,
..
} => {
format!(
r#"
The shell tool is used to execute shell commands.
- When invoking the shell tool, your call will be running in a sandbox, and some shell commands will require escalated privileges:
- Types of actions that require escalated privileges:
- Writing files other than those in the writable roots
- writable roots:
{}{}
- Examples of commands that require escalated privileges:
- git commit
- npm install or pnpm install
- cargo build
- cargo test
- When invoking a command that will require escalated privileges:
- Provide the with_escalated_permissions parameter with the boolean value true
- Include a short, 1 sentence explanation for why we need to run with_escalated_permissions in the justification parameter."#,
writable_roots.iter().map(|wr| format!(" - {}", wr.to_string_lossy())).collect::<Vec<String>>().join("\n"),
if !network_access {
"\n - Commands that require network access\n"
} else {
""
}
)
}
SandboxPolicy::DangerFullAccess => {
"Runs a shell command and returns its output.".to_string()
}
SandboxPolicy::ReadOnly => {
r#"
The shell tool is used to execute shell commands.
- When invoking the shell tool, your call will be running in a sandbox, and some shell commands (including apply_patch) will require escalated permissions:
- Types of actions that require escalated privileges:
- Writing files
- Applying patches
- Examples of commands that require escalated privileges:
- apply_patch
- git commit
- npm install or pnpm install
- cargo build
- cargo test
- When invoking a command that will require escalated privileges:
- Provide the with_escalated_permissions parameter with the boolean value true
- Include a short, 1 sentence explanation for why we need to run with_escalated_permissions in the justification parameter"#.to_string()
}
};
let description = SHELL_TOOL_DESCRIPTION.to_string();
OpenAiTool::Function(ResponsesApiTool {
name: "shell".to_string(),
@@ -300,7 +295,6 @@ The shell tool is used to execute shell commands.
},
})
}
fn create_view_image_tool() -> OpenAiTool {
// Support only local filesystem path.
let mut properties = BTreeMap::new();
@@ -534,23 +528,27 @@ pub(crate) fn get_openai_tools(
) -> Vec<OpenAiTool> {
let mut tools: Vec<OpenAiTool> = Vec::new();
match &config.shell_type {
ConfigShellToolType::DefaultShell => {
tools.push(create_shell_tool());
}
ConfigShellToolType::ShellWithRequest { sandbox_policy } => {
tools.push(create_shell_tool_for_sandbox(sandbox_policy));
}
ConfigShellToolType::LocalShell => {
tools.push(OpenAiTool::LocalShell {});
}
ConfigShellToolType::StreamableShell => {
tools.push(OpenAiTool::Function(
crate::exec_command::create_exec_command_tool_for_responses_api(),
));
tools.push(OpenAiTool::Function(
crate::exec_command::create_write_stdin_tool_for_responses_api(),
));
if config.experimental_unified_exec_tool {
tools.push(create_unified_exec_tool());
} else {
match &config.shell_type {
ConfigShellToolType::DefaultShell => {
tools.push(create_shell_tool());
}
ConfigShellToolType::ShellWithRequest => {
tools.push(create_shell_tool_for_sandbox());
}
ConfigShellToolType::LocalShell => {
tools.push(OpenAiTool::LocalShell {});
}
ConfigShellToolType::StreamableShell => {
tools.push(OpenAiTool::Function(
crate::exec_command::create_exec_command_tool_for_responses_api(),
));
tools.push(OpenAiTool::Function(
crate::exec_command::create_write_stdin_tool_for_responses_api(),
));
}
}
}
@@ -577,10 +575,8 @@ pub(crate) fn get_openai_tools(
if config.include_view_image_tool {
tools.push(create_view_image_tool());
}
if let Some(mcp_tools) = mcp_tools {
// Ensure deterministic ordering to maximize prompt cache hits.
// HashMap iteration order is non-deterministic, so sort by fully-qualified tool name.
let mut entries: Vec<(String, mcp_types::Tool)> = mcp_tools.into_iter().collect();
entries.sort_by(|a, b| a.0.cmp(&b.0));
@@ -636,18 +632,18 @@ mod tests {
let config = ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy: AskForApproval::Never,
sandbox_policy: SandboxPolicy::ReadOnly,
include_plan_tool: true,
include_apply_patch_tool: false,
include_web_search_request: true,
use_streamable_shell_tool: false,
include_view_image_tool: true,
experimental_unified_exec_tool: true,
});
let tools = get_openai_tools(&config, Some(HashMap::new()));
assert_eq_tool_names(
&tools,
&["local_shell", "update_plan", "web_search", "view_image"],
&["unified_exec", "update_plan", "web_search", "view_image"],
);
}
@@ -657,18 +653,18 @@ mod tests {
let config = ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy: AskForApproval::Never,
sandbox_policy: SandboxPolicy::ReadOnly,
include_plan_tool: true,
include_apply_patch_tool: false,
include_web_search_request: true,
use_streamable_shell_tool: false,
include_view_image_tool: true,
experimental_unified_exec_tool: true,
});
let tools = get_openai_tools(&config, Some(HashMap::new()));
assert_eq_tool_names(
&tools,
&["shell", "update_plan", "web_search", "view_image"],
&["unified_exec", "update_plan", "web_search", "view_image"],
);
}
@@ -678,12 +674,12 @@ mod tests {
let config = ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy: AskForApproval::Never,
sandbox_policy: SandboxPolicy::ReadOnly,
include_plan_tool: false,
include_apply_patch_tool: false,
include_web_search_request: true,
use_streamable_shell_tool: false,
include_view_image_tool: true,
experimental_unified_exec_tool: true,
});
let tools = get_openai_tools(
&config,
@@ -726,7 +722,7 @@ mod tests {
assert_eq_tool_names(
&tools,
&[
"shell",
"unified_exec",
"web_search",
"view_image",
"test_server/do_something_cool",
@@ -783,12 +779,12 @@ mod tests {
let config = ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy: AskForApproval::Never,
sandbox_policy: SandboxPolicy::ReadOnly,
include_plan_tool: false,
include_apply_patch_tool: false,
include_web_search_request: false,
use_streamable_shell_tool: false,
include_view_image_tool: true,
experimental_unified_exec_tool: true,
});
// Intentionally construct a map with keys that would sort alphabetically.
@@ -841,11 +837,11 @@ mod tests {
]);
let tools = get_openai_tools(&config, Some(tools_map));
// Expect shell first, followed by MCP tools sorted by fully-qualified name.
// Expect unified_exec first, followed by MCP tools sorted by fully-qualified name.
assert_eq_tool_names(
&tools,
&[
"shell",
"unified_exec",
"view_image",
"test_server/cool",
"test_server/do",
@@ -860,12 +856,12 @@ mod tests {
let config = ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy: AskForApproval::Never,
sandbox_policy: SandboxPolicy::ReadOnly,
include_plan_tool: false,
include_apply_patch_tool: false,
include_web_search_request: true,
use_streamable_shell_tool: false,
include_view_image_tool: true,
experimental_unified_exec_tool: true,
});
let tools = get_openai_tools(
@@ -893,7 +889,7 @@ mod tests {
assert_eq_tool_names(
&tools,
&["shell", "web_search", "view_image", "dash/search"],
&["unified_exec", "web_search", "view_image", "dash/search"],
);
assert_eq!(
@@ -922,12 +918,12 @@ mod tests {
let config = ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy: AskForApproval::Never,
sandbox_policy: SandboxPolicy::ReadOnly,
include_plan_tool: false,
include_apply_patch_tool: false,
include_web_search_request: true,
use_streamable_shell_tool: false,
include_view_image_tool: true,
experimental_unified_exec_tool: true,
});
let tools = get_openai_tools(
@@ -953,7 +949,7 @@ mod tests {
assert_eq_tool_names(
&tools,
&["shell", "web_search", "view_image", "dash/paginate"],
&["unified_exec", "web_search", "view_image", "dash/paginate"],
);
assert_eq!(
tools[3],
@@ -979,12 +975,12 @@ mod tests {
let config = ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy: AskForApproval::Never,
sandbox_policy: SandboxPolicy::ReadOnly,
include_plan_tool: false,
include_apply_patch_tool: false,
include_web_search_request: true,
use_streamable_shell_tool: false,
include_view_image_tool: true,
experimental_unified_exec_tool: true,
});
let tools = get_openai_tools(
@@ -1008,7 +1004,10 @@ mod tests {
)])),
);
assert_eq_tool_names(&tools, &["shell", "web_search", "view_image", "dash/tags"]);
assert_eq_tool_names(
&tools,
&["unified_exec", "web_search", "view_image", "dash/tags"],
);
assert_eq!(
tools[3],
OpenAiTool::Function(ResponsesApiTool {
@@ -1036,12 +1035,12 @@ mod tests {
let config = ToolsConfig::new(&ToolsConfigParams {
model_family: &model_family,
approval_policy: AskForApproval::Never,
sandbox_policy: SandboxPolicy::ReadOnly,
include_plan_tool: false,
include_apply_patch_tool: false,
include_web_search_request: true,
use_streamable_shell_tool: false,
include_view_image_tool: true,
experimental_unified_exec_tool: true,
});
let tools = get_openai_tools(
@@ -1065,7 +1064,10 @@ mod tests {
)])),
);
assert_eq_tool_names(&tools, &["shell", "web_search", "view_image", "dash/value"]);
assert_eq_tool_names(
&tools,
&["unified_exec", "web_search", "view_image", "dash/value"],
);
assert_eq!(
tools[3],
OpenAiTool::Function(ResponsesApiTool {
@@ -1086,13 +1088,7 @@ mod tests {
#[test]
fn test_shell_tool_for_sandbox_workspace_write() {
let sandbox_policy = SandboxPolicy::WorkspaceWrite {
writable_roots: vec!["workspace".into()],
network_access: false,
exclude_tmpdir_env_var: false,
exclude_slash_tmp: false,
};
let tool = super::create_shell_tool_for_sandbox(&sandbox_policy);
let tool = super::create_shell_tool_for_sandbox();
let OpenAiTool::Function(ResponsesApiTool {
description, name, ..
}) = &tool
@@ -1101,29 +1097,13 @@ mod tests {
};
assert_eq!(name, "shell");
let expected = r#"
The shell tool is used to execute shell commands.
- When invoking the shell tool, your call will be running in a sandbox, and some shell commands will require escalated privileges:
- Types of actions that require escalated privileges:
- Writing files other than those in the writable roots
- writable roots:
- workspace
- Commands that require network access
- Examples of commands that require escalated privileges:
- git commit
- npm install or pnpm install
- cargo build
- cargo test
- When invoking a command that will require escalated privileges:
- Provide the with_escalated_permissions parameter with the boolean value true
- Include a short, 1 sentence explanation for why we need to run with_escalated_permissions in the justification parameter."#;
let expected = super::SHELL_TOOL_DESCRIPTION;
assert_eq!(description, expected);
}
#[test]
fn test_shell_tool_for_sandbox_readonly() {
let tool = super::create_shell_tool_for_sandbox(&SandboxPolicy::ReadOnly);
let tool = super::create_shell_tool_for_sandbox();
let OpenAiTool::Function(ResponsesApiTool {
description, name, ..
}) = &tool
@@ -1132,27 +1112,13 @@ The shell tool is used to execute shell commands.
};
assert_eq!(name, "shell");
let expected = r#"
The shell tool is used to execute shell commands.
- When invoking the shell tool, your call will be running in a sandbox, and some shell commands (including apply_patch) will require escalated permissions:
- Types of actions that require escalated privileges:
- Writing files
- Applying patches
- Examples of commands that require escalated privileges:
- apply_patch
- git commit
- npm install or pnpm install
- cargo build
- cargo test
- When invoking a command that will require escalated privileges:
- Provide the with_escalated_permissions parameter with the boolean value true
- Include a short, 1 sentence explanation for why we need to run with_escalated_permissions in the justification parameter"#;
let expected = super::SHELL_TOOL_DESCRIPTION;
assert_eq!(description, expected);
}
#[test]
fn test_shell_tool_for_sandbox_danger_full_access() {
let tool = super::create_shell_tool_for_sandbox(&SandboxPolicy::DangerFullAccess);
let tool = super::create_shell_tool_for_sandbox();
let OpenAiTool::Function(ResponsesApiTool {
description, name, ..
}) = &tool
@@ -1161,6 +1127,7 @@ The shell tool is used to execute shell commands.
};
assert_eq!(name, "shell");
assert_eq!(description, "Runs a shell command and returns its output.");
let expected = super::SHELL_TOOL_DESCRIPTION;
assert_eq!(description, expected);
}
}

View File

@@ -65,6 +65,6 @@ pub(crate) fn should_persist_event_msg(ev: &EventMsg) -> bool {
| EventMsg::PlanUpdate(_)
| EventMsg::TurnAborted(_)
| EventMsg::ShutdownComplete
| EventMsg::ConversationHistory(_) => false,
| EventMsg::ConversationPath(_) => false,
}
}

View File

@@ -77,7 +77,13 @@ pub enum RolloutRecorderParams {
enum RolloutCmd {
AddItems(Vec<RolloutItem>),
Shutdown { ack: oneshot::Sender<()> },
/// Ensure all prior writes are processed; respond when flushed.
Flush {
ack: oneshot::Sender<()>,
},
Shutdown {
ack: oneshot::Sender<()>,
},
}
impl RolloutRecorderParams {
@@ -185,6 +191,17 @@ impl RolloutRecorder {
.map_err(|e| IoError::other(format!("failed to queue rollout items: {e}")))
}
/// Flush all queued writes and wait until they are committed by the writer task.
pub async fn flush(&self) -> std::io::Result<()> {
let (tx, rx) = oneshot::channel();
self.tx
.send(RolloutCmd::Flush { ack: tx })
.await
.map_err(|e| IoError::other(format!("failed to queue rollout flush: {e}")))?;
rx.await
.map_err(|e| IoError::other(format!("failed waiting for rollout flush: {e}")))
}
pub(crate) async fn get_rollout_history(path: &Path) -> std::io::Result<InitialHistory> {
info!("Resuming rollout from {path:?}");
tracing::error!("Resuming rollout from {path:?}");
@@ -211,11 +228,11 @@ impl RolloutRecorder {
match serde_json::from_value::<RolloutLine>(v.clone()) {
Ok(rollout_line) => match rollout_line.item {
RolloutItem::SessionMeta(session_meta_line) => {
tracing::error!(
"Parsed conversation ID from rollout file: {:?}",
session_meta_line.meta.id
);
conversation_id = Some(session_meta_line.meta.id);
// Use the FIRST SessionMeta encountered in the file as the canonical
// conversation id and main session information. Keep all items intact.
if conversation_id.is_none() {
conversation_id = Some(session_meta_line.meta.id);
}
items.push(RolloutItem::SessionMeta(session_meta_line));
}
RolloutItem::ResponseItem(item) => {
@@ -251,6 +268,10 @@ impl RolloutRecorder {
}))
}
pub(crate) fn get_rollout_path(&self) -> PathBuf {
self.rollout_path.clone()
}
pub async fn shutdown(&self) -> std::io::Result<()> {
let (tx_done, rx_done) = oneshot::channel();
match self.tx.send(RolloutCmd::Shutdown { ack: tx_done }).await {
@@ -351,6 +372,14 @@ async fn rollout_writer(
}
}
}
RolloutCmd::Flush { ack } => {
// Ensure underlying file is flushed and then ack.
if let Err(e) = writer.file.flush().await {
let _ = ack.send(());
return Err(e);
}
let _ = ack.send(());
}
RolloutCmd::Shutdown { ack } => {
let _ = ack.send(());
}

View File

@@ -1,36 +1,18 @@
use serde::Deserialize;
use serde::Serialize;
use shlex;
use std::path::Path;
use std::path::PathBuf;
use std::sync::Arc;
use tracing::trace;
use uuid::Uuid;
#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
/// This structure cannot derive Clone or this will break the Drop implementation.
pub struct ShellSnapshot {
pub(crate) path: PathBuf,
}
impl ShellSnapshot {
pub fn new(path: PathBuf) -> Self {
Self { path }
}
}
impl Drop for ShellSnapshot {
fn drop(&mut self) {
delete_shell_snapshot(&self.path);
}
#[derive(Debug, PartialEq, Eq, Clone, Serialize, Deserialize)]
pub struct ZshShell {
shell_path: String,
zshrc_path: String,
}
#[derive(Debug, PartialEq, Eq, Clone, Serialize, Deserialize)]
pub struct PosixShell {
pub(crate) shell_path: String,
pub(crate) rc_path: String,
#[serde(skip_serializing, skip_deserializing)]
pub(crate) shell_snapshot: Option<Arc<ShellSnapshot>>,
pub struct BashShell {
shell_path: String,
bashrc_path: String,
}
#[derive(Debug, PartialEq, Eq, Clone, Serialize, Deserialize)]
@@ -41,7 +23,8 @@ pub struct PowerShellConfig {
#[derive(Debug, PartialEq, Eq, Clone, Serialize, Deserialize)]
pub enum Shell {
Posix(PosixShell),
Zsh(ZshShell),
Bash(BashShell),
PowerShell(PowerShellConfig),
Unknown,
}
@@ -49,27 +32,11 @@ pub enum Shell {
impl Shell {
pub fn format_default_shell_invocation(&self, command: Vec<String>) -> Option<Vec<String>> {
match self {
Shell::Posix(shell) => {
let joined = strip_bash_lc(&command)
.or_else(|| shlex::try_join(command.iter().map(|s| s.as_str())).ok())?;
let mut source_path = Path::new(&shell.rc_path);
let session_cmd = if let Some(shell_snapshot) = &shell.shell_snapshot
&& shell_snapshot.path.exists()
{
source_path = shell_snapshot.path.as_path();
"-c".to_string()
} else {
"-lc".to_string()
};
let source_path_str = source_path.to_string_lossy().to_string();
let quoted_source_path = shlex::try_quote(&source_path_str).ok()?;
let rc_command =
format!("[ -f {quoted_source_path} ] && . {quoted_source_path}; ({joined})");
Some(vec![shell.shell_path.clone(), session_cmd, rc_command])
Shell::Zsh(zsh) => {
format_shell_invocation_with_rc(&command, &zsh.shell_path, &zsh.zshrc_path)
}
Shell::Bash(bash) => {
format_shell_invocation_with_rc(&command, &bash.shell_path, &bash.bashrc_path)
}
Shell::PowerShell(ps) => {
// If model generated a bash command, prefer a detected bash fallback
@@ -122,20 +89,33 @@ impl Shell {
pub fn name(&self) -> Option<String> {
match self {
Shell::Posix(shell) => Path::new(&shell.shell_path)
Shell::Zsh(zsh) => std::path::Path::new(&zsh.shell_path)
.file_name()
.map(|s| s.to_string_lossy().to_string()),
Shell::Bash(bash) => std::path::Path::new(&bash.shell_path)
.file_name()
.map(|s| s.to_string_lossy().to_string()),
Shell::PowerShell(ps) => Some(ps.exe.clone()),
Shell::Unknown => None,
}
}
}
pub fn get_snapshot(&self) -> Option<Arc<ShellSnapshot>> {
match self {
Shell::Posix(shell) => shell.shell_snapshot.clone(),
_ => None,
}
}
fn format_shell_invocation_with_rc(
command: &Vec<String>,
shell_path: &str,
rc_path: &str,
) -> Option<Vec<String>> {
let joined = strip_bash_lc(command)
.or_else(|| shlex::try_join(command.iter().map(|s| s.as_str())).ok())?;
let rc_command = if std::path::Path::new(rc_path).exists() {
format!("source {rc_path} && ({joined})")
} else {
joined
};
Some(vec![shell_path.to_string(), "-lc".to_string(), rc_command])
}
fn strip_bash_lc(command: &Vec<String>) -> Option<String> {
@@ -152,7 +132,7 @@ fn strip_bash_lc(command: &Vec<String>) -> Option<String> {
}
#[cfg(unix)]
async fn detect_default_user_shell(session_id: Uuid, codex_home: &Path) -> Shell {
fn detect_default_user_shell() -> Shell {
use libc::getpwuid;
use libc::getuid;
use std::ffi::CStr;
@@ -167,45 +147,31 @@ async fn detect_default_user_shell(session_id: Uuid, codex_home: &Path) -> Shell
.into_owned();
let home_path = CStr::from_ptr((*pw).pw_dir).to_string_lossy().into_owned();
let rc_path = if shell_path.ends_with("/zsh") {
format!("{home_path}/.zshrc")
} else if shell_path.ends_with("/bash") {
format!("{home_path}/.bashrc")
} else {
return Shell::Unknown;
};
let snapshot_path = snapshots::ensure_posix_snapshot(
&shell_path,
&rc_path,
Path::new(&home_path),
codex_home,
session_id,
)
.await;
if snapshot_path.is_none() {
trace!("failed to prepare posix snapshot; using live profile");
if shell_path.ends_with("/zsh") {
return Shell::Zsh(ZshShell {
shell_path,
zshrc_path: format!("{home_path}/.zshrc"),
});
}
let shell_snapshot =
snapshot_path.map(|snapshot| Arc::new(ShellSnapshot::new(snapshot)));
return Shell::Posix(PosixShell {
shell_path,
rc_path,
shell_snapshot,
});
if shell_path.ends_with("/bash") {
return Shell::Bash(BashShell {
shell_path,
bashrc_path: format!("{home_path}/.bashrc"),
});
}
}
}
Shell::Unknown
}
#[cfg(unix)]
pub async fn default_user_shell(session_id: Uuid, codex_home: &Path) -> Shell {
detect_default_user_shell(session_id, codex_home).await
pub async fn default_user_shell() -> Shell {
detect_default_user_shell()
}
#[cfg(target_os = "windows")]
pub async fn default_user_shell(_session_id: Uuid, _codex_home: &Path) -> Shell {
pub async fn default_user_shell() -> Shell {
use tokio::process::Command;
// Prefer PowerShell 7+ (`pwsh`) if available, otherwise fall back to Windows PowerShell.
@@ -245,158 +211,42 @@ pub async fn default_user_shell(_session_id: Uuid, _codex_home: &Path) -> Shell
}
#[cfg(all(not(target_os = "windows"), not(unix)))]
pub async fn default_user_shell(_session_id: Uuid, _codex_home: &Path) -> Shell {
pub async fn default_user_shell() -> Shell {
Shell::Unknown
}
#[cfg(unix)]
mod snapshots {
use super::*;
fn zsh_profile_paths(home: &Path) -> Vec<PathBuf> {
[".zshenv", ".zprofile", ".zshrc", ".zlogin"]
.into_iter()
.map(|name| home.join(name))
.collect()
}
fn posix_profile_source_script(home: &Path) -> String {
zsh_profile_paths(home)
.into_iter()
.map(|profile| {
let profile_string = profile.to_string_lossy().into_owned();
let quoted = shlex::try_quote(&profile_string)
.map(|cow| cow.into_owned())
.unwrap_or(profile_string.clone());
format!("[ -f {quoted} ] && . {quoted}")
})
.collect::<Vec<_>>()
.join("; ")
}
pub(crate) async fn ensure_posix_snapshot(
shell_path: &str,
rc_path: &str,
home: &Path,
codex_home: &Path,
session_id: Uuid,
) -> Option<PathBuf> {
let snapshot_path = codex_home.join(format!("shell_snapshots/snapshot_{session_id}.zsh"));
// Check if an update in the profile requires to re-generate the snapshot.
let snapshot_is_stale = async {
let snapshot_metadata = tokio::fs::metadata(&snapshot_path).await.ok()?;
let snapshot_modified = snapshot_metadata.modified().ok()?;
for profile in zsh_profile_paths(home) {
let Ok(profile_metadata) = tokio::fs::metadata(&profile).await else {
continue;
};
let Ok(profile_modified) = profile_metadata.modified() else {
return Some(true);
};
if profile_modified > snapshot_modified {
return Some(true);
}
}
Some(false)
}
.await
.unwrap_or(true);
if !snapshot_is_stale {
return Some(snapshot_path);
}
match regenerate_posix_snapshot(shell_path, rc_path, home, &snapshot_path).await {
Ok(()) => Some(snapshot_path),
Err(err) => {
tracing::warn!("failed to generate posix snapshot: {err}");
None
}
}
}
async fn regenerate_posix_snapshot(
shell_path: &str,
rc_path: &str,
home: &Path,
snapshot_path: &Path,
) -> std::io::Result<()> {
// Use `emulate -L sh` instead of `set -o posix` so we work on zsh builds
// that disable that option. Guard `alias -p` with `|| true` so the script
// keeps a zero exit status even if aliases are disabled.
let mut capture_script = String::new();
let profile_sources = posix_profile_source_script(home);
if !profile_sources.is_empty() {
capture_script.push_str(&format!("{profile_sources}; "));
}
let zshrc = home.join(rc_path);
capture_script.push_str(
&format!(". {}; setopt posixbuiltins; export -p; {{ alias | sed 's/^/alias /'; }} 2>/dev/null || true", zshrc.display()),
);
let output = tokio::process::Command::new(shell_path)
.arg("-lc")
.arg(capture_script)
.env("HOME", home)
.output()
.await?;
if !output.status.success() {
return Err(std::io::Error::other(format!(
"snapshot capture exited with status {}",
output.status
)));
}
let mut contents = String::from("# Generated by Codex. Do not edit.\n");
contents.push_str(&String::from_utf8_lossy(&output.stdout));
contents.push('\n');
if let Some(parent) = snapshot_path.parent() {
tokio::fs::create_dir_all(parent).await?;
}
let tmp_path = snapshot_path.with_extension("tmp");
tokio::fs::write(&tmp_path, contents).await?;
// Restrict the snapshot to user read/write so that environment variables or aliases
// that may contain secrets are not exposed to other users on the system.
use std::os::unix::fs::PermissionsExt;
let permissions = std::fs::Permissions::from_mode(0o600);
tokio::fs::set_permissions(&tmp_path, permissions).await?;
tokio::fs::rename(&tmp_path, snapshot_path).await?;
Ok(())
}
}
pub(crate) fn delete_shell_snapshot(path: &Path) {
if let Err(err) = std::fs::remove_file(path) {
trace!("failed to delete shell snapshot {path:?}: {err}");
}
}
#[cfg(test)]
#[cfg(unix)]
mod tests {
use super::*;
use std::process::Command;
use std::path::PathBuf;
#[tokio::test]
async fn test_current_shell_detects_zsh() {
let shell = Command::new("sh")
.arg("-c")
.arg("echo $SHELL")
.output()
.unwrap();
let home = std::env::var("HOME").unwrap();
let shell_path = String::from_utf8_lossy(&shell.stdout).trim().to_string();
if shell_path.ends_with("/zsh") {
assert_eq!(
default_user_shell().await,
Shell::Zsh(ZshShell {
shell_path: shell_path.to_string(),
zshrc_path: format!("{home}/.zshrc",),
})
);
}
}
#[tokio::test]
async fn test_run_with_profile_zshrc_not_exists() {
let shell = Shell::Posix(PosixShell {
let shell = Shell::Zsh(ZshShell {
shell_path: "/bin/zsh".to_string(),
rc_path: "/does/not/exist/.zshrc".to_string(),
shell_snapshot: None,
zshrc_path: "/does/not/exist/.zshrc".to_string(),
});
let actual_cmd = shell.format_default_shell_invocation(vec!["myecho".to_string()]);
assert_eq!(
@@ -404,7 +254,24 @@ mod tests {
Some(vec![
"/bin/zsh".to_string(),
"-lc".to_string(),
"[ -f /does/not/exist/.zshrc ] && . /does/not/exist/.zshrc; (myecho)".to_string(),
"myecho".to_string()
])
);
}
#[tokio::test]
async fn test_run_with_profile_bashrc_not_exists() {
let shell = Shell::Bash(BashShell {
shell_path: "/bin/bash".to_string(),
bashrc_path: "/does/not/exist/.bashrc".to_string(),
});
let actual_cmd = shell.format_default_shell_invocation(vec!["myecho".to_string()]);
assert_eq!(
actual_cmd,
Some(vec![
"/bin/bash".to_string(),
"-lc".to_string(),
"myecho".to_string()
])
);
}
@@ -416,11 +283,7 @@ mod tests {
let cases = vec![
(
vec!["myecho"],
vec![
shell_path,
"-lc",
"[ -f BASHRC_PATH ] && . BASHRC_PATH; (myecho)",
],
vec![shell_path, "-lc", "source BASHRC_PATH && (myecho)"],
Some("It works!\n"),
),
(
@@ -428,7 +291,7 @@ mod tests {
vec![
shell_path,
"-lc",
"[ -f BASHRC_PATH ] && . BASHRC_PATH; (echo 'single' \"double\")",
"source BASHRC_PATH && (echo 'single' \"double\")",
],
Some("single double\n"),
),
@@ -454,10 +317,9 @@ mod tests {
"#,
)
.unwrap();
let shell = Shell::Posix(PosixShell {
let shell = Shell::Bash(BashShell {
shell_path: shell_path.to_string(),
rc_path: bashrc_path.to_str().unwrap().to_string(),
shell_snapshot: None,
bashrc_path: bashrc_path.to_str().unwrap().to_string(),
});
let actual_cmd = shell
@@ -507,82 +369,6 @@ mod tests {
#[cfg(target_os = "macos")]
mod macos_tests {
use super::*;
use crate::shell::snapshots::ensure_posix_snapshot;
#[tokio::test]
async fn test_snapshot_generation_uses_session_id_and_cleanup() {
let shell_path = "/bin/zsh";
let temp_home = tempfile::tempdir().unwrap();
let codex_home = tempfile::tempdir().unwrap();
std::fs::write(
temp_home.path().join(".zshrc"),
"export SNAPSHOT_TEST_VAR=1\nalias snapshot_test_alias='echo hi'\n",
)
.unwrap();
let session_id = Uuid::new_v4();
let snapshot_path = ensure_posix_snapshot(
shell_path,
".zshrc",
temp_home.path(),
codex_home.path(),
session_id,
)
.await
.expect("snapshot path");
let filename = snapshot_path
.file_name()
.unwrap()
.to_string_lossy()
.to_string();
assert!(filename.contains(&session_id.to_string()));
assert!(snapshot_path.exists());
let snapshot_path_second = ensure_posix_snapshot(
shell_path,
".zshrc",
temp_home.path(),
codex_home.path(),
session_id,
)
.await
.expect("snapshot path");
assert_eq!(snapshot_path, snapshot_path_second);
let contents = std::fs::read_to_string(&snapshot_path).unwrap();
assert!(contents.contains("alias snapshot_test_alias='echo hi'"));
assert!(contents.contains("SNAPSHOT_TEST_VAR=1"));
delete_shell_snapshot(&snapshot_path);
assert!(!snapshot_path.exists());
}
#[test]
fn format_default_shell_invocation_prefers_snapshot_when_available() {
let temp_dir = tempfile::tempdir().unwrap();
let snapshot_path = temp_dir.path().join("snapshot.zsh");
std::fs::write(&snapshot_path, "export SNAPSHOT_READY=1").unwrap();
let shell = Shell::Posix(PosixShell {
shell_path: "/bin/zsh".to_string(),
rc_path: {
let path = temp_dir.path().join(".zshrc");
std::fs::write(&path, "# test zshrc").unwrap();
path.to_string_lossy().to_string()
},
shell_snapshot: Some(Arc::new(ShellSnapshot::new(snapshot_path.clone()))),
});
let invocation = shell.format_default_shell_invocation(vec!["echo".to_string()]);
let expected_command = vec!["/bin/zsh".to_string(), "-c".to_string(), {
let snapshot_path = snapshot_path.to_string_lossy();
format!("[ -f {snapshot_path} ] && . {snapshot_path}; (echo)")
}];
assert_eq!(invocation, Some(expected_command));
}
#[tokio::test]
async fn test_run_with_profile_escaping_and_execution() {
@@ -591,20 +377,12 @@ mod macos_tests {
let cases = vec![
(
vec!["myecho"],
vec![
shell_path,
"-lc",
"[ -f ZSHRC_PATH ] && . ZSHRC_PATH; (myecho)",
],
vec![shell_path, "-lc", "source ZSHRC_PATH && (myecho)"],
Some("It works!\n"),
),
(
vec!["myecho"],
vec![
shell_path,
"-lc",
"[ -f ZSHRC_PATH ] && . ZSHRC_PATH; (myecho)",
],
vec![shell_path, "-lc", "source ZSHRC_PATH && (myecho)"],
Some("It works!\n"),
),
(
@@ -612,7 +390,7 @@ mod macos_tests {
vec![
shell_path,
"-lc",
"[ -f ZSHRC_PATH ] && . ZSHRC_PATH; (bash -c \"echo 'single' \\\"double\\\"\")",
"source ZSHRC_PATH && (bash -c \"echo 'single' \\\"double\\\"\")",
],
Some("single double\n"),
),
@@ -621,7 +399,7 @@ mod macos_tests {
vec![
shell_path,
"-lc",
"[ -f ZSHRC_PATH ] && . ZSHRC_PATH; (echo 'single' \"double\")",
"source ZSHRC_PATH && (echo 'single' \"double\")",
],
Some("single double\n"),
),
@@ -648,10 +426,9 @@ mod macos_tests {
"#,
)
.unwrap();
let shell = Shell::Posix(PosixShell {
let shell = Shell::Zsh(ZshShell {
shell_path: shell_path.to_string(),
rc_path: zshrc_path.to_str().unwrap().to_string(),
shell_snapshot: None,
zshrc_path: zshrc_path.to_str().unwrap().to_string(),
});
let actual_cmd = shell

View File

@@ -0,0 +1,180 @@
//! Utilities for truncating large chunks of output while preserving a prefix
//! and suffix on UTF-8 boundaries.
/// Truncate the middle of a UTF-8 string to at most `max_bytes` bytes,
/// preserving the beginning and the end. Returns the possibly truncated
/// string and `Some(original_token_count)` (estimated at 4 bytes/token)
/// if truncation occurred; otherwise returns the original string and `None`.
pub(crate) fn truncate_middle(s: &str, max_bytes: usize) -> (String, Option<u64>) {
if s.len() <= max_bytes {
return (s.to_string(), None);
}
let est_tokens = (s.len() as u64).div_ceil(4);
if max_bytes == 0 {
return (format!("{est_tokens} tokens truncated…"), Some(est_tokens));
}
fn truncate_on_boundary(input: &str, max_len: usize) -> &str {
if input.len() <= max_len {
return input;
}
let mut end = max_len;
while end > 0 && !input.is_char_boundary(end) {
end -= 1;
}
&input[..end]
}
fn pick_prefix_end(s: &str, left_budget: usize) -> usize {
if let Some(head) = s.get(..left_budget)
&& let Some(i) = head.rfind('\n')
{
return i + 1;
}
truncate_on_boundary(s, left_budget).len()
}
fn pick_suffix_start(s: &str, right_budget: usize) -> usize {
let start_tail = s.len().saturating_sub(right_budget);
if let Some(tail) = s.get(start_tail..)
&& let Some(i) = tail.find('\n')
{
return start_tail + i + 1;
}
let mut idx = start_tail.min(s.len());
while idx < s.len() && !s.is_char_boundary(idx) {
idx += 1;
}
idx
}
let mut guess_tokens = est_tokens;
for _ in 0..4 {
let marker = format!("{guess_tokens} tokens truncated…");
let marker_len = marker.len();
let keep_budget = max_bytes.saturating_sub(marker_len);
if keep_budget == 0 {
return (format!("{est_tokens} tokens truncated…"), Some(est_tokens));
}
let left_budget = keep_budget / 2;
let right_budget = keep_budget - left_budget;
let prefix_end = pick_prefix_end(s, left_budget);
let mut suffix_start = pick_suffix_start(s, right_budget);
if suffix_start < prefix_end {
suffix_start = prefix_end;
}
let kept_content_bytes = prefix_end + (s.len() - suffix_start);
let truncated_content_bytes = s.len().saturating_sub(kept_content_bytes);
let new_tokens = (truncated_content_bytes as u64).div_ceil(4);
if new_tokens == guess_tokens {
let mut out = String::with_capacity(marker_len + kept_content_bytes + 1);
out.push_str(&s[..prefix_end]);
out.push_str(&marker);
out.push('\n');
out.push_str(&s[suffix_start..]);
return (out, Some(est_tokens));
}
guess_tokens = new_tokens;
}
let marker = format!("{guess_tokens} tokens truncated…");
let marker_len = marker.len();
let keep_budget = max_bytes.saturating_sub(marker_len);
if keep_budget == 0 {
return (format!("{est_tokens} tokens truncated…"), Some(est_tokens));
}
let left_budget = keep_budget / 2;
let right_budget = keep_budget - left_budget;
let prefix_end = pick_prefix_end(s, left_budget);
let suffix_start = pick_suffix_start(s, right_budget);
let mut out = String::with_capacity(marker_len + prefix_end + (s.len() - suffix_start) + 1);
out.push_str(&s[..prefix_end]);
out.push_str(&marker);
out.push('\n');
out.push_str(&s[suffix_start..]);
(out, Some(est_tokens))
}
#[cfg(test)]
mod tests {
use super::truncate_middle;
#[test]
fn truncate_middle_no_newlines_fallback() {
let s = "abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*";
let max_bytes = 32;
let (out, original) = truncate_middle(s, max_bytes);
assert!(out.starts_with("abc"));
assert!(out.contains("tokens truncated"));
assert!(out.ends_with("XYZ*"));
assert_eq!(original, Some((s.len() as u64).div_ceil(4)));
}
#[test]
fn truncate_middle_prefers_newline_boundaries() {
let mut s = String::new();
for i in 1..=20 {
s.push_str(&format!("{i:03}\n"));
}
assert_eq!(s.len(), 80);
let max_bytes = 64;
let (out, tokens) = truncate_middle(&s, max_bytes);
assert!(out.starts_with("001\n002\n003\n004\n"));
assert!(out.contains("tokens truncated"));
assert!(out.ends_with("017\n018\n019\n020\n"));
assert_eq!(tokens, Some(20));
}
#[test]
fn truncate_middle_handles_utf8_content() {
let s = "😀😀😀😀😀😀😀😀😀😀\nsecond line with ascii text\n";
let max_bytes = 32;
let (out, tokens) = truncate_middle(s, max_bytes);
assert!(out.contains("tokens truncated"));
assert!(!out.contains('\u{fffd}'));
assert_eq!(tokens, Some((s.len() as u64).div_ceil(4)));
}
#[test]
fn truncate_middle_prefers_newline_boundaries_2() {
// Build a multi-line string of 20 numbered lines (each "NNN\n").
let mut s = String::new();
for i in 1..=20 {
s.push_str(&format!("{i:03}\n"));
}
// Total length: 20 lines * 4 bytes per line = 80 bytes.
assert_eq!(s.len(), 80);
// Choose a cap that forces truncation while leaving room for
// a few lines on each side after accounting for the marker.
let max_bytes = 64;
// Expect exact output: first 4 lines, marker, last 4 lines, and correct token estimate (80/4 = 20).
assert_eq!(
truncate_middle(&s, max_bytes),
(
r#"001
002
003
004
…12 tokens truncated…
017
018
019
020
"#
.to_string(),
Some(20)
)
);
}
}

View File

@@ -0,0 +1,22 @@
use thiserror::Error;
#[derive(Debug, Error)]
pub(crate) enum UnifiedExecError {
#[error("Failed to create unified exec session: {pty_error}")]
CreateSession {
#[source]
pty_error: anyhow::Error,
},
#[error("Unknown session id {session_id}")]
UnknownSessionId { session_id: i32 },
#[error("failed to write to stdin")]
WriteToStdin,
#[error("missing command line for unified exec request")]
MissingCommandLine,
}
impl UnifiedExecError {
pub(crate) fn create_session(error: anyhow::Error) -> Self {
Self::CreateSession { pty_error: error }
}
}

View File

@@ -0,0 +1,653 @@
use portable_pty::CommandBuilder;
use portable_pty::PtySize;
use portable_pty::native_pty_system;
use std::collections::HashMap;
use std::collections::VecDeque;
use std::io::ErrorKind;
use std::io::Read;
use std::sync::Arc;
use std::sync::Mutex as StdMutex;
use std::sync::atomic::AtomicBool;
use std::sync::atomic::AtomicI32;
use std::sync::atomic::Ordering;
use tokio::sync::Mutex;
use tokio::sync::Notify;
use tokio::sync::mpsc;
use tokio::task::JoinHandle;
use tokio::time::Duration;
use tokio::time::Instant;
use crate::exec_command::ExecCommandSession;
use crate::truncate::truncate_middle;
mod errors;
pub(crate) use errors::UnifiedExecError;
const DEFAULT_TIMEOUT_MS: u64 = 1_000;
const MAX_TIMEOUT_MS: u64 = 60_000;
const UNIFIED_EXEC_OUTPUT_MAX_BYTES: usize = 128 * 1024; // 128 KiB
#[derive(Debug)]
pub(crate) struct UnifiedExecRequest<'a> {
pub session_id: Option<i32>,
pub input_chunks: &'a [String],
pub timeout_ms: Option<u64>,
}
#[derive(Debug, Clone, PartialEq)]
pub(crate) struct UnifiedExecResult {
pub session_id: Option<i32>,
pub output: String,
}
#[derive(Debug, Default)]
pub(crate) struct UnifiedExecSessionManager {
next_session_id: AtomicI32,
sessions: Mutex<HashMap<i32, ManagedUnifiedExecSession>>,
}
#[derive(Debug)]
struct ManagedUnifiedExecSession {
session: ExecCommandSession,
output_buffer: OutputBuffer,
/// Notifies waiters whenever new output has been appended to
/// `output_buffer`, allowing clients to poll for fresh data.
output_notify: Arc<Notify>,
output_task: JoinHandle<()>,
}
#[derive(Debug, Default)]
struct OutputBufferState {
chunks: VecDeque<Vec<u8>>,
total_bytes: usize,
}
impl OutputBufferState {
fn push_chunk(&mut self, chunk: Vec<u8>) {
self.total_bytes = self.total_bytes.saturating_add(chunk.len());
self.chunks.push_back(chunk);
let mut excess = self
.total_bytes
.saturating_sub(UNIFIED_EXEC_OUTPUT_MAX_BYTES);
while excess > 0 {
match self.chunks.front_mut() {
Some(front) if excess >= front.len() => {
excess -= front.len();
self.total_bytes = self.total_bytes.saturating_sub(front.len());
self.chunks.pop_front();
}
Some(front) => {
front.drain(..excess);
self.total_bytes = self.total_bytes.saturating_sub(excess);
break;
}
None => break,
}
}
}
fn drain(&mut self) -> Vec<Vec<u8>> {
let drained: Vec<Vec<u8>> = self.chunks.drain(..).collect();
self.total_bytes = 0;
drained
}
}
type OutputBuffer = Arc<Mutex<OutputBufferState>>;
type OutputHandles = (OutputBuffer, Arc<Notify>);
impl ManagedUnifiedExecSession {
fn new(session: ExecCommandSession) -> Self {
let output_buffer = Arc::new(Mutex::new(OutputBufferState::default()));
let output_notify = Arc::new(Notify::new());
let mut receiver = session.output_receiver();
let buffer_clone = Arc::clone(&output_buffer);
let notify_clone = Arc::clone(&output_notify);
let output_task = tokio::spawn(async move {
while let Ok(chunk) = receiver.recv().await {
let mut guard = buffer_clone.lock().await;
guard.push_chunk(chunk);
drop(guard);
notify_clone.notify_waiters();
}
});
Self {
session,
output_buffer,
output_notify,
output_task,
}
}
fn writer_sender(&self) -> mpsc::Sender<Vec<u8>> {
self.session.writer_sender()
}
fn output_handles(&self) -> OutputHandles {
(
Arc::clone(&self.output_buffer),
Arc::clone(&self.output_notify),
)
}
fn has_exited(&self) -> bool {
self.session.has_exited()
}
}
impl Drop for ManagedUnifiedExecSession {
fn drop(&mut self) {
self.output_task.abort();
}
}
impl UnifiedExecSessionManager {
pub async fn handle_request(
&self,
request: UnifiedExecRequest<'_>,
) -> Result<UnifiedExecResult, UnifiedExecError> {
let (timeout_ms, timeout_warning) = match request.timeout_ms {
Some(requested) if requested > MAX_TIMEOUT_MS => (
MAX_TIMEOUT_MS,
Some(format!(
"Warning: requested timeout {requested}ms exceeds maximum of {MAX_TIMEOUT_MS}ms; clamping to {MAX_TIMEOUT_MS}ms.\n"
)),
),
Some(requested) => (requested, None),
None => (DEFAULT_TIMEOUT_MS, None),
};
let mut new_session: Option<ManagedUnifiedExecSession> = None;
let session_id;
let writer_tx;
let output_buffer;
let output_notify;
if let Some(existing_id) = request.session_id {
let mut sessions = self.sessions.lock().await;
match sessions.get(&existing_id) {
Some(session) => {
if session.has_exited() {
sessions.remove(&existing_id);
return Err(UnifiedExecError::UnknownSessionId {
session_id: existing_id,
});
}
let (buffer, notify) = session.output_handles();
session_id = existing_id;
writer_tx = session.writer_sender();
output_buffer = buffer;
output_notify = notify;
}
None => {
return Err(UnifiedExecError::UnknownSessionId {
session_id: existing_id,
});
}
}
drop(sessions);
} else {
let command = request.input_chunks.to_vec();
let new_id = self.next_session_id.fetch_add(1, Ordering::SeqCst);
let session = create_unified_exec_session(&command).await?;
let managed_session = ManagedUnifiedExecSession::new(session);
let (buffer, notify) = managed_session.output_handles();
writer_tx = managed_session.writer_sender();
output_buffer = buffer;
output_notify = notify;
session_id = new_id;
new_session = Some(managed_session);
};
if request.session_id.is_some() {
let joined_input = request.input_chunks.join(" ");
if !joined_input.is_empty() && writer_tx.send(joined_input.into_bytes()).await.is_err()
{
return Err(UnifiedExecError::WriteToStdin);
}
}
let mut collected: Vec<u8> = Vec::with_capacity(4096);
let start = Instant::now();
let deadline = start + Duration::from_millis(timeout_ms);
loop {
let drained_chunks;
let mut wait_for_output = None;
{
let mut guard = output_buffer.lock().await;
drained_chunks = guard.drain();
if drained_chunks.is_empty() {
wait_for_output = Some(output_notify.notified());
}
}
if drained_chunks.is_empty() {
let remaining = deadline.saturating_duration_since(Instant::now());
if remaining == Duration::ZERO {
break;
}
let notified = wait_for_output.unwrap_or_else(|| output_notify.notified());
tokio::pin!(notified);
tokio::select! {
_ = &mut notified => {}
_ = tokio::time::sleep(remaining) => break,
}
continue;
}
for chunk in drained_chunks {
collected.extend_from_slice(&chunk);
}
if Instant::now() >= deadline {
break;
}
}
let (output, _maybe_tokens) = truncate_middle(
&String::from_utf8_lossy(&collected),
UNIFIED_EXEC_OUTPUT_MAX_BYTES,
);
let output = if let Some(warning) = timeout_warning {
format!("{warning}{output}")
} else {
output
};
let should_store_session = if let Some(session) = new_session.as_ref() {
!session.has_exited()
} else if request.session_id.is_some() {
let mut sessions = self.sessions.lock().await;
if let Some(existing) = sessions.get(&session_id) {
if existing.has_exited() {
sessions.remove(&session_id);
false
} else {
true
}
} else {
false
}
} else {
true
};
if should_store_session {
if let Some(session) = new_session {
self.sessions.lock().await.insert(session_id, session);
}
Ok(UnifiedExecResult {
session_id: Some(session_id),
output,
})
} else {
Ok(UnifiedExecResult {
session_id: None,
output,
})
}
}
}
async fn create_unified_exec_session(
command: &[String],
) -> Result<ExecCommandSession, UnifiedExecError> {
if command.is_empty() {
return Err(UnifiedExecError::MissingCommandLine);
}
let pty_system = native_pty_system();
let pair = pty_system
.openpty(PtySize {
rows: 24,
cols: 80,
pixel_width: 0,
pixel_height: 0,
})
.map_err(UnifiedExecError::create_session)?;
// Safe thanks to the check at the top of the function.
let mut command_builder = CommandBuilder::new(command[0].clone());
for arg in &command[1..] {
command_builder.arg(arg);
}
let mut child = pair
.slave
.spawn_command(command_builder)
.map_err(UnifiedExecError::create_session)?;
let killer = child.clone_killer();
let (writer_tx, mut writer_rx) = mpsc::channel::<Vec<u8>>(128);
let (output_tx, _) = tokio::sync::broadcast::channel::<Vec<u8>>(256);
let mut reader = pair
.master
.try_clone_reader()
.map_err(UnifiedExecError::create_session)?;
let output_tx_clone = output_tx.clone();
let reader_handle = tokio::task::spawn_blocking(move || {
let mut buf = [0u8; 8192];
loop {
match reader.read(&mut buf) {
Ok(0) => break,
Ok(n) => {
let _ = output_tx_clone.send(buf[..n].to_vec());
}
Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
Err(ref e) if e.kind() == ErrorKind::WouldBlock => {
std::thread::sleep(Duration::from_millis(5));
continue;
}
Err(_) => break,
}
}
});
let writer = pair
.master
.take_writer()
.map_err(UnifiedExecError::create_session)?;
let writer = Arc::new(StdMutex::new(writer));
let writer_handle = tokio::spawn({
let writer = writer.clone();
async move {
while let Some(bytes) = writer_rx.recv().await {
let writer = writer.clone();
let _ = tokio::task::spawn_blocking(move || {
if let Ok(mut guard) = writer.lock() {
use std::io::Write;
let _ = guard.write_all(&bytes);
let _ = guard.flush();
}
})
.await;
}
}
});
let exit_status = Arc::new(AtomicBool::new(false));
let wait_exit_status = Arc::clone(&exit_status);
let wait_handle = tokio::task::spawn_blocking(move || {
let _ = child.wait();
wait_exit_status.store(true, Ordering::SeqCst);
});
Ok(ExecCommandSession::new(
writer_tx,
output_tx,
killer,
reader_handle,
writer_handle,
wait_handle,
exit_status,
))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn push_chunk_trims_only_excess_bytes() {
let mut buffer = OutputBufferState::default();
buffer.push_chunk(vec![b'a'; UNIFIED_EXEC_OUTPUT_MAX_BYTES]);
buffer.push_chunk(vec![b'b']);
buffer.push_chunk(vec![b'c']);
assert_eq!(buffer.total_bytes, UNIFIED_EXEC_OUTPUT_MAX_BYTES);
assert_eq!(buffer.chunks.len(), 3);
assert_eq!(
buffer.chunks.front().unwrap().len(),
UNIFIED_EXEC_OUTPUT_MAX_BYTES - 2
);
assert_eq!(buffer.chunks.pop_back().unwrap(), vec![b'c']);
assert_eq!(buffer.chunks.pop_back().unwrap(), vec![b'b']);
}
#[cfg(unix)]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn unified_exec_persists_across_requests_jif() -> Result<(), UnifiedExecError> {
let manager = UnifiedExecSessionManager::default();
let open_shell = manager
.handle_request(UnifiedExecRequest {
session_id: None,
input_chunks: &["bash".to_string(), "-i".to_string()],
timeout_ms: Some(1_500),
})
.await?;
let session_id = open_shell.session_id.expect("expected session_id");
manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_id),
input_chunks: &[
"export".to_string(),
"CODEX_INTERACTIVE_SHELL_VAR=codex\n".to_string(),
],
timeout_ms: Some(2_500),
})
.await?;
let out_2 = manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_id),
input_chunks: &["echo $CODEX_INTERACTIVE_SHELL_VAR\n".to_string()],
timeout_ms: Some(1_500),
})
.await?;
assert!(out_2.output.contains("codex"));
Ok(())
}
#[cfg(unix)]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn multi_unified_exec_sessions() -> Result<(), UnifiedExecError> {
let manager = UnifiedExecSessionManager::default();
let shell_a = manager
.handle_request(UnifiedExecRequest {
session_id: None,
input_chunks: &["/bin/bash".to_string(), "-i".to_string()],
timeout_ms: Some(1_500),
})
.await?;
let session_a = shell_a.session_id.expect("expected session id");
manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_a),
input_chunks: &["export CODEX_INTERACTIVE_SHELL_VAR=codex\n".to_string()],
timeout_ms: Some(1_500),
})
.await?;
let out_2 = manager
.handle_request(UnifiedExecRequest {
session_id: None,
input_chunks: &[
"echo".to_string(),
"$CODEX_INTERACTIVE_SHELL_VAR\n".to_string(),
],
timeout_ms: Some(1_500),
})
.await?;
assert!(!out_2.output.contains("codex"));
let out_3 = manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_a),
input_chunks: &["echo $CODEX_INTERACTIVE_SHELL_VAR\n".to_string()],
timeout_ms: Some(1_500),
})
.await?;
assert!(out_3.output.contains("codex"));
Ok(())
}
#[cfg(unix)]
#[tokio::test]
async fn unified_exec_timeouts() -> Result<(), UnifiedExecError> {
let manager = UnifiedExecSessionManager::default();
let open_shell = manager
.handle_request(UnifiedExecRequest {
session_id: None,
input_chunks: &["bash".to_string(), "-i".to_string()],
timeout_ms: Some(1_500),
})
.await?;
let session_id = open_shell.session_id.expect("expected session id");
manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_id),
input_chunks: &[
"export".to_string(),
"CODEX_INTERACTIVE_SHELL_VAR=codex\n".to_string(),
],
timeout_ms: Some(1_500),
})
.await?;
let out_2 = manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_id),
input_chunks: &["sleep 5 && echo $CODEX_INTERACTIVE_SHELL_VAR\n".to_string()],
timeout_ms: Some(10),
})
.await?;
assert!(!out_2.output.contains("codex"));
tokio::time::sleep(Duration::from_secs(7)).await;
let empty = Vec::new();
let out_3 = manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_id),
input_chunks: &empty,
timeout_ms: Some(100),
})
.await?;
assert!(out_3.output.contains("codex"));
Ok(())
}
#[cfg(unix)]
#[tokio::test]
async fn requests_with_large_timeout_are_capped() -> Result<(), UnifiedExecError> {
let manager = UnifiedExecSessionManager::default();
let result = manager
.handle_request(UnifiedExecRequest {
session_id: None,
input_chunks: &["echo".to_string(), "codex".to_string()],
timeout_ms: Some(120_000),
})
.await?;
assert!(result.output.starts_with(
"Warning: requested timeout 120000ms exceeds maximum of 60000ms; clamping to 60000ms.\n"
));
assert!(result.output.contains("codex"));
Ok(())
}
#[cfg(unix)]
#[tokio::test]
async fn completed_commands_do_not_persist_sessions() -> Result<(), UnifiedExecError> {
let manager = UnifiedExecSessionManager::default();
let result = manager
.handle_request(UnifiedExecRequest {
session_id: None,
input_chunks: &["/bin/echo".to_string(), "codex".to_string()],
timeout_ms: Some(1_500),
})
.await?;
assert!(result.session_id.is_none());
assert!(result.output.contains("codex"));
assert!(manager.sessions.lock().await.is_empty());
Ok(())
}
#[cfg(unix)]
#[tokio::test]
async fn correct_path_resolution() -> Result<(), UnifiedExecError> {
let manager = UnifiedExecSessionManager::default();
let result = manager
.handle_request(UnifiedExecRequest {
session_id: None,
input_chunks: &["echo".to_string(), "codex".to_string()],
timeout_ms: Some(1_500),
})
.await?;
assert!(result.session_id.is_none());
assert!(result.output.contains("codex"));
assert!(manager.sessions.lock().await.is_empty());
Ok(())
}
#[cfg(unix)]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reusing_completed_session_returns_unknown_session() -> Result<(), UnifiedExecError> {
let manager = UnifiedExecSessionManager::default();
let open_shell = manager
.handle_request(UnifiedExecRequest {
session_id: None,
input_chunks: &["/bin/bash".to_string(), "-i".to_string()],
timeout_ms: Some(1_500),
})
.await?;
let session_id = open_shell.session_id.expect("expected session id");
manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_id),
input_chunks: &["exit\n".to_string()],
timeout_ms: Some(1_500),
})
.await?;
tokio::time::sleep(Duration::from_millis(200)).await;
let err = manager
.handle_request(UnifiedExecRequest {
session_id: Some(session_id),
input_chunks: &[],
timeout_ms: Some(100),
})
.await
.expect_err("expected unknown session error");
match err {
UnifiedExecError::UnknownSessionId { session_id: err_id } => {
assert_eq!(err_id, session_id);
}
other => panic!("expected UnknownSessionId, got {other:?}"),
}
assert!(!manager.sessions.lock().await.contains_key(&session_id));
Ok(())
}
}

View File

@@ -277,7 +277,25 @@ async fn resume_includes_initial_messages_and_sends_prior_items() {
"content": [{ "type": "input_text", "text": "hello" }]
}
]);
assert_eq!(request_body["input"], expected_input);
let input_array = request_body
.get("input")
.and_then(|v| v.as_array())
.cloned()
.expect("input array in request body");
let filtered: Vec<serde_json::Value> = input_array
.into_iter()
.filter(|item| {
let text = item
.get("content")
.and_then(|c| c.as_array())
.and_then(|a| a.first())
.and_then(|o| o.get("text"))
.and_then(|t| t.as_str())
.unwrap_or("");
!text.contains("<environment_context>")
})
.collect();
assert_eq!(serde_json::json!(filtered), expected_input);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -950,34 +968,6 @@ async fn history_dedupes_streamed_and_final_messages_across_turns() {
assert_eq!(requests.len(), 3, "expected 3 requests (one per turn)");
// Replace full-array compare with tail-only raw JSON compare using a single hard-coded value.
let r3_tail_expected = json!([
{
"type": "message",
"role": "user",
"content": [{"type":"input_text","text":"U1"}]
},
{
"type": "message",
"role": "assistant",
"content": [{"type":"output_text","text":"Hey there!\n"}]
},
{
"type": "message",
"role": "user",
"content": [{"type":"input_text","text":"U2"}]
},
{
"type": "message",
"role": "assistant",
"content": [{"type":"output_text","text":"Hey there!\n"}]
},
{
"type": "message",
"role": "user",
"content": [{"type":"input_text","text":"U3"}]
}
]);
let r3_input_array = requests[2]
.body_json::<serde_json::Value>()
.unwrap()
@@ -985,12 +975,60 @@ async fn history_dedupes_streamed_and_final_messages_across_turns() {
.and_then(|v| v.as_array())
.cloned()
.expect("r3 missing input array");
// skipping earlier context and developer messages
let tail_len = r3_tail_expected.as_array().unwrap().len();
let actual_tail = &r3_input_array[r3_input_array.len() - tail_len..];
// We only assert on the last 5 items of the input history for request 3.
// With per-turn environment context injected, the last 5 should be:
// [env_ctx, U2, assistant("Hey there!\n"), env_ctx, U3]
let actual_tail = &r3_input_array[r3_input_array.len() - 5..];
// env_ctx 1
assert_eq!(actual_tail[0]["type"], serde_json::json!("message"));
assert_eq!(actual_tail[0]["role"], serde_json::json!("user"));
let env_text_1 = &actual_tail[0]["content"][0]["text"];
assert!(
env_text_1
.as_str()
.expect("env text should be string")
.contains("<environment_context>")
);
// U2
assert_eq!(
serde_json::Value::Array(actual_tail.to_vec()),
r3_tail_expected,
"request 3 tail mismatch",
actual_tail[1],
serde_json::json!({
"type": "message",
"role": "user",
"content": [ { "type": "input_text", "text": "U2" } ]
})
);
// assistant response
assert_eq!(
actual_tail[2],
serde_json::json!({
"type": "message",
"role": "assistant",
"content": [ { "type": "output_text", "text": "Hey there!\n" } ]
})
);
// env_ctx 2
assert_eq!(actual_tail[3]["type"], serde_json::json!("message"));
assert_eq!(actual_tail[3]["role"], serde_json::json!("user"));
let env_text_2 = &actual_tail[3]["content"][0]["text"];
assert!(
env_text_2
.as_str()
.expect("env text should be string")
.contains("<environment_context>")
);
// U3
assert_eq!(
actual_tail[4],
serde_json::json!({
"type": "message",
"role": "user",
"content": [ { "type": "input_text", "text": "U3" } ]
})
);
}

View File

@@ -1,12 +1,16 @@
use codex_core::CodexAuth;
use codex_core::ContentItem;
use codex_core::ConversationManager;
use codex_core::ModelProviderInfo;
use codex_core::NewConversation;
use codex_core::ResponseItem;
use codex_core::built_in_model_providers;
use codex_core::protocol::ConversationHistoryResponseEvent;
use codex_core::protocol::ConversationPathResponseEvent;
use codex_core::protocol::EventMsg;
use codex_core::protocol::InputItem;
use codex_core::protocol::Op;
use codex_core::protocol::RolloutItem;
use codex_core::protocol::RolloutLine;
use core_test_support::load_default_config_for_test;
use core_test_support::wait_for_event;
use tempfile::TempDir;
@@ -71,84 +75,121 @@ async fn fork_conversation_twice_drops_to_first_message() {
let _ = wait_for_event(&codex, |ev| matches!(ev, EventMsg::TaskComplete(_))).await;
}
// Request history from the base conversation.
codex.submit(Op::GetHistory).await.unwrap();
// Request history from the base conversation to obtain rollout path.
codex.submit(Op::GetPath).await.unwrap();
let base_history =
wait_for_event(&codex, |ev| matches!(ev, EventMsg::ConversationHistory(_))).await;
// Capture entries from the base history and compute expected prefixes after each fork.
let entries_after_three = match &base_history {
EventMsg::ConversationHistory(ConversationHistoryResponseEvent { entries, .. }) => {
entries.clone()
}
wait_for_event(&codex, |ev| matches!(ev, EventMsg::ConversationPath(_))).await;
let base_path = match &base_history {
EventMsg::ConversationPath(ConversationPathResponseEvent { path, .. }) => path.clone(),
_ => panic!("expected ConversationHistory event"),
};
// History layout for this test:
// [0] user instructions,
// [1] environment context,
// [2] "first" user message,
// [3] "second" user message,
// [4] "third" user message.
// Fork 1: drops the last user message and everything after.
let expected_after_first = vec![
entries_after_three[0].clone(),
entries_after_three[1].clone(),
entries_after_three[2].clone(),
entries_after_three[3].clone(),
];
// GetHistory flushes before returning the path; no wait needed.
// Fork 2: drops the last user message and everything after.
// [0] user instructions,
// [1] environment context,
// [2] "first" user message,
let expected_after_second = vec![
entries_after_three[0].clone(),
entries_after_three[1].clone(),
entries_after_three[2].clone(),
];
// Helper: read rollout items (excluding SessionMeta) from a JSONL path.
let read_items = |p: &std::path::Path| -> Vec<RolloutItem> {
let text = std::fs::read_to_string(p).expect("read rollout file");
let mut items: Vec<RolloutItem> = Vec::new();
for line in text.lines() {
if line.trim().is_empty() {
continue;
}
let v: serde_json::Value = serde_json::from_str(line).expect("jsonl line");
let rl: RolloutLine = serde_json::from_value(v).expect("rollout line");
match rl.item {
RolloutItem::SessionMeta(_) => {}
other => items.push(other),
}
}
items
};
// Fork once with n=1 → drops the last user message and everything after.
// Compute expected prefixes after each fork by truncating base rollout at nth-from-last user input.
let base_items = read_items(&base_path);
let find_user_input_positions = |items: &[RolloutItem]| -> Vec<usize> {
let mut pos = Vec::new();
for (i, it) in items.iter().enumerate() {
if let RolloutItem::ResponseItem(ResponseItem::Message { role, content, .. }) = it
&& role == "user"
{
// Consider any user message as an input boundary; recorder stores both EventMsg and ResponseItem.
// We specifically look for input items, which are represented as ContentItem::InputText.
if content
.iter()
.any(|c| matches!(c, ContentItem::InputText { .. }))
{
pos.push(i);
}
}
}
pos
};
let user_inputs = find_user_input_positions(&base_items);
// After dropping last user input (n=1), cut strictly before that input if present, else empty.
let cut1 = user_inputs
.get(user_inputs.len().saturating_sub(1))
.copied()
.unwrap_or(0);
let expected_after_first: Vec<RolloutItem> = base_items[..cut1].to_vec();
// After dropping again (n=1 on fork1), compute expected relative to fork1's rollout.
// Fork once with n=1 → drops the last user input and everything after.
let NewConversation {
conversation: codex_fork1,
..
} = conversation_manager
.fork_conversation(entries_after_three.clone(), 1, config_for_fork.clone())
.fork_conversation(1, config_for_fork.clone(), base_path.clone())
.await
.expect("fork 1");
codex_fork1.submit(Op::GetHistory).await.unwrap();
codex_fork1.submit(Op::GetPath).await.unwrap();
let fork1_history = wait_for_event(&codex_fork1, |ev| {
matches!(ev, EventMsg::ConversationHistory(_))
matches!(ev, EventMsg::ConversationPath(_))
})
.await;
let entries_after_first_fork = match &fork1_history {
EventMsg::ConversationHistory(ConversationHistoryResponseEvent { entries, .. }) => {
assert!(matches!(
fork1_history,
EventMsg::ConversationHistory(ConversationHistoryResponseEvent { ref entries, .. }) if *entries == expected_after_first
));
entries.clone()
}
let fork1_path = match &fork1_history {
EventMsg::ConversationPath(ConversationPathResponseEvent { path, .. }) => path.clone(),
_ => panic!("expected ConversationHistory event after first fork"),
};
// GetHistory on fork1 flushed; the file is ready.
let fork1_items = read_items(&fork1_path);
pretty_assertions::assert_eq!(
serde_json::to_value(&fork1_items).unwrap(),
serde_json::to_value(&expected_after_first).unwrap()
);
// Fork again with n=1 → drops the (new) last user message, leaving only the first.
let NewConversation {
conversation: codex_fork2,
..
} = conversation_manager
.fork_conversation(entries_after_first_fork.clone(), 1, config_for_fork.clone())
.fork_conversation(1, config_for_fork.clone(), fork1_path.clone())
.await
.expect("fork 2");
codex_fork2.submit(Op::GetHistory).await.unwrap();
codex_fork2.submit(Op::GetPath).await.unwrap();
let fork2_history = wait_for_event(&codex_fork2, |ev| {
matches!(ev, EventMsg::ConversationHistory(_))
matches!(ev, EventMsg::ConversationPath(_))
})
.await;
assert!(matches!(
fork2_history,
EventMsg::ConversationHistory(ConversationHistoryResponseEvent { ref entries, .. }) if *entries == expected_after_second
));
let fork2_path = match &fork2_history {
EventMsg::ConversationPath(ConversationPathResponseEvent { path, .. }) => path.clone(),
_ => panic!("expected ConversationHistory event after second fork"),
};
// GetHistory on fork2 flushed; the file is ready.
let fork1_items = read_items(&fork1_path);
let fork1_user_inputs = find_user_input_positions(&fork1_items);
let cut_last_on_fork1 = fork1_user_inputs
.get(fork1_user_inputs.len().saturating_sub(1))
.copied()
.unwrap_or(0);
let expected_after_second: Vec<RolloutItem> = fork1_items[..cut_last_on_fork1].to_vec();
let fork2_items = read_items(&fork2_path);
pretty_assertions::assert_eq!(
serde_json::to_value(&fork2_items).unwrap(),
serde_json::to_value(&expected_after_second).unwrap()
);
}

View File

@@ -17,7 +17,6 @@ use core_test_support::load_default_config_for_test;
use core_test_support::load_sse_fixture_with_id;
use core_test_support::wait_for_event;
use tempfile::TempDir;
use uuid::Uuid;
use wiremock::Mock;
use wiremock::MockServer;
use wiremock::ResponseTemplate;
@@ -192,7 +191,8 @@ async fn prompt_tools_are_consistent_across_requests() {
let expected_instructions: &str = include_str!("../../prompt.md");
// our internal implementation is responsible for keeping tools in sync
// with the OpenAI schema, so we just verify the tool presence here
let expected_tools_names: &[&str] = &["shell", "update_plan", "apply_patch", "view_image"];
let expected_tools_names: &[&str] =
&["unified_exec", "update_plan", "apply_patch", "view_image"];
let body0 = requests[0].body_json::<serde_json::Value>().unwrap();
assert_eq!(
body0["instructions"],
@@ -270,9 +270,9 @@ async fn prefixes_context_and_instructions_once_and_consistently_across_requests
let requests = server.received_requests().await.unwrap();
assert_eq!(requests.len(), 2, "expected two POST requests");
let shell = default_user_shell(Uuid::new_v4(), codex_home.path()).await;
let shell = default_user_shell().await;
let expected_env_text = format!(
let expected_env_text_init = format!(
r#"<environment_context>
<cwd>{}</cwd>
<approval_policy>on-request</approval_policy>
@@ -285,13 +285,28 @@ async fn prefixes_context_and_instructions_once_and_consistently_across_requests
None => String::new(),
}
);
// Per-turn environment context omits the shell tag.
let expected_env_text_turn = format!(
r#"<environment_context>
<cwd>{}</cwd>
<approval_policy>on-request</approval_policy>
<sandbox_mode>read-only</sandbox_mode>
<network_access>restricted</network_access>
</environment_context>"#,
cwd.path().to_string_lossy(),
);
let expected_ui_text =
"<user_instructions>\n\nbe consistent and helpful\n\n</user_instructions>";
let expected_env_msg = serde_json::json!({
let expected_env_msg_init = serde_json::json!({
"type": "message",
"role": "user",
"content": [ { "type": "input_text", "text": expected_env_text } ]
"content": [ { "type": "input_text", "text": expected_env_text_init } ]
});
let expected_env_msg_turn = serde_json::json!({
"type": "message",
"role": "user",
"content": [ { "type": "input_text", "text": expected_env_text_turn } ]
});
let expected_ui_msg = serde_json::json!({
"type": "message",
@@ -307,7 +322,12 @@ async fn prefixes_context_and_instructions_once_and_consistently_across_requests
let body1 = requests[0].body_json::<serde_json::Value>().unwrap();
assert_eq!(
body1["input"],
serde_json::json!([expected_ui_msg, expected_env_msg, expected_user_message_1])
serde_json::json!([
expected_ui_msg,
expected_env_msg_init,
expected_env_msg_turn,
expected_user_message_1
])
);
let expected_user_message_2 = serde_json::json!({
@@ -319,7 +339,7 @@ async fn prefixes_context_and_instructions_once_and_consistently_across_requests
let expected_body2 = serde_json::json!(
[
body1["input"].as_array().unwrap().as_slice(),
[expected_user_message_2].as_slice(),
[expected_env_msg_turn, expected_user_message_2].as_slice(),
]
.concat()
);
@@ -427,11 +447,17 @@ async fn overrides_turn_context_but_keeps_cached_prefix_and_key_constant() {
// After overriding the turn context, the environment context should be emitted again
// reflecting the new approval policy and sandbox settings. Omit cwd because it did
// not change.
let expected_env_text_2 = r#"<environment_context>
let expected_env_text_2 = format!(
r#"<environment_context>
<approval_policy>never</approval_policy>
<sandbox_mode>workspace-write</sandbox_mode>
<network_access>enabled</network_access>
</environment_context>"#;
<writable_roots>
<root>{}</root>
</writable_roots>
</environment_context>"#,
writable.path().to_string_lossy()
);
let expected_env_msg_2 = serde_json::json!({
"type": "message",
"role": "user",
@@ -541,10 +567,24 @@ async fn per_turn_overrides_keep_cached_prefix_and_key_constant() {
"role": "user",
"content": [ { "type": "input_text", "text": "hello 2" } ]
});
let expected_env_text_2 = format!(
r#"<environment_context>
<cwd>{}</cwd>
<approval_policy>never</approval_policy>
<sandbox_mode>workspace-write</sandbox_mode>
<network_access>enabled</network_access>
</environment_context>"#,
new_cwd.path().to_string_lossy()
);
let expected_env_msg_2 = serde_json::json!({
"type": "message",
"role": "user",
"content": [ { "type": "input_text", "text": expected_env_text_2 } ]
});
let expected_body2 = serde_json::json!(
[
body1["input"].as_array().unwrap().as_slice(),
[expected_user_message_2].as_slice(),
[expected_env_msg_2, expected_user_message_2].as_slice(),
]
.concat()
);

View File

@@ -559,7 +559,7 @@ impl EventProcessor for EventProcessorWithHumanOutput {
}
},
EventMsg::ShutdownComplete => return CodexStatus::Shutdown,
EventMsg::ConversationHistory(_) => {}
EventMsg::ConversationPath(_) => {}
EventMsg::UserMessage(_) => {}
}
CodexStatus::Running

View File

@@ -17,10 +17,10 @@ use anyhow::Context;
use anyhow::Result;
use codex_mcp_client::McpClient;
use mcp_types::ClientCapabilities;
use mcp_types::Implementation;
use mcp_types::InitializeRequestParams;
use mcp_types::ListToolsRequestParams;
use mcp_types::MCP_SCHEMA_VERSION;
use mcp_types::McpClientInfo;
use tracing_subscriber::EnvFilter;
#[tokio::main]
@@ -60,10 +60,13 @@ async fn main() -> Result<()> {
sampling: None,
elicitation: None,
},
client_info: McpClientInfo {
client_info: Implementation {
name: "codex-mcp-client".to_owned(),
version: env!("CARGO_PKG_VERSION").to_owned(),
title: Some("Codex".to_string()),
// This field is used by Codex when it is an MCP server: it should
// not be used when Codex is an MCP client.
user_agent: None,
},
protocol_version: MCP_SCHEMA_VERSION.to_owned(),
};

View File

@@ -40,6 +40,7 @@ uuid = { version = "1", features = ["serde", "v4"] }
[dev-dependencies]
assert_cmd = "2"
base64 = "0.22"
mcp_test_support = { path = "tests/common" }
os_info = "3.12.0"
pretty_assertions = "1.4.1"

View File

@@ -11,6 +11,8 @@ use codex_core::NewConversation;
use codex_core::RolloutRecorder;
use codex_core::SessionMeta;
use codex_core::auth::CLIENT_ID;
use codex_core::auth::get_auth_file;
use codex_core::auth::try_read_auth_json;
use codex_core::config::Config;
use codex_core::config::ConfigOverrides;
use codex_core::config::ConfigToml;
@@ -67,6 +69,7 @@ use codex_protocol::mcp_protocol::SendUserMessageResponse;
use codex_protocol::mcp_protocol::SendUserTurnParams;
use codex_protocol::mcp_protocol::SendUserTurnResponse;
use codex_protocol::mcp_protocol::ServerNotification;
use codex_protocol::mcp_protocol::UserInfoResponse;
use codex_protocol::mcp_protocol::UserSavedConfig;
use codex_protocol::models::ContentItem;
use codex_protocol::models::ResponseItem;
@@ -187,6 +190,9 @@ impl CodexMessageProcessor {
ClientRequest::GetUserAgent { request_id } => {
self.get_user_agent(request_id).await;
}
ClientRequest::UserInfo { request_id } => {
self.get_user_info(request_id).await;
}
ClientRequest::ExecOneOffCommand { request_id, params } => {
self.exec_one_off_command(request_id, params).await;
}
@@ -439,6 +445,18 @@ impl CodexMessageProcessor {
self.outgoing.send_response(request_id, response).await;
}
async fn get_user_info(&self, request_id: RequestId) {
// Read alleged user email from auth.json (best-effort; not verified).
let auth_path = get_auth_file(&self.config.codex_home);
let alleged_user_email = match try_read_auth_json(&auth_path) {
Ok(auth) => auth.tokens.and_then(|t| t.id_token.email),
Err(_) => None,
};
let response = UserInfoResponse { alleged_user_email };
self.outgoing.send_response(request_id, response).await;
}
async fn exec_one_off_command(&self, request_id: RequestId, params: ExecOneOffCommandParams) {
tracing::debug!("ExecOneOffCommand params: {params:?}");

View File

@@ -277,7 +277,7 @@ async fn run_codex_tool_session_inner(
| EventMsg::GetHistoryEntryResponse(_)
| EventMsg::PlanUpdate(_)
| EventMsg::TurnAborted(_)
| EventMsg::ConversationHistory(_)
| EventMsg::ConversationPath(_)
| EventMsg::UserMessage(_)
| EventMsg::ShutdownComplete => {
// For now, we do not do anything extra for these

View File

@@ -234,11 +234,11 @@ impl MessageProcessor {
},
instructions: None,
protocol_version: params.protocol_version.clone(),
server_info: mcp_types::McpServerInfo {
server_info: mcp_types::Implementation {
name: "codex-mcp-server".to_string(),
version: env!("CARGO_PKG_VERSION").to_string(),
title: Some("Codex".to_string()),
user_agent: get_codex_user_agent(),
user_agent: Some(get_codex_user_agent()),
},
};

View File

@@ -26,13 +26,13 @@ use codex_protocol::mcp_protocol::SendUserTurnParams;
use mcp_types::CallToolRequestParams;
use mcp_types::ClientCapabilities;
use mcp_types::Implementation;
use mcp_types::InitializeRequestParams;
use mcp_types::JSONRPC_VERSION;
use mcp_types::JSONRPCMessage;
use mcp_types::JSONRPCNotification;
use mcp_types::JSONRPCRequest;
use mcp_types::JSONRPCResponse;
use mcp_types::McpClientInfo;
use mcp_types::ModelContextProtocolNotification;
use mcp_types::ModelContextProtocolRequest;
use mcp_types::RequestId;
@@ -54,6 +54,18 @@ pub struct McpProcess {
impl McpProcess {
pub async fn new(codex_home: &Path) -> anyhow::Result<Self> {
Self::new_with_env(codex_home, &[]).await
}
/// Creates a new MCP process, allowing tests to override or remove
/// specific environment variables for the child process only.
///
/// Pass a tuple of (key, Some(value)) to set/override, or (key, None) to
/// remove a variable from the child's environment.
pub async fn new_with_env(
codex_home: &Path,
env_overrides: &[(&str, Option<&str>)],
) -> anyhow::Result<Self> {
// Use assert_cmd to locate the binary path and then switch to tokio::process::Command
let std_cmd = StdCommand::cargo_bin("codex-mcp-server")
.context("should find binary for codex-mcp-server")?;
@@ -68,6 +80,17 @@ impl McpProcess {
cmd.env("CODEX_HOME", codex_home);
cmd.env("RUST_LOG", "debug");
for (k, v) in env_overrides {
match v {
Some(val) => {
cmd.env(k, val);
}
None => {
cmd.env_remove(k);
}
}
}
let mut process = cmd
.kill_on_drop(true)
.spawn()
@@ -111,10 +134,11 @@ impl McpProcess {
roots: None,
sampling: None,
},
client_info: McpClientInfo {
client_info: Implementation {
name: "elicitation test".into(),
title: Some("Elicitation Test".into()),
version: "0.0.0".into(),
user_agent: None,
},
protocol_version: mcp_types::MCP_SCHEMA_VERSION.into(),
};
@@ -271,6 +295,11 @@ impl McpProcess {
self.send_request("getUserAgent", None).await
}
/// Send a `userInfo` JSON-RPC request.
pub async fn send_user_info_request(&mut self) -> anyhow::Result<i64> {
self.send_request("userInfo", None).await
}
/// Send a `listConversations` JSON-RPC request.
pub async fn send_list_conversations_request(
&mut self,

View File

@@ -41,7 +41,7 @@ async fn get_auth_status_no_auth() {
let codex_home = TempDir::new().unwrap_or_else(|e| panic!("create tempdir: {e}"));
create_config_toml(codex_home.path()).expect("write config.toml");
let mut mcp = McpProcess::new(codex_home.path())
let mut mcp = McpProcess::new_with_env(codex_home.path(), &[("OPENAI_API_KEY", None)])
.await
.expect("spawn mcp process");
timeout(DEFAULT_READ_TIMEOUT, mcp.initialize())

View File

@@ -46,7 +46,7 @@ async fn logout_chatgpt_removes_auth() {
login_with_api_key(codex_home.path(), "sk-test-key").expect("seed api key");
assert!(codex_home.path().join("auth.json").exists());
let mut mcp = McpProcess::new(codex_home.path())
let mut mcp = McpProcess::new_with_env(codex_home.path(), &[("OPENAI_API_KEY", None)])
.await
.expect("spawn mcp process");
timeout(DEFAULT_READ_TIMEOUT, mcp.initialize())

View File

@@ -10,3 +10,4 @@ mod list_resume;
mod login;
mod send_message;
mod user_agent;
mod user_info;

View File

@@ -0,0 +1,78 @@
use std::time::Duration;
use anyhow::Context;
use base64::Engine;
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
use codex_core::auth::AuthDotJson;
use codex_core::auth::get_auth_file;
use codex_core::auth::write_auth_json;
use codex_core::token_data::IdTokenInfo;
use codex_core::token_data::TokenData;
use codex_protocol::mcp_protocol::UserInfoResponse;
use mcp_test_support::McpProcess;
use mcp_test_support::to_response;
use mcp_types::JSONRPCResponse;
use mcp_types::RequestId;
use pretty_assertions::assert_eq;
use serde_json::json;
use tempfile::TempDir;
use tokio::time::timeout;
const DEFAULT_READ_TIMEOUT: Duration = Duration::from_secs(10);
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn user_info_returns_email_from_auth_json() {
let codex_home = TempDir::new().expect("create tempdir");
let auth_path = get_auth_file(codex_home.path());
let mut id_token = IdTokenInfo::default();
id_token.email = Some("user@example.com".to_string());
id_token.raw_jwt = encode_id_token_with_email("user@example.com").expect("encode id token");
let auth = AuthDotJson {
openai_api_key: None,
tokens: Some(TokenData {
id_token,
access_token: "access".to_string(),
refresh_token: "refresh".to_string(),
account_id: None,
}),
last_refresh: None,
};
write_auth_json(&auth_path, &auth).expect("write auth.json");
let mut mcp = McpProcess::new(codex_home.path())
.await
.expect("spawn mcp process");
timeout(DEFAULT_READ_TIMEOUT, mcp.initialize())
.await
.expect("initialize timeout")
.expect("initialize request");
let request_id = mcp.send_user_info_request().await.expect("send userInfo");
let response: JSONRPCResponse = timeout(
DEFAULT_READ_TIMEOUT,
mcp.read_stream_until_response_message(RequestId::Integer(request_id)),
)
.await
.expect("userInfo timeout")
.expect("userInfo response");
let received: UserInfoResponse = to_response(response).expect("deserialize userInfo response");
let expected = UserInfoResponse {
alleged_user_email: Some("user@example.com".to_string()),
};
assert_eq!(received, expected);
}
fn encode_id_token_with_email(email: &str) -> anyhow::Result<String> {
let header_b64 = URL_SAFE_NO_PAD.encode(
serde_json::to_vec(&json!({ "alg": "none", "typ": "JWT" }))
.context("serialize jwt header")?,
);
let payload =
serde_json::to_vec(&json!({ "email": email })).context("serialize jwt payload")?;
let payload_b64 = URL_SAFE_NO_PAD.encode(payload);
Ok(format!("{header_b64}.{payload_b64}.signature"))
}

View File

@@ -265,8 +265,11 @@ class StructField:
name: str
type_name: str
serde: str | None = None
comment: str | None = None
def append(self, out: list[str], supports_const: bool) -> None:
if self.comment:
out.append(f" // {self.comment}\n")
if self.serde:
out.append(f" {self.serde}\n")
if self.viz == "const":
@@ -312,6 +315,18 @@ def define_struct(
else:
fields.append(StructField("pub", rs_prop.name, prop_type, rs_prop.serde))
# Special-case: add Codex-specific user_agent to Implementation
if name == "Implementation":
fields.append(
StructField(
"pub",
"user_agent",
"Option<String>",
'#[serde(default, skip_serializing_if = "Option::is_none")]',
"This is an extra field that the Codex MCP server sends as part of InitializeResult.",
)
)
if implements_request_trait(name):
add_trait_impl(name, "ModelContextProtocolRequest", fields, out)
elif implements_notification_trait(name):

View File

@@ -482,21 +482,14 @@ pub struct ImageContent {
/// Describes the name and version of an MCP implementation, with an optional title for UI representation.
#[derive(Debug, Clone, PartialEq, Deserialize, Serialize, TS)]
pub struct McpClientInfo {
pub struct Implementation {
pub name: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
pub version: String,
}
/// Describes the name and version of an MCP implementation, with an optional title for UI representation.
#[derive(Debug, Clone, PartialEq, Deserialize, Serialize, TS)]
pub struct McpServerInfo {
pub name: String,
// This is an extra field that the Codex MCP server sends as part of InitializeResult.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
pub version: String,
pub user_agent: String,
pub user_agent: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Deserialize, Serialize, TS)]
@@ -512,7 +505,7 @@ impl ModelContextProtocolRequest for InitializeRequest {
pub struct InitializeRequestParams {
pub capabilities: ClientCapabilities,
#[serde(rename = "clientInfo")]
pub client_info: McpClientInfo,
pub client_info: Implementation,
#[serde(rename = "protocolVersion")]
pub protocol_version: String,
}
@@ -526,7 +519,7 @@ pub struct InitializeResult {
#[serde(rename = "protocolVersion")]
pub protocol_version: String,
#[serde(rename = "serverInfo")]
pub server_info: McpServerInfo,
pub server_info: Implementation,
}
impl From<InitializeResult> for serde_json::Value {

View File

@@ -1,10 +1,10 @@
use mcp_types::ClientCapabilities;
use mcp_types::ClientRequest;
use mcp_types::Implementation;
use mcp_types::InitializeRequestParams;
use mcp_types::JSONRPC_VERSION;
use mcp_types::JSONRPCMessage;
use mcp_types::JSONRPCRequest;
use mcp_types::McpClientInfo;
use mcp_types::RequestId;
use serde_json::json;
@@ -58,10 +58,11 @@ fn deserialize_initialize_request() {
sampling: None,
elicitation: None,
},
client_info: McpClientInfo {
client_info: Implementation {
name: "acme-client".into(),
title: Some("Acme".to_string()),
version: "1.2.3".into(),
user_agent: None,
},
protocol_version: "2025-06-18".into(),
}

View File

@@ -39,6 +39,7 @@ pub fn generate_ts(out_dir: &Path, prettier: Option<&Path>) -> Result<()> {
codex_protocol::mcp_protocol::ExecCommandApprovalResponse::export_all_to(out_dir)?;
codex_protocol::mcp_protocol::GetUserSavedConfigResponse::export_all_to(out_dir)?;
codex_protocol::mcp_protocol::GetUserAgentResponse::export_all_to(out_dir)?;
codex_protocol::mcp_protocol::UserInfoResponse::export_all_to(out_dir)?;
// All notification types reachable from this enum will be generated by
// induction, so they do not need to be listed individually.

View File

@@ -152,6 +152,10 @@ pub enum ClientRequest {
#[serde(rename = "id")]
request_id: RequestId,
},
UserInfo {
#[serde(rename = "id")]
request_id: RequestId,
},
/// Execute a command (argv vector) under the server's sandbox.
ExecOneOffCommand {
#[serde(rename = "id")]
@@ -374,6 +378,16 @@ pub struct GetUserAgentResponse {
pub user_agent: String,
}
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, TS)]
#[serde(rename_all = "camelCase")]
pub struct UserInfoResponse {
/// Note: `alleged_user_email` is not currently verified. We read it from
/// the local auth.json, which the user could theoretically modify. In the
/// future, we may add logic to verify the email against the server before
/// returning it.
pub alleged_user_email: Option<String>,
}
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, TS)]
#[serde(rename_all = "camelCase")]
pub struct GetUserSavedConfigResponse {

View File

@@ -115,7 +115,6 @@ pub enum ResponseItem {
status: Option<String>,
action: WebSearchAction,
},
#[serde(other)]
Other,
}

View File

@@ -149,7 +149,7 @@ pub enum Op {
/// Request the full in-memory conversation transcript for the current session.
/// Reply is delivered via `EventMsg::ConversationHistory`.
GetHistory,
GetPath,
/// Request the list of MCP tools available across all configured servers.
/// Reply is delivered via `EventMsg::McpListToolsResponse`.
@@ -499,7 +499,7 @@ pub enum EventMsg {
/// Notification that the agent is shutting down.
ShutdownComplete,
ConversationHistory(ConversationHistoryResponseEvent),
ConversationPath(ConversationPathResponseEvent),
}
// Individual event payload types matching each `EventMsg` variant.
@@ -801,9 +801,9 @@ pub struct WebSearchEndEvent {
/// Response payload for `Op::GetHistory` containing the current session's
/// in-memory transcript.
#[derive(Debug, Clone, Deserialize, Serialize, TS)]
pub struct ConversationHistoryResponseEvent {
pub struct ConversationPathResponseEvent {
pub conversation_id: ConversationId,
pub entries: Vec<ResponseItem>,
pub path: PathBuf,
}
#[derive(Debug, Clone, Deserialize, Serialize, TS)]

View File

@@ -79,7 +79,7 @@ tokio-stream = "0.1.17"
tracing = { version = "0.1.41", features = ["log"] }
tracing-appender = "0.2.3"
tracing-subscriber = { version = "0.3.19", features = ["env-filter"] }
tui-markdown = "0.3.3"
pulldown-cmark = "0.10"
unicode-segmentation = "1.12.0"
unicode-width = "0.1"
url = "2"

View File

@@ -1,9 +1,11 @@
use std::path::PathBuf;
use crate::app::App;
use crate::backtrack_helpers;
use crate::pager_overlay::Overlay;
use crate::tui;
use crate::tui::TuiEvent;
use codex_core::protocol::ConversationHistoryResponseEvent;
use codex_core::protocol::ConversationPathResponseEvent;
use codex_protocol::mcp_protocol::ConversationId;
use color_eyre::eyre::Result;
use crossterm::event::KeyCode;
@@ -98,7 +100,7 @@ impl App {
) {
self.backtrack.pending = Some((base_id, drop_last_messages, prefill));
self.app_event_tx.send(crate::app_event::AppEvent::CodexOp(
codex_core::protocol::Op::GetHistory,
codex_core::protocol::Op::GetPath,
));
}
@@ -265,7 +267,7 @@ impl App {
pub(crate) async fn on_conversation_history_for_backtrack(
&mut self,
tui: &mut tui::Tui,
ev: ConversationHistoryResponseEvent,
ev: ConversationPathResponseEvent,
) -> Result<()> {
if let Some((base_id, _, _)) = self.backtrack.pending.as_ref()
&& ev.conversation_id == *base_id
@@ -281,14 +283,14 @@ impl App {
async fn fork_and_switch_to_new_conversation(
&mut self,
tui: &mut tui::Tui,
ev: ConversationHistoryResponseEvent,
ev: ConversationPathResponseEvent,
drop_count: usize,
prefill: String,
) {
let cfg = self.chat_widget.config_ref().clone();
// Perform the fork via a thin wrapper for clarity/testability.
let result = self
.perform_fork(ev.entries.clone(), drop_count, cfg.clone())
.perform_fork(ev.path.clone(), drop_count, cfg.clone())
.await;
match result {
Ok(new_conv) => {
@@ -301,13 +303,11 @@ impl App {
/// Thin wrapper around ConversationManager::fork_conversation.
async fn perform_fork(
&self,
entries: Vec<codex_protocol::models::ResponseItem>,
path: PathBuf,
drop_count: usize,
cfg: codex_core::config::Config,
) -> codex_core::error::Result<codex_core::NewConversation> {
self.server
.fork_conversation(entries, drop_count, cfg)
.await
self.server.fork_conversation(drop_count, cfg, path).await
}
/// Install a forked conversation into the ChatWidget and update UI to reflect selection.

View File

@@ -1,4 +1,4 @@
use codex_core::protocol::ConversationHistoryResponseEvent;
use codex_core::protocol::ConversationPathResponseEvent;
use codex_core::protocol::Event;
use codex_file_search::FileMatch;
@@ -58,5 +58,5 @@ pub(crate) enum AppEvent {
UpdateSandboxPolicy(SandboxPolicy),
/// Forwarded conversation history snapshot from the current conversation.
ConversationHistory(ConversationHistoryResponseEvent),
ConversationHistory(ConversationPathResponseEvent),
}

View File

@@ -0,0 +1,15 @@
use std::io::Read;
use std::io::{self};
fn main() {
let mut input = String::new();
if let Err(err) = io::stdin().read_to_string(&mut input) {
eprintln!("failed to read stdin: {err}");
std::process::exit(1);
}
let parser = pulldown_cmark::Parser::new(&input);
for event in parser {
println!("{event:?}");
}
}

View File

@@ -1083,7 +1083,7 @@ impl ChatWidget {
self.on_user_message_event(ev);
}
}
EventMsg::ConversationHistory(ev) => {
EventMsg::ConversationPath(ev) => {
self.app_event_tx
.send(crate::app_event::AppEvent::ConversationHistory(ev));
}

View File

@@ -0,0 +1,18 @@
---
source: tui/src/chatwidget/tests.rs
expression: visual
---
> -- Indented code block (4 spaces)
SELECT *
FROM "users"
WHERE "email" LIKE '%@example.com';
```sh
printf 'fenced within fenced\n'
```
{
// comment allowed in jsonc
"path": "C:\\Program Files\\App",
"regex": "^foo.*(bar)?$"
}

View File

@@ -1756,3 +1756,123 @@ fn chatwidget_exec_and_status_layout_vt100_snapshot() {
let visual = vt_lines.join("\n");
assert_snapshot!(visual);
}
// E2E vt100 snapshot for complex markdown with indented and nested fenced code blocks
#[test]
fn chatwidget_markdown_code_blocks_vt100_snapshot() {
let (mut chat, mut rx, _op_rx) = make_chatwidget_manual();
// Simulate a final agent message via streaming deltas instead of a single message
chat.handle_codex_event(Event {
id: "t1".into(),
msg: EventMsg::TaskStarted(TaskStartedEvent {
model_context_window: None,
}),
});
// Build a vt100 visual from the history insertions only (no UI overlay)
let width: u16 = 80;
let height: u16 = 50;
let backend = ratatui::backend::TestBackend::new(width, height);
let mut term = crate::custom_terminal::Terminal::with_options(backend).expect("terminal");
// Place viewport at the last line so that history lines insert above it
term.set_viewport_area(Rect::new(0, height - 1, width, 1));
let mut ansi: Vec<u8> = Vec::new();
// Simulate streaming via AgentMessageDelta in 2-character chunks (no final AgentMessage).
let source: &str = r#"
-- Indented code block (4 spaces)
SELECT *
FROM "users"
WHERE "email" LIKE '%@example.com';
````markdown
```sh
printf 'fenced within fenced\n'
```
````
```jsonc
{
// comment allowed in jsonc
"path": "C:\\Program Files\\App",
"regex": "^foo.*(bar)?$"
}
```
"#;
let mut it = source.chars();
loop {
let mut delta = String::new();
match it.next() {
Some(c) => delta.push(c),
None => break,
}
if let Some(c2) = it.next() {
delta.push(c2);
}
chat.handle_codex_event(Event {
id: "t1".into(),
msg: EventMsg::AgentMessageDelta(AgentMessageDeltaEvent { delta }),
});
// Drive commit ticks and drain emitted history lines into the vt100 buffer.
loop {
chat.on_commit_tick();
let mut inserted_any = false;
while let Ok(app_ev) = rx.try_recv() {
if let AppEvent::InsertHistoryCell(cell) = app_ev {
let lines = cell.display_lines(width);
crate::insert_history::insert_history_lines_to_writer(
&mut term, &mut ansi, lines,
);
inserted_any = true;
}
}
if !inserted_any {
break;
}
}
}
// Finalize the stream without sending a final AgentMessage, to flush any tail.
chat.handle_codex_event(Event {
id: "t1".into(),
msg: EventMsg::TaskComplete(TaskCompleteEvent {
last_agent_message: None,
}),
});
for lines in drain_insert_history(&mut rx) {
crate::insert_history::insert_history_lines_to_writer(&mut term, &mut ansi, lines);
}
let mut parser = vt100::Parser::new(height, width, 0);
parser.process(&ansi);
let mut vt_lines: Vec<String> = (0..height)
.map(|row| {
let mut s = String::with_capacity(width as usize);
for col in 0..width {
if let Some(cell) = parser.screen().cell(row, col) {
if let Some(ch) = cell.contents().chars().next() {
s.push(ch);
} else {
s.push(' ');
}
} else {
s.push(' ');
}
}
s.trim_end().to_string()
})
.collect();
// Compact trailing blank rows for a stable snapshot
while matches!(vt_lines.last(), Some(l) if l.trim().is_empty()) {
vt_lines.pop();
}
let visual = vt_lines.join("\n");
assert_snapshot!(visual);
}

View File

@@ -1 +0,0 @@

View File

@@ -97,7 +97,17 @@ pub fn insert_history_lines_to_writer<B, W>(
for line in wrapped {
queue!(writer, Print("\r\n")).ok();
write_spans(writer, &line).ok();
// Merge line-level style into each span so that ANSI colors reflect
// line styles (e.g., blockquotes with green fg).
let merged_spans: Vec<Span> = line
.spans
.iter()
.map(|s| Span {
style: s.style.patch(line.style),
content: s.content.clone(),
})
.collect();
write_spans(writer, merged_spans.iter()).ok();
}
queue!(writer, ResetScrollRegion).ok();
@@ -264,6 +274,10 @@ where
#[cfg(test)]
mod tests {
use super::*;
use crate::markdown_render::render_markdown_text;
use ratatui::layout::Rect;
use ratatui::style::Color;
use vt100::Parser;
#[test]
fn writes_bold_then_regular_spans() {
@@ -292,4 +306,240 @@ mod tests {
String::from_utf8(expected).unwrap()
);
}
#[test]
fn vt100_blockquote_line_emits_green_fg() {
// Set up a small off-screen terminal
let width: u16 = 40;
let height: u16 = 10;
let backend = ratatui::backend::TestBackend::new(width, height);
let mut term = crate::custom_terminal::Terminal::with_options(backend).expect("terminal");
// Place viewport on the last line so history inserts scroll upward
let viewport = Rect::new(0, height - 1, width, 1);
term.set_viewport_area(viewport);
// Build a blockquote-like line: apply line-level green style and prefix "> "
let mut line: Line<'static> = Line::from(vec!["> ".into(), "Hello world".into()]);
line = line.style(Color::Green);
let mut ansi: Vec<u8> = Vec::new();
insert_history_lines_to_writer(&mut term, &mut ansi, vec![line]);
// Parse ANSI using vt100 and assert at least one non-default fg color appears
let mut parser = Parser::new(height, width, 0);
parser.process(&ansi);
let mut saw_colored = false;
'outer: for row in 0..height {
for col in 0..width {
if let Some(cell) = parser.screen().cell(row, col)
&& cell.has_contents()
&& cell.fgcolor() != vt100::Color::Default
{
saw_colored = true;
break 'outer;
}
}
}
assert!(
saw_colored,
"expected at least one colored cell in vt100 output"
);
}
#[test]
fn vt100_blockquote_wrap_preserves_color_on_all_wrapped_lines() {
// Force wrapping by using a narrow viewport width and a long blockquote line.
let width: u16 = 20;
let height: u16 = 8;
let backend = ratatui::backend::TestBackend::new(width, height);
let mut term = crate::custom_terminal::Terminal::with_options(backend).expect("terminal");
// Viewport is the last line so history goes directly above it.
let viewport = Rect::new(0, height - 1, width, 1);
term.set_viewport_area(viewport);
// Create a long blockquote with a distinct prefix and enough text to wrap.
let mut line: Line<'static> = Line::from(vec![
"> ".into(),
"This is a long quoted line that should wrap".into(),
]);
line = line.style(Color::Green);
let mut ansi: Vec<u8> = Vec::new();
insert_history_lines_to_writer(&mut term, &mut ansi, vec![line]);
// Parse and inspect the final screen buffer.
let mut parser = Parser::new(height, width, 0);
parser.process(&ansi);
let screen = parser.screen();
// Collect rows that are non-empty; these should correspond to our wrapped lines.
let mut non_empty_rows: Vec<u16> = Vec::new();
for row in 0..height {
let mut any = false;
for col in 0..width {
if let Some(cell) = screen.cell(row, col)
&& cell.has_contents()
&& cell.contents() != "\0"
&& cell.contents() != " "
{
any = true;
break;
}
}
if any {
non_empty_rows.push(row);
}
}
// Expect at least two rows due to wrapping.
assert!(
non_empty_rows.len() >= 2,
"expected wrapped output to span >=2 rows, got {non_empty_rows:?}",
);
// For each non-empty row, ensure all non-space cells are using a non-default fg color.
for row in non_empty_rows {
for col in 0..width {
if let Some(cell) = screen.cell(row, col) {
let contents = cell.contents();
if !contents.is_empty() && contents != " " {
assert!(
cell.fgcolor() != vt100::Color::Default,
"expected non-default fg on row {row} col {col}, got {:?}",
cell.fgcolor()
);
}
}
}
}
}
#[test]
fn vt100_colored_prefix_then_plain_text_resets_color() {
let width: u16 = 40;
let height: u16 = 6;
let backend = ratatui::backend::TestBackend::new(width, height);
let mut term = crate::custom_terminal::Terminal::with_options(backend).expect("terminal");
let viewport = Rect::new(0, height - 1, width, 1);
term.set_viewport_area(viewport);
// First span colored, rest plain.
let line: Line<'static> = Line::from(vec![
Span::styled("1. ", ratatui::style::Style::default().fg(Color::LightBlue)),
Span::raw("Hello world"),
]);
let mut ansi: Vec<u8> = Vec::new();
insert_history_lines_to_writer(&mut term, &mut ansi, vec![line]);
let mut parser = Parser::new(height, width, 0);
parser.process(&ansi);
let screen = parser.screen();
// Find the first non-empty row; verify first three cells are colored, following cells default.
'rows: for row in 0..height {
let mut has_text = false;
for col in 0..width {
if let Some(cell) = screen.cell(row, col)
&& cell.has_contents()
&& cell.contents() != " "
{
has_text = true;
break;
}
}
if !has_text {
continue;
}
// Expect "1. Hello world" starting at col 0.
for col in 0..3 {
let cell = screen.cell(row, col).unwrap();
assert!(
cell.fgcolor() != vt100::Color::Default,
"expected colored prefix at col {col}, got {:?}",
cell.fgcolor()
);
}
for col in 3..(3 + "Hello world".len() as u16) {
let cell = screen.cell(row, col).unwrap();
assert_eq!(
cell.fgcolor(),
vt100::Color::Default,
"expected default color for plain text at col {col}, got {:?}",
cell.fgcolor()
);
}
break 'rows;
}
}
#[test]
fn vt100_deep_nested_mixed_list_third_level_marker_is_colored() {
// Markdown with five levels (ordered → unordered → ordered → unordered → unordered).
let md = "1. First\n - Second level\n 1. Third level (ordered)\n - Fourth level (bullet)\n - Fifth level to test indent consistency\n";
let text = render_markdown_text(md);
let lines: Vec<Line<'static>> = text.lines.clone();
let width: u16 = 60;
let height: u16 = 12;
let backend = ratatui::backend::TestBackend::new(width, height);
let mut term = crate::custom_terminal::Terminal::with_options(backend).expect("terminal");
let viewport = ratatui::layout::Rect::new(0, height - 1, width, 1);
term.set_viewport_area(viewport);
let mut ansi: Vec<u8> = Vec::new();
insert_history_lines_to_writer(&mut term, &mut ansi, lines);
let mut parser = Parser::new(height, width, 0);
parser.process(&ansi);
let screen = parser.screen();
// Reconstruct screen rows as strings to locate the 3rd level line.
let mut rows: Vec<String> = Vec::with_capacity(height as usize);
for row in 0..height {
let mut s = String::with_capacity(width as usize);
for col in 0..width {
if let Some(cell) = screen.cell(row, col) {
if let Some(ch) = cell.contents().chars().next() {
s.push(ch);
} else {
s.push(' ');
}
} else {
s.push(' ');
}
}
rows.push(s.trim_end().to_string());
}
let needle = "1. Third level (ordered)";
let row_idx = rows
.iter()
.position(|r| r.contains(needle))
.unwrap_or_else(|| {
panic!("expected to find row containing {needle:?}, have rows: {rows:?}")
});
let col_start = rows[row_idx].find(needle).unwrap() as u16; // column where '1' starts
// Verify that the numeric marker ("1.") at the third level is colored
// (non-default fg) and the content after the following space resets to default.
for c in [col_start, col_start + 1] {
let cell = screen.cell(row_idx as u16, c).unwrap();
assert!(
cell.fgcolor() != vt100::Color::Default,
"expected colored 3rd-level marker at row {row_idx} col {c}, got {:?}",
cell.fgcolor()
);
}
let content_col = col_start + 3; // skip '1', '.', and the space
if let Some(cell) = screen.cell(row_idx as u16, content_col) {
assert_eq!(
cell.fgcolor(),
vt100::Color::Default,
"expected default color for 3rd-level content at row {row_idx} col {content_col}, got {:?}",
cell.fgcolor()
);
}
}
}

View File

@@ -45,6 +45,7 @@ pub mod insert_history;
mod key_hint;
pub mod live_wrap;
mod markdown;
mod markdown_render;
mod markdown_stream;
pub mod onboarding;
mod pager_overlay;
@@ -61,11 +62,6 @@ mod user_approval_widget;
mod version;
mod wrapping;
// Internal vt100-based replay tests live as a separate source file to keep them
// close to the widget code. Include them in unit tests.
#[cfg(test)]
mod chatwidget_stream_tests;
#[cfg(not(debug_assertions))]
mod updates;

View File

@@ -1,8 +1,6 @@
use crate::citation_regex::CITATION_REGEX;
use codex_core::config::Config;
use codex_core::config_types::UriBasedFileOpener;
use ratatui::text::Line;
use std::borrow::Cow;
use std::path::Path;
pub(crate) fn append_markdown(
@@ -19,238 +17,13 @@ fn append_markdown_with_opener_and_cwd(
file_opener: UriBasedFileOpener,
cwd: &Path,
) {
// Historically, we fed the entire `markdown_source` into the renderer in
// one pass. However, fenced code blocks sometimes lost leading whitespace
// when formatted by the markdown renderer/highlighter. To preserve code
// block content exactly, split the source into "text" and "code" segments:
// - Render non-code text through `tui_markdown` (with citation rewrite).
// - Render code block content verbatim as plain lines without additional
// formatting, preserving leading spaces.
for seg in split_text_and_fences(markdown_source) {
match seg {
Segment::Text(s) => {
let processed = rewrite_file_citations(&s, file_opener, cwd);
let rendered = tui_markdown::from_str(&processed);
crate::render::line_utils::push_owned_lines(&rendered.lines, lines);
}
Segment::Code { content, .. } => {
// Emit the code content exactly as-is, line by line.
// We don't attempt syntax highlighting to avoid whitespace bugs.
for line in content.split_inclusive('\n') {
// split_inclusive keeps the trailing \n; we want lines without it.
let line = if let Some(stripped) = line.strip_suffix('\n') {
stripped
} else {
line
};
let owned_line: Line<'static> = line.to_string().into();
lines.push(owned_line);
}
}
}
}
}
/// Rewrites file citations in `src` into markdown hyperlinks using the
/// provided `scheme` (`vscode`, `cursor`, etc.). The resulting URI follows the
/// format expected by VS Code-compatible file openers:
///
/// ```text
/// <scheme>://file<ABS_PATH>:<LINE>
/// ```
fn rewrite_file_citations<'a>(
src: &'a str,
file_opener: UriBasedFileOpener,
cwd: &Path,
) -> Cow<'a, str> {
// Map enum values to the corresponding URI scheme strings.
let scheme: &str = match file_opener.get_scheme() {
Some(scheme) => scheme,
None => return Cow::Borrowed(src),
};
CITATION_REGEX.replace_all(src, |caps: &regex_lite::Captures<'_>| {
let file = &caps[1];
let start_line = &caps[2];
// Resolve the path against `cwd` when it is relative.
let absolute_path = {
let p = Path::new(file);
let absolute_path = if p.is_absolute() {
path_clean::clean(p)
} else {
path_clean::clean(cwd.join(p))
};
// VS Code expects forward slashes even on Windows because URIs use
// `/` as the path separator.
absolute_path.to_string_lossy().replace('\\', "/")
};
// Render as a normal markdown link so the downstream renderer emits
// the hyperlink escape sequence (when supported by the terminal).
//
// In practice, sometimes multiple citations for the same file, but with a
// different line number, are shown sequentially, so we:
// - include the line number in the label to disambiguate them
// - add a space after the link to make it easier to read
format!("[{file}:{start_line}]({scheme}://file{absolute_path}:{start_line}) ")
})
}
// use shared helper from `line_utils`
// Minimal code block splitting.
// - Recognizes fenced blocks opened by ``` or ~~~ (allowing leading whitespace).
// The opening fence may include a language string which we ignore.
// The closing fence must be on its own line (ignoring surrounding whitespace).
// - Additionally recognizes indented code blocks that begin after a blank line
// with a line starting with at least 4 spaces or a tab, and continue for
// consecutive lines that are blank or also indented by >= 4 spaces or a tab.
enum Segment {
Text(String),
Code {
_lang: Option<String>,
content: String,
},
}
fn split_text_and_fences(src: &str) -> Vec<Segment> {
let mut segments = Vec::new();
let mut curr_text = String::new();
#[derive(Copy, Clone, PartialEq)]
enum CodeMode {
None,
Fenced,
Indented,
}
let mut code_mode = CodeMode::None;
let mut fence_token = "";
let mut code_lang: Option<String> = None;
let mut code_content = String::new();
// We intentionally do not require a preceding blank line for indented code blocks,
// since streamed model output often omits it. This favors preserving indentation.
for line in src.split_inclusive('\n') {
let line_no_nl = line.strip_suffix('\n');
let trimmed_start = match line_no_nl {
Some(l) => l.trim_start(),
None => line.trim_start(),
};
if code_mode == CodeMode::None {
let open = if trimmed_start.starts_with("```") {
Some("```")
} else if trimmed_start.starts_with("~~~") {
Some("~~~")
} else {
None
};
if let Some(tok) = open {
// Flush pending text segment.
if !curr_text.is_empty() {
segments.push(Segment::Text(curr_text.clone()));
curr_text.clear();
}
fence_token = tok;
// Capture language after the token on this line (before newline).
let after = &trimmed_start[tok.len()..];
let lang = after.trim();
code_lang = if lang.is_empty() {
None
} else {
Some(lang.to_string())
};
code_mode = CodeMode::Fenced;
code_content.clear();
// Do not include the opening fence line in output.
continue;
}
// Check for start of an indented code block: only after a blank line
// (or at the beginning), and the line must start with >=4 spaces or a tab.
let raw_line = match line_no_nl {
Some(l) => l,
None => line,
};
let leading_spaces = raw_line.chars().take_while(|c| *c == ' ').count();
let starts_with_tab = raw_line.starts_with('\t');
// Consider any line that begins with >=4 spaces or a tab to start an
// indented code block. This favors preserving indentation even when a
// preceding blank line is omitted (common in streamed model output).
let starts_indented_code = (leading_spaces >= 4) || starts_with_tab;
if starts_indented_code {
// Flush pending text and begin an indented code block.
if !curr_text.is_empty() {
segments.push(Segment::Text(curr_text.clone()));
curr_text.clear();
}
code_mode = CodeMode::Indented;
code_content.clear();
code_content.push_str(line);
// Inside code now; do not treat this line as normal text.
continue;
}
// Normal text line.
curr_text.push_str(line);
} else {
match code_mode {
CodeMode::Fenced => {
// inside fenced code: check for closing fence on its own line
let trimmed = match line_no_nl {
Some(l) => l.trim(),
None => line.trim(),
};
if trimmed == fence_token {
// End code block: emit segment without fences
segments.push(Segment::Code {
_lang: code_lang.take(),
content: code_content.clone(),
});
code_content.clear();
code_mode = CodeMode::None;
fence_token = "";
continue;
}
// Accumulate code content exactly as-is.
code_content.push_str(line);
}
CodeMode::Indented => {
// Continue while the line is blank, or starts with >=4 spaces, or a tab.
let raw_line = match line_no_nl {
Some(l) => l,
None => line,
};
let is_blank = raw_line.trim().is_empty();
let leading_spaces = raw_line.chars().take_while(|c| *c == ' ').count();
let starts_with_tab = raw_line.starts_with('\t');
if is_blank || leading_spaces >= 4 || starts_with_tab {
code_content.push_str(line);
} else {
// Close the indented code block and reprocess this line as normal text.
segments.push(Segment::Code {
_lang: None,
content: code_content.clone(),
});
code_content.clear();
code_mode = CodeMode::None;
// Now handle current line as text.
curr_text.push_str(line);
}
}
CodeMode::None => unreachable!(),
}
}
}
if code_mode != CodeMode::None {
// Unterminated code fence: treat accumulated content as a code segment.
segments.push(Segment::Code {
_lang: code_lang.take(),
content: code_content.clone(),
});
} else if !curr_text.is_empty() {
segments.push(Segment::Text(curr_text.clone()));
}
segments
// Render via pulldown-cmark and rewrite citations during traversal (outside code blocks).
let rendered = crate::markdown_render::render_markdown_text_with_citations(
markdown_source,
file_opener.get_scheme(),
cwd,
);
crate::render::line_utils::push_owned_lines(&rendered.lines, lines);
}
#[cfg(test)]
@@ -258,88 +31,6 @@ mod tests {
use super::*;
use pretty_assertions::assert_eq;
#[test]
fn citation_is_rewritten_with_absolute_path() {
let markdown = "See 【F:/src/main.rs†L42-L50】 for details.";
let cwd = Path::new("/workspace");
let result = rewrite_file_citations(markdown, UriBasedFileOpener::VsCode, cwd);
assert_eq!(
"See [/src/main.rs:42](vscode://file/src/main.rs:42) for details.",
result
);
}
#[test]
fn citation_is_rewritten_with_relative_path() {
let markdown = "Refer to 【F:lib/mod.rs†L5】 here.";
let cwd = Path::new("/home/user/project");
let result = rewrite_file_citations(markdown, UriBasedFileOpener::Windsurf, cwd);
assert_eq!(
"Refer to [lib/mod.rs:5](windsurf://file/home/user/project/lib/mod.rs:5) here.",
result
);
}
#[test]
fn citation_followed_by_space_so_they_do_not_run_together() {
let markdown = "References on lines 【F:src/foo.rs†L24】【F:src/foo.rs†L42】";
let cwd = Path::new("/home/user/project");
let result = rewrite_file_citations(markdown, UriBasedFileOpener::VsCode, cwd);
assert_eq!(
"References on lines [src/foo.rs:24](vscode://file/home/user/project/src/foo.rs:24) [src/foo.rs:42](vscode://file/home/user/project/src/foo.rs:42) ",
result
);
}
#[test]
fn citation_unchanged_without_file_opener() {
let markdown = "Look at 【F:file.rs†L1】.";
let cwd = Path::new("/");
let unchanged = rewrite_file_citations(markdown, UriBasedFileOpener::VsCode, cwd);
// The helper itself always rewrites this test validates behaviour of
// append_markdown when `file_opener` is None.
let mut out = Vec::new();
append_markdown_with_opener_and_cwd(markdown, &mut out, UriBasedFileOpener::None, cwd);
// Convert lines back to string for comparison.
let rendered: String = out
.iter()
.flat_map(|l| l.spans.iter())
.map(|s| s.content.clone())
.collect::<Vec<_>>()
.join("");
assert_eq!(markdown, rendered);
// Ensure helper rewrites.
assert_ne!(markdown, unchanged);
}
#[test]
fn fenced_code_blocks_preserve_leading_whitespace() {
let src = "```\n indented\n\t\twith tabs\n four spaces\n```\n";
let cwd = Path::new("/");
let mut out = Vec::new();
append_markdown_with_opener_and_cwd(src, &mut out, UriBasedFileOpener::None, cwd);
let rendered: Vec<String> = out
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
rendered,
vec![
" indented".to_string(),
"\t\twith tabs".to_string(),
" four spaces".to_string()
]
);
}
#[test]
fn citations_not_rewritten_inside_code_blocks() {
let src = "Before 【F:/x.rs†L1】\n```\nInside 【F:/x.rs†L2】\n```\nAfter 【F:/x.rs†L3】\n";
@@ -355,19 +46,31 @@ mod tests {
.collect::<String>()
})
.collect();
// Expect first and last lines rewritten, middle line unchanged.
assert!(rendered[0].contains("vscode://file"));
assert_eq!(rendered[1], "Inside 【F:/x.rs†L2】");
assert!(matches!(rendered.last(), Some(s) if s.contains("vscode://file")));
// Expect a line containing the inside text unchanged.
assert!(rendered.iter().any(|s| s.contains("Inside 【F:/x.rs†L2】")));
// And first/last sections rewritten.
assert!(
rendered
.first()
.map(|s| s.contains("vscode://file"))
.unwrap_or(false)
);
assert!(
rendered
.last()
.map(|s| s.contains("vscode://file"))
.unwrap_or(false)
);
}
#[test]
fn indented_code_blocks_preserve_leading_whitespace() {
let src = "Before\n code 1\n\tcode with tab\n code 2\nAfter\n";
// Basic sanity: indented code with surrounding blank lines should produce the indented line.
let src = "Before\n\n code 1\n\nAfter\n";
let cwd = Path::new("/");
let mut out = Vec::new();
append_markdown_with_opener_and_cwd(src, &mut out, UriBasedFileOpener::None, cwd);
let rendered: Vec<String> = out
let lines: Vec<String> = out
.iter()
.map(|l| {
l.spans
@@ -376,16 +79,7 @@ mod tests {
.collect::<String>()
})
.collect();
assert_eq!(
rendered,
vec![
"Before".to_string(),
" code 1".to_string(),
"\tcode with tab".to_string(),
" code 2".to_string(),
"After".to_string()
]
);
assert_eq!(lines, vec!["Before", "", " code 1", "", "After"]);
}
#[test]
@@ -403,11 +97,17 @@ mod tests {
.collect::<String>()
})
.collect();
// Expect first and last lines rewritten, and the indented code line present
// unchanged (citations inside not rewritten). We do not assert on blank
// separator lines since the markdown renderer may normalize them.
assert!(rendered.iter().any(|s| s.contains("vscode://file")));
assert!(rendered.iter().any(|s| s == " Inside 【F:/x.rs†L2】"));
assert!(
rendered
.iter()
.any(|s| s.contains("Start") && s.contains("vscode://file"))
);
assert!(
rendered
.iter()
.any(|s| s.contains("End") && s.contains("vscode://file"))
);
assert!(rendered.iter().any(|s| s.contains("Inside 【F:/x.rs†L2】")));
}
#[test]
@@ -435,27 +135,6 @@ mod tests {
);
}
#[test]
fn tui_markdown_splits_ordered_marker_and_text() {
// With marker and content on the same line, tui_markdown keeps it as one line
// even in the surrounding section context.
let rendered = tui_markdown::from_str("Loose vs. tight list items:\n1. Tight item\n");
let lines: Vec<String> = rendered
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert!(
lines.iter().any(|w| w == "1. Tight item"),
"expected single line '1. Tight item' in context: {lines:?}"
);
}
#[test]
fn append_markdown_matches_tui_markdown_for_ordered_item() {
use codex_core::config_types::UriBasedFileOpener;
@@ -480,72 +159,6 @@ mod tests {
assert_eq!(lines, vec!["1. Tight item".to_string()]);
}
#[test]
fn tui_markdown_shape_for_loose_tight_section() {
// Use the exact source from the session deltas used in tests.
let source = r#"
Loose vs. tight list items:
1. Tight item
2. Another tight item
3.
Loose item
"#;
let rendered = tui_markdown::from_str(source);
let lines: Vec<String> = rendered
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
// Join into a single string and assert the exact shape we observe
// from tui_markdown in this larger context (marker and content split).
let joined = {
let mut s = String::new();
for (i, l) in lines.iter().enumerate() {
s.push_str(l);
if i + 1 < lines.len() {
s.push('\n');
}
}
s
};
let expected = r#"Loose vs. tight list items:
1.
Tight item
2.
Another tight item
3.
Loose item"#;
assert_eq!(
joined, expected,
"unexpected tui_markdown shape: {joined:?}"
);
}
#[test]
fn split_text_and_fences_keeps_ordered_list_line_as_text() {
// No fences here; expect a single Text segment containing the full input.
let src = "Loose vs. tight list items:\n1. Tight item\n";
let segs = super::split_text_and_fences(src);
assert_eq!(
segs.len(),
1,
"expected single text segment, got {}",
segs.len()
);
match &segs[0] {
super::Segment::Text(s) => assert_eq!(s, src),
_ => panic!("expected Text segment for non-fence input"),
}
}
#[test]
fn append_markdown_keeps_ordered_list_line_unsplit_in_context() {
use codex_core::config_types::UriBasedFileOpener;

View File

@@ -0,0 +1,566 @@
use crate::citation_regex::CITATION_REGEX;
use pulldown_cmark::CodeBlockKind;
use pulldown_cmark::CowStr;
use pulldown_cmark::Event;
use pulldown_cmark::HeadingLevel;
use pulldown_cmark::Options;
use pulldown_cmark::Parser;
use pulldown_cmark::Tag;
use pulldown_cmark::TagEnd;
use ratatui::style::Style;
use ratatui::style::Stylize;
use ratatui::text::Line;
use ratatui::text::Span;
use ratatui::text::Text;
use std::borrow::Cow;
use std::path::Path;
#[derive(Clone, Debug)]
struct IndentContext {
prefix: Vec<Span<'static>>,
marker: Option<Vec<Span<'static>>>,
is_list: bool,
}
impl IndentContext {
fn new(prefix: Vec<Span<'static>>, marker: Option<Vec<Span<'static>>>, is_list: bool) -> Self {
Self {
prefix,
marker,
is_list,
}
}
}
#[allow(dead_code)]
pub(crate) fn render_markdown_text(input: &str) -> Text<'static> {
let mut options = Options::empty();
options.insert(Options::ENABLE_STRIKETHROUGH);
let parser = Parser::new_ext(input, options);
let mut w = Writer::new(parser, None, None);
w.run();
w.text
}
pub(crate) fn render_markdown_text_with_citations(
input: &str,
scheme: Option<&str>,
cwd: &Path,
) -> Text<'static> {
let mut options = Options::empty();
options.insert(Options::ENABLE_STRIKETHROUGH);
let parser = Parser::new_ext(input, options);
let mut w = Writer::new(
parser,
scheme.map(|s| s.to_string()),
Some(cwd.to_path_buf()),
);
w.run();
w.text
}
struct Writer<'a, I>
where
I: Iterator<Item = Event<'a>>,
{
iter: I,
text: Text<'static>,
inline_styles: Vec<Style>,
indent_stack: Vec<IndentContext>,
list_indices: Vec<Option<u64>>,
link: Option<String>,
needs_newline: bool,
pending_marker_line: bool,
in_paragraph: bool,
scheme: Option<String>,
cwd: Option<std::path::PathBuf>,
in_code_block: bool,
}
impl<'a, I> Writer<'a, I>
where
I: Iterator<Item = Event<'a>>,
{
fn new(iter: I, scheme: Option<String>, cwd: Option<std::path::PathBuf>) -> Self {
Self {
iter,
text: Text::default(),
inline_styles: Vec::new(),
indent_stack: Vec::new(),
list_indices: Vec::new(),
link: None,
needs_newline: false,
pending_marker_line: false,
in_paragraph: false,
scheme,
cwd,
in_code_block: false,
}
}
fn run(&mut self) {
while let Some(ev) = self.iter.next() {
self.handle_event(ev);
}
}
fn handle_event(&mut self, event: Event<'a>) {
match event {
Event::Start(tag) => self.start_tag(tag),
Event::End(tag) => self.end_tag(tag),
Event::Text(text) => self.text(text),
Event::Code(code) => self.code(code),
Event::SoftBreak => self.soft_break(),
Event::HardBreak => self.hard_break(),
Event::Rule => {
if !self.text.lines.is_empty() {
self.push_blank_line();
}
self.push_line(Line::from("———"));
self.needs_newline = true;
}
Event::Html(html) => self.html(html, false),
Event::InlineHtml(html) => self.html(html, true),
Event::FootnoteReference(_) => {}
Event::TaskListMarker(_) => {}
}
}
fn start_tag(&mut self, tag: Tag<'a>) {
match tag {
Tag::Paragraph => self.start_paragraph(),
Tag::Heading { level, .. } => self.start_heading(level),
Tag::BlockQuote => self.start_blockquote(),
Tag::CodeBlock(kind) => {
let indent = match kind {
CodeBlockKind::Fenced(_) => None,
CodeBlockKind::Indented => Some(Span::from(" ".repeat(4))),
};
let lang = match kind {
CodeBlockKind::Fenced(lang) => Some(lang.to_string()),
CodeBlockKind::Indented => None,
};
self.start_codeblock(lang, indent)
}
Tag::List(start) => self.start_list(start),
Tag::Item => self.start_item(),
Tag::Emphasis => self.push_inline_style(Style::new().italic()),
Tag::Strong => self.push_inline_style(Style::new().bold()),
Tag::Strikethrough => self.push_inline_style(Style::new().crossed_out()),
Tag::Link { dest_url, .. } => self.push_link(dest_url.to_string()),
Tag::HtmlBlock
| Tag::FootnoteDefinition(_)
| Tag::Table(_)
| Tag::TableHead
| Tag::TableRow
| Tag::TableCell
| Tag::Image { .. }
| Tag::MetadataBlock(_) => {}
}
}
fn end_tag(&mut self, tag: TagEnd) {
match tag {
TagEnd::Paragraph => self.end_paragraph(),
TagEnd::Heading(_) => self.end_heading(),
TagEnd::BlockQuote => self.end_blockquote(),
TagEnd::CodeBlock => self.end_codeblock(),
TagEnd::List(_) => self.end_list(),
TagEnd::Item => {
self.indent_stack.pop();
self.pending_marker_line = false;
}
TagEnd::Emphasis | TagEnd::Strong | TagEnd::Strikethrough => self.pop_inline_style(),
TagEnd::Link => self.pop_link(),
TagEnd::HtmlBlock
| TagEnd::FootnoteDefinition
| TagEnd::Table
| TagEnd::TableHead
| TagEnd::TableRow
| TagEnd::TableCell
| TagEnd::Image
| TagEnd::MetadataBlock(_) => {}
}
}
fn start_paragraph(&mut self) {
if self.needs_newline {
self.push_blank_line();
}
self.push_line(Line::default());
self.needs_newline = false;
self.in_paragraph = true;
}
fn end_paragraph(&mut self) {
self.needs_newline = true;
self.in_paragraph = false;
self.pending_marker_line = false;
}
fn start_heading(&mut self, level: HeadingLevel) {
if self.needs_newline {
self.push_line(Line::default());
self.needs_newline = false;
}
let heading_style = match level {
HeadingLevel::H1 => Style::new().bold().underlined(),
HeadingLevel::H2 => Style::new().bold(),
HeadingLevel::H3 => Style::new().bold().italic(),
HeadingLevel::H4 => Style::new().italic(),
HeadingLevel::H5 => Style::new().italic(),
HeadingLevel::H6 => Style::new().italic(),
};
let content = format!("{} ", "#".repeat(level as usize));
self.push_line(Line::from(vec![Span::styled(content, heading_style)]));
self.push_inline_style(heading_style);
self.needs_newline = false;
}
fn end_heading(&mut self) {
self.needs_newline = true;
self.pop_inline_style();
}
fn start_blockquote(&mut self) {
if self.needs_newline {
self.push_blank_line();
self.needs_newline = false;
}
self.indent_stack
.push(IndentContext::new(vec![Span::from("> ")], None, false));
}
fn end_blockquote(&mut self) {
self.indent_stack.pop();
self.needs_newline = true;
}
fn text(&mut self, text: CowStr<'a>) {
if self.pending_marker_line {
self.push_line(Line::default());
}
self.pending_marker_line = false;
if self.in_code_block
&& !self.needs_newline
&& self
.text
.lines
.last()
.map(|line| !line.spans.is_empty())
.unwrap_or(false)
{
self.push_line(Line::default());
}
for (i, line) in text.lines().enumerate() {
if self.needs_newline {
self.push_line(Line::default());
self.needs_newline = false;
}
if i > 0 {
self.push_line(Line::default());
}
let mut content = line.to_string();
if !self.in_code_block
&& let (Some(scheme), Some(cwd)) = (&self.scheme, &self.cwd)
{
let cow = rewrite_file_citations_with_scheme(&content, Some(scheme.as_str()), cwd);
if let std::borrow::Cow::Owned(s) = cow {
content = s;
}
}
let span = Span::styled(
content,
self.inline_styles.last().copied().unwrap_or_default(),
);
self.push_span(span);
}
self.needs_newline = false;
}
fn code(&mut self, code: CowStr<'a>) {
if self.pending_marker_line {
self.push_line(Line::default());
self.pending_marker_line = false;
}
let span = Span::from(code.into_string()).dim();
self.push_span(span);
}
fn html(&mut self, html: CowStr<'a>, inline: bool) {
self.pending_marker_line = false;
for (i, line) in html.lines().enumerate() {
if self.needs_newline {
self.push_line(Line::default());
self.needs_newline = false;
}
if i > 0 {
self.push_line(Line::default());
}
let style = self.inline_styles.last().copied().unwrap_or_default();
self.push_span(Span::styled(line.to_string(), style));
}
self.needs_newline = !inline;
}
fn hard_break(&mut self) {
self.push_line(Line::default());
}
fn soft_break(&mut self) {
self.push_line(Line::default());
}
fn start_list(&mut self, index: Option<u64>) {
if self.list_indices.is_empty() && self.needs_newline {
self.push_line(Line::default());
}
self.list_indices.push(index);
}
fn end_list(&mut self) {
self.list_indices.pop();
self.needs_newline = true;
}
fn start_item(&mut self) {
self.pending_marker_line = true;
let depth = self.list_indices.len();
let is_ordered = self
.list_indices
.last()
.map(|index| index.is_some())
.unwrap_or(false);
let width = depth * 4 - 3;
let marker = if let Some(last_index) = self.list_indices.last_mut() {
match last_index {
None => Some(vec![Span::from(" ".repeat(width - 1) + "- ")]),
Some(index) => {
*index += 1;
Some(vec![format!("{:width$}. ", *index - 1).light_blue()])
}
}
} else {
None
};
let indent_prefix = if depth == 0 {
Vec::new()
} else {
let indent_len = if is_ordered { width + 2 } else { width + 1 };
vec![Span::from(" ".repeat(indent_len))]
};
self.indent_stack
.push(IndentContext::new(indent_prefix, marker, true));
self.needs_newline = false;
}
fn start_codeblock(&mut self, _lang: Option<String>, indent: Option<Span<'static>>) {
if !self.text.lines.is_empty() {
self.push_blank_line();
}
self.in_code_block = true;
self.indent_stack.push(IndentContext::new(
vec![indent.unwrap_or_default()],
None,
false,
));
// let opener = match lang {
// Some(l) if !l.is_empty() => format!("```{l}"),
// _ => "```".to_string(),
// };
// self.push_line(opener.into());
self.needs_newline = true;
}
fn end_codeblock(&mut self) {
// self.push_line("```".into());
self.needs_newline = true;
self.in_code_block = false;
self.indent_stack.pop();
}
fn push_inline_style(&mut self, style: Style) {
let current = self.inline_styles.last().copied().unwrap_or_default();
let merged = current.patch(style);
self.inline_styles.push(merged);
}
fn pop_inline_style(&mut self) {
self.inline_styles.pop();
}
fn push_link(&mut self, dest_url: String) {
self.link = Some(dest_url);
}
fn pop_link(&mut self) {
if let Some(link) = self.link.take() {
self.push_span(" (".into());
self.push_span(link.cyan().underlined());
self.push_span(")".into());
}
}
fn push_line(&mut self, line: Line<'static>) {
let mut line = line;
let was_pending = self.pending_marker_line;
let mut spans = self.current_prefix_spans();
spans.append(&mut line.spans);
let blockquote_active = self
.indent_stack
.iter()
.any(|ctx| ctx.prefix.iter().any(|s| s.content.contains('>')));
let style = if blockquote_active {
Style::new().green()
} else {
line.style
};
self.text.lines.push(Line::from_iter(spans).style(style));
if was_pending {
self.pending_marker_line = false;
}
}
fn push_span(&mut self, span: Span<'static>) {
if let Some(last) = self.text.lines.last_mut() {
last.push_span(span);
} else {
self.push_line(Line::from(vec![span]));
}
}
fn push_blank_line(&mut self) {
if self.indent_stack.iter().all(|ctx| ctx.is_list) {
self.text.lines.push(Line::default());
} else {
self.push_line(Line::default());
}
}
fn current_prefix_spans(&self) -> Vec<Span<'static>> {
let mut prefix: Vec<Span<'static>> = Vec::new();
let last_marker_index = if self.pending_marker_line {
self.indent_stack
.iter()
.enumerate()
.rev()
.find_map(|(i, ctx)| if ctx.marker.is_some() { Some(i) } else { None })
} else {
None
};
let last_list_index = self.indent_stack.iter().rposition(|ctx| ctx.is_list);
for (i, ctx) in self.indent_stack.iter().enumerate() {
if self.pending_marker_line {
if Some(i) == last_marker_index
&& let Some(marker) = &ctx.marker
{
prefix.extend(marker.iter().cloned());
continue;
}
if ctx.is_list && last_marker_index.is_some_and(|idx| idx > i) {
continue;
}
} else if ctx.is_list && Some(i) != last_list_index {
continue;
}
prefix.extend(ctx.prefix.iter().cloned());
}
prefix
}
}
pub(crate) fn rewrite_file_citations_with_scheme<'a>(
src: &'a str,
scheme_opt: Option<&str>,
cwd: &Path,
) -> Cow<'a, str> {
let scheme: &str = match scheme_opt {
Some(s) => s,
None => return Cow::Borrowed(src),
};
CITATION_REGEX.replace_all(src, |caps: &regex_lite::Captures<'_>| {
let file = &caps[1];
let start_line = &caps[2];
// Resolve the path against `cwd` when it is relative.
let absolute_path = {
let p = Path::new(file);
let absolute_path = if p.is_absolute() {
path_clean::clean(p)
} else {
path_clean::clean(cwd.join(p))
};
// VS Code expects forward slashes even on Windows because URIs use
// `/` as the path separator.
absolute_path.to_string_lossy().replace('\\', "/")
};
// Render as a normal markdown link so the downstream renderer emits
// the hyperlink escape sequence (when supported by the terminal).
//
// In practice, sometimes multiple citations for the same file, but with a
// different line number, are shown sequentially, so we:
// - include the line number in the label to disambiguate them
// - add a space after the link to make it easier to read
format!("[{file}:{start_line}]({scheme}://file{absolute_path}:{start_line}) ")
})
}
#[cfg(test)]
mod markdown_render_tests {
include!("markdown_render_tests.rs");
}
#[cfg(test)]
mod tests {
use super::*;
use pretty_assertions::assert_eq;
#[test]
fn citation_is_rewritten_with_absolute_path() {
let markdown = "See 【F:/src/main.rs†L42-L50】 for details.";
let cwd = Path::new("/workspace");
let result = rewrite_file_citations_with_scheme(markdown, Some("vscode"), cwd);
assert_eq!(
"See [/src/main.rs:42](vscode://file/src/main.rs:42) for details.",
result
);
}
#[test]
fn citation_followed_by_space_so_they_do_not_run_together() {
let markdown = "References on lines 【F:src/foo.rs†L24】【F:src/foo.rs†L42】";
let cwd = Path::new("/home/user/project");
let result = rewrite_file_citations_with_scheme(markdown, Some("vscode"), cwd);
assert_eq!(
"References on lines [src/foo.rs:24](vscode://file/home/user/project/src/foo.rs:24) [src/foo.rs:42](vscode://file/home/user/project/src/foo.rs:42) ",
result
);
}
#[test]
fn citation_unchanged_without_file_opener() {
let markdown = "Look at 【F:file.rs†L1】.";
let cwd = Path::new("/");
let unchanged = rewrite_file_citations_with_scheme(markdown, Some("vscode"), cwd);
// The helper itself always rewrites this test validates behaviour of
// append_markdown when `file_opener` is None.
let rendered = render_markdown_text_with_citations(markdown, None, cwd);
// Convert lines back to string for comparison.
let rendered: String = rendered
.lines
.iter()
.flat_map(|l| l.spans.iter())
.map(|s| s.content.clone())
.collect::<Vec<_>>()
.join("");
assert_eq!(markdown, rendered);
// Ensure helper rewrites.
assert_ne!(markdown, unchanged);
}
}

View File

@@ -0,0 +1,995 @@
use pretty_assertions::assert_eq;
use ratatui::style::Stylize;
use ratatui::text::Line;
use ratatui::text::Span;
use ratatui::text::Text;
use crate::markdown_render::render_markdown_text;
use insta::assert_snapshot;
#[test]
fn empty() {
assert_eq!(render_markdown_text(""), Text::default());
}
#[test]
fn paragraph_single() {
assert_eq!(
render_markdown_text("Hello, world!"),
Text::from("Hello, world!")
);
}
#[test]
fn paragraph_soft_break() {
assert_eq!(
render_markdown_text("Hello\nWorld"),
Text::from_iter(["Hello", "World"])
);
}
#[test]
fn paragraph_multiple() {
assert_eq!(
render_markdown_text("Paragraph 1\n\nParagraph 2"),
Text::from_iter(["Paragraph 1", "", "Paragraph 2"])
);
}
#[test]
fn headings() {
let md = "# Heading 1\n## Heading 2\n### Heading 3\n#### Heading 4\n##### Heading 5\n###### Heading 6\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["# ".bold().underlined(), "Heading 1".bold().underlined()]),
Line::default(),
Line::from_iter(["## ".bold(), "Heading 2".bold()]),
Line::default(),
Line::from_iter(["### ".bold().italic(), "Heading 3".bold().italic()]),
Line::default(),
Line::from_iter(["#### ".italic(), "Heading 4".italic()]),
Line::default(),
Line::from_iter(["##### ".italic(), "Heading 5".italic()]),
Line::default(),
Line::from_iter(["###### ".italic(), "Heading 6".italic()]),
]);
assert_eq!(text, expected);
}
#[test]
fn blockquote_single() {
let text = render_markdown_text("> Blockquote");
let expected = Text::from(Line::from_iter(["> ", "Blockquote"]).green());
assert_eq!(text, expected);
}
#[test]
fn blockquote_soft_break() {
// Soft break via lazy continuation should render as a new line in blockquotes.
let text = render_markdown_text("> This is a blockquote\nwith a soft break\n");
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
lines,
vec![
"> This is a blockquote".to_string(),
"> with a soft break".to_string()
]
);
}
#[test]
fn blockquote_multiple_with_break() {
let text = render_markdown_text("> Blockquote 1\n\n> Blockquote 2\n");
let expected = Text::from_iter([
Line::from_iter(["> ", "Blockquote 1"]).green(),
Line::default(),
Line::from_iter(["> ", "Blockquote 2"]).green(),
]);
assert_eq!(text, expected);
}
#[test]
fn blockquote_three_paragraphs_short_lines() {
let md = "> one\n>\n> two\n>\n> three\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["> ", "one"]).green(),
Line::from_iter(["> "]).green(),
Line::from_iter(["> ", "two"]).green(),
Line::from_iter(["> "]).green(),
Line::from_iter(["> ", "three"]).green(),
]);
assert_eq!(text, expected);
}
#[test]
fn blockquote_nested_two_levels() {
let md = "> Level 1\n>> Level 2\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["> ", "Level 1"]).green(),
Line::from_iter(["> "]).green(),
Line::from_iter(["> ", "> ", "Level 2"]).green(),
]);
assert_eq!(text, expected);
}
#[test]
fn blockquote_with_list_items() {
let md = "> - item 1\n> - item 2\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["> ", "- ", "item 1"]).green(),
Line::from_iter(["> ", "- ", "item 2"]).green(),
]);
assert_eq!(text, expected);
}
#[test]
fn blockquote_with_ordered_list() {
let md = "> 1. first\n> 2. second\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(vec![
Span::from("> "),
"1. ".light_blue(),
Span::from("first"),
])
.green(),
Line::from_iter(vec![
Span::from("> "),
"2. ".light_blue(),
Span::from("second"),
])
.green(),
]);
assert_eq!(text, expected);
}
#[test]
fn blockquote_list_then_nested_blockquote() {
let md = "> - parent\n> > child\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["> ", "- ", "parent"]).green(),
Line::from_iter(["> ", " ", "> ", "child"]).green(),
]);
assert_eq!(text, expected);
}
#[test]
fn list_item_with_inline_blockquote_on_same_line() {
let md = "1. > quoted\n";
let text = render_markdown_text(md);
let mut lines = text.lines.iter();
let first = lines.next().expect("one line");
// Expect content to include the ordered marker, a space, "> ", and the text
let s: String = first.spans.iter().map(|sp| sp.content.clone()).collect();
assert_eq!(s, "1. > quoted");
}
#[test]
fn blockquote_surrounded_by_blank_lines() {
let md = "foo\n\n> bar\n\nbaz\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
lines,
vec![
"foo".to_string(),
"".to_string(),
"> bar".to_string(),
"".to_string(),
"baz".to_string(),
]
);
}
#[test]
fn blockquote_in_ordered_list_on_next_line() {
// Blockquote begins on a new line within an ordered list item; it should
// render inline on the same marker line.
let md = "1.\n > quoted\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["1. > quoted".to_string()]);
}
#[test]
fn blockquote_in_unordered_list_on_next_line() {
// Blockquote begins on a new line within an unordered list item; it should
// render inline on the same marker line.
let md = "-\n > quoted\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["- > quoted".to_string()]);
}
#[test]
fn blockquote_two_paragraphs_inside_ordered_list_has_blank_line() {
// Two blockquote paragraphs inside a list item should be separated by a blank line.
let md = "1.\n > para 1\n >\n > para 2\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
lines,
vec![
"1. > para 1".to_string(),
" > ".to_string(),
" > para 2".to_string(),
],
"expected blockquote content to stay aligned after list marker"
);
}
#[test]
fn blockquote_inside_nested_list() {
let md = "1. A\n - B\n > inner\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["1. A", " - B", " > inner"]);
}
#[test]
fn list_item_text_then_blockquote() {
let md = "1. before\n > quoted\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["1. before", " > quoted"]);
}
#[test]
fn list_item_blockquote_then_text() {
let md = "1.\n > quoted\n after\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["1. > quoted", " > after"]);
}
#[test]
fn list_item_text_blockquote_text() {
let md = "1. before\n > quoted\n after\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["1. before", " > quoted", " > after"]);
}
#[test]
fn blockquote_with_heading_and_paragraph() {
let md = "> # Heading\n> paragraph text\n";
let text = render_markdown_text(md);
// Validate on content shape; styling is handled elsewhere
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
lines,
vec![
"> # Heading".to_string(),
"> ".to_string(),
"> paragraph text".to_string(),
]
);
}
#[test]
fn blockquote_heading_inherits_heading_style() {
let text = render_markdown_text("> # test header\n> in blockquote\n");
assert_eq!(
text.lines,
[
Line::from_iter([
"> ".into(),
"# ".bold().underlined(),
"test header".bold().underlined(),
])
.green(),
Line::from_iter(["> "]).green(),
Line::from_iter(["> ", "in blockquote"]).green(),
]
);
}
#[test]
fn blockquote_with_code_block() {
let md = "> ```\n> code\n> ```\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["> code".to_string()]);
}
#[test]
fn blockquote_with_multiline_code_block() {
let md = "> ```\n> first\n> second\n> ```\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["> first", "> second"]);
}
#[test]
fn nested_blockquote_with_inline_and_fenced_code() {
/*
let md = \"> Nested quote with code:\n\
> > Inner quote and `inline code`\n\
> >\n\
> > ```\n\
> > # fenced code inside a quote\n\
> > echo \"hello from a quote\"\n\
> > ```\n";
*/
let md = r#"> Nested quote with code:
> > Inner quote and `inline code`
> >
> > ```
> > # fenced code inside a quote
> > echo "hello from a quote"
> > ```
"#;
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
lines,
vec![
"> Nested quote with code:".to_string(),
"> ".to_string(),
"> > Inner quote and inline code".to_string(),
"> > ".to_string(),
"> > # fenced code inside a quote".to_string(),
"> > echo \"hello from a quote\"".to_string(),
]
);
}
#[test]
fn list_unordered_single() {
let text = render_markdown_text("- List item 1\n");
let expected = Text::from_iter([Line::from_iter(["- ", "List item 1"])]);
assert_eq!(text, expected);
}
#[test]
fn list_unordered_multiple() {
let text = render_markdown_text("- List item 1\n- List item 2\n");
let expected = Text::from_iter([
Line::from_iter(["- ", "List item 1"]),
Line::from_iter(["- ", "List item 2"]),
]);
assert_eq!(text, expected);
}
#[test]
fn list_ordered() {
let text = render_markdown_text("1. List item 1\n2. List item 2\n");
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "List item 1".into()]),
Line::from_iter(["2. ".light_blue(), "List item 2".into()]),
]);
assert_eq!(text, expected);
}
#[test]
fn list_nested() {
let text = render_markdown_text("- List item 1\n - Nested list item 1\n");
let expected = Text::from_iter([
Line::from_iter(["- ", "List item 1"]),
Line::from_iter([" - ", "Nested list item 1"]),
]);
assert_eq!(text, expected);
}
#[test]
fn list_ordered_custom_start() {
let text = render_markdown_text("3. First\n4. Second\n");
let expected = Text::from_iter([
Line::from_iter(["3. ".light_blue(), "First".into()]),
Line::from_iter(["4. ".light_blue(), "Second".into()]),
]);
assert_eq!(text, expected);
}
#[test]
fn nested_unordered_in_ordered() {
let md = "1. Outer\n - Inner A\n - Inner B\n2. Next\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "Outer".into()]),
Line::from_iter([" - ", "Inner A"]),
Line::from_iter([" - ", "Inner B"]),
Line::from_iter(["2. ".light_blue(), "Next".into()]),
]);
assert_eq!(text, expected);
}
#[test]
fn nested_ordered_in_unordered() {
let md = "- Outer\n 1. One\n 2. Two\n- Last\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["- ", "Outer"]),
Line::from_iter([" 1. ".light_blue(), "One".into()]),
Line::from_iter([" 2. ".light_blue(), "Two".into()]),
Line::from_iter(["- ", "Last"]),
]);
assert_eq!(text, expected);
}
#[test]
fn loose_list_item_multiple_paragraphs() {
let md = "1. First paragraph\n\n Second paragraph of same item\n\n2. Next item\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "First paragraph".into()]),
Line::default(),
Line::from_iter([" ", "Second paragraph of same item"]),
Line::from_iter(["2. ".light_blue(), "Next item".into()]),
]);
assert_eq!(text, expected);
}
#[test]
fn tight_item_with_soft_break() {
let md = "- item line1\n item line2\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["- ", "item line1"]),
Line::from_iter([" ", "item line2"]),
]);
assert_eq!(text, expected);
}
#[test]
fn deeply_nested_mixed_three_levels() {
let md = "1. A\n - B\n 1. C\n2. D\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "A".into()]),
Line::from_iter([" - ", "B"]),
Line::from_iter([" 1. ".light_blue(), "C".into()]),
Line::from_iter(["2. ".light_blue(), "D".into()]),
]);
assert_eq!(text, expected);
}
#[test]
fn loose_items_due_to_blank_line_between_items() {
let md = "1. First\n\n2. Second\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "First".into()]),
Line::from_iter(["2. ".light_blue(), "Second".into()]),
]);
assert_eq!(text, expected);
}
#[test]
fn mixed_tight_then_loose_in_one_list() {
let md = "1. Tight\n\n2.\n Loose\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "Tight".into()]),
Line::from_iter(["2. ".light_blue(), "Loose".into()]),
]);
assert_eq!(text, expected);
}
#[test]
fn ordered_item_with_indented_continuation_is_tight() {
let md = "1. Foo\n Bar\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "Foo".into()]),
Line::from_iter([" ", "Bar"]),
]);
assert_eq!(text, expected);
}
#[test]
fn inline_code() {
let text = render_markdown_text("Example of `Inline code`");
let expected = Line::from_iter(["Example of ".into(), "Inline code".dim()]).into();
assert_eq!(text, expected);
}
#[test]
fn strong() {
assert_eq!(
render_markdown_text("**Strong**"),
Text::from(Line::from("Strong".bold()))
);
}
#[test]
fn emphasis() {
assert_eq!(
render_markdown_text("*Emphasis*"),
Text::from(Line::from("Emphasis".italic()))
);
}
#[test]
fn strikethrough() {
assert_eq!(
render_markdown_text("~~Strikethrough~~"),
Text::from(Line::from("Strikethrough".crossed_out()))
);
}
#[test]
fn strong_emphasis() {
let text = render_markdown_text("**Strong *emphasis***");
let expected = Text::from(Line::from_iter([
"Strong ".bold(),
"emphasis".bold().italic(),
]));
assert_eq!(text, expected);
}
#[test]
fn link() {
let text = render_markdown_text("[Link](https://example.com)");
let expected = Text::from(Line::from_iter([
"Link".into(),
" (".into(),
"https://example.com".cyan().underlined(),
")".into(),
]));
assert_eq!(text, expected);
}
#[test]
fn code_block_unhighlighted() {
let text = render_markdown_text("```rust\nfn main() {}\n```\n");
let expected = Text::from_iter([Line::from_iter(["", "fn main() {}"])]);
assert_eq!(text, expected);
}
#[test]
fn code_block_multiple_lines_root() {
let md = "```\nfirst\nsecond\n```\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["", "first"]),
Line::from_iter(["", "second"]),
]);
assert_eq!(text, expected);
}
#[test]
fn code_block_indented() {
let md = " function greet() {\n console.log(\"Hi\");\n }\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter([" ", "function greet() {"]),
Line::from_iter([" ", " console.log(\"Hi\");"]),
Line::from_iter([" ", "}"]),
]);
assert_eq!(text, expected);
}
#[test]
fn horizontal_rule_renders_em_dashes() {
let md = "Before\n\n---\n\nAfter\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["Before", "", "———", "", "After"]);
}
#[test]
fn code_block_with_inner_triple_backticks_outer_four() {
let md = r#"````text
Here is a code block that shows another fenced block:
```md
# Inside fence
- bullet
- `inline code`
```
````
"#;
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
lines,
vec![
"Here is a code block that shows another fenced block:".to_string(),
String::new(),
"```md".to_string(),
"# Inside fence".to_string(),
"- bullet".to_string(),
"- `inline code`".to_string(),
"```".to_string(),
]
);
}
#[test]
fn code_block_inside_unordered_list_item_is_indented() {
let md = "- Item\n\n ```\n code line\n ```\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["- Item", "", " code line"]);
}
#[test]
fn code_block_multiple_lines_inside_unordered_list() {
let md = "- Item\n\n ```\n first\n second\n ```\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["- Item", "", " first", " second"]);
}
#[test]
fn code_block_inside_unordered_list_item_multiple_lines() {
let md = "- Item\n\n ```\n first\n second\n ```\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(lines, vec!["- Item", "", " first", " second"]);
}
#[test]
fn markdown_render_complex_snapshot() {
let md = r#"# H1: Markdown Streaming Test
Intro paragraph with bold **text**, italic *text*, and inline code `x=1`.
Combined bold-italic ***both*** and escaped asterisks \*literal\*.
Auto-link: <https://example.com> and reference link [ref][r1].
Link with title: [hover me](https://example.com "Example") and mailto <mailto:test@example.com>.
Image: ![alt text](https://example.com/img.png "Title")
> Blockquote level 1
>> Blockquote level 2 with `inline code`
- Unordered list item 1
- Nested bullet with italics _inner_
- Unordered list item 2 with ~~strikethrough~~
1. Ordered item one
2. Ordered item two with sublist:
1) Alt-numbered subitem
- [ ] Task: unchecked
- [x] Task: checked with link [home](https://example.org)
---
Table below (alignment test):
| Left | Center | Right |
|:-----|:------:|------:|
| a | b | c |
Inline HTML: <sup>sup</sup> and <sub>sub</sub>.
HTML block:
<div style="border:1px solid #ccc;padding:2px">inline block</div>
Escapes: \_underscores\_, backslash \\, ticks ``code with `backtick` inside``.
Emoji shortcodes: :sparkles: :tada: (if supported).
Hard break test (line ends with two spaces)
Next line should be close to previous.
Footnote reference here[^1] and another[^longnote].
Horizontal rule with asterisks:
***
Fenced code block (JSON):
```json
{ "a": 1, "b": [true, false] }
```
Fenced code with tildes and triple backticks inside:
~~~markdown
To close ``` you need tildes.
~~~
Indented code block:
for i in range(3): print(i)
Definition-like list:
Term
: Definition with `code`.
Character entities: &amp; &lt; &gt; &quot; &#39;
[^1]: This is the first footnote.
[^longnote]: A longer footnote with a link to [Rust](https://www.rust-lang.org/).
Escaped pipe in text: a \| b \| c.
URL with parentheses: [link](https://example.com/path_(with)_parens).
[r1]: https://example.com/ref "Reference link title"
"#;
let text = render_markdown_text(md);
// Convert to plain text lines for snapshot (ignore styles)
let rendered = text
.lines
.iter()
.map(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
})
.collect::<Vec<_>>()
.join("\n");
assert_snapshot!(rendered);
}
#[test]
fn ordered_item_with_code_block_and_nested_bullet() {
let md = "1. **item 1**\n\n2. **item 2**\n ```\n code\n ```\n - `PROCESS_START` (a `OnceLock<Instant>`) keeps the start time for the entire process.\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|line| {
line.spans
.iter()
.map(|span| span.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
lines,
vec![
"1. item 1".to_string(),
"2. item 2".to_string(),
String::new(),
" code".to_string(),
" - PROCESS_START (a OnceLock<Instant>) keeps the start time for the entire process.".to_string(),
]
);
}
#[test]
fn nested_five_levels_mixed_lists() {
let md = "1. First\n - Second level\n 1. Third level (ordered)\n - Fourth level (bullet)\n - Fifth level to test indent consistency\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "First".into()]),
Line::from_iter([" - ", "Second level"]),
Line::from_iter([" 1. ".light_blue(), "Third level (ordered)".into()]),
Line::from_iter([" - ", "Fourth level (bullet)"]),
Line::from_iter([
" - ",
"Fifth level to test indent consistency",
]),
]);
assert_eq!(text, expected);
}
#[test]
fn html_inline_is_verbatim() {
let md = "Hello <span>world</span>!";
let text = render_markdown_text(md);
let expected: Text = Line::from_iter(["Hello ", "<span>", "world", "</span>", "!"]).into();
assert_eq!(text, expected);
}
#[test]
fn html_block_is_verbatim_multiline() {
let md = "<div>\n <span>hi</span>\n</div>\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["<div>"]),
Line::from_iter([" <span>hi</span>"]),
Line::from_iter(["</div>"]),
]);
assert_eq!(text, expected);
}
#[test]
fn html_in_tight_ordered_item_soft_breaks_with_space() {
let md = "1. Foo\n <i>Bar</i>\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "Foo".into()]),
Line::from_iter([" ", "<i>", "Bar", "</i>"]),
]);
assert_eq!(text, expected);
}
#[test]
fn html_continuation_paragraph_in_unordered_item_indented() {
let md = "- Item\n\n <em>continued</em>\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["- ", "Item"]),
Line::default(),
Line::from_iter([" ", "<em>", "continued", "</em>"]),
]);
assert_eq!(text, expected);
}
#[test]
fn unordered_item_continuation_paragraph_is_indented() {
let md = "- Intro\n\n Continuation paragraph line 1\n Continuation paragraph line 2\n";
let text = render_markdown_text(md);
let lines: Vec<String> = text
.lines
.iter()
.map(|line| {
line.spans
.iter()
.map(|span| span.content.clone())
.collect::<String>()
})
.collect();
assert_eq!(
lines,
vec![
"- Intro".to_string(),
String::new(),
" Continuation paragraph line 1".to_string(),
" Continuation paragraph line 2".to_string(),
]
);
}
#[test]
fn ordered_item_continuation_paragraph_is_indented() {
let md = "1. Intro\n\n More details about intro\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "Intro".into()]),
Line::default(),
Line::from_iter([" ", "More details about intro"]),
]);
assert_eq!(text, expected);
}
#[test]
fn nested_item_continuation_paragraph_is_indented() {
let md = "1. A\n - B\n\n Continuation for B\n2. C\n";
let text = render_markdown_text(md);
let expected = Text::from_iter([
Line::from_iter(["1. ".light_blue(), "A".into()]),
Line::from_iter([" - ", "B"]),
Line::default(),
Line::from_iter([" ", "Continuation for B"]),
Line::from_iter(["2. ".light_blue(), "C".into()]),
]);
assert_eq!(text, expected);
}

View File

@@ -4,8 +4,6 @@ use codex_core::config::Config;
use ratatui::text::Line;
use crate::markdown;
use crate::render::markdown_utils::is_inside_unclosed_fence;
use crate::render::markdown_utils::strip_empty_fenced_code_blocks;
/// Newline-gated accumulator that renders markdown and commits only fully
/// completed logical lines.
@@ -42,6 +40,7 @@ impl MarkdownStreamCollector {
}
pub fn push_delta(&mut self, delta: &str) {
tracing::trace!("push_delta: {delta:?}");
self.buffer.push_str(delta);
}
@@ -49,14 +48,15 @@ impl MarkdownStreamCollector {
/// since the last commit. When the buffer does not end with a newline, the
/// final rendered line is considered incomplete and is not emitted.
pub fn commit_complete_lines(&mut self, config: &Config) -> Vec<Line<'static>> {
// In non-test builds, unwrap an outer ```markdown fence during commit as well,
// so fence markers never appear in streamed history.
let source = unwrap_markdown_language_fence_if_enabled(self.buffer.clone());
let source = strip_empty_fenced_code_blocks(&source);
let source = self.buffer.clone();
let last_newline_idx = source.rfind('\n');
let source = if let Some(last_newline_idx) = last_newline_idx {
source[..=last_newline_idx].to_string()
} else {
return Vec::new();
};
let mut rendered: Vec<Line<'static>> = Vec::new();
markdown::append_markdown(&source, &mut rendered, config);
let mut complete_line_count = rendered.len();
if complete_line_count > 0
&& crate::render::line_utils::is_blank_line_spaces_only(
@@ -65,87 +65,12 @@ impl MarkdownStreamCollector {
{
complete_line_count -= 1;
}
// Heuristic: if the buffer ends with a double newline and the last non-blank
// rendered line looks like a list bullet with inline content (e.g., "- item"),
// defer committing that line. Subsequent context (e.g., another list item)
// can cause the renderer to split the bullet marker and text into separate
// logical lines ("- " then "item"), which would otherwise duplicate content.
if self.buffer.ends_with("\n\n") && complete_line_count > 0 {
let last = &rendered[complete_line_count - 1];
let mut text = String::new();
for s in &last.spans {
text.push_str(&s.content);
}
if text.starts_with("- ") && text.trim() != "-" {
complete_line_count = complete_line_count.saturating_sub(1);
}
}
if !self.buffer.ends_with('\n') {
complete_line_count = complete_line_count.saturating_sub(1);
// If we're inside an unclosed fenced code block, also drop the
// last rendered line to avoid committing a partial code line.
if is_inside_unclosed_fence(&source) {
complete_line_count = complete_line_count.saturating_sub(1);
}
// If the next (incomplete) line appears to begin a list item,
// also defer the previous completed line because the renderer may
// retroactively treat it as part of the list (e.g., ordered list item 1).
if let Some(last_nl) = source.rfind('\n') {
let tail = &source[last_nl + 1..];
if starts_with_list_marker(tail) {
complete_line_count = complete_line_count.saturating_sub(1);
}
}
}
// Conservatively withhold trailing list-like lines (unordered or ordered)
// because streaming mid-item can cause the renderer to later split or
// restructure them (e.g., duplicating content or separating the marker).
// Only defers lines at the end of the out slice so previously committed
// lines remain stable.
if complete_line_count > self.committed_line_count {
let mut safe_count = complete_line_count;
while safe_count > self.committed_line_count {
let l = &rendered[safe_count - 1];
let mut text = String::new();
for s in &l.spans {
text.push_str(&s.content);
}
let listish = is_potentially_volatile_list_line(&text);
if listish {
safe_count -= 1;
continue;
}
break;
}
complete_line_count = safe_count;
}
if self.committed_line_count >= complete_line_count {
return Vec::new();
}
let out_slice = &rendered[self.committed_line_count..complete_line_count];
// Strong correctness: while a fenced code block is open (no closing fence yet),
// do not emit any new lines from inside it. Wait until the fence closes to emit
// the entire block together. This avoids stray backticks and misformatted content.
if is_inside_unclosed_fence(&source) {
return Vec::new();
}
// Additional conservative hold-back: if exactly one short, plain word
// line would be emitted, defer it. This avoids committing a lone word
// that might become the first ordered-list item once the next delta
// arrives (e.g., next line starts with "2 " or "2. ").
if out_slice.len() == 1 {
let mut s = String::new();
for sp in &out_slice[0].spans {
s.push_str(&sp.content);
}
if is_short_plain_word(&s) {
return Vec::new();
}
}
let out = out_slice.to_vec();
self.committed_line_count = complete_line_count;
@@ -157,12 +82,19 @@ impl MarkdownStreamCollector {
/// for rendering. Optionally unwraps ```markdown language fences in
/// non-test builds.
pub fn finalize_and_drain(&mut self, config: &Config) -> Vec<Line<'static>> {
let mut source: String = self.buffer.clone();
let raw_buffer = self.buffer.clone();
let mut source: String = raw_buffer.clone();
if !source.ends_with('\n') {
source.push('\n');
}
let source = unwrap_markdown_language_fence_if_enabled(source);
let source = strip_empty_fenced_code_blocks(&source);
tracing::debug!(
raw_len = raw_buffer.len(),
source_len = source.len(),
"markdown finalize (raw length: {}, rendered length: {})",
raw_buffer.len(),
source.len()
);
tracing::trace!("markdown finalize (raw source):\n---\n{source}\n---");
let mut rendered: Vec<Line<'static>> = Vec::new();
markdown::append_markdown(&source, &mut rendered, config);
@@ -179,122 +111,6 @@ impl MarkdownStreamCollector {
}
}
#[inline]
fn is_potentially_volatile_list_line(text: &str) -> bool {
let t = text.trim_end();
if t == "-" || t == "*" || t == "- " || t == "* " {
return true;
}
if t.starts_with("- ") || t.starts_with("* ") {
return true;
}
// ordered list like "1. " or "23. "
let mut it = t.chars().peekable();
let mut saw_digit = false;
while let Some(&ch) = it.peek() {
if ch.is_ascii_digit() {
saw_digit = true;
it.next();
continue;
}
break;
}
if saw_digit && it.peek() == Some(&'.') {
// consume '.'
it.next();
if it.peek() == Some(&' ') {
return true;
}
}
false
}
#[inline]
fn starts_with_list_marker(text: &str) -> bool {
let t = text.trim_start();
if t.starts_with("- ") || t.starts_with("* ") || t.starts_with("-\t") || t.starts_with("*\t") {
return true;
}
// ordered list marker like "1 ", "1. ", "23 ", "23. "
let mut it = t.chars().peekable();
let mut saw_digit = false;
while let Some(&ch) = it.peek() {
if ch.is_ascii_digit() {
saw_digit = true;
it.next();
} else {
break;
}
}
if !saw_digit {
return false;
}
match it.peek() {
Some('.') => {
it.next();
matches!(it.peek(), Some(' '))
}
Some(' ') => true,
_ => false,
}
}
#[inline]
fn is_short_plain_word(s: &str) -> bool {
let t = s.trim();
if t.is_empty() || t.len() > 5 {
return false;
}
t.chars().all(|c| c.is_alphanumeric())
}
/// fence helpers are provided by `crate::render::markdown_utils`
#[cfg(test)]
fn unwrap_markdown_language_fence_if_enabled(s: String) -> String {
// In tests, keep content exactly as provided to simplify assertions.
s
}
#[cfg(not(test))]
fn unwrap_markdown_language_fence_if_enabled(s: String) -> String {
// Best-effort unwrap of a single outer fenced markdown block.
// Recognizes common forms like ```markdown, ```md (any case), optional
// surrounding whitespace, and flexible trailing newlines/CRLF.
// If the block is not recognized, return the input unchanged.
let lines = s.lines().collect::<Vec<_>>();
if lines.len() < 2 {
return s;
}
// Identify opening fence and language.
let open = lines.first().map(|l| l.trim_start()).unwrap_or("");
if !open.starts_with("```") {
return s;
}
let lang = open.trim_start_matches("```").trim();
let is_markdown_lang = lang.eq_ignore_ascii_case("markdown") || lang.eq_ignore_ascii_case("md");
if !is_markdown_lang {
return s;
}
// Find the last non-empty line and ensure it is a closing fence.
let mut last_idx = lines.len() - 1;
while last_idx > 0 && lines[last_idx].trim().is_empty() {
last_idx -= 1;
}
if lines[last_idx].trim() != "```" {
return s;
}
// Reconstruct the inner content between the fences.
let mut out = String::new();
for l in lines.iter().take(last_idx).skip(1) {
out.push_str(l);
out.push('\n');
}
out
}
pub(crate) struct StepResult {
pub history: Vec<Line<'static>>, // lines to insert into history this step
}
@@ -373,6 +189,7 @@ mod tests {
use super::*;
use codex_core::config::Config;
use codex_core::config::ConfigOverrides;
use ratatui::style::Color;
fn test_config() -> Config {
let overrides = ConfigOverrides {
@@ -406,6 +223,125 @@ mod tests {
assert_eq!(out.len(), 1);
}
#[test]
fn e2e_stream_blockquote_simple_is_green() {
let cfg = test_config();
let out = super::simulate_stream_markdown_for_tests(&["> Hello\n"], true, &cfg);
assert_eq!(out.len(), 1);
let l = &out[0];
assert_eq!(
l.style.fg,
Some(Color::Green),
"expected blockquote line fg green, got {:?}",
l.style.fg
);
}
#[test]
fn e2e_stream_blockquote_nested_is_green() {
let cfg = test_config();
let out =
super::simulate_stream_markdown_for_tests(&["> Level 1\n>> Level 2\n"], true, &cfg);
// Filter out any blank lines that may be inserted at paragraph starts.
let non_blank: Vec<_> = out
.into_iter()
.filter(|l| {
let s = l
.spans
.iter()
.map(|sp| sp.content.clone())
.collect::<Vec<_>>()
.join("");
let t = s.trim();
// Ignore quote-only blank lines like ">" inserted at paragraph boundaries.
!(t.is_empty() || t == ">")
})
.collect();
assert_eq!(non_blank.len(), 2);
assert_eq!(non_blank[0].style.fg, Some(Color::Green));
assert_eq!(non_blank[1].style.fg, Some(Color::Green));
}
#[test]
fn e2e_stream_blockquote_with_list_items_is_green() {
let cfg = test_config();
let out =
super::simulate_stream_markdown_for_tests(&["> - item 1\n> - item 2\n"], true, &cfg);
assert_eq!(out.len(), 2);
assert_eq!(out[0].style.fg, Some(Color::Green));
assert_eq!(out[1].style.fg, Some(Color::Green));
}
#[test]
fn e2e_stream_nested_mixed_lists_ordered_marker_is_light_blue() {
let cfg = test_config();
let md = [
"1. First\n",
" - Second level\n",
" 1. Third level (ordered)\n",
" - Fourth level (bullet)\n",
" - Fifth level to test indent consistency\n",
];
let out = super::simulate_stream_markdown_for_tests(&md, true, &cfg);
// Find the line that contains the third-level ordered text
let find_idx = out.iter().position(|l| {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<String>()
.contains("Third level (ordered)")
});
let idx = find_idx.expect("expected third-level ordered line");
let line = &out[idx];
// Expect at least one span on this line to be styled light blue
let has_light_blue = line
.spans
.iter()
.any(|s| s.style.fg == Some(ratatui::style::Color::LightBlue));
assert!(
has_light_blue,
"expected an ordered-list marker span with light blue fg on: {line:?}"
);
}
#[test]
fn e2e_stream_blockquote_wrap_preserves_green_style() {
let cfg = test_config();
let long = "> This is a very long quoted line that should wrap across multiple columns to verify style preservation.";
let out = super::simulate_stream_markdown_for_tests(&[long, "\n"], true, &cfg);
// Wrap to a narrow width to force multiple output lines.
let wrapped = crate::wrapping::word_wrap_lines(
out.iter().collect::<Vec<_>>(),
crate::wrapping::RtOptions::new(24),
);
// Filter out purely blank lines
let non_blank: Vec<_> = wrapped
.into_iter()
.filter(|l| {
let s = l
.spans
.iter()
.map(|sp| sp.content.clone())
.collect::<Vec<_>>()
.join("");
!s.trim().is_empty()
})
.collect();
assert!(
non_blank.len() >= 2,
"expected wrapped blockquote to span multiple lines"
);
for (i, l) in non_blank.iter().enumerate() {
assert_eq!(
l.style.fg,
Some(Color::Green),
"wrapped line {} should preserve green style, got {:?}",
i,
l.style.fg
);
}
}
#[test]
fn heading_starts_on_new_line_when_following_paragraph() {
let cfg = test_config();
@@ -490,7 +426,7 @@ mod tests {
.collect();
assert_eq!(
s1,
vec!["Sounds good!", ""],
vec!["Sounds good!"],
"expected paragraph followed by blank separator before heading chunk"
);
@@ -509,7 +445,7 @@ mod tests {
.collect();
assert_eq!(
s2,
vec!["## Adding Bird subcommand"],
vec!["", "## Adding Bird subcommand"],
"expected the heading line only on the final commit"
);
@@ -531,18 +467,6 @@ mod tests {
vec!["Hello."],
"unexpected markdown lines: {rendered_strings:?}"
);
let line_to_string = |l: &ratatui::text::Line<'_>| -> String {
l.spans
.iter()
.map(|s| s.content.clone())
.collect::<Vec<_>>()
.join("")
};
assert_eq!(line_to_string(&out1[0]), "Sounds good!");
assert_eq!(line_to_string(&out1[1]), "");
assert_eq!(line_to_string(&out2[0]), "## Adding Bird subcommand");
}
fn lines_to_plain_strings(lines: &[ratatui::text::Line<'_>]) -> Vec<String> {
@@ -560,35 +484,11 @@ mod tests {
#[test]
fn lists_and_fences_commit_without_duplication() {
let cfg = test_config();
// List case
let deltas = vec!["- a\n- ", "b\n- c\n"];
let streamed = simulate_stream_markdown_for_tests(&deltas, true, &cfg);
let streamed_str = lines_to_plain_strings(&streamed);
let mut rendered_all: Vec<ratatui::text::Line<'static>> = Vec::new();
crate::markdown::append_markdown("- a\n- b\n- c\n", &mut rendered_all, &cfg);
let rendered_all_str = lines_to_plain_strings(&rendered_all);
assert_eq!(
streamed_str, rendered_all_str,
"list streaming should equal full render without duplication"
);
assert_streamed_equals_full(&["- a\n- ", "b\n- c\n"]);
// Fenced code case: stream in small chunks
let deltas2 = vec!["```", "\nco", "de 1\ncode 2\n", "```\n"];
let streamed2 = simulate_stream_markdown_for_tests(&deltas2, true, &cfg);
let streamed2_str = lines_to_plain_strings(&streamed2);
let mut rendered_all2: Vec<ratatui::text::Line<'static>> = Vec::new();
crate::markdown::append_markdown("```\ncode 1\ncode 2\n```\n", &mut rendered_all2, &cfg);
let rendered_all2_str = lines_to_plain_strings(&rendered_all2);
assert_eq!(
streamed2_str, rendered_all2_str,
"fence streaming should equal full render without duplication"
);
assert_streamed_equals_full(&["```", "\nco", "de 1\ncode 2\n", "```\n"]);
}
#[test]
@@ -622,6 +522,56 @@ mod tests {
);
}
#[test]
fn e2e_stream_deep_nested_third_level_marker_is_light_blue() {
let cfg = test_config();
let md = "1. First\n - Second level\n 1. Third level (ordered)\n - Fourth level (bullet)\n - Fifth level to test indent consistency\n";
let streamed = super::simulate_stream_markdown_for_tests(&[md], true, &cfg);
let streamed_strs = lines_to_plain_strings(&streamed);
// Locate the third-level line in the streamed output; avoid relying on exact indent.
let target_suffix = "1. Third level (ordered)";
let mut found = None;
for line in &streamed {
let s: String = line.spans.iter().map(|sp| sp.content.clone()).collect();
if s.contains(target_suffix) {
found = Some(line.clone());
break;
}
}
let line = found.unwrap_or_else(|| {
panic!("expected to find the third-level ordered list line; got: {streamed_strs:?}")
});
// The marker (including indent and "1.") is expected to be in the first span
// and colored LightBlue; following content should be default color.
assert!(
!line.spans.is_empty(),
"expected non-empty spans for the third-level line"
);
let marker_span = &line.spans[0];
assert_eq!(
marker_span.style.fg,
Some(Color::LightBlue),
"expected LightBlue 3rd-level ordered marker, got {:?}",
marker_span.style.fg
);
// Find the first non-empty non-space content span and verify it is default color.
let mut content_fg = None;
for sp in &line.spans[1..] {
let t = sp.content.trim();
if !t.is_empty() {
content_fg = Some(sp.style.fg);
break;
}
}
assert_eq!(
content_fg.flatten(),
None,
"expected default color for 3rd-level content, got {content_fg:?}"
);
}
#[test]
fn empty_fenced_block_is_dropped_and_separator_preserved_before_heading() {
let cfg = test_config();
@@ -768,16 +718,12 @@ mod tests {
let expected = vec![
"Loose vs. tight list items:".to_string(),
"".to_string(),
"1. ".to_string(),
"Tight item".to_string(),
"2. ".to_string(),
"Another tight item".to_string(),
"3. ".to_string(),
"Loose item with its own paragraph.".to_string(),
"1. Tight item".to_string(),
"2. Another tight item".to_string(),
"3. Loose item with its own paragraph.".to_string(),
"".to_string(),
"This paragraph belongs to the same list item.".to_string(),
"4. ".to_string(),
"Second loose item with a nested list after a blank line.".to_string(),
" This paragraph belongs to the same list item.".to_string(),
"4. Second loose item with a nested list after a blank line.".to_string(),
" - Nested bullet under a loose item".to_string(),
" - Another nested bullet".to_string(),
];
@@ -788,63 +734,39 @@ mod tests {
}
// Targeted tests derived from fuzz findings. Each asserts streamed == full render.
#[test]
fn fuzz_class_bare_dash_then_task_item() {
fn assert_streamed_equals_full(deltas: &[&str]) {
let cfg = test_config();
// Case similar to: ["two\n", "- \n* [x] done "]
let deltas = vec!["two\n", "- \n* [x] done \n"];
let streamed = simulate_stream_markdown_for_tests(&deltas, true, &cfg);
let streamed = simulate_stream_markdown_for_tests(deltas, true, &cfg);
let streamed_strs = lines_to_plain_strings(&streamed);
let full: String = deltas.iter().copied().collect();
let mut rendered: Vec<ratatui::text::Line<'static>> = Vec::new();
crate::markdown::append_markdown(&full, &mut rendered, &cfg);
let rendered_strs = lines_to_plain_strings(&rendered);
assert_eq!(streamed_strs, rendered_strs);
assert_eq!(streamed_strs, rendered_strs, "full:\n---\n{full}\n---");
}
#[test]
fn fuzz_class_bullet_duplication_variant_1() {
let cfg = test_config();
// Case similar to: ["aph.\n- let one\n- bull", "et two\n\n second paragraph "]
let deltas = vec!["aph.\n- let one\n- bull", "et two\n\n second paragraph \n"];
let streamed = simulate_stream_markdown_for_tests(&deltas, true, &cfg);
let streamed_strs = lines_to_plain_strings(&streamed);
let full: String = deltas.iter().copied().collect();
let mut rendered: Vec<ratatui::text::Line<'static>> = Vec::new();
crate::markdown::append_markdown(&full, &mut rendered, &cfg);
let rendered_strs = lines_to_plain_strings(&rendered);
assert_eq!(streamed_strs, rendered_strs);
assert_streamed_equals_full(&[
"aph.\n- let one\n- bull",
"et two\n\n second paragraph \n",
]);
}
#[test]
fn fuzz_class_bullet_duplication_variant_2() {
let cfg = test_config();
// Case similar to: ["- e\n c", "e\n- bullet two\n\n second paragraph in bullet two\n"]
let deltas = vec![
assert_streamed_equals_full(&[
"- e\n c",
"e\n- bullet two\n\n second paragraph in bullet two\n",
];
let streamed = simulate_stream_markdown_for_tests(&deltas, true, &cfg);
let streamed_strs = lines_to_plain_strings(&streamed);
let full: String = deltas.iter().copied().collect();
let mut rendered: Vec<ratatui::text::Line<'static>> = Vec::new();
crate::markdown::append_markdown(&full, &mut rendered, &cfg);
let rendered_strs = lines_to_plain_strings(&rendered);
assert_eq!(streamed_strs, rendered_strs);
]);
}
#[test]
fn fuzz_class_ordered_list_split_weirdness() {
let cfg = test_config();
// Case similar to: ["one\n2", " two\n- \n* [x] d"]
let deltas = vec!["one\n2", " two\n- \n* [x] d\n"];
let streamed = simulate_stream_markdown_for_tests(&deltas, true, &cfg);
let streamed_strs = lines_to_plain_strings(&streamed);
let full: String = deltas.iter().copied().collect();
let mut rendered: Vec<ratatui::text::Line<'static>> = Vec::new();
crate::markdown::append_markdown(&full, &mut rendered, &cfg);
let rendered_strs = lines_to_plain_strings(&rendered);
assert_eq!(streamed_strs, rendered_strs);
fn streaming_html_block_then_text_matches_full() {
assert_streamed_equals_full(&[
"HTML block:\n",
"<div>inline block</div>\n",
"more stuff\n",
]);
}
}

View File

@@ -1,72 +0,0 @@
/// Returns true if the provided text contains an unclosed fenced code block
/// (opened by ``` or ~~~, closed by a matching fence on its own line).
pub fn is_inside_unclosed_fence(s: &str) -> bool {
let mut open = false;
for line in s.lines() {
let t = line.trim_start();
if t.starts_with("```") || t.starts_with("~~~") {
if !open {
open = true;
} else {
// closing fence on same pattern toggles off
open = false;
}
}
}
open
}
/// Remove fenced code blocks that contain no content (whitespace-only) to avoid
/// streaming empty code blocks like ```lang\n``` or ```\n```.
pub fn strip_empty_fenced_code_blocks(s: &str) -> String {
// Only remove complete fenced blocks that contain no non-whitespace content.
// Leave all other content unchanged to avoid affecting partial streams.
let lines: Vec<&str> = s.lines().collect();
let mut out = String::with_capacity(s.len());
let mut i = 0usize;
while i < lines.len() {
let line = lines[i];
let trimmed_start = line.trim_start();
let fence_token = if trimmed_start.starts_with("```") {
"```"
} else if trimmed_start.starts_with("~~~") {
"~~~"
} else {
""
};
if !fence_token.is_empty() {
// Find a matching closing fence on its own line.
let mut j = i + 1;
let mut has_content = false;
let mut found_close = false;
while j < lines.len() {
let l = lines[j];
if l.trim() == fence_token {
found_close = true;
break;
}
if !l.trim().is_empty() {
has_content = true;
}
j += 1;
}
if found_close && !has_content {
// Drop i..=j and insert at most a single blank separator line.
if !out.ends_with('\n') {
out.push('\n');
}
i = j + 1;
continue;
}
// Not an empty fenced block; emit as-is.
out.push_str(line);
out.push('\n');
i += 1;
} else {
out.push_str(line);
out.push('\n');
i += 1;
}
}
out
}

View File

@@ -1,3 +1,2 @@
pub mod highlight;
pub mod line_utils;
pub mod markdown_utils;

View File

@@ -0,0 +1,62 @@
---
source: tui/src/markdown_render_tests.rs
expression: rendered
---
# H1: Markdown Streaming Test
Intro paragraph with bold text, italic text, and inline code x=1.
Combined bold-italic both and escaped asterisks *literal*.
Auto-link: https://example.com (https://example.com) and reference link [ref][r1].
Link with title: hover me (https://example.com) and mailto mailto:test@example.com (mailto:test@example.com).
Image: alt text
> Blockquote level 1
>
> > Blockquote level 2 with inline code
- Unordered list item 1
- Nested bullet with italics inner
- Unordered list item 2 with strikethrough
1. Ordered item one
2. Ordered item two with sublist:
1. Alt-numbered subitem
- [ ] Task: unchecked
- [x] Task: checked with link home (https://example.org)
———
Table below (alignment test):
| Left | Center | Right |
|:-----|:------:|------:|
| a | b | c |
Inline HTML: <sup>sup</sup> and <sub>sub</sub>.
HTML block:
<div style="border:1px solid #ccc;padding:2px">inline block</div>
Escapes: \_underscores\_, backslash \\, ticks ``code with `backtick` inside``.
Emoji shortcodes: :sparkles: :tada: (if supported).
Hard break test (line ends with two spaces)
Next line should be close to previous.
Footnote reference here[^1] and another[^longnote].
Horizontal rule with asterisks:
***
Fenced code block (JSON):
```json
{ "a": 1, "b": [true, false] }
```
Fenced code with tildes and triple backticks inside:
~~~markdown
To close ``` you need tildes.
~~~
Indented code block:
for i in range(3): print(i)
Definition-like list:
Term
: Definition with `code`.
Character entities: &amp; &lt; &gt; &quot; &#39;
[^1]: This is the first footnote.
[^longnote]: A longer footnote with a link to [Rust](https://www.rust-lang.org/).
Escaped pipe in text: a \| b \| c.
URL with parentheses: [link](https://example.com/path_(with)_parens).
[r1]: https://example.com/ref "Reference link title"

View File

@@ -380,16 +380,12 @@ mod tests {
let expected = vec![
"Loose vs. tight list items:".to_string(),
"".to_string(),
"1. ".to_string(),
"Tight item".to_string(),
"2. ".to_string(),
"Another tight item".to_string(),
"3. ".to_string(),
"Loose item with its own paragraph.".to_string(),
"1. Tight item".to_string(),
"2. Another tight item".to_string(),
"3. Loose item with its own paragraph.".to_string(),
"".to_string(),
"This paragraph belongs to the same list item.".to_string(),
"4. ".to_string(),
"Second loose item with a nested list after a blank line.".to_string(),
" This paragraph belongs to the same list item.".to_string(),
"4. Second loose item with a nested list after a blank line.".to_string(),
" - Nested bullet under a loose item".to_string(),
" - Another nested bullet".to_string(),
];

View File

@@ -187,10 +187,17 @@ where
// Build first wrapped line with initial indent.
let mut first_line = rt_opts.initial_indent.clone();
first_line.style = first_line.style.patch(line.style);
{
let mut sliced = slice_line_spans(line, &span_bounds, first_line_range);
let sliced = slice_line_spans(line, &span_bounds, first_line_range);
let mut spans = first_line.spans;
spans.append(&mut sliced.spans);
spans.append(
&mut sliced
.spans
.into_iter()
.map(|s| s.patch_style(line.style))
.collect(),
);
first_line.spans = spans;
out.push(first_line);
}
@@ -209,10 +216,17 @@ where
continue;
}
let mut subsequent_line = rt_opts.subsequent_indent.clone();
subsequent_line.style = subsequent_line.style.patch(line.style);
let offset_range = (r.start + base)..(r.end + base);
let mut sliced = slice_line_spans(line, &span_bounds, &offset_range);
let sliced = slice_line_spans(line, &span_bounds, &offset_range);
let mut spans = subsequent_line.spans;
spans.append(&mut sliced.spans);
spans.append(
&mut sliced
.spans
.into_iter()
.map(|s| s.patch_style(line.style))
.collect(),
);
subsequent_line.spans = spans;
out.push(subsequent_line);
}

View File

@@ -42,7 +42,7 @@ they'll be committed to your working directory.
### Example prompts
Below are a few bite-size examples you can copy-paste. Replace the text in quotes with your own task. See the [prompting guide](https://github.com/openai/codex/blob/main/codex-cli/examples/prompting_guide.md) for more tips and usage patterns.
Below are a few bite-size examples you can copy-paste. Replace the text in quotes with your own task.
| ✨ | What you type | What happens |
| --- | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |