Compare commits

...

7 Commits

Author SHA1 Message Date
Xin Lin
b0d1b40792 fix: plugin bundle archive handling for upload and install 2026-05-21 22:53:31 -07:00
rreichel3-oai
b14f11d3d2 [codex] Enable Node env proxy for managed network proxy (#23905)
## Summary
- set `NODE_USE_ENV_PROXY=1` when Codex applies managed network proxy
environment overrides
- keep the Node opt-in in the proxy environment key set used by
shell/runtime env handling
- cover the new env var in the focused network proxy env test

## Why
Codex already sets HTTP proxy environment variables for child processes
when the managed network proxy is active. Node's built-in network
behavior needs the `NODE_USE_ENV_PROXY` opt-in to honor those env vars,
so Node-based skill scripts can otherwise skip the managed proxy path
and fail under restricted network access.

## Validation
- `just fmt` in `codex-rs`
- `cargo test -p codex-network-proxy` in `codex-rs`
2026-05-22 01:27:25 -04:00
anp-oai
c83ba22359 Allow parallel MCP tool calls when annotated readOnly (#23750)
## Summary
- Treat MCP tools with `readOnlyHint: true` as parallel-safe even when
`supports_parallel_tool_calls` is unset or `false`.
- Keep server-level `supports_parallel_tool_calls` as an additive
override for non-read-only tools.
- Add focused unit coverage for the MCP handler eligibility decision.
- Update RMCP integration coverage to keep the serial baseline on a
mutable tool, verify read-only concurrency without server opt-in, and
preserve the server opt-in concurrency path separately.

## Testing
- `just fmt`
- `cargo test -p codex-core --lib tools::handlers::mcp::tests::`
- `cargo test -p codex-core --test all
stdio_mcp_read_only_tool_calls_run_concurrently_without_server_opt_in`
- `cargo test -p codex-core --test all
stdio_mcp_parallel_tool_calls_opt_in_runs_concurrently`
- `cargo test -p codex-rmcp-client`
2026-05-21 20:40:34 -07:00
Celia Chen
464ab40dfa feat: best-effort compact large tool schemas (#23904)
## Why

The `dev/cc/ref-def` branch preserves richer JSON Schema detail for
connector tools, including `$defs` and nested shapes. That improves
fidelity, but it pushes the largest connector schemas well past the
intended tool-schema budget. This PR adds a best-effort compaction pass
for unusually large tool input schemas so the p99 and max tails stay
small while ordinary schemas are left alone.

## What Changed

- Added best-effort large-schema compaction in
`codex-rs/tools/src/json_schema.rs` after schema sanitization and
definition pruning.
- Compaction runs as a waterfall only while the compact JSON budget
proxy is exceeded:
  1. Strip schema `description` metadata.
  2. Drop root `$defs` / `definitions`.
  3. Collapse deep nested complex schema objects to `{}`.
- Kept top-level argument names and immediate schema shape where
possible.

## Corpus Results

Scope: 2,025 schemas under `golden_schemas`, all parsed successfully.
Token count is `o200k_base` over compact JSON from
`parse_tool_input_schema`.

| Percentile | Before `origin/main` `4dbca61e20` | After branch
`dev/cc/ref-def` `f9bf071758` | After this PR |
|---|---:|---:|---:|
| p0 | 9 | 9 | 9 |
| p10 | 59 | 63 | 63 |
| p25 | 81 | 86 | 86 |
| p50 | 114 | 127 | 125 |
| p75 | 174 | 205 | 202 |
| p90 | 295 | 335 | 322 |
| p95 | 391 | 526 | 422 |
| p99 | 794 | 1,303 | 689 |
| max | 2,836 | 3,337 | 887 |

After this PR, `0 / 2,025` schemas are over 1k tokens.

### Compaction Savings

These are cumulative waterfall stages over the same corpus. Later passes
only run for schemas that are still over the compact JSON budget proxy.

| Stage | Total tokens | Step savings | Schemas changed by step |
|---|---:|---:|---:|
| No compaction | 391,862 | - | - |
| Strip schema `description` metadata | 350,961 | 40,901 | 66 |
| Drop root `$defs` / `definitions` | 340,683 | 10,278 | 13 |
| Collapse deep complex schemas to `{}` | 335,875 | 4,808 | 6 |
2026-05-22 01:26:17 +00:00
sayan-oai
7e802b22f1 Expose conversation history to extension tools (#23963)
## Why

Extension tools that need conversation context should be able to read it
from the live tool invocation instead of reaching into thread
persistence themselves.

## What changed

- Add a `ConversationHistory` snapshot to extension `ToolCall`s and
populate it from the current raw in-memory response history.
- Expose all history items at this boundary so each extension can filter
and bound the subset it needs before consuming or forwarding it.
- Cover the adapter and registry dispatch paths and update existing
extension tests that construct `ToolCall` literals.

## Test plan

- `cargo test -p codex-tools`
- `cargo test -p codex-extension-api`
- `cargo test -p codex-goal-extension`
- `cargo test -p codex-memories-extension`
- `cargo test -p codex-core passes_turn_fields_to_extension_call`
- `cargo test -p codex-core
extension_tool_executors_are_model_visible_and_dispatchable`
2026-05-22 01:11:47 +00:00
Celia Chen
0cec508148 feat: support local refs and defs in tool input schemas (#23357)
# Why

Some connector tool input schemas use local JSON Schema references and
definition tables to avoid duplicating large nested shapes. Codex
previously lowered these schemas into the supported subset in a way that
could discard `$ref`-only schema objects and lose the corresponding
definitions, which made non-strict tool registration less faithful than
the original connector schema.

This keeps the existing minimal-lowering policy: Codex still does not
raw-pass through arbitrary JSON Schema, but it now preserves local
reference structure that fits the Responses-compatible subset and prunes
definition entries that cannot be reached by following `$ref`s from the
root schema after sanitization, including refs found transitively inside
other reachable definitions. The pruning matters because Responses
parses definition tables even when entries are unused, so keeping dead
definitions wastes prompt tokens.

# What changed

- Added `$ref`, `$defs`, and legacy `definitions` fields to the tool
`JsonSchema` representation.
- Updated `parse_tool_input_schema` lowering so `$ref`-only schema
objects survive sanitization instead of becoming `{}`.
- Sanitized definition tables recursively and dropped malformed
definition tables so non-strict registration degrades gracefully.
- Added reachability pruning for root definition tables by starting from
refs outside definition tables, then following refs inside reachable
definitions.
- Added JSON Pointer decoding for local definition refs such as
`#/$defs/Foo~1Bar`.

# Verification
ran local golden-schema probes against representative connector schemas
to validate behavior on real generated schemas:

| Golden schema | Before bytes | After bytes | `$defs` before -> after |
`$ref` before -> after | Result |
|---|---:|---:|---:|---:|---|
| `google_calendar/create_space` | 7111 | 4526 | 7 -> 7 | 7 -> 7 | all
definitions preserved because all are reachable |
| `figma/apply_file_variable_changes` | 4609 | 999 | 8 -> 5 | 8 -> 5 |
unused defs pruned after unsupported `oneOf` shapes lower away |
| `snowflake/list_catalog_integrations` | 1380 | 404 | 3 -> 0 | 0 -> 0 |
all defs pruned because none are referenced |
| `dropbox/create_shared_link` | 8894 | 1836 | 14 -> 4 | 9 -> 4 | only
defs reachable from the root schema after sanitization are retained,
including transitively through other retained defs |

Token increase across golden schema due to this change:
<img width="817" height="366" alt="Screenshot 2026-05-19 at 1 47 04 PM"
src="https://github.com/user-attachments/assets/d5c80fe9-da85-41e6-8ac7-a01d1e0b0f71"
/>
2026-05-22 00:32:14 +00:00
Eric Traut
5a6e905994 Fix auto-review permission profile override (#23956)
## Summary
The auto-review runtime sync path was assigning a raw
`PermissionProfile` into `runtime_permission_profile_override`, whose
field now expects `RuntimePermissionProfileOverride`. That broke the TUI
Bazel build.

This changes the assignment to store
`RuntimePermissionProfileOverride::from_config(&self.config)`, matching
the other runtime override paths and preserving the active profile and
network metadata with the permission profile.
2026-05-21 16:52:36 -07:00
24 changed files with 1981 additions and 306 deletions

1
MODULE.bazel.lock generated
View File

@@ -1150,6 +1150,7 @@
"jni_0.21.1": "{\"dependencies\":[{\"name\":\"cesu8\",\"req\":\"^1.1.0\"},{\"name\":\"cfg-if\",\"req\":\"^1.0.0\"},{\"name\":\"combine\",\"req\":\"^4.1.0\"},{\"name\":\"java-locator\",\"optional\":true,\"req\":\"^0.1\"},{\"name\":\"jni-sys\",\"req\":\"^0.3.0\"},{\"name\":\"libloading\",\"optional\":true,\"req\":\"^0.7\"},{\"name\":\"log\",\"req\":\"^0.4.4\"},{\"name\":\"thiserror\",\"req\":\"^1.0.20\"},{\"kind\":\"dev\",\"name\":\"assert_matches\",\"req\":\"^1.5.0\"},{\"kind\":\"dev\",\"name\":\"lazy_static\",\"req\":\"^1\"},{\"kind\":\"dev\",\"name\":\"rusty-fork\",\"req\":\"^0.3.0\"},{\"kind\":\"build\",\"name\":\"walkdir\",\"req\":\"^2\"},{\"features\":[\"Win32_Globalization\"],\"name\":\"windows-sys\",\"req\":\"^0.45.0\",\"target\":\"cfg(windows)\"},{\"kind\":\"dev\",\"name\":\"bytemuck\",\"req\":\"^1.13.0\",\"target\":\"cfg(windows)\"}],\"features\":{\"default\":[],\"invocation\":[\"java-locator\",\"libloading\"]}}",
"jobserver_0.1.34": "{\"dependencies\":[{\"features\":[\"std\"],\"name\":\"getrandom\",\"req\":\"^0.3.2\",\"target\":\"cfg(windows)\"},{\"name\":\"libc\",\"req\":\"^0.2.171\",\"target\":\"cfg(unix)\"},{\"features\":[\"fs\"],\"kind\":\"dev\",\"name\":\"nix\",\"req\":\"^0.28.0\",\"target\":\"cfg(unix)\"},{\"kind\":\"dev\",\"name\":\"tempfile\",\"req\":\"^3.10.1\"}],\"features\":{}}",
"js-sys_0.3.85": "{\"dependencies\":[{\"default_features\":false,\"name\":\"once_cell\",\"req\":\"^1.12\"},{\"default_features\":false,\"name\":\"wasm-bindgen\",\"req\":\"=0.2.108\"}],\"features\":{\"default\":[\"std\"],\"std\":[\"wasm-bindgen/std\"]}}",
"jsonptr_0.7.1": "{\"dependencies\":[{\"features\":[\"fancy\"],\"name\":\"miette\",\"optional\":true,\"req\":\"^7.4.0\"},{\"kind\":\"dev\",\"name\":\"quickcheck\",\"req\":\"^1.0.3\"},{\"kind\":\"dev\",\"name\":\"quickcheck_macros\",\"req\":\"^1.0.0\"},{\"features\":[\"alloc\"],\"name\":\"serde\",\"optional\":true,\"req\":\"^1.0.203\"},{\"features\":[\"alloc\"],\"name\":\"serde_json\",\"optional\":true,\"req\":\"^1.0.119\"},{\"name\":\"syn\",\"optional\":true,\"req\":\"^1.0.109\",\"target\":\"cfg(any())\"},{\"name\":\"toml\",\"optional\":true,\"req\":\"^0.8\"}],\"features\":{\"assign\":[],\"default\":[\"std\",\"serde\",\"json\",\"resolve\",\"assign\",\"delete\"],\"delete\":[\"resolve\"],\"json\":[\"dep:serde_json\",\"serde\"],\"miette\":[\"dep:miette\",\"std\"],\"resolve\":[],\"std\":[\"serde/std\",\"serde_json?/std\"],\"toml\":[\"dep:toml\",\"serde\",\"std\"]}}",
"jsonwebtoken_9.3.1": "{\"dependencies\":[{\"name\":\"base64\",\"req\":\"^0.22\"},{\"default_features\":false,\"kind\":\"dev\",\"name\":\"criterion\",\"req\":\"^0.4\",\"target\":\"cfg(all(target_arch = \\\"wasm32\\\", not(any(target_os = \\\"emscripten\\\", target_os = \\\"wasi\\\"))))\"},{\"kind\":\"dev\",\"name\":\"criterion\",\"req\":\"^0.4\",\"target\":\"cfg(not(all(target_arch = \\\"wasm32\\\", not(any(target_os = \\\"emscripten\\\", target_os = \\\"wasi\\\")))))\"},{\"name\":\"js-sys\",\"req\":\"^0.3\",\"target\":\"cfg(target_arch = \\\"wasm32\\\")\"},{\"name\":\"pem\",\"optional\":true,\"req\":\"^3\"},{\"features\":[\"std\"],\"name\":\"ring\",\"req\":\"^0.17.4\",\"target\":\"cfg(not(target_arch = \\\"wasm32\\\"))\"},{\"features\":[\"std\",\"wasm32_unknown_unknown_js\"],\"name\":\"ring\",\"req\":\"^0.17.4\",\"target\":\"cfg(target_arch = \\\"wasm32\\\")\"},{\"features\":[\"derive\"],\"name\":\"serde\",\"req\":\"^1.0\"},{\"name\":\"serde_json\",\"req\":\"^1.0\"},{\"name\":\"simple_asn1\",\"optional\":true,\"req\":\"^0.6\"},{\"features\":[\"wasm-bindgen\"],\"kind\":\"dev\",\"name\":\"time\",\"req\":\"^0.3\",\"target\":\"cfg(all(target_arch = \\\"wasm32\\\", not(any(target_os = \\\"emscripten\\\", target_os = \\\"wasi\\\"))))\"},{\"kind\":\"dev\",\"name\":\"time\",\"req\":\"^0.3\",\"target\":\"cfg(not(all(target_arch = \\\"wasm32\\\", not(any(target_os = \\\"emscripten\\\", target_os = \\\"wasi\\\")))))\"},{\"kind\":\"dev\",\"name\":\"wasm-bindgen-test\",\"req\":\"^0.3.1\"}],\"features\":{\"default\":[\"use_pem\"],\"use_pem\":[\"pem\",\"simple_asn1\"]}}",
"keyring_3.6.3": "{\"dependencies\":[{\"kind\":\"dev\",\"name\":\"base64\",\"req\":\"^0.22\"},{\"name\":\"byteorder\",\"optional\":true,\"req\":\"^1.2\",\"target\":\"cfg(target_os = \\\"windows\\\")\"},{\"features\":[\"derive\",\"wrap_help\"],\"kind\":\"dev\",\"name\":\"clap\",\"req\":\"^4\"},{\"name\":\"dbus-secret-service\",\"optional\":true,\"req\":\"^4.0.0-rc.1\",\"target\":\"cfg(target_os = \\\"openbsd\\\")\"},{\"name\":\"dbus-secret-service\",\"optional\":true,\"req\":\"^4.0.0-rc.2\",\"target\":\"cfg(target_os = \\\"linux\\\")\"},{\"name\":\"dbus-secret-service\",\"optional\":true,\"req\":\"^4.0.1\",\"target\":\"cfg(target_os = \\\"freebsd\\\")\"},{\"kind\":\"dev\",\"name\":\"doc-comment\",\"req\":\"^0.3\"},{\"kind\":\"dev\",\"name\":\"env_logger\",\"req\":\"^0.11.5\"},{\"kind\":\"dev\",\"name\":\"fastrand\",\"req\":\"^2\"},{\"features\":[\"std\"],\"name\":\"linux-keyutils\",\"optional\":true,\"req\":\"^0.2\",\"target\":\"cfg(target_os = \\\"linux\\\")\"},{\"name\":\"log\",\"req\":\"^0.4.22\"},{\"name\":\"openssl\",\"optional\":true,\"req\":\"^0.10.66\"},{\"kind\":\"dev\",\"name\":\"rpassword\",\"req\":\"^7\"},{\"kind\":\"dev\",\"name\":\"rprompt\",\"req\":\"^2\"},{\"name\":\"secret-service\",\"optional\":true,\"req\":\"^4\",\"target\":\"cfg(target_os = \\\"freebsd\\\")\"},{\"name\":\"secret-service\",\"optional\":true,\"req\":\"^4\",\"target\":\"cfg(target_os = \\\"linux\\\")\"},{\"name\":\"secret-service\",\"optional\":true,\"req\":\"^4\",\"target\":\"cfg(target_os = \\\"openbsd\\\")\"},{\"name\":\"security-framework\",\"optional\":true,\"req\":\"^2\",\"target\":\"cfg(target_os = \\\"ios\\\")\"},{\"name\":\"security-framework\",\"optional\":true,\"req\":\"^3\",\"target\":\"cfg(target_os = \\\"macos\\\")\"},{\"kind\":\"dev\",\"name\":\"whoami\",\"req\":\"^1.5\"},{\"features\":[\"Win32_Foundation\",\"Win32_Security_Credentials\"],\"name\":\"windows-sys\",\"optional\":true,\"req\":\"^0.60\",\"target\":\"cfg(target_os = \\\"windows\\\")\"},{\"name\":\"zbus\",\"optional\":true,\"req\":\"^4\",\"target\":\"cfg(target_os = \\\"freebsd\\\")\"},{\"name\":\"zbus\",\"optional\":true,\"req\":\"^4\",\"target\":\"cfg(target_os = \\\"linux\\\")\"},{\"name\":\"zbus\",\"optional\":true,\"req\":\"^4\",\"target\":\"cfg(target_os = \\\"openbsd\\\")\"},{\"name\":\"zeroize\",\"req\":\"^1.8.1\",\"target\":\"cfg(target_os = \\\"windows\\\")\"}],\"features\":{\"apple-native\":[\"dep:security-framework\"],\"async-io\":[\"zbus?/async-io\"],\"async-secret-service\":[\"dep:secret-service\",\"dep:zbus\"],\"crypto-openssl\":[\"dbus-secret-service?/crypto-openssl\",\"secret-service?/crypto-openssl\"],\"crypto-rust\":[\"dbus-secret-service?/crypto-rust\",\"secret-service?/crypto-rust\"],\"linux-native\":[\"dep:linux-keyutils\"],\"linux-native-async-persistent\":[\"linux-native\",\"async-secret-service\"],\"linux-native-sync-persistent\":[\"linux-native\",\"sync-secret-service\"],\"sync-secret-service\":[\"dep:dbus-secret-service\"],\"tokio\":[\"zbus?/tokio\"],\"vendored\":[\"dbus-secret-service?/vendored\",\"openssl?/vendored\"],\"windows-native\":[\"dep:windows-sys\",\"dep:byteorder\"]}}",
"kqueue-sys_1.0.4": "{\"dependencies\":[{\"name\":\"bitflags\",\"req\":\"^1.2.1\"},{\"name\":\"libc\",\"req\":\"^0.2.74\"}],\"features\":{}}",

8
codex-rs/Cargo.lock generated
View File

@@ -3773,12 +3773,14 @@ dependencies = [
"codex-utils-output-truncation",
"codex-utils-pty",
"codex-utils-string",
"jsonptr",
"pretty_assertions",
"rmcp",
"serde",
"serde_json",
"thiserror 2.0.18",
"tracing",
"urlencoding",
]
[[package]]
@@ -8125,6 +8127,12 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "jsonptr"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a5a3cc660ba5d72bce0b3bb295bf20847ccbb40fd423f3f05b61273672e561fe"
[[package]]
name = "jsonwebtoken"
version = "9.3.1"

View File

@@ -301,6 +301,7 @@ indexmap = "2.12.0"
insta = "1.46.3"
inventory = "0.3.19"
itertools = "0.14.0"
jsonptr = { version = "0.7.1", default-features = false }
jsonwebtoken = "9.3.1"
keyring = { version = "3.6", default-features = false }
landlock = "0.4.4"

View File

@@ -6,6 +6,7 @@ pub mod marketplace;
pub mod marketplace_add;
pub mod marketplace_remove;
pub mod marketplace_upgrade;
mod plugin_bundle_archive;
pub mod remote;
pub mod remote_bundle;
pub mod remote_legacy;

View File

@@ -0,0 +1,315 @@
use flate2::Compression;
use flate2::read::GzDecoder;
use flate2::write::GzEncoder;
use std::fmt;
use std::fs;
use std::io;
use std::io::Read;
use std::io::Write;
use std::path::Path;
use std::path::PathBuf;
use tar::Archive;
#[derive(Debug, thiserror::Error)]
pub(crate) enum PluginBundlePackError {
#[error("invalid plugin path `{path}`: {reason}")]
InvalidPluginPath { path: PathBuf, reason: String },
#[error("plugin archive would be {bytes} bytes, exceeding maximum size of {max_bytes} bytes")]
ArchiveTooLarge { bytes: usize, max_bytes: usize },
#[error("failed to archive plugin bundle: {source}")]
Io {
#[source]
source: io::Error,
},
}
#[derive(Debug, thiserror::Error)]
pub(crate) enum PluginBundleUnpackError {
#[error(
"plugin bundle extracted size would be {bytes} bytes, exceeding maximum total size of {max_bytes} bytes"
)]
ExtractedBundleTooLarge { bytes: u64, max_bytes: u64 },
#[error("{context}: {source}")]
Io {
context: &'static str,
#[source]
source: io::Error,
},
#[error("{0}")]
InvalidBundle(String),
}
impl PluginBundleUnpackError {
fn io(context: &'static str, source: io::Error) -> Self {
Self::Io { context, source }
}
}
pub(crate) fn pack_plugin_bundle_tar_gz(
plugin_path: &Path,
max_bytes: usize,
) -> Result<Vec<u8>, PluginBundlePackError> {
if !plugin_path.is_dir() {
return Err(PluginBundlePackError::InvalidPluginPath {
path: plugin_path.to_path_buf(),
reason: "expected a plugin directory".to_string(),
});
}
if !plugin_path.join(".codex-plugin/plugin.json").is_file() {
return Err(PluginBundlePackError::InvalidPluginPath {
path: plugin_path.to_path_buf(),
reason: "missing .codex-plugin/plugin.json".to_string(),
});
}
let encoder = GzEncoder::new(SizeLimitedBuffer::new(max_bytes), Compression::default());
let mut archive = tar::Builder::new(encoder);
append_plugin_tree(&mut archive, plugin_path, plugin_path).map_err(archive_io_error)?;
let encoder = archive.into_inner().map_err(archive_io_error)?;
encoder
.finish()
.map(SizeLimitedBuffer::into_inner)
.map_err(archive_io_error)
}
fn append_plugin_tree<W: Write>(
archive: &mut tar::Builder<W>,
plugin_root: &Path,
current: &Path,
) -> io::Result<()> {
let mut entries = fs::read_dir(current)?.collect::<Result<Vec<_>, io::Error>>()?;
entries.sort_by_key(fs::DirEntry::file_name);
for entry in entries {
let path = entry.path();
let file_type = entry.file_type()?;
let relative_path = path.strip_prefix(plugin_root).map_err(|err| {
io::Error::other(format!(
"failed to compute plugin archive path for `{}`: {err}",
path.display()
))
})?;
if file_type.is_dir() {
archive.append_dir(relative_path, &path)?;
append_plugin_tree(archive, plugin_root, &path)?;
} else if file_type.is_file() {
archive.append_path_with_name(&path, relative_path)?;
} else {
return Err(io::Error::other(format!(
"unsupported plugin archive entry type: {}",
path.display()
)));
}
}
Ok(())
}
fn archive_io_error(source: io::Error) -> PluginBundlePackError {
if let Some(limit) = source
.get_ref()
.and_then(|err| err.downcast_ref::<ArchiveSizeLimitExceeded>())
{
return PluginBundlePackError::ArchiveTooLarge {
bytes: limit.bytes,
max_bytes: limit.max_bytes,
};
}
PluginBundlePackError::Io { source }
}
pub(crate) fn unpack_plugin_bundle_tar_gz(
bytes: &[u8],
destination: &Path,
max_total_bytes: u64,
) -> Result<(), PluginBundleUnpackError> {
fs::create_dir_all(destination).map_err(|source| {
PluginBundleUnpackError::io(
"failed to create plugin bundle extraction directory",
source,
)
})?;
let archive = GzDecoder::new(std::io::Cursor::new(bytes));
let mut archive = Archive::new(archive);
unpack_plugin_bundle_tar(&mut archive, destination, max_total_bytes)
}
fn unpack_plugin_bundle_tar<R: Read>(
archive: &mut Archive<R>,
destination: &Path,
max_total_bytes: u64,
) -> Result<(), PluginBundleUnpackError> {
let mut extracted_bytes = 0u64;
let entries = archive.entries().map_err(|source| {
PluginBundleUnpackError::io("failed to read plugin bundle tar", source)
})?;
for entry in entries {
let mut entry = entry.map_err(|source| {
PluginBundleUnpackError::io("failed to read plugin bundle tar entry", source)
})?;
let entry_type = entry.header().entry_type();
let entry_size = entry.size();
let entry_path = entry
.path()
.map_err(|source| {
PluginBundleUnpackError::io("failed to read plugin bundle tar entry path", source)
})?
.into_owned();
let output_path = checked_tar_output_path(destination, &entry_path)?;
if entry_type.is_dir() {
fs::create_dir_all(&output_path).map_err(|source| {
PluginBundleUnpackError::io("failed to create plugin bundle directory", source)
})?;
continue;
}
if entry_type.is_file() {
enforce_total_extracted_size(entry_size, &mut extracted_bytes, max_total_bytes)?;
let Some(parent) = output_path.parent() else {
return Err(PluginBundleUnpackError::InvalidBundle(format!(
"plugin bundle output path has no parent: {}",
output_path.display()
)));
};
fs::create_dir_all(parent).map_err(|source| {
PluginBundleUnpackError::io("failed to create plugin bundle directory", source)
})?;
entry.unpack(&output_path).map_err(|source| {
PluginBundleUnpackError::io("failed to unpack plugin bundle entry", source)
})?;
continue;
}
if entry_type.is_hard_link() || entry_type.is_symlink() {
return Err(PluginBundleUnpackError::InvalidBundle(format!(
"plugin bundle tar entry `{}` is a link",
entry_path.display()
)));
}
return Err(PluginBundleUnpackError::InvalidBundle(format!(
"plugin bundle tar entry `{}` has unsupported type {:?}",
entry_path.display(),
entry_type
)));
}
Ok(())
}
fn checked_tar_output_path(
destination: &Path,
entry_name: &Path,
) -> Result<PathBuf, PluginBundleUnpackError> {
let mut output_path = destination.to_path_buf();
let mut has_component = false;
for component in entry_name.components() {
match component {
std::path::Component::Normal(component) => {
has_component = true;
output_path.push(component);
}
std::path::Component::CurDir => {}
std::path::Component::ParentDir
| std::path::Component::RootDir
| std::path::Component::Prefix(_) => {
return Err(PluginBundleUnpackError::InvalidBundle(format!(
"plugin bundle tar entry `{}` escapes extraction root",
entry_name.display()
)));
}
}
}
if !has_component {
return Err(PluginBundleUnpackError::InvalidBundle(
"plugin bundle tar entry has an empty path".to_string(),
));
}
Ok(output_path)
}
fn enforce_total_extracted_size(
entry_size: u64,
extracted_bytes: &mut u64,
max_total_bytes: u64,
) -> Result<(), PluginBundleUnpackError> {
let next_total = extracted_bytes.checked_add(entry_size).ok_or(
PluginBundleUnpackError::ExtractedBundleTooLarge {
bytes: u64::MAX,
max_bytes: max_total_bytes,
},
)?;
if next_total > max_total_bytes {
return Err(PluginBundleUnpackError::ExtractedBundleTooLarge {
bytes: next_total,
max_bytes: max_total_bytes,
});
}
*extracted_bytes = next_total;
Ok(())
}
struct SizeLimitedBuffer {
bytes: Vec<u8>,
max_bytes: usize,
}
impl SizeLimitedBuffer {
fn new(max_bytes: usize) -> Self {
Self {
bytes: Vec::new(),
max_bytes,
}
}
fn into_inner(self) -> Vec<u8> {
self.bytes
}
}
impl Write for SizeLimitedBuffer {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let next_len = self.bytes.len().checked_add(buf.len()).ok_or_else(|| {
io::Error::other(ArchiveSizeLimitExceeded {
bytes: usize::MAX,
max_bytes: self.max_bytes,
})
})?;
if next_len > self.max_bytes {
return Err(io::Error::other(ArchiveSizeLimitExceeded {
bytes: next_len,
max_bytes: self.max_bytes,
}));
}
self.bytes.extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
#[derive(Debug)]
struct ArchiveSizeLimitExceeded {
bytes: usize,
max_bytes: usize,
}
impl fmt::Display for ArchiveSizeLimitExceeded {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(
f,
"archive would be {} bytes, exceeding maximum size of {} bytes",
self.bytes, self.max_bytes
)
}
}
impl std::error::Error for ArchiveSizeLimitExceeded {}

View File

@@ -1,18 +1,15 @@
use super::*;
use crate::plugin_bundle_archive::PluginBundlePackError;
use crate::plugin_bundle_archive::pack_plugin_bundle_tar_gz;
use codex_login::CodexAuth;
use codex_login::default_client::build_reqwest_client;
use codex_utils_absolute_path::AbsolutePathBuf;
use flate2::Compression;
use flate2::write::GzEncoder;
use reqwest::RequestBuilder;
use reqwest::StatusCode;
use serde::Deserialize;
use serde::Serialize;
use std::collections::BTreeMap;
use std::fmt;
use std::fs;
use std::io;
use std::io::Write;
use std::path::Path;
use tracing::warn;
@@ -477,140 +474,20 @@ fn archive_plugin_for_upload_with_limit(
plugin_path: &Path,
max_bytes: usize,
) -> Result<Vec<u8>, RemotePluginCatalogError> {
if !plugin_path.is_dir() {
return Err(RemotePluginCatalogError::InvalidPluginPath {
pack_plugin_bundle_tar_gz(plugin_path, max_bytes).map_err(|err| match err {
PluginBundlePackError::InvalidPluginPath { path, reason } => {
RemotePluginCatalogError::InvalidPluginPath { path, reason }
}
PluginBundlePackError::ArchiveTooLarge { bytes, max_bytes } => {
RemotePluginCatalogError::ArchiveTooLarge { bytes, max_bytes }
}
PluginBundlePackError::Io { source } => RemotePluginCatalogError::Archive {
path: plugin_path.to_path_buf(),
reason: "expected a plugin directory".to_string(),
});
}
if !plugin_path.join(".codex-plugin/plugin.json").is_file() {
return Err(RemotePluginCatalogError::InvalidPluginPath {
path: plugin_path.to_path_buf(),
reason: "missing .codex-plugin/plugin.json".to_string(),
});
}
let encoder = GzEncoder::new(SizeLimitedBuffer::new(max_bytes), Compression::default());
let mut archive = tar::Builder::new(encoder);
append_plugin_tree(&mut archive, plugin_path, plugin_path)
.map_err(|source| archive_error(plugin_path, source))?;
let encoder = archive
.into_inner()
.map_err(|source| archive_error(plugin_path, source))?;
encoder
.finish()
.map(SizeLimitedBuffer::into_inner)
.map_err(|source| archive_error(plugin_path, source))
source,
},
})
}
fn append_plugin_tree<W: Write>(
archive: &mut tar::Builder<W>,
plugin_root: &Path,
current: &Path,
) -> io::Result<()> {
let mut entries = fs::read_dir(current)?.collect::<Result<Vec<_>, io::Error>>()?;
entries.sort_by_key(fs::DirEntry::file_name);
for entry in entries {
let path = entry.path();
let file_type = entry.file_type()?;
let relative_path = path.strip_prefix(plugin_root).map_err(|err| {
io::Error::other(format!(
"failed to compute plugin archive path for `{}`: {err}",
path.display()
))
})?;
if file_type.is_dir() {
archive.append_dir(relative_path, &path)?;
append_plugin_tree(archive, plugin_root, &path)?;
} else if file_type.is_file() {
archive.append_path_with_name(&path, relative_path)?;
} else {
return Err(io::Error::other(format!(
"unsupported plugin archive entry type: {}",
path.display()
)));
}
}
Ok(())
}
fn archive_error(plugin_path: &Path, source: io::Error) -> RemotePluginCatalogError {
if let Some(limit) = source
.get_ref()
.and_then(|err| err.downcast_ref::<ArchiveSizeLimitExceeded>())
{
return RemotePluginCatalogError::ArchiveTooLarge {
bytes: limit.bytes,
max_bytes: limit.max_bytes,
};
}
RemotePluginCatalogError::Archive {
path: plugin_path.to_path_buf(),
source,
}
}
struct SizeLimitedBuffer {
bytes: Vec<u8>,
max_bytes: usize,
}
impl SizeLimitedBuffer {
fn new(max_bytes: usize) -> Self {
Self {
bytes: Vec::new(),
max_bytes,
}
}
fn into_inner(self) -> Vec<u8> {
self.bytes
}
}
impl Write for SizeLimitedBuffer {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let next_len = self.bytes.len().checked_add(buf.len()).ok_or_else(|| {
io::Error::other(ArchiveSizeLimitExceeded {
bytes: usize::MAX,
max_bytes: self.max_bytes,
})
})?;
if next_len > self.max_bytes {
return Err(io::Error::other(ArchiveSizeLimitExceeded {
bytes: next_len,
max_bytes: self.max_bytes,
}));
}
self.bytes.extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
#[derive(Debug)]
struct ArchiveSizeLimitExceeded {
bytes: usize,
max_bytes: usize,
}
impl fmt::Display for ArchiveSizeLimitExceeded {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(
f,
"archive would be {} bytes, exceeding maximum size of {} bytes",
self.bytes, self.max_bytes
)
}
}
impl std::error::Error for ArchiveSizeLimitExceeded {}
async fn send_and_expect_status(
request: RequestBuilder,
url_for_error: &str,

View File

@@ -326,6 +326,34 @@ fn archive_plugin_for_upload_places_manifest_at_archive_root() {
);
}
#[test]
fn archive_plugin_for_upload_round_trips_through_plugin_bundle_archive_with_long_paths() {
let temp_dir = TempDir::new().unwrap();
let plugin_path = write_test_plugin(temp_dir.path(), "demo-plugin");
let long_skill_path = Path::new("skills")
.join(["segment"; 40].join("/"))
.join("SKILL.md");
write_file(&plugin_path.join(&long_skill_path), "# Long path skill\n");
let archive_bytes = archive_plugin_for_upload(&plugin_path).unwrap();
let destination = TempDir::new().unwrap();
crate::plugin_bundle_archive::unpack_plugin_bundle_tar_gz(
&archive_bytes,
destination.path(),
/*max_total_bytes*/ 1024 * 1024,
)
.expect("extract shared plugin archive");
assert_eq!(
fs::read_to_string(destination.path().join(".codex-plugin/plugin.json")).unwrap(),
r#"{"name":"demo-plugin"}"#
);
assert_eq!(
fs::read_to_string(destination.path().join(long_skill_path)).unwrap(),
"# Long path skill\n"
);
}
#[tokio::test]
async fn save_remote_plugin_share_updates_existing_workspace_plugin() {
let codex_home = TempDir::new().unwrap();

View File

@@ -1,3 +1,5 @@
use crate::plugin_bundle_archive::PluginBundleUnpackError;
use crate::plugin_bundle_archive::unpack_plugin_bundle_tar_gz;
use crate::remote::REMOTE_GLOBAL_MARKETPLACE_NAME;
use crate::store::PluginInstallResult;
use crate::store::PluginStore;
@@ -8,17 +10,14 @@ use codex_plugin::PluginId;
use codex_plugin::PluginIdError;
use codex_utils_absolute_path::AbsolutePathBuf;
use codex_utils_plugins::find_plugin_manifest_path;
use flate2::read::GzDecoder;
use reqwest::Response;
use reqwest::StatusCode;
use serde_json::Value as JsonValue;
use std::fs;
use std::io;
use std::io::Read;
use std::path::Path;
use std::path::PathBuf;
use std::time::Duration;
use tar::Archive;
use url::Host;
use url::Url;
@@ -542,146 +541,17 @@ fn extract_plugin_bundle_tar_gz_with_limits(
destination: &Path,
max_total_bytes: u64,
) -> Result<(), RemotePluginBundleInstallError> {
fs::create_dir_all(destination).map_err(|source| {
RemotePluginBundleInstallError::io(
"failed to create remote plugin bundle extraction directory",
source,
)
})?;
let archive = GzDecoder::new(std::io::Cursor::new(bytes));
let mut archive = Archive::new(archive);
extract_plugin_bundle_tar(&mut archive, destination, max_total_bytes)
}
fn extract_plugin_bundle_tar<R: Read>(
archive: &mut Archive<R>,
destination: &Path,
max_total_bytes: u64,
) -> Result<(), RemotePluginBundleInstallError> {
let mut extracted_bytes = 0u64;
let entries = archive.entries().map_err(|source| {
RemotePluginBundleInstallError::io("failed to read remote plugin bundle tar", source)
})?;
let entries = entries.raw(true);
for entry in entries {
let mut entry = entry.map_err(|source| {
RemotePluginBundleInstallError::io(
"failed to read remote plugin bundle tar entry",
source,
)
})?;
let entry_type = entry.header().entry_type();
let entry_size = entry.size();
let entry_path = entry.path().map_err(|source| {
RemotePluginBundleInstallError::io(
"failed to read remote plugin bundle tar entry path",
source,
)
})?;
let entry_path = entry_path.into_owned();
let output_path = checked_tar_output_path(destination, &entry_path)?;
if entry_type.is_dir() {
fs::create_dir_all(&output_path).map_err(|source| {
RemotePluginBundleInstallError::io(
"failed to create remote plugin bundle directory",
source,
)
})?;
continue;
unpack_plugin_bundle_tar_gz(bytes, destination, max_total_bytes).map_err(|err| match err {
PluginBundleUnpackError::ExtractedBundleTooLarge { bytes, max_bytes } => {
RemotePluginBundleInstallError::ExtractedBundleTooLarge { bytes, max_bytes }
}
if entry_type.is_file() {
enforce_total_extracted_size(entry_size, &mut extracted_bytes, max_total_bytes)?;
let Some(parent) = output_path.parent() else {
return Err(RemotePluginBundleInstallError::InvalidBundle(format!(
"remote plugin bundle output path has no parent: {}",
output_path.display()
)));
};
fs::create_dir_all(parent).map_err(|source| {
RemotePluginBundleInstallError::io(
"failed to create remote plugin bundle directory",
source,
)
})?;
entry.unpack(&output_path).map_err(|source| {
RemotePluginBundleInstallError::io(
"failed to unpack remote plugin bundle entry",
source,
)
})?;
continue;
PluginBundleUnpackError::Io { context, source } => {
RemotePluginBundleInstallError::io(context, source)
}
if entry_type.is_hard_link() || entry_type.is_symlink() {
return Err(RemotePluginBundleInstallError::InvalidBundle(format!(
"remote plugin bundle tar entry `{}` is a link",
entry_path.display()
)));
PluginBundleUnpackError::InvalidBundle(message) => {
RemotePluginBundleInstallError::InvalidBundle(message)
}
return Err(RemotePluginBundleInstallError::InvalidBundle(format!(
"remote plugin bundle tar entry `{}` has unsupported type {:?}",
entry_path.display(),
entry_type
)));
}
Ok(())
}
fn checked_tar_output_path(
destination: &Path,
entry_name: &Path,
) -> Result<PathBuf, RemotePluginBundleInstallError> {
let mut output_path = destination.to_path_buf();
let mut has_component = false;
for component in entry_name.components() {
match component {
std::path::Component::Normal(component) => {
has_component = true;
output_path.push(component);
}
std::path::Component::CurDir => {}
std::path::Component::ParentDir
| std::path::Component::RootDir
| std::path::Component::Prefix(_) => {
return Err(RemotePluginBundleInstallError::InvalidBundle(format!(
"remote plugin bundle tar entry `{}` escapes extraction root",
entry_name.display()
)));
}
}
}
if !has_component {
return Err(RemotePluginBundleInstallError::InvalidBundle(
"remote plugin bundle tar entry has an empty path".to_string(),
));
}
Ok(output_path)
}
fn enforce_total_extracted_size(
entry_size: u64,
extracted_bytes: &mut u64,
max_total_bytes: u64,
) -> Result<(), RemotePluginBundleInstallError> {
let next_total = extracted_bytes.checked_add(entry_size).ok_or(
RemotePluginBundleInstallError::ExtractedBundleTooLarge {
bytes: u64::MAX,
max_bytes: max_total_bytes,
},
)?;
if next_total > max_total_bytes {
return Err(RemotePluginBundleInstallError::ExtractedBundleTooLarge {
bytes: next_total,
max_bytes: max_total_bytes,
});
}
*extracted_bytes = next_total;
Ok(())
})
}
fn find_extracted_plugin_root(
@@ -706,6 +576,7 @@ mod tests {
use flate2::Compression;
use flate2::write::GzEncoder;
use pretty_assertions::assert_eq;
use std::io::Write;
use tempfile::tempdir;
const REMOTE_PLUGIN_ID: &str = "plugins~Plugin_00000000000000000000000000000000";
@@ -830,7 +701,7 @@ mod tests {
)
.expect_err("invalid tar.gz should be rejected");
assert!(format!("{err}").contains("failed to read remote plugin bundle tar"));
assert!(format!("{err}").contains("failed to read plugin bundle tar"));
}
#[test]
@@ -961,8 +832,11 @@ mod tests {
#[test]
fn extraction_rejects_tar_path_traversal() {
let destination = tempdir().expect("tempdir");
let err = checked_tar_output_path(destination.path(), Path::new("../evil.txt"))
.expect_err("tar path traversal should be rejected");
let err = extract_plugin_bundle_tar_gz(
&tar_gz_bytes_with_raw_path("../evil.txt", b"evil", /*mode*/ 0o644),
destination.path(),
)
.expect_err("tar path traversal should be rejected");
assert!(format!("{err}").contains("escapes extraction root"));
}
@@ -987,20 +861,20 @@ mod tests {
}
#[test]
fn extraction_rejects_pax_metadata_entries() {
fn extraction_supports_gnu_long_name_entries() {
let destination = tempdir().expect("tempdir");
let err = extract_plugin_bundle_tar_gz(
&tar_gz_bytes_with_entry_type(
tar::EntryType::XHeader,
"PaxHeaders.0/linear",
b"18 path=linear\n",
/*mode*/ 0o644,
),
let long_path = format!("{}/file.txt", ["segment"; 40].join("/"));
extract_plugin_bundle_tar_gz(
&tar_gz_bytes(&[(long_path.as_str(), b"long", /*mode*/ 0o644)]),
destination.path(),
)
.expect_err("pax metadata entries should be rejected");
.expect("extract bundle with GNU long name entry");
assert!(format!("{err}").contains("unsupported type"));
assert_eq!(
std::fs::read(destination.path().join(long_path)).expect("read extracted file"),
b"long"
);
}
#[cfg(unix)]
@@ -1051,16 +925,25 @@ mod tests {
finish_tar_gz(tar)
}
fn tar_gz_bytes_with_entry_type(
entry_type: tar::EntryType,
path: &str,
contents: &[u8],
mode: u32,
) -> Vec<u8> {
let encoder = GzEncoder::new(Vec::new(), Compression::default());
let mut tar = tar::Builder::new(encoder);
append_tar_entry(&mut tar, entry_type, path, contents, mode);
finish_tar_gz(tar)
fn tar_gz_bytes_with_raw_path(path: &str, contents: &[u8], mode: u32) -> Vec<u8> {
let mut header = tar::Header::new_gnu();
header.set_entry_type(tar::EntryType::Regular);
header.set_size(contents.len() as u64);
header.set_mode(mode);
header.as_mut_bytes()[..path.len()].copy_from_slice(path.as_bytes());
header.set_cksum();
let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
encoder
.write_all(header.as_bytes())
.expect("write tar header");
encoder.write_all(contents).expect("write tar contents");
let padding = (512 - (contents.len() % 512)) % 512;
encoder
.write_all(&vec![0; padding])
.expect("write tar padding");
encoder.write_all(&[0; 1024]).expect("write tar terminator");
encoder.finish().expect("finish gzip")
}
fn append_tar_entry<W: std::io::Write>(

View File

@@ -126,6 +126,11 @@ impl ContextManager {
&self.items
}
/// Returns raw items in the history and consumes the snapshot.
pub(crate) fn into_raw_items(self) -> Vec<ResponseItem> {
self.items
}
pub(crate) fn history_version(&self) -> u64 {
self.history_version
}

View File

@@ -1,5 +1,6 @@
use std::sync::Arc;
use codex_tools::ConversationHistory;
use codex_tools::ToolCall as ExtensionToolCall;
use codex_tools::ToolName;
use codex_tools::ToolSpec;
@@ -53,7 +54,7 @@ impl ToolExecutor<ToolInvocation> for ExtensionToolAdapter {
&self,
invocation: ToolInvocation,
) -> Result<Box<dyn ToolOutput>, FunctionCallError> {
self.0.handle(to_extension_call(&invocation)).await
self.0.handle(to_extension_call(&invocation).await).await
}
}
@@ -86,12 +87,15 @@ impl CoreToolRuntime for ExtensionToolAdapter {
}
}
fn to_extension_call(invocation: &ToolInvocation) -> ExtensionToolCall {
async fn to_extension_call(invocation: &ToolInvocation) -> ExtensionToolCall {
let conversation_history =
ConversationHistory::new(invocation.session.clone_history().await.into_raw_items());
ExtensionToolCall {
turn_id: invocation.turn.sub_id.clone(),
call_id: invocation.call_id.clone(),
tool_name: invocation.tool_name.clone(),
truncation_policy: invocation.turn.truncation_policy,
conversation_history,
payload: invocation.payload.clone(),
}
}
@@ -108,6 +112,8 @@ fn extension_tool_hook_input(arguments: &str) -> Value {
mod tests {
use std::sync::Arc;
use codex_protocol::models::ContentItem;
use codex_protocol::models::ResponseItem;
use pretty_assertions::assert_eq;
use serde_json::json;
use tokio::sync::Mutex;
@@ -236,6 +242,17 @@ mod tests {
let (session, turn) = crate::session::tests::make_session_and_context().await;
let turn_id = turn.sub_id.clone();
let truncation_policy = turn.truncation_policy;
let history_item = ResponseItem::Message {
id: None,
role: "user".to_string(),
content: vec![ContentItem::InputText {
text: "extension history".to_string(),
}],
phase: None,
};
session
.record_into_history(std::slice::from_ref(&history_item), &turn)
.await;
let invocation = ToolInvocation {
session: session.into(),
turn: turn.into(),
@@ -261,6 +278,10 @@ mod tests {
codex_tools::ToolName::plain("extension_echo")
);
assert_eq!(captured_call.truncation_policy, truncation_policy);
assert_eq!(
captured_call.conversation_history.items(),
std::slice::from_ref(&history_item)
);
match captured_call.payload {
ToolPayload::Function { arguments } => {
assert_eq!(arguments, json!({ "message": "hello" }).to_string());

View File

@@ -49,7 +49,16 @@ impl ToolExecutor<ToolInvocation> for McpHandler {
}
fn supports_parallel_tool_calls(&self) -> bool {
// Correctly implemented MCP servers should tolerate parallel calls to
// tools that advertise themselves as read-only.
self.tool_info.supports_parallel_tool_calls
|| self
.tool_info
.tool
.annotations
.as_ref()
.and_then(|annotations| annotations.read_only_hint)
.unwrap_or(false)
}
async fn handle(
@@ -443,6 +452,44 @@ mod tests {
assert_eq!(mcp_hook_tool_input(" "), json!({}));
}
#[test]
fn mcp_read_only_hint_supports_parallel_calls_without_server_opt_in() {
let mut read_only_info = tool_info("foo", "mcp__foo__", "read");
read_only_info.tool.annotations = Some(rmcp::model::ToolAnnotations::new().read_only(true));
assert!(
McpHandler::new(read_only_info)
.expect("MCP tool spec should build")
.supports_parallel_tool_calls()
);
}
#[test]
fn mcp_parallel_calls_require_read_only_hint_or_server_opt_in() {
let missing_hint_info = tool_info("foo", "mcp__foo__", "unannotated");
assert!(
!McpHandler::new(missing_hint_info)
.expect("MCP tool spec should build")
.supports_parallel_tool_calls()
);
let mut writable_info = tool_info("foo", "mcp__foo__", "write");
writable_info.tool.annotations = Some(rmcp::model::ToolAnnotations::new().read_only(false));
assert!(
!McpHandler::new(writable_info)
.expect("MCP tool spec should build")
.supports_parallel_tool_calls()
);
let mut server_opt_in_info = tool_info("foo", "mcp__foo__", "server_opt_in");
server_opt_in_info.supports_parallel_tool_calls = true;
assert!(
McpHandler::new(server_opt_in_info)
.expect("MCP tool spec should build")
.supports_parallel_tool_calls()
);
}
fn tool_info(server_name: &str, callable_namespace: &str, tool_name: &str) -> ToolInfo {
ToolInfo {
server_name: server_name.to_string(),

View File

@@ -11,6 +11,7 @@ use codex_extension_api::ResponsesApiTool;
use codex_extension_api::ToolCall as ExtensionToolCall;
use codex_extension_api::ToolExecutor;
use codex_protocol::dynamic_tools::DynamicToolSpec;
use codex_protocol::models::ContentItem;
use codex_protocol::models::FunctionCallOutputBody;
use codex_protocol::models::ResponseInputItem;
use codex_protocol::models::ResponseItem;
@@ -81,6 +82,7 @@ impl ToolExecutor<ExtensionToolCall> for ExtensionEchoExecutor {
Ok(Box::new(codex_tools::JsonToolOutput::new(json!({
"arguments": arguments,
"callId": call.call_id,
"conversationHistory": call.conversation_history.items(),
"ok": true,
}))))
}
@@ -327,6 +329,17 @@ fn mcp_tool_info(
async fn extension_tool_executors_are_model_visible_and_dispatchable() -> anyhow::Result<()> {
let (mut session, turn) = make_session_and_context().await;
session.services.extensions = extension_tool_test_registry();
let history_item = ResponseItem::Message {
id: None,
role: "user".to_string(),
content: vec![ContentItem::InputText {
text: "extension history".to_string(),
}],
phase: None,
};
session
.record_into_history(std::slice::from_ref(&history_item), &turn)
.await;
let router = ToolRouter::from_turn_context(
&turn,
@@ -384,6 +397,7 @@ async fn extension_tool_executors_are_model_visible_and_dispatchable() -> anyhow
json!({
"arguments": { "message": "hello" },
"callId": "call-extension",
"conversationHistory": [history_item],
"ok": true,
})
);

View File

@@ -101,10 +101,28 @@ fn read_only_user_turn_with_model(
fixture: &TestCodex,
text: impl Into<String>,
model: String,
) -> Op {
user_turn_with_permission_profile(fixture, text, model, PermissionProfile::read_only())
}
fn auto_approved_user_turn(fixture: &TestCodex, text: impl Into<String>) -> Op {
user_turn_with_permission_profile(
fixture,
text,
fixture.session_configured.model.clone(),
PermissionProfile::Disabled,
)
}
fn user_turn_with_permission_profile(
fixture: &TestCodex,
text: impl Into<String>,
model: String,
permission_profile: PermissionProfile,
) -> Op {
let cwd = fixture.cwd.path().to_path_buf();
let (sandbox_policy, permission_profile) =
turn_permission_fields(PermissionProfile::read_only(), cwd.as_path());
turn_permission_fields(permission_profile, cwd.as_path());
Op::UserInput {
items: vec![UserInput::Text {
text: text.into(),
@@ -840,7 +858,10 @@ async fn stdio_mcp_parallel_tool_calls_default_false_runs_serially() -> anyhow::
.await?;
fixture
.codex
.submit(read_only_user_turn(
// Keep this baseline on the mutable sync tool so read-only hints do not
// make the call parallel-safe. Bypass read-only turn permissions so
// approval behavior does not block the scheduling assertion.
.submit(auto_approved_user_turn(
&fixture,
"call the rmcp sync tool twice",
))
@@ -899,6 +920,102 @@ async fn stdio_mcp_parallel_tool_calls_default_false_runs_serially() -> anyhow::
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn stdio_mcp_read_only_tool_calls_run_concurrently_without_server_opt_in()
-> anyhow::Result<()> {
skip_if_no_network!(Ok(()));
let server = responses::start_mock_server().await;
let first_call_id = "sync-read-only-1";
let second_call_id = "sync-read-only-2";
let server_name = "rmcp";
let namespace = format!("mcp__{server_name}__");
// The stdio MCP test server holds each sync call at this barrier until both
// calls arrive. A serial scheduler times out inside the server instead of
// returning the structured `{ "result": "ok" }` result asserted below.
let args = json!({
"sleep_after_ms": 100,
"barrier": {
"id": "stdio-mcp-read-only-tool-calls",
"participants": 2,
"timeout_ms": 1_000
}
})
.to_string();
mount_sse_once(
&server,
responses::sse(vec![
responses::ev_response_created("resp-1"),
responses::ev_function_call_with_namespace(
first_call_id,
&namespace,
"sync_readonly",
&args,
),
responses::ev_function_call_with_namespace(
second_call_id,
&namespace,
"sync_readonly",
&args,
),
responses::ev_completed("resp-1"),
]),
)
.await;
let final_mock = mount_sse_once(
&server,
responses::sse(vec![
responses::ev_assistant_message("msg-1", "rmcp sync tools completed successfully."),
responses::ev_completed("resp-2"),
]),
)
.await;
let rmcp_test_server_bin = remote_aware_stdio_server_bin()?;
let fixture = test_codex()
.with_config(move |config| {
insert_mcp_server(
config,
server_name,
stdio_transport(rmcp_test_server_bin, /*env*/ None, Vec::new()),
TestMcpServerOptions {
environment_id: remote_aware_environment_id(),
tool_timeout_sec: Some(Duration::from_secs(2)),
..Default::default()
},
);
})
.build_with_remote_env(&server)
.await?;
fixture
.codex
.submit(read_only_user_turn(
&fixture,
"call the rmcp sync_readonly tool twice",
))
.await?;
wait_for_event(&fixture.codex, |ev| matches!(ev, EventMsg::TurnComplete(_))).await;
let request = final_mock.single_request();
for call_id in [first_call_id, second_call_id] {
let output_text = request
.function_call_output_text(call_id)
.expect("function_call_output present for rmcp sync call");
let wrapped_payload = split_wall_time_wrapped_output(&output_text);
let output_json: Value = serde_json::from_str(wrapped_payload)
.expect("wrapped MCP output should preserve structured JSON");
assert_eq!(output_json, json!({ "result": "ok" }));
}
server.verify().await;
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn stdio_mcp_parallel_tool_calls_opt_in_runs_concurrently() -> anyhow::Result<()> {
skip_if_no_network!(Ok(()));
@@ -957,7 +1074,10 @@ async fn stdio_mcp_parallel_tool_calls_opt_in_runs_concurrently() -> anyhow::Res
.await?;
fixture
.codex
.submit(read_only_user_turn(
// Exercise the server opt-in with the mutable sync tool rather than the
// read-only sync_readonly tool. Bypass read-only turn permissions so
// approval behavior does not block the scheduling assertion.
.submit(auto_approved_user_turn(
&fixture,
"call the rmcp sync tool twice",
))

View File

@@ -10,6 +10,7 @@ pub use capabilities::NoopExtensionEventSink;
pub use capabilities::NoopResponseItemInjector;
pub use capabilities::ResponseItemInjectionFuture;
pub use capabilities::ResponseItemInjector;
pub use codex_tools::ConversationHistory;
pub use codex_tools::FunctionCallError;
pub use codex_tools::JsonToolOutput;
pub use codex_tools::ResponsesApiTool;

View File

@@ -625,6 +625,7 @@ fn tool_call(tool_name: &str, call_id: &str, arguments: serde_json::Value) -> To
call_id: call_id.to_string(),
tool_name: codex_extension_api::ToolName::plain(tool_name),
truncation_policy: TruncationPolicy::Bytes(1024),
conversation_history: codex_extension_api::ConversationHistory::default(),
payload: ToolPayload::Function {
arguments: arguments.to_string(),
},

View File

@@ -139,6 +139,7 @@ async fn read_tool_reads_memory_file() {
call_id: "call-1".to_string(),
tool_name: memory_tool_name(crate::READ_TOOL_NAME),
truncation_policy: TruncationPolicy::Bytes(1024),
conversation_history: codex_extension_api::ConversationHistory::default(),
payload: payload.clone(),
})
.await
@@ -183,6 +184,7 @@ async fn search_tool_accepts_multiple_queries() {
call_id: "call-1".to_string(),
tool_name: memory_tool_name(crate::SEARCH_TOOL_NAME),
truncation_policy: TruncationPolicy::Bytes(1024),
conversation_history: codex_extension_api::ConversationHistory::default(),
payload: payload.clone(),
})
.await
@@ -253,6 +255,7 @@ async fn search_tool_accepts_windowed_all_match_mode() {
call_id: "call-1".to_string(),
tool_name: memory_tool_name(crate::SEARCH_TOOL_NAME),
truncation_policy: TruncationPolicy::Bytes(1024),
conversation_history: codex_extension_api::ConversationHistory::default(),
payload: payload.clone(),
})
.await
@@ -303,6 +306,7 @@ async fn search_tool_rejects_legacy_single_query() {
call_id: "call-1".to_string(),
tool_name: memory_tool_name(crate::SEARCH_TOOL_NAME),
truncation_policy: TruncationPolicy::Bytes(1024),
conversation_history: codex_extension_api::ConversationHistory::default(),
payload,
})
.await;

View File

@@ -366,12 +366,14 @@ pub const ALL_PROXY_ENV_KEYS: &[&str] = &["ALL_PROXY", "all_proxy"];
pub const PROXY_ACTIVE_ENV_KEY: &str = "CODEX_NETWORK_PROXY_ACTIVE";
pub const ALLOW_LOCAL_BINDING_ENV_KEY: &str = "CODEX_NETWORK_ALLOW_LOCAL_BINDING";
const ELECTRON_GET_USE_PROXY_ENV_KEY: &str = "ELECTRON_GET_USE_PROXY";
const NODE_USE_ENV_PROXY_ENV_KEY: &str = "NODE_USE_ENV_PROXY";
#[cfg(any(target_os = "macos", test))]
const GIT_SSH_COMMAND_ENV_KEY: &str = "GIT_SSH_COMMAND";
pub const PROXY_ENV_KEYS: &[&str] = &[
PROXY_ACTIVE_ENV_KEY,
ALLOW_LOCAL_BINDING_ENV_KEY,
ELECTRON_GET_USE_PROXY_ENV_KEY,
NODE_USE_ENV_PROXY_ENV_KEY,
"HTTP_PROXY",
"HTTPS_PROXY",
"http_proxy",
@@ -525,6 +527,8 @@ fn apply_proxy_env_overrides(
ELECTRON_GET_USE_PROXY_ENV_KEY.to_string(),
"true".to_string(),
);
// Node.js built-in HTTP clients only honor proxy environment variables when this is enabled.
env.insert(NODE_USE_ENV_PROXY_ENV_KEY.to_string(), "1".to_string());
// Keep HTTP_PROXY/HTTPS_PROXY as HTTP endpoints. A lot of clients break if
// those vars contain SOCKS URLs. We only switch ALL_PROXY here.
@@ -1016,6 +1020,7 @@ mod tests {
env.get(ELECTRON_GET_USE_PROXY_ENV_KEY),
Some(&"true".to_string())
);
assert_eq!(env.get(NODE_USE_ENV_PROXY_ENV_KEY), Some(&"1".to_string()));
#[cfg(target_os = "macos")]
assert_eq!(
env.get(GIT_SSH_COMMAND_ENV_KEY),

View File

@@ -70,6 +70,7 @@ impl TestToolServer {
Self::echo_dash_tool(),
Self::cwd_tool(),
Self::sync_tool(),
Self::sync_readonly_tool(),
Self::image_tool(),
Self::image_scenario_tool(),
sandbox_meta_tool,
@@ -205,6 +206,12 @@ impl TestToolServer {
}))
.expect("sync tool output schema should deserialize");
tool.output_schema = Some(Arc::new(output_schema));
tool
}
fn sync_readonly_tool() -> Tool {
let mut tool = Self::sync_tool();
tool.name = Cow::Borrowed("sync_readonly");
tool.annotations = Some(ToolAnnotations::new().read_only(true));
tool
}
@@ -551,6 +558,10 @@ impl ServerHandler for TestToolServer {
let args = Self::parse_call_args::<SyncArgs>(&request, "sync")?;
Self::sync_result(args).await
}
"sync_readonly" => {
let args = Self::parse_call_args::<SyncArgs>(&request, "sync_readonly")?;
Self::sync_result(args).await
}
other => Err(McpError::invalid_params(
format!("unknown tool: {other}"),
None,

View File

@@ -17,6 +17,7 @@ codex-utils-absolute-path = { workspace = true }
codex-utils-output-truncation = { workspace = true }
codex-utils-pty = { workspace = true }
codex-utils-string = { workspace = true }
jsonptr = { workspace = true }
rmcp = { workspace = true, default-features = false, features = [
"base64",
"macros",
@@ -27,6 +28,7 @@ serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }
urlencoding = { workspace = true }
[dev-dependencies]
pretty_assertions = { workspace = true }

View File

@@ -3,6 +3,10 @@ use serde::Serialize;
use serde_json::Value as JsonValue;
use serde_json::json;
use std::collections::BTreeMap;
use std::collections::BTreeSet;
const DEFINITION_TABLE_KEYS: [&str; 2] = ["$defs", "definitions"];
const SCHEMA_CHILD_KEYS: [&str; 2] = ["items", "anyOf"];
/// Primitive JSON Schema type names we support in tool definitions.
///
@@ -33,6 +37,8 @@ pub enum JsonSchemaType {
/// Generic JSON-Schema subset needed for our tool definitions.
#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)]
pub struct JsonSchema {
#[serde(rename = "$ref", skip_serializing_if = "Option::is_none")]
pub schema_ref: Option<String>,
#[serde(rename = "type", skip_serializing_if = "Option::is_none")]
pub schema_type: Option<JsonSchemaType>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -52,6 +58,10 @@ pub struct JsonSchema {
pub additional_properties: Option<AdditionalProperties>,
#[serde(rename = "anyOf", skip_serializing_if = "Option::is_none")]
pub any_of: Option<Vec<JsonSchema>>,
#[serde(rename = "$defs", skip_serializing_if = "Option::is_none")]
pub defs: Option<BTreeMap<String, JsonSchema>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub definitions: Option<BTreeMap<String, JsonSchema>>,
}
impl JsonSchema {
@@ -149,6 +159,8 @@ impl From<JsonSchema> for AdditionalProperties {
pub fn parse_tool_input_schema(input_schema: &JsonValue) -> Result<JsonSchema, serde_json::Error> {
let mut input_schema = input_schema.clone();
sanitize_json_schema(&mut input_schema);
prune_unreachable_definitions(&mut input_schema);
compact_large_tool_schema(&mut input_schema);
let schema: JsonSchema = serde_json::from_value(input_schema)?;
if matches!(
schema.schema_type,
@@ -159,10 +171,220 @@ pub fn parse_tool_input_schema(input_schema: &JsonValue) -> Result<JsonSchema, s
Ok(schema)
}
// Use compact normalized JSON bytes as a cheap local proxy for the 1k-token
// schema budget.
const MAX_COMPACT_TOOL_SCHEMA_BYTES: usize = 4_000;
const MAX_COMPACT_TOOL_SCHEMA_DEPTH: usize = 2;
/// Shrink unusually large tool schemas while preserving the top-level argument
/// surface. Compaction is best-effort rather than a hard cap: it runs only
/// after schema sanitization/pruning and applies increasingly lossy passes
/// while the schema remains over budget.
fn compact_large_tool_schema(value: &mut JsonValue) {
for pass in LARGE_SCHEMA_COMPACTION_PASSES {
if compact_schema_fits_budget(value) {
break;
}
pass(value);
}
}
type LargeSchemaCompactionPass = fn(&mut JsonValue);
const LARGE_SCHEMA_COMPACTION_PASSES: &[LargeSchemaCompactionPass] = &[
strip_schema_descriptions,
drop_schema_definitions,
collapse_deep_schema_objects_from_root,
];
fn collapse_deep_schema_objects_from_root(value: &mut JsonValue) {
collapse_deep_schema_objects(value, /*depth*/ 0);
}
fn compact_schema_fits_budget(value: &JsonValue) -> bool {
compact_normalized_schema_len(value) <= MAX_COMPACT_TOOL_SCHEMA_BYTES
}
fn compact_normalized_schema_len(value: &JsonValue) -> usize {
serde_json::from_value::<JsonSchema>(value.clone())
.and_then(|schema| serde_json::to_vec(&schema))
.map(|json| json.len())
.unwrap_or(0)
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum DefinitionTraversal {
Include,
Skip,
}
fn for_each_schema_child(
map: &serde_json::Map<String, JsonValue>,
definition_traversal: DefinitionTraversal,
visitor: &mut impl FnMut(&JsonValue),
) {
if let Some(properties) = map.get("properties")
&& let Some(properties_map) = properties.as_object()
{
for value in properties_map.values() {
visitor(value);
}
}
for key in SCHEMA_CHILD_KEYS {
if let Some(value) = map.get(key) {
visitor(value);
}
}
if let Some(additional_properties) = map.get("additionalProperties")
&& !matches!(additional_properties, JsonValue::Bool(_))
{
visitor(additional_properties);
}
if definition_traversal == DefinitionTraversal::Include {
for key in DEFINITION_TABLE_KEYS {
if let Some(definitions) = map.get(key)
&& let Some(definitions_map) = definitions.as_object()
{
for value in definitions_map.values() {
visitor(value);
}
}
}
}
}
fn strip_schema_descriptions(value: &mut JsonValue) {
match value {
JsonValue::Array(values) => {
for value in values {
strip_schema_descriptions(value);
}
}
JsonValue::Object(map) => {
map.remove("description");
for_each_schema_child_mut(map, DefinitionTraversal::Include, &mut |value| {
strip_schema_descriptions(value);
});
}
_ => {}
}
}
fn for_each_schema_child_mut(
map: &mut serde_json::Map<String, JsonValue>,
definition_traversal: DefinitionTraversal,
visitor: &mut impl FnMut(&mut JsonValue),
) {
if let Some(properties) = map.get_mut("properties")
&& let Some(properties_map) = properties.as_object_mut()
{
for value in properties_map.values_mut() {
visitor(value);
}
}
for key in SCHEMA_CHILD_KEYS {
if let Some(value) = map.get_mut(key) {
visitor(value);
}
}
if let Some(additional_properties) = map.get_mut("additionalProperties")
&& !matches!(additional_properties, JsonValue::Bool(_))
{
visitor(additional_properties);
}
if definition_traversal == DefinitionTraversal::Include {
for key in DEFINITION_TABLE_KEYS {
if let Some(definitions) = map.get_mut(key)
&& let Some(definitions_map) = definitions.as_object_mut()
{
for value in definitions_map.values_mut() {
visitor(value);
}
}
}
}
}
/// Replace local definition refs with empty schemas before dropping root
/// definition tables, so downstream behavior does not depend on how a schema
/// parser handles refs to missing definitions.
fn drop_schema_definitions(value: &mut JsonValue) {
rewrite_definition_refs_to_empty_schemas(value);
let JsonValue::Object(map) = value else {
return;
};
for key in DEFINITION_TABLE_KEYS {
map.remove(key);
}
}
fn rewrite_definition_refs_to_empty_schemas(value: &mut JsonValue) {
match value {
JsonValue::Array(values) => {
for value in values {
rewrite_definition_refs_to_empty_schemas(value);
}
}
JsonValue::Object(map) => {
if map
.get("$ref")
.and_then(JsonValue::as_str)
.and_then(parse_local_definition_ref)
.is_some()
{
*value = json!({});
return;
}
for_each_schema_child_mut(map, DefinitionTraversal::Skip, &mut |value| {
rewrite_definition_refs_to_empty_schemas(value);
});
}
_ => {}
}
}
fn collapse_deep_schema_objects(value: &mut JsonValue, depth: usize) {
match value {
JsonValue::Array(values) => {
for value in values {
collapse_deep_schema_objects(value, depth);
}
}
JsonValue::Object(map) => {
if depth >= MAX_COMPACT_TOOL_SCHEMA_DEPTH && is_complex_schema_object(map) {
*value = json!({});
return;
}
for_each_schema_child_mut(map, DefinitionTraversal::Skip, &mut |value| {
collapse_deep_schema_objects(value, depth + 1);
});
}
_ => {}
}
}
fn is_complex_schema_object(map: &serde_json::Map<String, JsonValue>) -> bool {
SCHEMA_CHILD_KEYS.iter().any(|key| map.contains_key(*key))
|| map.contains_key("properties")
|| map.contains_key("additionalProperties")
|| map.contains_key("$ref")
}
/// Sanitize a JSON Schema (as serde_json::Value) so it can fit our limited
/// schema representation. This function:
/// - Ensures every typed schema object has a `"type"` when required.
/// - Preserves explicit `anyOf`.
/// - Preserves `$ref` and reachable local `$defs` / `definitions`.
/// - Collapses `const` into single-value `enum`.
/// - Fills required child fields for object/array schema types, including
/// nullable unions, with permissive defaults when absent.
@@ -200,6 +422,9 @@ fn sanitize_json_schema(value: &mut JsonValue) {
if let Some(value) = map.get_mut("anyOf") {
sanitize_json_schema(value);
}
for table in DEFINITION_TABLE_KEYS {
sanitize_schema_table(map, table);
}
if let Some(const_value) = map.remove("const") {
map.insert("enum".to_string(), JsonValue::Array(vec![const_value]));
@@ -207,7 +432,7 @@ fn sanitize_json_schema(value: &mut JsonValue) {
let mut schema_types = normalized_schema_types(map);
if schema_types.is_empty() && map.contains_key("anyOf") {
if schema_types.is_empty() && (map.contains_key("$ref") || map.contains_key("anyOf")) {
return;
}
@@ -241,6 +466,29 @@ fn sanitize_json_schema(value: &mut JsonValue) {
}
}
/// Sanitize a schema definition table before deserializing into `JsonSchema`.
///
/// Definition tables must be objects. Codex keeps valid definition tables and
/// recursively applies the same compatibility lowering used for inline schemas,
/// but drops malformed tables so `strict: false` tool registration degrades
/// gracefully instead of failing on an unreachable or invalid definition table.
fn sanitize_schema_table(map: &mut serde_json::Map<String, JsonValue>, key: &str) {
let should_remove = match map.get_mut(key) {
Some(JsonValue::Object(definitions)) => {
for definition in definitions.values_mut() {
sanitize_json_schema(definition);
}
false
}
Some(_) => true,
None => false,
};
if should_remove {
map.remove(key);
}
}
fn ensure_default_children_for_schema_types(
map: &mut serde_json::Map<String, JsonValue>,
schema_types: &[JsonSchemaPrimitiveType],
@@ -257,6 +505,143 @@ fn ensure_default_children_for_schema_types(
}
}
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
struct DefinitionPointer {
table: &'static str,
name: String,
}
/// Prune unused root definition entries to avoid sending tokens for definitions
/// the tool schema never references.
fn prune_unreachable_definitions(value: &mut JsonValue) {
let reachable = collect_reachable_definitions(value);
let JsonValue::Object(map) = value else {
return;
};
for table in DEFINITION_TABLE_KEYS {
prune_schema_table(map, table, &reachable);
}
}
fn prune_schema_table(
map: &mut serde_json::Map<String, JsonValue>,
table: &'static str,
reachable: &BTreeSet<DefinitionPointer>,
) {
let Some(JsonValue::Object(definitions)) = map.get_mut(table) else {
return;
};
definitions.retain(|name, _| {
reachable.contains(&DefinitionPointer {
table,
name: name.clone(),
})
});
if definitions.is_empty() {
map.remove(table);
}
}
fn collect_reachable_definitions(value: &JsonValue) -> BTreeSet<DefinitionPointer> {
let mut reachable = BTreeSet::new();
let mut pending = Vec::new();
collect_refs_outside_definitions(value, &mut pending);
while let Some(pointer) = pending.pop() {
if !reachable.insert(pointer.clone()) {
continue;
}
if let Some(definition) = definition_for_pointer(value, &pointer) {
collect_refs(definition, &mut pending);
}
}
reachable
}
fn collect_refs_outside_definitions(value: &JsonValue, refs: &mut Vec<DefinitionPointer>) {
match value {
JsonValue::Array(values) => {
for value in values {
collect_refs_outside_definitions(value, refs);
}
}
JsonValue::Object(map) => {
collect_ref_from_map(map, refs);
for_each_schema_child(map, DefinitionTraversal::Skip, &mut |value| {
collect_refs_outside_definitions(value, refs);
});
}
_ => {}
}
}
fn collect_refs(value: &JsonValue, refs: &mut Vec<DefinitionPointer>) {
match value {
JsonValue::Array(values) => {
for value in values {
collect_refs(value, refs);
}
}
JsonValue::Object(map) => {
collect_ref_from_map(map, refs);
for value in map.values() {
collect_refs(value, refs);
}
}
_ => {}
}
}
fn collect_ref_from_map(
map: &serde_json::Map<String, JsonValue>,
refs: &mut Vec<DefinitionPointer>,
) {
if let Some(JsonValue::String(schema_ref)) = map.get("$ref")
&& let Some(pointer) = parse_local_definition_ref(schema_ref)
{
refs.push(pointer);
}
}
fn definition_for_pointer<'a>(
value: &'a JsonValue,
pointer: &DefinitionPointer,
) -> Option<&'a JsonValue> {
let JsonValue::Object(map) = value else {
return None;
};
map.get(pointer.table)
.and_then(JsonValue::as_object)
.and_then(|definitions| definitions.get(&pointer.name))
}
fn parse_local_definition_ref(schema_ref: &str) -> Option<DefinitionPointer> {
let fragment = schema_ref.strip_prefix('#')?;
let pointer = urlencoding::decode(fragment).ok()?;
let pointer = jsonptr::Pointer::parse(pointer.as_ref()).ok()?;
let (table_token, pointer) = pointer.split_front()?;
let table = table_token.decoded();
let table = DEFINITION_TABLE_KEYS
.into_iter()
.find(|candidate| table.as_ref() == *candidate)?;
// Responses API non-strict mode accepts nested local refs such as
// `#/$defs/User/properties/name`, so keep the parent definition reachable.
let (name, _) = pointer.split_front()?;
Some(DefinitionPointer {
table,
name: name.decoded().into_owned(),
})
}
fn normalized_schema_types(
map: &serde_json::Map<String, JsonValue>,
) -> Vec<JsonSchemaPrimitiveType> {

View File

@@ -779,6 +779,393 @@ fn parse_tool_input_schema_preserves_explicit_enum_type_union() {
);
}
fn many_string_properties(count: usize) -> serde_json::Map<String, serde_json::Value> {
(0..count)
.map(|index| {
(
format!("field_{index:03}"),
serde_json::json!({ "type": "string" }),
)
})
.collect()
}
#[test]
fn parse_large_tool_input_schema_stops_after_descriptions_when_under_budget() {
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"description": "x".repeat(4_500),
"properties": {
"metadata": {
"$ref": "#/$defs/metadata"
}
},
"$defs": {
"metadata": {
"type": "string",
"description": "Metadata value"
}
}
}))
.expect("parse schema");
assert_eq!(
serde_json::to_value(schema).expect("serialize schema"),
serde_json::json!({
"type": "object",
"properties": {
"metadata": {
"$ref": "#/$defs/metadata"
}
},
"$defs": {
"metadata": {
"type": "string"
}
}
})
);
}
#[test]
fn parse_large_tool_input_schema_ignores_dropped_metadata_for_budget() {
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"event": {
"type": "object",
"title": "Calendar event",
"properties": {
"recurrence": {
"type": "object",
"examples": [
{
"payload": "x".repeat(4_500)
}
],
"properties": {
"pattern": {
"type": "string",
"title": "Recurrence pattern"
}
}
}
}
}
}
}))
.expect("parse schema");
assert_eq!(
serde_json::to_value(schema).expect("serialize schema"),
serde_json::json!({
"type": "object",
"properties": {
"event": {
"type": "object",
"properties": {
"recurrence": {
"type": "object",
"properties": {
"pattern": {
"type": "string"
}
}
}
}
}
}
})
);
}
#[test]
fn parse_large_tool_input_schema_stops_after_dropping_root_definitions_when_under_budget() {
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"description": "x".repeat(4_500),
"properties": {
"event": {
"type": "object",
"description": "Calendar event",
"properties": {
"recurrence": {
"type": "object",
"description": "Recurrence settings",
"properties": {
"pattern": {
"type": "string",
"description": "Recurrence pattern"
}
}
}
}
},
"metadata": {
"$ref": "#/$defs/metadata"
}
},
"$defs": {
"metadata": {
"type": "object",
"description": "metadata object",
"properties": many_string_properties(/*count*/ 300)
}
}
}))
.expect("parse schema");
assert_eq!(
serde_json::to_value(schema).expect("serialize schema"),
serde_json::json!({
"type": "object",
"properties": {
"event": {
"type": "object",
"properties": {
"recurrence": {
"type": "object",
"properties": {
"pattern": {
"type": "string"
}
}
}
}
},
"metadata": {}
}
})
);
}
#[test]
fn parse_large_tool_input_schema_strips_descriptions_without_removing_description_property() {
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"description": "x".repeat(4_500),
"properties": {
"description": {
"type": "string",
"description": "User-facing description value"
},
"metadata": {
"type": "object",
"description": "Metadata object",
"properties": {
"label": {
"type": "string",
"description": "Metadata label"
}
}
},
"tags": {
"type": "array",
"description": "Tag list",
"items": {
"type": "string",
"description": "Tag value"
}
},
"extras": {
"type": "object",
"additionalProperties": {
"type": "string",
"description": "Extra value"
}
},
"choice": {
"description": "Choice value",
"anyOf": [
{
"type": "string",
"description": "String choice"
},
{
"type": "number",
"description": "Number choice"
}
]
}
}
}))
.expect("parse schema");
assert_eq!(
serde_json::to_value(schema).expect("serialize schema"),
serde_json::json!({
"type": "object",
"properties": {
"choice": {
"anyOf": [
{
"type": "string"
},
{
"type": "number"
}
]
},
"description": {
"type": "string"
},
"extras": {
"type": "object",
"properties": {},
"additionalProperties": {
"type": "string"
}
},
"metadata": {
"type": "object",
"properties": {
"label": {
"type": "string"
}
}
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
}
}
})
);
}
#[test]
fn parse_large_tool_input_schema_preserves_object_enum_literal_descriptions() {
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"description": "x".repeat(4_500),
"properties": {
"choice": {
"enum": [
{
"description": "first literal",
"id": 1
},
{
"description": "second literal",
"id": 2
}
]
}
}
}))
.expect("parse schema");
assert_eq!(
serde_json::to_value(schema).expect("serialize schema"),
serde_json::json!({
"type": "object",
"properties": {
"choice": {
"type": "string",
"enum": [
{
"description": "first literal",
"id": 1
},
{
"description": "second literal",
"id": 2
}
]
}
}
})
);
}
#[test]
fn collapse_deep_schema_objects_traverses_schema_children() {
let mut schema = serde_json::json!({
"type": "object",
"properties": {
"object_parent": {
"type": "object",
"properties": {
"complex": {
"type": "object",
"properties": {
"leaf": { "type": "string" }
}
},
"scalar": {
"type": "string"
}
}
},
"array_parent": {
"type": "array",
"items": {
"type": "object",
"properties": {
"leaf": { "type": "string" }
}
}
},
"map_parent": {
"type": "object",
"additionalProperties": {
"type": "object",
"properties": {
"leaf": { "type": "string" }
}
}
},
"union_parent": {
"anyOf": [
{
"type": "object",
"properties": {
"leaf": { "type": "string" }
}
},
{ "type": "string" }
]
}
}
});
super::collapse_deep_schema_objects(&mut schema, /*depth*/ 0);
assert_eq!(
schema,
serde_json::json!({
"type": "object",
"properties": {
"object_parent": {
"type": "object",
"properties": {
"complex": {},
"scalar": {
"type": "string"
}
}
},
"array_parent": {
"type": "array",
"items": {}
},
"map_parent": {
"type": "object",
"additionalProperties": {}
},
"union_parent": {
"anyOf": [
{},
{ "type": "string" }
]
}
}
})
);
}
#[test]
fn parse_tool_input_schema_preserves_string_enum_constraints() {
// Example schema shape:
@@ -848,3 +1235,538 @@ fn parse_tool_input_schema_preserves_string_enum_constraints() {
)
);
}
#[test]
fn parse_tool_input_schema_preserves_refs_and_prunes_unreachable_defs() {
// Example schema shape:
// {
// "type": "object",
// "properties": { "user": { "$ref": "#/$defs/User" } },
// "$defs": {
// "User": { "type": "object", "properties": { "name": { "type": "string" } } },
// "Unused": { "type": "string" }
// }
// }
//
// Expected normalization behavior:
// - Local `$ref` is preserved as a schema hint.
// - Reachable `$defs` entries stay attached to the root schema.
// - Unreachable `$defs` entries are pruned.
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"user": {"$ref": "#/$defs/User"}
},
"$defs": {
"User": {
"type": "object",
"properties": {
"name": {"type": "string"}
}
},
"Unused": {"type": "string"}
}
}))
.expect("parse schema");
assert_eq!(
schema,
JsonSchema {
schema_type: Some(JsonSchemaType::Single(JsonSchemaPrimitiveType::Object)),
properties: Some(BTreeMap::from([(
"user".to_string(),
JsonSchema {
schema_ref: Some("#/$defs/User".to_string()),
..Default::default()
},
)])),
defs: Some(BTreeMap::from([(
"User".to_string(),
JsonSchema::object(
BTreeMap::from([(
"name".to_string(),
JsonSchema::string(/*description*/ None),
)]),
/*required*/ None,
/*additional_properties*/ None,
),
)])),
..Default::default()
}
);
}
#[test]
fn parse_tool_input_schema_preserves_refs_from_properties_named_def_tables() {
// Example schema shape:
// {
// "type": "object",
// "properties": {
// "$defs": { "$ref": "#/$defs/User" }
// },
// "$defs": { "User": { "type": "string" }, "Unused": { "type": "boolean" } }
// }
//
// Expected normalization behavior:
// - A property named like the `$defs` keyword is treated as a user field
// while traversing `properties`.
// - Refs from that property schema still mark root definitions reachable.
// - Unreferenced root definitions are still pruned.
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"$defs": {"$ref": "#/$defs/User"}
},
"$defs": {
"User": {"type": "string"},
"Unused": {"type": "boolean"}
}
}))
.expect("parse schema");
assert_eq!(
schema,
JsonSchema {
schema_type: Some(JsonSchemaType::Single(JsonSchemaPrimitiveType::Object)),
properties: Some(BTreeMap::from([(
"$defs".to_string(),
JsonSchema {
schema_ref: Some("#/$defs/User".to_string()),
..Default::default()
},
)])),
defs: Some(BTreeMap::from([(
"User".to_string(),
JsonSchema::string(/*description*/ None),
)])),
..Default::default()
}
);
}
#[test]
fn parse_tool_input_schema_collects_refs_from_schema_child_keywords() {
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"items_holder": {
"type": "array",
"items": {"$ref": "#/$defs/Item"}
},
"map_holder": {
"type": "object",
"additionalProperties": {"$ref": "#/$defs/Extra"}
},
"choice": {
"anyOf": [
{"$ref": "#/$defs/Choice"},
{"type": "string"}
]
}
},
"$defs": {
"Choice": {"type": "boolean"},
"Extra": {"type": "number"},
"Item": {"type": "string"},
"Unused": {"type": "null"}
}
}))
.expect("parse schema");
assert_eq!(
serde_json::to_value(schema).expect("serialize schema"),
serde_json::json!({
"type": "object",
"properties": {
"choice": {
"anyOf": [
{"$ref": "#/$defs/Choice"},
{"type": "string"}
]
},
"items_holder": {
"type": "array",
"items": {"$ref": "#/$defs/Item"}
},
"map_holder": {
"type": "object",
"properties": {},
"additionalProperties": {"$ref": "#/$defs/Extra"}
}
},
"$defs": {
"Choice": {"type": "boolean"},
"Extra": {"type": "number"},
"Item": {"type": "string"}
}
})
);
}
#[test]
fn parse_tool_input_schema_handles_cyclic_local_refs() {
// Example schema shape:
// {
// "type": "object",
// "properties": { "node": { "$ref": "#/$defs/Node" } },
// "$defs": {
// "Node": {
// "type": "object",
// "properties": { "next": { "$ref": "#/$defs/Node" } }
// }
// }
// }
//
// Expected normalization behavior:
// - Recursive refs are preserved.
// - Pruning traversal terminates after visiting each local target once.
// - Responses API handles this recursive local-ref shape correctly.
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"node": {"$ref": "#/$defs/Node"}
},
"$defs": {
"Node": {
"type": "object",
"properties": {
"next": {"$ref": "#/$defs/Node"}
}
}
}
}))
.expect("parse schema");
assert_eq!(
schema,
JsonSchema {
schema_type: Some(JsonSchemaType::Single(JsonSchemaPrimitiveType::Object)),
properties: Some(BTreeMap::from([(
"node".to_string(),
JsonSchema {
schema_ref: Some("#/$defs/Node".to_string()),
..Default::default()
},
)])),
defs: Some(BTreeMap::from([(
"Node".to_string(),
JsonSchema::object(
BTreeMap::from([(
"next".to_string(),
JsonSchema {
schema_ref: Some("#/$defs/Node".to_string()),
..Default::default()
},
)]),
/*required*/ None,
/*additional_properties*/ None,
),
)])),
..Default::default()
}
);
}
#[test]
fn parse_tool_input_schema_preserves_legacy_definitions() {
// Example schema shape:
// {
// "type": "object",
// "properties": { "user": { "$ref": "#/definitions/User" } },
// "definitions": {
// "User": { "type": "object", "properties": { "profile": { "$ref": "#/definitions/Profile" } } },
// "Profile": { "type": "object", "properties": { "name": { "type": "string" } } }
// }
// }
//
// Expected normalization behavior:
// - Codex preserves legacy `definitions`.
// - Reachability follows refs through the legacy definition table.
// - Unreachable legacy definition entries are pruned.
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"user": {"$ref": "#/definitions/User"}
},
"definitions": {
"User": {
"type": "object",
"properties": {
"profile": {"$ref": "#/definitions/Profile"}
}
},
"Profile": {
"type": "object",
"properties": {
"name": {"type": "string"}
}
},
"Unused": {"type": "string"}
}
}))
.expect("parse schema");
assert_eq!(
schema,
JsonSchema {
schema_type: Some(JsonSchemaType::Single(JsonSchemaPrimitiveType::Object)),
properties: Some(BTreeMap::from([(
"user".to_string(),
JsonSchema {
schema_ref: Some("#/definitions/User".to_string()),
..Default::default()
},
)])),
definitions: Some(BTreeMap::from([
(
"Profile".to_string(),
JsonSchema::object(
BTreeMap::from([(
"name".to_string(),
JsonSchema::string(/*description*/ None),
)]),
/*required*/ None,
/*additional_properties*/ None,
),
),
(
"User".to_string(),
JsonSchema::object(
BTreeMap::from([(
"profile".to_string(),
JsonSchema {
schema_ref: Some("#/definitions/Profile".to_string()),
..Default::default()
},
)]),
/*required*/ None,
/*additional_properties*/ None,
),
),
])),
..Default::default()
}
);
}
#[test]
fn parse_tool_input_schema_preserves_unresolved_and_external_refs() {
// Example schema shape:
// {
// "type": "object",
// "properties": {
// "missing": { "$ref": "#/$defs/Missing" },
// "remote": { "$ref": "https://example.com/schema.json" }
// },
// "$defs": { "Unused": { "type": "string" } }
// }
//
// Expected normalization behavior:
// - Unresolved local refs and external refs are preserved.
// - Unreachable local definitions are still pruned.
// - Responses API handles these refs correctly during downstream validation.
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"missing": {"$ref": "#/$defs/Missing"},
"remote": {"$ref": "https://example.com/schema.json"}
},
"$defs": {
"Unused": {"type": "string"}
}
}))
.expect("parse schema");
assert_eq!(
schema,
JsonSchema {
schema_type: Some(JsonSchemaType::Single(JsonSchemaPrimitiveType::Object)),
properties: Some(BTreeMap::from([
(
"missing".to_string(),
JsonSchema {
schema_ref: Some("#/$defs/Missing".to_string()),
..Default::default()
},
),
(
"remote".to_string(),
JsonSchema {
schema_ref: Some("https://example.com/schema.json".to_string()),
..Default::default()
},
),
])),
..Default::default()
}
);
}
#[test]
fn parse_tool_input_schema_preserves_nested_defs_ref_parent() {
// Example schema shape:
// {
// "type": "object",
// "properties": { "name": { "$ref": "#/$defs/User/properties/name" } },
// "$defs": {
// "User": { "type": "object", "properties": { "name": { "type": "string" } } },
// "name": { "type": "string" },
// "Unused": { "type": "boolean" }
// }
// }
//
// Expected normalization behavior:
// - The nested JSON Pointer ref remains unchanged.
// - The parent root definition is retained so the local ref does not dangle.
// - Unreferenced root definitions are still pruned.
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"name": {"$ref": "#/$defs/User/properties/name"}
},
"$defs": {
"User": {
"type": "object",
"properties": {
"name": {"type": "string"}
}
},
"name": {"type": "string"},
"Unused": {"type": "boolean"}
}
}))
.expect("parse schema");
assert_eq!(
schema,
JsonSchema {
schema_type: Some(JsonSchemaType::Single(JsonSchemaPrimitiveType::Object)),
properties: Some(BTreeMap::from([(
"name".to_string(),
JsonSchema {
schema_ref: Some("#/$defs/User/properties/name".to_string()),
..Default::default()
},
)])),
defs: Some(BTreeMap::from([(
"User".to_string(),
JsonSchema::object(
BTreeMap::from([(
"name".to_string(),
JsonSchema::string(/*description*/ None),
)]),
/*required*/ None,
/*additional_properties*/ None,
),
)])),
..Default::default()
}
);
}
#[test]
fn parse_tool_input_schema_preserves_percent_encoded_definition_refs() {
// Example schema shape:
// {
// "type": "object",
// "properties": {
// "user": { "$ref": "#/$defs/User%20Name" },
// "profile": { "$ref": "#/%24defs/Profile%7E0Name" }
// },
// "$defs": {
// "User Name": { "type": "string" },
// "Profile~Name": { "type": "string" },
// "Unused": { "type": "boolean" }
// }
// }
//
// Expected normalization behavior:
// - URI fragment percent encoding is decoded before JSON Pointer `~`
// escaping, per RFC 6901 section 6.
// - The original `$ref` strings are preserved, but their definition
// targets are recognized as reachable and retained.
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"user": {"$ref": "#/$defs/User%20Name"},
"profile": {"$ref": "#/%24defs/Profile%7E0Name"}
},
"$defs": {
"User Name": {"type": "string"},
"Profile~Name": {"type": "string"},
"Unused": {"type": "boolean"}
}
}))
.expect("parse schema");
assert_eq!(
schema,
JsonSchema {
schema_type: Some(JsonSchemaType::Single(JsonSchemaPrimitiveType::Object)),
properties: Some(BTreeMap::from([
(
"profile".to_string(),
JsonSchema {
schema_ref: Some("#/%24defs/Profile%7E0Name".to_string()),
..Default::default()
},
),
(
"user".to_string(),
JsonSchema {
schema_ref: Some("#/$defs/User%20Name".to_string()),
..Default::default()
},
),
])),
defs: Some(BTreeMap::from([
(
"Profile~Name".to_string(),
JsonSchema::string(/*description*/ None),
),
(
"User Name".to_string(),
JsonSchema::string(/*description*/ None),
),
])),
..Default::default()
}
);
}
#[test]
fn parse_tool_input_schema_drops_malformed_definition_tables() {
// Example schema shape:
// {
// "type": "object",
// "properties": { "user": { "$ref": "#/$defs/User" } },
// "$defs": ["not", "an", "object"]
// }
//
// Expected normalization behavior:
// - Malformed `$defs` tables are dropped instead of rejecting the schema.
// - The unresolved local ref remains visible to the model.
let schema = parse_tool_input_schema(&serde_json::json!({
"type": "object",
"properties": {
"user": {"$ref": "#/$defs/User"}
},
"$defs": ["not", "an", "object"]
}))
.expect("parse schema");
assert_eq!(
schema,
JsonSchema {
schema_type: Some(JsonSchemaType::Single(JsonSchemaPrimitiveType::Object)),
properties: Some(BTreeMap::from([(
"user".to_string(),
JsonSchema {
schema_ref: Some("#/$defs/User".to_string()),
..Default::default()
},
)])),
..Default::default()
}
);
}

View File

@@ -57,6 +57,7 @@ pub use responses_api::dynamic_tool_to_responses_api_tool;
pub use responses_api::mcp_tool_to_deferred_responses_api_tool;
pub use responses_api::mcp_tool_to_responses_api_tool;
pub use responses_api::tool_definition_to_responses_api_tool;
pub use tool_call::ConversationHistory;
pub use tool_call::ToolCall;
pub use tool_config::ShellCommandBackendConfig;
pub use tool_config::ToolEnvironmentMode;

View File

@@ -1,7 +1,27 @@
use crate::FunctionCallError;
use crate::ToolName;
use crate::ToolPayload;
use codex_protocol::models::ResponseItem;
use codex_utils_output_truncation::TruncationPolicy;
use std::sync::Arc;
/// Raw response history snapshot available when an extension tool is invoked.
#[derive(Clone, Debug, Default)]
pub struct ConversationHistory {
items: Arc<[ResponseItem]>,
}
impl ConversationHistory {
pub fn new(items: Vec<ResponseItem>) -> Self {
Self {
items: items.into(),
}
}
pub fn items(&self) -> &[ResponseItem] {
&self.items
}
}
// TODO: this is temporary and will disappear in the next PR (as we make codex-extension-api generic on Invocation.
#[derive(Clone, Debug)]
@@ -10,6 +30,7 @@ pub struct ToolCall {
pub call_id: String,
pub tool_name: ToolName,
pub truncation_policy: TruncationPolicy,
pub conversation_history: ConversationHistory,
pub payload: ToolPayload,
}

View File

@@ -910,7 +910,8 @@ impl App {
return;
}
self.runtime_permission_profile_override = Some(permission_profile);
self.runtime_permission_profile_override =
Some(RuntimePermissionProfileOverride::from_config(&self.config));
self.sync_active_thread_permission_settings_to_cached_session()
.await;