Load requirements from Codex Backend. It only does this for enterprise
customers signed in with ChatGPT.
Todo in follow-up PRs:
* Add to app-server and exec too
* Switch from fail-open to fail-closed on failure
In a [recent PR](https://github.com/openai/codex/pull/9182), I made some
improvements to config error messages so errors didn't leave app server
clients in a dead state. This is a follow-on PR to make these error
messages more readable and actionable for both TUI and GUI users. For
example, see #9668 where the user was understandably confused about the
source of the problem and how to fix it.
The improved error message:
1. Clearly identifies the config file where the error was found (which
is more important now that we support layered configs)
2. Provides a line and column number of the error
3. Displays the line where the error occurred and underlines it
For example, if my `config.toml` includes the following:
```toml
[features]
collaboration_modes = "true"
```
Here's the current CLI error message:
```
Error loading config.toml: invalid type: string "true", expected a boolean in `features`
```
And here's the improved message:
```
Error loading config.toml:
/Users/etraut/.codex/config.toml:43:23: invalid type: string "true", expected a boolean
|
43 | collaboration_modes = "true"
| ^^^^^^
```
The bulk of the new logic is contained within a new module
`config_loader/diagnostics.rs` that is responsible for calculating the
text range for a given toml path (which is more involved than I would
have expected).
In addition, this PR adds the file name and text range to the
`ConfigWarningNotification` app server struct. This allows GUI clients
to present the user with a better error message and an optional link to
open the errant config file. This was a suggestion from @.bolinfest when
he reviewed my previous PR.
## Summary
#9555 is the start of a rename, so I'm starting to standardize here.
Sets up `model_instructions` templating with a strongly-typed object for
injecting a personality block into the model instructions.
## Testing
- [x] Added tests
- [x] Ran locally
Config includes multiple code execution entrypoints.
Now, we load the config from predetermined locations first
(~/.codex/config.toml etc), use those to learn which folders are
'trusted', and only load additional config from the CWD if it is
trusted.
**Before:**
```
Error loading configuration: value `Never` is not in the allowed set [OnRequest]
```
**After:**
```
Error loading configuration: invalid value for `approval_policy`: `Never` is not in the
allowed set [OnRequest] (set by MDM com.openai.codex:requirements_toml_base64)
```
Done by introducing a new struct `ConfigRequirementsWithSources` onto
which we `merge_unset_fields` now. Also introduces a pair of requirement
value and its `RequirementSource` (inspired by `ConfigLayerSource`):
```rust
pub struct Sourced<T> {
pub value: T,
pub source: RequirementSource,
}
```
Load managed requirements from MDM key `requirements_toml_base64`.
Tested on my Mac (using `defaults` to set the preference, though this
would be set by MDM in production):
```
➜ codex git:(gt/mdm-requirements) defaults read com.openai.codex requirements_toml_base64 | base64 -d
allowed_approval_policies = ["on-request"]
➜ codex git:(gt/mdm-requirements) just c --yolo
cargo run --bin codex -- "$@"
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.26s
Running `target/debug/codex --yolo`
Error loading configuration: value `Never` is not in the allowed set [OnRequest]
error: Recipe `codex` failed on line 11 with exit code 1
➜ codex git:(gt/mdm-requirements) defaults delete com.openai.codex requirements_toml_base64
➜ codex git:(gt/mdm-requirements) just c --yolo
cargo run --bin codex -- "$@"
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.24s
Running `target/debug/codex --yolo`
╭──────────────────────────────────────────────────────────╮
│ >_ OpenAI Codex (v0.0.0) │
│ │
│ model: codex-auto-balanced medium /model to change │
│ directory: ~/code/codex/codex-rs │
╰──────────────────────────────────────────────────────────╯
Tip: Start a fresh idea with /new; the previous session stays in history.
```
This adds logic to load `/etc/codex/config.toml` and associate it with
`ConfigLayerSource::System` on UNIX. I refactored the code so it shares
logic with the creation of the `ConfigLayerSource::User` layer.
This is necessary so that `$CODEX_HOME/skills` and `$CODEX_HOME/rules`
still get loaded even if `$CODEX_HOME/config.toml` does not exist. See
#8453.
For now, it is possible to omit this layer when creating a dummy
`ConfigLayerStack` in a test. We can revisit that later, if it turns out
to be the right thing to do.
- allow configuring `project_root_markers` in `config.toml`
(user/system/MDM) to control project discovery beyond `.git`
- honor the markers after merging pre-project layers; default to
`[".git"]` when unset and skip ancestor walk when set to an empty array
- document the option and add coverage for alternate markers in config
loader tests
- We now support `.codex/config.toml` in repo (from `cwd` up to the
first `.git` found, if any) as layers in `ConfigLayerStack`. A new
`ConfigLayerSource::Project` variant was added to support this.
- In doing this work, I realized that we were resolving relative paths
in `config.toml` after merging everything into one `toml::Value`, which
is wrong: paths should be relativized with respect to the folder
containing the `config.toml` that was deserialized. This PR introduces a
deserialize/re-serialize strategy to account for this in
`resolve_config_paths()`. (This is why `Serialize` is added to so many
types as part of this PR.)
- Added tests to verify this new behavior.
---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with [ReviewStack](https://reviewstack.dev/openai/codex/pull/8354).
* #8359
* __->__ #8354
`load_config_layers_state()` should load config from a
`.codex/config.toml` in any folder between the `cwd` for a thread and
the project root. Though in order to do that,
`load_config_layers_state()` needs to know what the `cwd` is, so this PR
does the work to thread the `cwd` through for existing callsites.
A notable exception is the `/config` endpoint in app server for which a
`cwd` is not guaranteed to be associated with the query, so the `cwd`
param is `Option<AbsolutePathBuf>` to account for this case.
The logic to make use of the `cwd` will be done in a follow-up PR.
This adds support for `allowed_sandbox_modes` in `requirements.toml` and
provides legacy support for constraining sandbox modes in
`managed_config.toml`. This is converted to `Constrained<SandboxPolicy>`
in `ConfigRequirements` and applied to `Config` such that constraints
are enforced throughout the harness.
Note that, because `managed_config.toml` is deprecated, we do not add
support for the new `external-sandbox` variant recently introduced in
https://github.com/openai/codex/pull/8290. As noted, that variant is not
supported in `config.toml` today, but can be configured programmatically
via app server.
This implements the new config design where config _requirements_ are
loaded separately (and with a special schema) as compared to config
_settings_. In particular, on UNIX, with this PR, you could define
`/etc/codex/requirements.toml` with:
```toml
allowed_approval_policies = ["never", "on-request"]
```
to enforce that `Config.approval_policy` must be one of those two values
when Codex runs.
We plan to expand the set of things that can be restricted by
`/etc/codex/requirements.toml` in short order.
Note that requirements can come from several sources:
- new MDM key on macOS (not implemented yet)
- `/etc/codex/requirements.toml`
- re-interpretation of legacy MDM key on macOS
(`com.openai.codex/config_toml_base64`)
- re-interpretation of legacy `/etc/codex/managed_config.toml`
So our resolution strategy is to load TOML data from those sources, in
order. Later TOMLs are "merged" into previous TOMLs, but any field that
is already set cannot be overwritten. See
`ConfigRequirementsToml::merge_unset_fields()`.
This is a significant change to how layers of configuration are applied.
In particular, the `ConfigLayerStack` now has two important fields:
- `layers: Vec<ConfigLayerEntry>`
- `requirements: ConfigRequirements`
We merge `TomlValue`s across the layers, but they are subject to
`ConfigRequirements` before creating a `Config`.
How I would review this PR:
- start with `codex-rs/app-server-protocol/src/protocol/v2.rs` and note
the new variants added to the `ConfigLayerSource` enum:
`LegacyManagedConfigTomlFromFile` and `LegacyManagedConfigTomlFromMdm`
- note that `ConfigLayerSource` now has a `precedence()` method and
implements `PartialOrd`
- `codex-rs/core/src/config_loader/layer_io.rs` is responsible for
loading "admin" preferences from `/etc/codex/managed_config.toml` and
MDM. Because `/etc/codex/managed_config.toml` is now deprecated in favor
of `/etc/codex/requirements.toml` and `/etc/codex/config.toml`, we now
include some extra information on the `LoadedConfigLayers` returned in
`layer_io.rs`.
- `codex-rs/core/src/config_loader/mod.rs` has major changes to
`load_config_layers_state()`, which is what produces `ConfigLayerStack`.
The docstring has the new specification and describes the various layers
that will be loaded and the precedence order.
- It uses the information from `LoaderOverrides` "twice," both in the
spirit of legacy support:
- We use one instances to derive an instance of `ConfigRequirements`.
Currently, the only field in `managed_config.toml` that contributes to
`ConfigRequirements` is `approval_policy`. This PR introduces
`Constrained::allow_only()` to support this.
- We use a clone of `LoaderOverrides` to derive
`ConfigLayerSource::LegacyManagedConfigTomlFromFile` and
`ConfigLayerSource::LegacyManagedConfigTomlFromMdm` layers, as
appropriate. As before, this ends up being a "best effort" at enterprise
controls, but is enforcement is not guaranteed like it is for
`ConfigRequirements`.
- Now we only create a "user" layer if `$CODEX_HOME/config.toml` exists.
(Previously, a user layer was always created for `ConfigLayerStack`.)
- Similarly, we only add a "session flags" layer if there are CLI
overrides.
- `config_loader/state.rs` contains the updated implementation for
`ConfigLayerStack`. Note the public API is largely the same as before,
but the implementation is quite different. We leverage the fact that
`ConfigLayerSource` is now `PartialOrd` to ensure layers are in the
correct order.
- A `Config` constructed via `ConfigBuilder.build()` will use
`load_config_layers_state()` to create the `ConfigLayerStack` and use
the associated `ConfigRequirements` when constructing the `Config`
object.
- That said, a `Config` constructed via
`Config::load_from_base_config_with_overrides()` does _not_ yet use
`ConfigBuilder`, so it creates a `ConfigRequirements::default()` instead
of loading a proper `ConfigRequirements`. I will fix this in a
subsequent PR.
Then the following files are mostly test changes:
```
codex-rs/app-server/tests/suite/v2/config_rpc.rs
codex-rs/core/src/config/service.rs
codex-rs/core/src/config_loader/tests.rs
```
Again, because we do not always include "user" and "session flags"
layers when the contents are empty, `ConfigLayerStack` sometimes has
fewer layers than before (and the precedence order changed slightly),
which is the main reason integration tests changed.
