mirror of
https://github.com/openai/codex.git
synced 2026-05-14 08:12:36 +00:00
xli-codex/generate-python-sdk
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Channing Conger
|
36460387ec |
Enable V8 sandboxing for source-built builds (#21146)
## Summary
This is the first PR in the V8 in-process sandboxing rollout.
It adds the build-system and Rust feature plumbing needed to support
sandboxed V8 builds, then enables sandboxing by default for the
source-built Bazel V8 path that we control directly. It deliberately
keeps the published `rusty_v8` artifact workflows on their current
non-sandboxed contract so this PR can land and ship independently before
we change any released artifacts.
## Rollout plan
- [x] **PR 1: land sandbox plumbing and default source-built Bazel V8 to
sandboxed mode**
- [ ] **PR 2: publish sandbox-enabled release artifacts and add
compatibility validation**
- Produce sandboxed artifact pairs for every released Cargo target that
does not already use the source-built Bazel path.
- Add CI coverage that consumes those sandboxed artifacts and verifies:
- `codex-v8-poc` reports sandbox enabled
- `codex-code-mode` builds/tests against the sandboxed path
- [ ] **PR 3: switch release consumers to sandboxed artifacts by
default**
- Update released artifact selectors/checksums.
- Enable the Rust `v8_enable_sandbox` feature in the default release
path.
- Make the sandboxed artifact family the normal path for published
builds.
- [ ] **PR 4: remove rollout-only compatibility paths**
- Remove the temporary non-sandbox release compatibility config once the
new default has shipped and baked.
- Keep the invariant tests permanently.
|
||
|
Michael Bolin
|
466798aa83 |
ci: cross-compile Windows Bazel tests (#20585)
## Status This is the Bazel PR-CI cross-compilation follow-up to #20485. It is intentionally split from the Cargo/cargo-xwin release-build PoC so #20485 can stay as the historical release-build exploration. The unrelated async-utils test cleanup has been moved to #20686, so this PR is focused on the Windows Bazel CI path. The intended tradeoff is now explicit in `.github/workflows/bazel.yml`: pull requests get the fast Windows cross-compiled Bazel test leg, while post-merge pushes to `main` run both that fast cross leg and a fully native Windows Bazel test leg. The native main-only job keeps full V8/code-mode coverage and gets a 40-minute timeout because it is less latency-sensitive than PR CI. All other Bazel jobs remain at 30 minutes. ## Why Windows Bazel PR CI currently does the expensive part of the build on Windows. A native Windows Bazel test job on `main` completed in about 28m12s, leaving very little headroom under the 30-minute job timeout and making Windows the slowest PR signal. #20485 showed that Windows cross-compilation can be materially faster for Cargo release builds, but PR CI needs Bazel because Bazel owns our test sharding, flaky-test retries, and integration-test layout. This PR applies the same high-level shape we already use for macOS Bazel CI: compile with remote Linux execution, then run platform-specific tests on the platform runner. The compromise is deliberately signal-aware: code-mode/V8 changes are rare enough that PR CI can accept losing the direct V8/code-mode smoke-test signal temporarily, while `main` still runs the native Windows job post-merge to catch that class of regression. A follow-up PR should investigate making the cross-built Windows gnullvm V8 archive pass the direct V8/code-mode tests so this tradeoff can eventually go away. ## What Changed - Adds a `ci-windows-cross` Bazel config that targets `x86_64-pc-windows-gnullvm`, uses Linux RBE for build actions, and keeps `TestRunner` actions local on the Windows runner. - Adds explicit Windows platform definitions for `windows_x86_64_gnullvm`, `windows_x86_64_msvc`, and a bridge toolchain that lets gnullvm test targets execute under the Windows MSVC host platform. - Updates the Windows Bazel PR test leg to opt into the cross-compile path via `--windows-cross-compile` and `--remote-download-toplevel`. - Adds a `test-windows-native-main` job that runs only for `push` events on `refs/heads/main`, uses the native Windows Bazel path, includes V8/code-mode smoke tests, and has `timeout-minutes: 40`. - Keeps fork/community PRs without `BUILDBUDDY_API_KEY` on the previous local Windows MSVC-host fallback, including `--host_platform=//:local_windows_msvc` and `--jobs=8`. - Preserves the existing integration-test shape on non-gnullvm platforms, while generating Windows-cross wrapper targets only for `windows_gnullvm`. - Resolves `CARGO_BIN_EXE_*` values from runfiles at test runtime, avoiding hard-coded Cargo paths and duplicate test runfiles. - Extends the V8 Bazel patches enough for the `x86_64-pc-windows-gnullvm` target and Linux remote execution path. - Makes the Windows sandbox test cwd derive from `INSTA_WORKSPACE_ROOT` at runtime when Bazel provides it, because cross-compiled binaries may contain Linux compile-time paths. - Keeps the direct V8/code-mode unit smoke tests out of the Windows cross PR path for now while native Windows CI continues to cover them post-merge. ## Command Shape The fast Windows PR test leg invokes the normal Bazel CI wrapper like this: ```shell ./.github/scripts/run-bazel-ci.sh \ --print-failed-action-summary \ --print-failed-test-logs \ --windows-cross-compile \ --remote-download-toplevel \ -- \ test \ --test_tag_filters=-argument-comment-lint \ --test_verbose_timeout_warnings \ --build_metadata=COMMIT_SHA=${GITHUB_SHA} \ -- \ //... \ -//third_party/v8:all \ -//codex-rs/code-mode:code-mode-unit-tests \ -//codex-rs/v8-poc:v8-poc-unit-tests ``` With the BuildBuddy secret available on Windows, the wrapper selects `--config=ci-windows-cross` and appends the important Windows-cross overrides after rc expansion: ```shell --host_platform=//:rbe --shell_executable=/bin/bash --action_env=PATH=/usr/bin:/bin --host_action_env=PATH=/usr/bin:/bin --test_env=PATH=${CODEX_BAZEL_WINDOWS_PATH} ``` The native post-merge Windows job intentionally omits `--windows-cross-compile` and does not exclude the V8/code-mode unit targets: ```shell ./.github/scripts/run-bazel-ci.sh \ --print-failed-action-summary \ --print-failed-test-logs \ -- \ test \ --test_tag_filters=-argument-comment-lint \ --test_verbose_timeout_warnings \ --build_metadata=COMMIT_SHA=${GITHUB_SHA} \ --build_metadata=TAG_windows_native_main=true \ -- \ //... \ -//third_party/v8:all ``` ## Research Notes The existing macOS Bazel CI config already uses the model we want here: build actions run remotely with `--strategy=remote`, but `TestRunner` actions execute on the macOS runner. This PR mirrors that pattern for Windows with `--strategy=TestRunner=local`. The important Bazel detail is that `rules_rs` is already targeting `x86_64-pc-windows-gnullvm` for Windows Bazel PR tests. This PR changes where the build actions execute; it does not switch the Bazel PR test target to Cargo, `cargo-nextest`, or the MSVC release target. Cargo release builds differ from this Bazel path for V8: the normal Windows Cargo release target is MSVC, and `rusty_v8` publishes prebuilt Windows MSVC `.lib.gz` archives. The Bazel PR path targets `windows-gnullvm`; `rusty_v8` does not publish a prebuilt Windows GNU/gnullvm archive, so this PR builds that archive in-tree. That Linux-RBE-built gnullvm archive currently crashes in direct V8/code-mode smoke tests, which is why the workflow keeps native Windows coverage on `main`. The less obvious Bazel detail is test wrapper selection. Bazel chooses the Windows test wrapper (`tw.exe`) from the test action execution platform, not merely from the Rust target triple. The outer `workspace_root_test` therefore declares the default test toolchain and uses the bridge toolchain above so the test action executes on Windows while its inner Rust binary is built for gnullvm. The V8 investigation exposed a Windows-client gotcha: even when an action execution platform is Linux RBE, Bazel can still derive the genrule shell path from the Windows client. That produced remote commands trying to run `C:\Program Files\Git\usr\bin\bash.exe` on Linux workers. The wrapper now passes `--shell_executable=/bin/bash` with `--host_platform=//:rbe` for the Windows cross path. The same Windows-client/Linux-RBE boundary also affected `third_party/v8:binding_cc`: a multiline genrule command can carry CRLF line endings into Linux remote bash, which failed as `$'\r'`. That genrule now keeps the `sed` command on one physical shell line while using an explicit Starlark join so the shell arguments stay readable. ## Verification Local checks included: ```shell bash -n .github/scripts/run-bazel-ci.sh bash -n workspace_root_test_launcher.sh.tpl ruby -e "require %q{yaml}; YAML.load_file(%q{.github/workflows/bazel.yml}); puts %q{ok}" RUNNER_OS=Linux ./scripts/list-bazel-clippy-targets.sh RUNNER_OS=Windows ./scripts/list-bazel-clippy-targets.sh RUNNER_OS=Linux ./tools/argument-comment-lint/list-bazel-targets.sh RUNNER_OS=Windows ./tools/argument-comment-lint/list-bazel-targets.sh ``` The Linux clippy and argument-comment target lists contain zero `*-windows-cross-bin` labels, while the Windows lists still include 47 Windows-cross internal test binaries. CI evidence: - Baseline native Windows Bazel test on `main`: success in about 28m12s, https://github.com/openai/codex/actions/runs/25206257208/job/73907325959 - Green Windows-cross Bazel run on the split PR before adding the main-only native leg: Windows test 9m16s, Windows release verify 5m10s, Windows clippy 4m43s, https://github.com/openai/codex/actions/runs/25231890068 - The latest SHA adds the explicit PR-vs-main tradeoff in `bazel.yml`; CI is rerunning on that focused diff. ## Follow-Up A subsequent PR should investigate making a cross-built Windows binary work with V8/code-mode enabled. Likely options are either making the Linux-RBE-built `windows-gnullvm` V8 archive correct at runtime, or evaluating whether a Bazel MSVC target/toolchain can reuse the same prebuilt MSVC `rusty_v8` archive shape that Cargo release builds already use. |
||
|
viyatb-oai
|
d9a385ac8c |
fix: pin inputs (#17471)
## Summary - Pin Rust git patch dependencies to immutable revisions and make cargo-deny reject unknown git and registry sources unless explicitly allowlisted. - Add checked-in SHA-256 coverage for the current rusty_v8 release assets, wire those hashes into Bazel, and verify CI override downloads before use. - Add rusty_v8 MODULE.bazel update/check tooling plus a Bazel CI guard so future V8 bumps cannot drift from the checked-in checksum manifest. - Pin release/lint cargo installs and all external GitHub Actions refs to immutable inputs. ## Future V8 bump flow Run these after updating the resolved `v8` crate version and checksum manifest: ```bash python3 .github/scripts/rusty_v8_bazel.py update-module-bazel python3 .github/scripts/rusty_v8_bazel.py check-module-bazel ``` The update command rewrites the matching `rusty_v8_<crate_version>` `http_file` SHA-256 values in `MODULE.bazel` from `third_party/v8/rusty_v8_<crate_version>.sha256`. The check command is also wired into Bazel CI to block drift. ## Notes - This intentionally excludes RustSec dependency upgrades and bubblewrap-related changes per request. - The branch was rebased onto the latest origin/main before opening the PR. ## Validation - cargo fetch --locked - cargo deny check advisories - cargo deny check - cargo deny check sources - python3 .github/scripts/rusty_v8_bazel.py check-module-bazel - python3 .github/scripts/rusty_v8_bazel.py update-module-bazel - python3 -m unittest discover -s .github/scripts -p 'test_rusty_v8_bazel.py' - python3 -m py_compile .github/scripts/rusty_v8_bazel.py .github/scripts/rusty_v8_module_bazel.py .github/scripts/test_rusty_v8_bazel.py - repo-wide GitHub Actions `uses:` audit: all external action refs are pinned to 40-character SHAs - yq eval on touched workflows and local actions - git diff --check - just bazel-lock-check ## Hash verification - Confirmed `MODULE.bazel` hashes match `third_party/v8/rusty_v8_146_4_0.sha256`. - Confirmed GitHub release asset digests for denoland/rusty_v8 `v146.4.0` and openai/codex `rusty-v8-v146.4.0` match the checked-in hashes. - Streamed and SHA-256 hashed all 10 `MODULE.bazel` rusty_v8 asset URLs locally; every downloaded byte stream matched both `MODULE.bazel` and the checked-in manifest. ## Pin verification - Confirmed signing-action pins match the peeled commits for their tag comments: `sigstore/cosign-installer@v3.7.0`, `azure/login@v2`, and `azure/trusted-signing-action@v0`. - Pinned the remaining tag-based action refs in Bazel CI/setup: `actions/setup-node@v6`, `facebook/install-dotslash@v2`, `bazelbuild/setup-bazelisk@v3`, and `actions/cache/restore@v5`. - Normalized all `bazelbuild/setup-bazelisk@v3` refs to the peeled commit behind the annotated tag. - Audited Cargo git dependencies: every manifest git dependency uses `rev` only, every `Cargo.lock` git source has `?rev=<sha>#<same-sha>`, and `cargo deny check sources` passes with `required-git-spec = "rev"`. - Shallow-fetched each distinct git dependency repo at its pinned SHA and verified Git reports each object as a commit. |
||
|
Michael Bolin
|
343d1af3da |
bazel: enable the full Windows gnullvm CI path (#15952)
## Why This PR is the current, consolidated follow-up to the earlier Windows Bazel attempt in #11229. The goal is no longer just to get a tiny Windows smoke job limping along: it is to make the ordinary Bazel CI path usable on `windows-latest` for `x86_64-pc-windows-gnullvm`, with the same broad `//...` test shape that macOS and Linux already use. The earlier smoke-list version of this work was useful as a foothold, but it was not a good long-term landing point. Windows Bazel kept surfacing real issues outside that allowlist: - GitHub's Windows runner exposed runfiles-manifest bugs such as `FINDSTR: Cannot open D:MANIFEST`, which broke Bazel test launchers even when the manifest file existed. - `rules_rs`, `rules_rust`, LLVM extraction, and Abseil still needed `windows-gnullvm`-specific fixes for our hermetic toolchain. - the V8 path needed more work than just turning the Windows matrix entry back on: `rusty_v8` does not ship Windows GNU artifacts in the same shape we need, and Bazel's in-tree V8 build needed a set of Windows GNU portability fixes. Windows performance pressure also pushed this toward a full solution instead of a permanent smoke suite. During this investigation we hit targets such as `//codex-rs/shell-command:shell-command-unit-tests` that were much more expensive on Windows because they repeatedly spawn real PowerShell parsers (see #16057 for one concrete example of that pressure). That made it much more valuable to get the real Windows Bazel path working than to keep iterating on a narrowly curated subset. The net result is that this PR now aims for the same CI contract on Windows that we already expect elsewhere: keep standalone `//third_party/v8:all` out of the ordinary Bazel lane, but allow V8 consumers under `//codex-rs/...` to build and test transitively through `//...`. ## What Changed ### CI and workflow wiring - re-enable the `windows-latest` / `x86_64-pc-windows-gnullvm` Bazel matrix entry in `.github/workflows/bazel.yml` - move the Windows Bazel output root to `D:\b` and enable `git config --global core.longpaths true` in `.github/actions/setup-bazel-ci/action.yml` - keep the ordinary Bazel target set on Windows aligned with macOS and Linux by running `//...` while excluding only standalone `//third_party/v8:all` targets from the normal lane ### Toolchain and module support for `windows-gnullvm` - patch `rules_rs` so `windows-gnullvm` is modeled as a distinct Windows exec/toolchain platform instead of collapsing into the generic Windows shape - patch `rules_rust` build-script environment handling so llvm-mingw build-script probes do not inherit unsupported `-fstack-protector*` flags - patch the LLVM module archive so it extracts cleanly on Windows and provides the MinGW libraries this toolchain needs - patch Abseil so its thread-local identity path matches the hermetic `windows-gnullvm` toolchain instead of taking an incompatible MinGW pthread path - keep both MSVC and GNU Windows targets in the generated Cargo metadata because the current V8 release-asset story still uses MSVC-shaped names in some places while the Bazel build targets the GNU ABI ### Windows test-launch and binary-behavior fixes - update `workspace_root_test_launcher.bat.tpl` to read the runfiles manifest directly instead of shelling out to `findstr`, which was the source of the `D:MANIFEST` failures on the GitHub Windows runner - thread a larger Windows GNU stack reserve through `defs.bzl` so Bazel-built binaries that pull in V8 behave correctly both under normal builds and under `bazel test` - remove the no-longer-needed Windows bootstrap sh-toolchain override from `.bazelrc` ### V8 / `rusty_v8` Windows GNU support - export and apply the new Windows GNU patch set from `patches/BUILD.bazel` / `MODULE.bazel` - patch the V8 module/rules/source layers so the in-tree V8 build can produce Windows GNU archives under Bazel - teach `third_party/v8/BUILD.bazel` to build Windows GNU static archives in-tree instead of aliasing them to the MSVC prebuilts - reuse the Linux release binding for the experimental Windows GNU path where `rusty_v8` does not currently publish a Windows GNU binding artifact ## Testing - the primary end-to-end validation for this work is the `Bazel` workflow plus `v8-canary`, since the hard parts are Windows-specific and depend on real GitHub runner behavior - before consolidation back onto this PR, the same net change passed the full Bazel matrix in [run 23675590471](https://github.com/openai/codex/actions/runs/23675590471) and passed `v8-canary` in [run 23675590453](https://github.com/openai/codex/actions/runs/23675590453) - those successful runs included the `windows-latest` / `x86_64-pc-windows-gnullvm` Bazel job with the ordinary `//...` path, not the earlier Windows smoke allowlist --- [//]: # (BEGIN SAPLING FOOTER) Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/openai/codex/pull/15952). * #16067 * __->__ #15952 |
||
|
Channing Conger
|
1350477150 |
Add v8-poc consumer of our new built v8 (#15203)
This adds a dummy v8-poc project that in Cargo links against our prebuilt binaries and the ones provided by rusty_v8 for non musl platforms. This demonstrates that we can successfully link and use v8 on all platforms that we want to target. In bazel things are slightly more complicated. Since the libraries as published have libc++ linked in already we end up with a lot of double linked symbols if we try to use them in bazel land. Instead we fall back to building rusty_v8 and v8 from source (cached of course) on the platforms we ship to. There is likely some compatibility drift in the windows bazel builder that we'll need to reconcile before we can re-enable them. I'm happy to be on the hook to unwind that. |
||
|
Channing Conger
|
ded7854f09 |
V8 Bazel Build (#15021)
Alternative approach, we use rusty_v8 for all platforms that its predefined, but lets build from source a musl v8 version with bazel for x86 and aarch64 only. We would need to release this on github and then use the release. |