mirror of
https://github.com/openai/codex.git
synced 2026-05-20 03:05:02 +00:00
7cdeab33d1
## Summary Move the rusty_v8 artifact production into hermetic Bazel path and bump the `v8` crate to `147.4.0` The new flow builds V8 release artifacts from source for Darwin and Linux targets, publishes both the current release-compatible artifacts and sandbox-enabled variants, and keeps Cargo consumers on prebuilt binaries by continuing to feed the `v8` crate the archive and generated binding files it already expects. ## Why We need control over V8 build-time features without giving up prebuilt artifacts for downstream Cargo builds. Upstream `rusty_v8` already supports source-only features such as `v8_enable_sandbox`, but its normal prebuilt release assets do not cover every feature combination we need. Building the artifacts ourselves lets us enable settings such as the V8 sandbox and pointer compression at artifact build time, then publish those outputs so ordinary Cargo builds can still consume prebuilts instead of compiling V8 locally. This keeps the fast consumer experience of prebuilt `rusty_v8` archives while giving us a reproducible path to ship featureful variants that upstream does not currently publish for us. ## Implementation Notes The Bazel graph in this PR is not copied wholesale from `rusty_v8`; `rusty_v8`'s normal source build is still GN/Ninja-based. Instead, this change starts from upstream V8's Bazel rules and adapts them to Codex's hermetic toolchains and dependency layout. Where we intentionally follow `rusty_v8`, we mirror its existing artifact contract: - the same `v8` crate version and generated binding expectations - the same sandbox feature relationship, where sandboxing requires pointer compression - the same custom libc++ model expected by Cargo's default `use_custom_libcxx` feature - the same release-style archive plus `src_binding` outputs consumed by the `v8` crate To preserve that contract, the Bazel release path pins the libc++, libc++abi, and llvm-libc revisions used by `rusty_v8 v147.4.0`, builds release artifacts with `--config=rusty-v8-upstream-libcxx`, and folds the matching runtime objects into the final static archive. ## Windows Windows is annoyingly handled differently. Codex's current hermetic Bazel Windows C++ platform is `windows-gnullvm` / `x86_64-w64-windows-gnu`, while upstream `rusty_v8` publishes Windows prebuilts for `*-pc-windows-msvc`. Those are different ABIs, so the Bazel graph cannot truthfully reproduce the upstream MSVC artifacts until we add a real MSVC-targeting C++ toolchain. For now: - Windows MSVC consumers continue to use upstream `rusty_v8` release archives. - Windows GNU targets are built in-tree so they link against a matching GNU ABI. - The canary workflow separately exercises upstream `rusty_v8` source builds for MSVC sandbox artifacts, but MSVC is not yet part of the Bazel-produced release matrix. ## Validation This PR is technically self validating through CI. I have already published it as a release tag so the artifacts from this branch are published to https://github.com/openai/codex/releases/tag/rusty-v8-v147.4.0 CI for this PR should therefore consume our own release targets. I have also locally tested for linux and darwin. --------- Co-authored-by: Codex <noreply@openai.com>
412 lines
14 KiB
YAML
412 lines
14 KiB
YAML
name: v8-canary
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- ".bazelrc"
|
|
- ".github/actions/setup-bazel-ci/**"
|
|
- ".github/scripts/rusty_v8_bazel.py"
|
|
- ".github/scripts/rusty_v8_module_bazel.py"
|
|
- ".github/workflows/rusty-v8-release.yml"
|
|
- ".github/workflows/v8-canary.yml"
|
|
- "MODULE.bazel"
|
|
- "MODULE.bazel.lock"
|
|
- "codex-rs/Cargo.toml"
|
|
- "patches/BUILD.bazel"
|
|
- "patches/llvm_*.patch"
|
|
- "patches/rules_cc_*.patch"
|
|
- "patches/v8_*.patch"
|
|
- "third_party/v8/**"
|
|
push:
|
|
branches:
|
|
- main
|
|
paths:
|
|
- ".bazelrc"
|
|
- ".github/actions/setup-bazel-ci/**"
|
|
- ".github/scripts/rusty_v8_bazel.py"
|
|
- ".github/scripts/rusty_v8_module_bazel.py"
|
|
- ".github/workflows/rusty-v8-release.yml"
|
|
- ".github/workflows/v8-canary.yml"
|
|
- "MODULE.bazel"
|
|
- "MODULE.bazel.lock"
|
|
- "codex-rs/Cargo.toml"
|
|
- "patches/BUILD.bazel"
|
|
- "patches/llvm_*.patch"
|
|
- "patches/rules_cc_*.patch"
|
|
- "patches/v8_*.patch"
|
|
- "third_party/v8/**"
|
|
workflow_dispatch:
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}::${{ github.event.pull_request.number > 0 && format('pr-{0}', github.event.pull_request.number) || github.ref_name }}
|
|
cancel-in-progress: ${{ github.ref_name != 'main' }}
|
|
|
|
jobs:
|
|
metadata:
|
|
runs-on: ubuntu-latest
|
|
outputs:
|
|
v8_version: ${{ steps.v8_version.outputs.version }}
|
|
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
persist-credentials: false
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
|
|
with:
|
|
python-version: "3.12"
|
|
|
|
- name: Resolve exact v8 crate version
|
|
id: v8_version
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
version="$(python3 .github/scripts/rusty_v8_bazel.py resolved-v8-crate-version)"
|
|
echo "version=${version}" >> "$GITHUB_OUTPUT"
|
|
|
|
build:
|
|
name: Build ${{ matrix.variant }} ${{ matrix.target }}
|
|
needs: metadata
|
|
runs-on: ${{ matrix.runner }}
|
|
permissions:
|
|
contents: read
|
|
actions: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- runner: ubuntu-24.04
|
|
bazel_config: ci-v8
|
|
platform: linux_amd64
|
|
sandbox: false
|
|
target: x86_64-unknown-linux-gnu
|
|
variant: release
|
|
- runner: ubuntu-24.04
|
|
bazel_config: ci-v8
|
|
platform: linux_amd64
|
|
sandbox: true
|
|
target: x86_64-unknown-linux-gnu
|
|
variant: ptrcomp-sandbox
|
|
- runner: ubuntu-24.04-arm
|
|
bazel_config: ci-v8
|
|
platform: linux_arm64
|
|
sandbox: false
|
|
target: aarch64-unknown-linux-gnu
|
|
variant: release
|
|
- runner: ubuntu-24.04-arm
|
|
bazel_config: ci-v8
|
|
platform: linux_arm64
|
|
sandbox: true
|
|
target: aarch64-unknown-linux-gnu
|
|
variant: ptrcomp-sandbox
|
|
- runner: macos-15-xlarge
|
|
bazel_config: ci-macos
|
|
platform: macos_amd64
|
|
sandbox: false
|
|
target: x86_64-apple-darwin
|
|
variant: release
|
|
- runner: macos-15-xlarge
|
|
bazel_config: ci-macos
|
|
platform: macos_amd64
|
|
sandbox: true
|
|
target: x86_64-apple-darwin
|
|
variant: ptrcomp-sandbox
|
|
- runner: macos-15-xlarge
|
|
bazel_config: ci-macos
|
|
platform: macos_arm64
|
|
sandbox: false
|
|
target: aarch64-apple-darwin
|
|
variant: release
|
|
- runner: macos-15-xlarge
|
|
bazel_config: ci-macos
|
|
platform: macos_arm64
|
|
sandbox: true
|
|
target: aarch64-apple-darwin
|
|
variant: ptrcomp-sandbox
|
|
- runner: ubuntu-24.04
|
|
bazel_config: ci-v8
|
|
platform: linux_amd64_musl
|
|
sandbox: false
|
|
target: x86_64-unknown-linux-musl
|
|
variant: release
|
|
- runner: ubuntu-24.04
|
|
bazel_config: ci-v8
|
|
platform: linux_amd64_musl
|
|
sandbox: true
|
|
target: x86_64-unknown-linux-musl
|
|
variant: ptrcomp-sandbox
|
|
- runner: ubuntu-24.04-arm
|
|
bazel_config: ci-v8
|
|
platform: linux_arm64_musl
|
|
sandbox: false
|
|
target: aarch64-unknown-linux-musl
|
|
variant: release
|
|
- runner: ubuntu-24.04-arm
|
|
bazel_config: ci-v8
|
|
platform: linux_arm64_musl
|
|
sandbox: true
|
|
target: aarch64-unknown-linux-musl
|
|
variant: ptrcomp-sandbox
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
persist-credentials: false
|
|
|
|
- name: Set up Bazel
|
|
uses: ./.github/actions/setup-bazel-ci
|
|
with:
|
|
target: ${{ matrix.target }}
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
|
|
with:
|
|
python-version: "3.12"
|
|
|
|
- name: Set up Rust toolchain for Cargo smoke
|
|
uses: dtolnay/rust-toolchain@a0b273b48ed29de4470960879e8381ff45632f26 # 1.93.0
|
|
with:
|
|
toolchain: "1.93.0"
|
|
|
|
- name: Build Bazel V8 release pair
|
|
env:
|
|
BUILDBUDDY_API_KEY: ${{ secrets.BUILDBUDDY_API_KEY }}
|
|
PLATFORM: ${{ matrix.platform }}
|
|
SANDBOX: ${{ matrix.sandbox }}
|
|
TARGET: ${{ matrix.target }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
target_suffix="${TARGET//-/_}"
|
|
pair_kind="release_pair"
|
|
if [[ "${SANDBOX}" == "true" ]]; then
|
|
pair_kind="sandbox_release_pair"
|
|
fi
|
|
pair_target="//third_party/v8:rusty_v8_${pair_kind}_${target_suffix}"
|
|
|
|
bazel_args=(
|
|
build
|
|
"--platforms=@llvm//platforms:${PLATFORM}"
|
|
--config=rusty-v8-upstream-libcxx
|
|
"${pair_target}"
|
|
--build_metadata=COMMIT_SHA=$(git rev-parse HEAD)
|
|
)
|
|
if [[ "${SANDBOX}" != "true" ]]; then
|
|
bazel_args+=(--config=v8-release-compat)
|
|
fi
|
|
|
|
bazel \
|
|
--noexperimental_remote_repo_contents_cache \
|
|
"${bazel_args[@]}" \
|
|
"--config=${{ matrix.bazel_config }}" \
|
|
"--remote_header=x-buildbuddy-api-key=${BUILDBUDDY_API_KEY}"
|
|
|
|
- name: Stage release pair
|
|
env:
|
|
BAZEL_CONFIG: ${{ matrix.bazel_config }}
|
|
BUILDBUDDY_API_KEY: ${{ secrets.BUILDBUDDY_API_KEY }}
|
|
PLATFORM: ${{ matrix.platform }}
|
|
SANDBOX: ${{ matrix.sandbox }}
|
|
TARGET: ${{ matrix.target }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
stage_args=(
|
|
--platform "${PLATFORM}"
|
|
--target "${TARGET}"
|
|
--output-dir "dist/${TARGET}"
|
|
--bazel-config "${BAZEL_CONFIG}"
|
|
)
|
|
if [[ "${SANDBOX}" == "true" ]]; then
|
|
stage_args+=(--sandbox)
|
|
else
|
|
stage_args+=(--bazel-config v8-release-compat)
|
|
fi
|
|
|
|
python3 .github/scripts/rusty_v8_bazel.py stage-release-pair "${stage_args[@]}"
|
|
|
|
- name: Smoke test staged artifact with Cargo
|
|
env:
|
|
SANDBOX: ${{ matrix.sandbox }}
|
|
TARGET: ${{ matrix.target }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
host_arch="$(uname -m)"
|
|
case "${TARGET}:${host_arch}" in
|
|
x86_64-apple-darwin:x86_64|aarch64-apple-darwin:arm64|x86_64-unknown-linux-gnu:x86_64|aarch64-unknown-linux-gnu:aarch64)
|
|
;;
|
|
*)
|
|
echo "Skipping non-native Cargo smoke for ${TARGET} on ${host_arch}."
|
|
exit 0
|
|
;;
|
|
esac
|
|
|
|
archive="$(find "dist/${TARGET}" -maxdepth 1 -type f -name 'librusty_v8_*.a.gz' -print -quit)"
|
|
binding="$(find "dist/${TARGET}" -maxdepth 1 -type f -name 'src_binding_*.rs' -print -quit)"
|
|
if [[ -z "${archive}" || -z "${binding}" ]]; then
|
|
echo "Missing staged archive or binding for ${TARGET}." >&2
|
|
exit 1
|
|
fi
|
|
|
|
cargo_args=(test -p codex-v8-poc)
|
|
if [[ "${SANDBOX}" == "true" ]]; then
|
|
cargo_args+=(--features sandbox)
|
|
fi
|
|
|
|
(
|
|
cd codex-rs
|
|
CARGO_TARGET_DIR="${RUNNER_TEMP}/rusty-v8-cargo-smoke-${TARGET}-${SANDBOX}" \
|
|
RUSTY_V8_ARCHIVE="${GITHUB_WORKSPACE}/${archive}" \
|
|
RUSTY_V8_SRC_BINDING_PATH="${GITHUB_WORKSPACE}/${binding}" \
|
|
cargo "${cargo_args[@]}"
|
|
)
|
|
|
|
- name: Upload staged artifacts
|
|
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
|
|
with:
|
|
name: v8-canary-${{ needs.metadata.outputs.v8_version }}-${{ matrix.variant }}-${{ matrix.target }}
|
|
path: dist/${{ matrix.target }}/*
|
|
|
|
build-windows-source:
|
|
name: Build ptrcomp-sandbox ${{ matrix.target }} from source
|
|
needs: metadata
|
|
runs-on: ${{ matrix.runner }}
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- runner: windows-2022
|
|
target: x86_64-pc-windows-msvc
|
|
- runner: windows-2022
|
|
target: aarch64-pc-windows-msvc
|
|
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
|
|
- name: Configure git for upstream checkout
|
|
shell: bash
|
|
run: git config --global core.symlinks true
|
|
|
|
- name: Check out upstream rusty_v8
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
with:
|
|
repository: denoland/rusty_v8
|
|
ref: v${{ needs.metadata.outputs.v8_version }}
|
|
path: upstream-rusty-v8
|
|
submodules: recursive
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
|
|
with:
|
|
python-version: "3.11"
|
|
architecture: x64
|
|
|
|
- name: Set up Codex Rust toolchain for Cargo smoke
|
|
uses: dtolnay/rust-toolchain@a0b273b48ed29de4470960879e8381ff45632f26 # 1.93.0
|
|
with:
|
|
toolchain: "1.93.0"
|
|
targets: ${{ matrix.target }}
|
|
|
|
- name: Install rusty_v8 Rust toolchain
|
|
env:
|
|
TARGET: ${{ matrix.target }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
rustup toolchain install 1.91.0 --profile minimal --no-self-update
|
|
rustup target add --toolchain 1.91.0 "${TARGET}"
|
|
|
|
- name: Write upstream submodule status
|
|
shell: bash
|
|
working-directory: upstream-rusty-v8
|
|
run: git submodule status --recursive > git_submodule_status.txt
|
|
|
|
- name: Restore upstream source-build cache
|
|
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5
|
|
with:
|
|
path: |
|
|
upstream-rusty-v8/target/sccache
|
|
upstream-rusty-v8/target/${{ matrix.target }}/release/gn_out
|
|
key: rusty-v8-source-${{ matrix.target }}-sandbox-${{ hashFiles('upstream-rusty-v8/Cargo.lock', 'upstream-rusty-v8/build.rs', 'upstream-rusty-v8/git_submodule_status.txt') }}
|
|
restore-keys: |
|
|
rusty-v8-source-${{ matrix.target }}-sandbox-
|
|
|
|
- name: Install and start sccache
|
|
shell: pwsh
|
|
env:
|
|
SCCACHE_CACHE_SIZE: 256M
|
|
SCCACHE_DIR: ${{ github.workspace }}/upstream-rusty-v8/target/sccache
|
|
SCCACHE_IDLE_TIMEOUT: 0
|
|
run: |
|
|
$version = "v0.8.2"
|
|
$platform = "x86_64-pc-windows-msvc"
|
|
$basename = "sccache-$version-$platform"
|
|
$url = "https://github.com/mozilla/sccache/releases/download/$version/$basename.tar.gz"
|
|
cd ~
|
|
curl -LO $url
|
|
tar -xzvf "$basename.tar.gz"
|
|
. $basename/sccache --start-server
|
|
echo "$(pwd)/$basename" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
|
|
|
|
- name: Install Chromium clang for ARM64 MSVC cross build
|
|
if: matrix.target == 'aarch64-pc-windows-msvc'
|
|
shell: bash
|
|
working-directory: upstream-rusty-v8
|
|
run: python3 tools/clang/scripts/update.py
|
|
|
|
- name: Build upstream rusty_v8 sandbox release pair
|
|
env:
|
|
SCCACHE_IDLE_TIMEOUT: 0
|
|
TARGET: ${{ matrix.target }}
|
|
V8_FROM_SOURCE: "1"
|
|
shell: bash
|
|
working-directory: upstream-rusty-v8
|
|
run: cargo +1.91.0 build --locked --release --target "${TARGET}" --features v8_enable_sandbox
|
|
|
|
- name: Stage upstream sandbox release pair
|
|
env:
|
|
TARGET: ${{ matrix.target }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
python3 .github/scripts/rusty_v8_bazel.py stage-upstream-release-pair \
|
|
--source-root upstream-rusty-v8 \
|
|
--target "${TARGET}" \
|
|
--output-dir "dist/${TARGET}" \
|
|
--sandbox
|
|
|
|
- name: Smoke link staged artifact with Cargo
|
|
env:
|
|
TARGET: ${{ matrix.target }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
archive="$(find "dist/${TARGET}" -maxdepth 1 -type f -name 'rusty_v8_*.lib.gz' -print -quit)"
|
|
binding="$(find "dist/${TARGET}" -maxdepth 1 -type f -name 'src_binding_*.rs' -print -quit)"
|
|
if [[ -z "${archive}" || -z "${binding}" ]]; then
|
|
echo "Missing staged archive or binding for ${TARGET}." >&2
|
|
exit 1
|
|
fi
|
|
|
|
(
|
|
cd codex-rs
|
|
RUSTY_V8_ARCHIVE="${GITHUB_WORKSPACE}/${archive}" \
|
|
RUSTY_V8_SRC_BINDING_PATH="${GITHUB_WORKSPACE}/${binding}" \
|
|
cargo +1.93.0 test -p codex-v8-poc --target "${TARGET}" --features sandbox --no-run
|
|
)
|
|
|
|
- name: Upload staged artifacts
|
|
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7
|
|
with:
|
|
name: v8-canary-${{ needs.metadata.outputs.v8_version }}-ptrcomp-sandbox-${{ matrix.target }}
|
|
path: dist/${{ matrix.target }}/*
|