mirror of
https://github.com/openai/codex.git
synced 2026-04-26 15:45:02 +00:00
## Why This PR is the current, consolidated follow-up to the earlier Windows Bazel attempt in #11229. The goal is no longer just to get a tiny Windows smoke job limping along: it is to make the ordinary Bazel CI path usable on `windows-latest` for `x86_64-pc-windows-gnullvm`, with the same broad `//...` test shape that macOS and Linux already use. The earlier smoke-list version of this work was useful as a foothold, but it was not a good long-term landing point. Windows Bazel kept surfacing real issues outside that allowlist: - GitHub's Windows runner exposed runfiles-manifest bugs such as `FINDSTR: Cannot open D:MANIFEST`, which broke Bazel test launchers even when the manifest file existed. - `rules_rs`, `rules_rust`, LLVM extraction, and Abseil still needed `windows-gnullvm`-specific fixes for our hermetic toolchain. - the V8 path needed more work than just turning the Windows matrix entry back on: `rusty_v8` does not ship Windows GNU artifacts in the same shape we need, and Bazel's in-tree V8 build needed a set of Windows GNU portability fixes. Windows performance pressure also pushed this toward a full solution instead of a permanent smoke suite. During this investigation we hit targets such as `//codex-rs/shell-command:shell-command-unit-tests` that were much more expensive on Windows because they repeatedly spawn real PowerShell parsers (see #16057 for one concrete example of that pressure). That made it much more valuable to get the real Windows Bazel path working than to keep iterating on a narrowly curated subset. The net result is that this PR now aims for the same CI contract on Windows that we already expect elsewhere: keep standalone `//third_party/v8:all` out of the ordinary Bazel lane, but allow V8 consumers under `//codex-rs/...` to build and test transitively through `//...`. ## What Changed ### CI and workflow wiring - re-enable the `windows-latest` / `x86_64-pc-windows-gnullvm` Bazel matrix entry in `.github/workflows/bazel.yml` - move the Windows Bazel output root to `D:\b` and enable `git config --global core.longpaths true` in `.github/actions/setup-bazel-ci/action.yml` - keep the ordinary Bazel target set on Windows aligned with macOS and Linux by running `//...` while excluding only standalone `//third_party/v8:all` targets from the normal lane ### Toolchain and module support for `windows-gnullvm` - patch `rules_rs` so `windows-gnullvm` is modeled as a distinct Windows exec/toolchain platform instead of collapsing into the generic Windows shape - patch `rules_rust` build-script environment handling so llvm-mingw build-script probes do not inherit unsupported `-fstack-protector*` flags - patch the LLVM module archive so it extracts cleanly on Windows and provides the MinGW libraries this toolchain needs - patch Abseil so its thread-local identity path matches the hermetic `windows-gnullvm` toolchain instead of taking an incompatible MinGW pthread path - keep both MSVC and GNU Windows targets in the generated Cargo metadata because the current V8 release-asset story still uses MSVC-shaped names in some places while the Bazel build targets the GNU ABI ### Windows test-launch and binary-behavior fixes - update `workspace_root_test_launcher.bat.tpl` to read the runfiles manifest directly instead of shelling out to `findstr`, which was the source of the `D:MANIFEST` failures on the GitHub Windows runner - thread a larger Windows GNU stack reserve through `defs.bzl` so Bazel-built binaries that pull in V8 behave correctly both under normal builds and under `bazel test` - remove the no-longer-needed Windows bootstrap sh-toolchain override from `.bazelrc` ### V8 / `rusty_v8` Windows GNU support - export and apply the new Windows GNU patch set from `patches/BUILD.bazel` / `MODULE.bazel` - patch the V8 module/rules/source layers so the in-tree V8 build can produce Windows GNU archives under Bazel - teach `third_party/v8/BUILD.bazel` to build Windows GNU static archives in-tree instead of aliasing them to the MSVC prebuilts - reuse the Linux release binding for the experimental Windows GNU path where `rusty_v8` does not currently publish a Windows GNU binding artifact ## Testing - the primary end-to-end validation for this work is the `Bazel` workflow plus `v8-canary`, since the hard parts are Windows-specific and depend on real GitHub runner behavior - before consolidation back onto this PR, the same net change passed the full Bazel matrix in [run 23675590471](https://github.com/openai/codex/actions/runs/23675590471) and passed `v8-canary` in [run 23675590453](https://github.com/openai/codex/actions/runs/23675590453) - those successful runs included the `windows-latest` / `x86_64-pc-windows-gnullvm` Bazel job with the ordinary `//...` path, not the earlier Windows smoke allowlist --- [//]: # (BEGIN SAPLING FOOTER) Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/openai/codex/pull/15952). * #16067 * __->__ #15952
353 lines
12 KiB
Python
353 lines
12 KiB
Python
load("@bazel_skylib//rules:copy_file.bzl", "copy_file")
|
|
load("@rules_cc//cc:cc_static_library.bzl", "cc_static_library")
|
|
load("@rules_cc//cc:defs.bzl", "cc_library")
|
|
|
|
package(default_visibility = ["//visibility:public"])
|
|
|
|
config_setting(
|
|
name = "platform_aarch64_unknown_linux_musl",
|
|
constraint_values = [
|
|
"@platforms//cpu:aarch64",
|
|
"@platforms//os:linux",
|
|
"@llvm//constraints/libc:musl",
|
|
],
|
|
)
|
|
|
|
config_setting(
|
|
name = "platform_x86_64_unknown_linux_musl",
|
|
constraint_values = [
|
|
"@platforms//cpu:x86_64",
|
|
"@platforms//os:linux",
|
|
"@llvm//constraints/libc:musl",
|
|
],
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_x86_64_apple_darwin",
|
|
actual = "@rusty_v8_146_4_0_x86_64_apple_darwin_archive//file",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_aarch64_apple_darwin",
|
|
actual = "@rusty_v8_146_4_0_aarch64_apple_darwin_archive//file",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_x86_64_unknown_linux_gnu",
|
|
actual = "@rusty_v8_146_4_0_x86_64_unknown_linux_gnu_archive//file",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_aarch64_unknown_linux_gnu",
|
|
actual = "@rusty_v8_146_4_0_aarch64_unknown_linux_gnu_archive//file",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_x86_64_unknown_linux_musl",
|
|
actual = "@rusty_v8_146_4_0_x86_64_unknown_linux_musl_archive//file",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_aarch64_unknown_linux_musl",
|
|
actual = "@rusty_v8_146_4_0_aarch64_unknown_linux_musl_archive//file",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_x86_64_pc_windows_msvc",
|
|
actual = "@rusty_v8_146_4_0_x86_64_pc_windows_msvc_archive//file",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_aarch64_pc_windows_msvc",
|
|
actual = "@rusty_v8_146_4_0_aarch64_pc_windows_msvc_archive//file",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_aarch64_pc_windows_gnullvm",
|
|
# `rusty_v8` only ships prebuilt Windows archives for MSVC. Build the
|
|
# GNU-flavored archive in-tree so windows-gnullvm consumers can link
|
|
# against a matching ABI instead of trying to reuse the MSVC release.
|
|
actual = ":v8_146_4_0_aarch64_pc_windows_gnullvm_bazel",
|
|
)
|
|
|
|
alias(
|
|
name = "v8_146_4_0_x86_64_pc_windows_gnullvm",
|
|
actual = ":v8_146_4_0_x86_64_pc_windows_gnullvm_bazel",
|
|
)
|
|
|
|
filegroup(
|
|
name = "src_binding_release_x86_64_apple_darwin",
|
|
srcs = ["@v8_crate_146_4_0//:src_binding_release_x86_64_apple_darwin"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "src_binding_release_aarch64_apple_darwin",
|
|
srcs = ["@v8_crate_146_4_0//:src_binding_release_aarch64_apple_darwin"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "src_binding_release_aarch64_unknown_linux_gnu",
|
|
srcs = ["@v8_crate_146_4_0//:src_binding_release_aarch64_unknown_linux_gnu"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "src_binding_release_x86_64_unknown_linux_gnu",
|
|
srcs = ["@v8_crate_146_4_0//:src_binding_release_x86_64_unknown_linux_gnu"],
|
|
)
|
|
|
|
alias(
|
|
name = "src_binding_release_x86_64_unknown_linux_musl",
|
|
actual = "@rusty_v8_146_4_0_x86_64_unknown_linux_musl_binding//file",
|
|
)
|
|
|
|
alias(
|
|
name = "src_binding_release_aarch64_unknown_linux_musl",
|
|
actual = "@rusty_v8_146_4_0_aarch64_unknown_linux_musl_binding//file",
|
|
)
|
|
|
|
filegroup(
|
|
name = "src_binding_release_x86_64_pc_windows_msvc",
|
|
srcs = ["@v8_crate_146_4_0//:src_binding_release_x86_64_pc_windows_msvc"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "src_binding_release_aarch64_pc_windows_msvc",
|
|
srcs = ["@v8_crate_146_4_0//:src_binding_release_aarch64_pc_windows_msvc"],
|
|
)
|
|
|
|
alias(
|
|
name = "src_binding_release_x86_64_pc_windows_gnullvm",
|
|
# `rusty_v8` does not publish a Windows GNU binding file. The generated
|
|
# binding only describes this V8 release's C++ API surface, so reuse the
|
|
# Linux release binding while the windows-gnullvm archive build is still
|
|
# experimental.
|
|
actual = ":src_binding_release_x86_64_unknown_linux_gnu",
|
|
)
|
|
|
|
alias(
|
|
name = "src_binding_release_aarch64_pc_windows_gnullvm",
|
|
actual = ":src_binding_release_aarch64_unknown_linux_gnu",
|
|
)
|
|
|
|
alias(
|
|
name = "rusty_v8_archive_for_target",
|
|
actual = select({
|
|
"@rules_rs//rs/experimental/platforms/config:aarch64-apple-darwin": ":v8_146_4_0_aarch64_apple_darwin_bazel",
|
|
"@rules_rs//rs/experimental/platforms/config:aarch64-pc-windows-gnullvm": ":v8_146_4_0_aarch64_pc_windows_gnullvm",
|
|
"@rules_rs//rs/experimental/platforms/config:aarch64-pc-windows-msvc": ":v8_146_4_0_aarch64_pc_windows_msvc",
|
|
"@rules_rs//rs/experimental/platforms/config:aarch64-unknown-linux-gnu": ":v8_146_4_0_aarch64_unknown_linux_gnu_bazel",
|
|
":platform_aarch64_unknown_linux_musl": ":v8_146_4_0_aarch64_unknown_linux_musl_release_base",
|
|
"@rules_rs//rs/experimental/platforms/config:x86_64-apple-darwin": ":v8_146_4_0_x86_64_apple_darwin_bazel",
|
|
"@rules_rs//rs/experimental/platforms/config:x86_64-pc-windows-gnullvm": ":v8_146_4_0_x86_64_pc_windows_gnullvm",
|
|
"@rules_rs//rs/experimental/platforms/config:x86_64-pc-windows-msvc": ":v8_146_4_0_x86_64_pc_windows_msvc",
|
|
"@rules_rs//rs/experimental/platforms/config:x86_64-unknown-linux-gnu": ":v8_146_4_0_x86_64_unknown_linux_gnu_bazel",
|
|
":platform_x86_64_unknown_linux_musl": ":v8_146_4_0_x86_64_unknown_linux_musl_release",
|
|
"//conditions:default": ":v8_146_4_0_x86_64_unknown_linux_gnu_bazel",
|
|
}),
|
|
)
|
|
|
|
alias(
|
|
name = "rusty_v8_binding_for_target",
|
|
actual = select({
|
|
"@rules_rs//rs/experimental/platforms/config:aarch64-apple-darwin": ":src_binding_release_aarch64_apple_darwin",
|
|
"@rules_rs//rs/experimental/platforms/config:aarch64-pc-windows-gnullvm": ":src_binding_release_aarch64_pc_windows_gnullvm",
|
|
"@rules_rs//rs/experimental/platforms/config:aarch64-pc-windows-msvc": ":src_binding_release_aarch64_pc_windows_msvc",
|
|
"@rules_rs//rs/experimental/platforms/config:aarch64-unknown-linux-gnu": ":src_binding_release_aarch64_unknown_linux_gnu",
|
|
":platform_aarch64_unknown_linux_musl": ":src_binding_release_aarch64_unknown_linux_musl",
|
|
"@rules_rs//rs/experimental/platforms/config:x86_64-apple-darwin": ":src_binding_release_x86_64_apple_darwin",
|
|
"@rules_rs//rs/experimental/platforms/config:x86_64-pc-windows-gnullvm": ":src_binding_release_x86_64_pc_windows_gnullvm",
|
|
"@rules_rs//rs/experimental/platforms/config:x86_64-pc-windows-msvc": ":src_binding_release_x86_64_pc_windows_msvc",
|
|
"@rules_rs//rs/experimental/platforms/config:x86_64-unknown-linux-gnu": ":src_binding_release_x86_64_unknown_linux_gnu",
|
|
":platform_x86_64_unknown_linux_musl": ":src_binding_release_x86_64_unknown_linux_musl",
|
|
"//conditions:default": ":src_binding_release_x86_64_unknown_linux_gnu",
|
|
}),
|
|
)
|
|
|
|
V8_COPTS = ["-std=c++20"]
|
|
|
|
V8_STATIC_LIBRARY_FEATURES = [
|
|
"-symbol_check",
|
|
"-validate-static-library",
|
|
]
|
|
|
|
genrule(
|
|
name = "binding_cc",
|
|
srcs = ["@v8_crate_146_4_0//:binding_cc"],
|
|
outs = ["binding.cc"],
|
|
# Keep this as a literal shell snippet. The string-concatenated form looked
|
|
# cleaner in Starlark but produced a broken `sed` invocation in CI.
|
|
cmd = """
|
|
sed \
|
|
-e '/#include "v8\\/src\\/flags\\/flags.h"/d' \
|
|
-e 's|"v8/src/libplatform/default-platform.h"|"src/libplatform/default-platform.h"|' \
|
|
-e 's| namespace i = v8::internal;| (void)usage;|' \
|
|
-e '/using HelpOptions = i::FlagList::HelpOptions;/d' \
|
|
-e '/HelpOptions help_options = HelpOptions(HelpOptions::kExit, usage);/d' \
|
|
-e 's| i::FlagList::SetFlagsFromCommandLine(argc, argv, true, help_options);| v8::V8::SetFlagsFromCommandLine(argc, argv, true);|' \
|
|
$(location @v8_crate_146_4_0//:binding_cc) > "$@"
|
|
""",
|
|
)
|
|
|
|
copy_file(
|
|
name = "support_h",
|
|
src = "@v8_crate_146_4_0//:support_h",
|
|
out = "support.h",
|
|
)
|
|
|
|
cc_library(
|
|
name = "v8_146_4_0_binding",
|
|
srcs = [":binding_cc"],
|
|
hdrs = [":support_h"],
|
|
copts = V8_COPTS,
|
|
deps = [
|
|
"@v8//:core_lib_icu",
|
|
"@v8//:rusty_v8_internal_headers",
|
|
],
|
|
)
|
|
|
|
cc_static_library(
|
|
name = "v8_146_4_0_aarch64_apple_darwin_bazel",
|
|
deps = [":v8_146_4_0_binding"],
|
|
features = V8_STATIC_LIBRARY_FEATURES,
|
|
)
|
|
|
|
cc_static_library(
|
|
name = "v8_146_4_0_aarch64_unknown_linux_gnu_bazel",
|
|
deps = [":v8_146_4_0_binding"],
|
|
features = V8_STATIC_LIBRARY_FEATURES,
|
|
)
|
|
|
|
cc_static_library(
|
|
name = "v8_146_4_0_aarch64_pc_windows_gnullvm_bazel",
|
|
deps = [":v8_146_4_0_binding"],
|
|
features = V8_STATIC_LIBRARY_FEATURES,
|
|
)
|
|
|
|
cc_static_library(
|
|
name = "v8_146_4_0_x86_64_apple_darwin_bazel",
|
|
deps = [":v8_146_4_0_binding"],
|
|
features = V8_STATIC_LIBRARY_FEATURES,
|
|
)
|
|
|
|
cc_static_library(
|
|
name = "v8_146_4_0_x86_64_pc_windows_gnullvm_bazel",
|
|
deps = [":v8_146_4_0_binding"],
|
|
features = V8_STATIC_LIBRARY_FEATURES,
|
|
)
|
|
|
|
cc_static_library(
|
|
name = "v8_146_4_0_x86_64_unknown_linux_gnu_bazel",
|
|
deps = [":v8_146_4_0_binding"],
|
|
features = V8_STATIC_LIBRARY_FEATURES,
|
|
)
|
|
|
|
cc_static_library(
|
|
name = "v8_146_4_0_aarch64_unknown_linux_musl_release_base",
|
|
deps = [":v8_146_4_0_binding"],
|
|
features = V8_STATIC_LIBRARY_FEATURES,
|
|
)
|
|
|
|
genrule(
|
|
name = "v8_146_4_0_aarch64_unknown_linux_musl_release",
|
|
srcs = [
|
|
":v8_146_4_0_aarch64_unknown_linux_musl_release_base",
|
|
"@llvm//runtimes/compiler-rt:clang_rt.builtins.static",
|
|
],
|
|
tools = [
|
|
"@llvm//tools:llvm-ar",
|
|
"@llvm//tools:llvm-ranlib",
|
|
],
|
|
outs = ["libv8_146_4_0_aarch64_unknown_linux_musl.a"],
|
|
cmd = """
|
|
cat > "$(@D)/merge.mri" <<'EOF'
|
|
create $@
|
|
addlib $(location :v8_146_4_0_aarch64_unknown_linux_musl_release_base)
|
|
addlib $(location @llvm//runtimes/compiler-rt:clang_rt.builtins.static)
|
|
save
|
|
end
|
|
EOF
|
|
$(location @llvm//tools:llvm-ar) -M < "$(@D)/merge.mri"
|
|
$(location @llvm//tools:llvm-ranlib) "$@"
|
|
""",
|
|
)
|
|
|
|
cc_static_library(
|
|
name = "v8_146_4_0_x86_64_unknown_linux_musl_release",
|
|
deps = [":v8_146_4_0_binding"],
|
|
features = V8_STATIC_LIBRARY_FEATURES,
|
|
)
|
|
|
|
filegroup(
|
|
name = "src_binding_release_aarch64_unknown_linux_musl_release",
|
|
srcs = ["@v8_crate_146_4_0//:src_binding_release_aarch64_unknown_linux_gnu"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "src_binding_release_x86_64_unknown_linux_musl_release",
|
|
srcs = ["@v8_crate_146_4_0//:src_binding_release_x86_64_unknown_linux_gnu"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "rusty_v8_release_pair_x86_64_apple_darwin",
|
|
srcs = [
|
|
":v8_146_4_0_x86_64_apple_darwin",
|
|
":src_binding_release_x86_64_apple_darwin",
|
|
],
|
|
)
|
|
|
|
filegroup(
|
|
name = "rusty_v8_release_pair_aarch64_apple_darwin",
|
|
srcs = [
|
|
":v8_146_4_0_aarch64_apple_darwin",
|
|
":src_binding_release_aarch64_apple_darwin",
|
|
],
|
|
)
|
|
|
|
filegroup(
|
|
name = "rusty_v8_release_pair_x86_64_unknown_linux_gnu",
|
|
srcs = [
|
|
":v8_146_4_0_x86_64_unknown_linux_gnu",
|
|
":src_binding_release_x86_64_unknown_linux_gnu",
|
|
],
|
|
)
|
|
|
|
filegroup(
|
|
name = "rusty_v8_release_pair_aarch64_unknown_linux_gnu",
|
|
srcs = [
|
|
":v8_146_4_0_aarch64_unknown_linux_gnu",
|
|
":src_binding_release_aarch64_unknown_linux_gnu",
|
|
],
|
|
)
|
|
|
|
filegroup(
|
|
name = "rusty_v8_release_pair_x86_64_unknown_linux_musl",
|
|
srcs = [
|
|
":v8_146_4_0_x86_64_unknown_linux_musl_release",
|
|
":src_binding_release_x86_64_unknown_linux_musl_release",
|
|
],
|
|
)
|
|
|
|
filegroup(
|
|
name = "rusty_v8_release_pair_aarch64_unknown_linux_musl",
|
|
srcs = [
|
|
":v8_146_4_0_aarch64_unknown_linux_musl_release",
|
|
":src_binding_release_aarch64_unknown_linux_musl_release",
|
|
],
|
|
)
|
|
|
|
filegroup(
|
|
name = "rusty_v8_release_pair_x86_64_pc_windows_msvc",
|
|
srcs = [
|
|
":v8_146_4_0_x86_64_pc_windows_msvc",
|
|
":src_binding_release_x86_64_pc_windows_msvc",
|
|
],
|
|
)
|
|
|
|
filegroup(
|
|
name = "rusty_v8_release_pair_aarch64_pc_windows_msvc",
|
|
srcs = [
|
|
":v8_146_4_0_aarch64_pc_windows_msvc",
|
|
":src_binding_release_aarch64_pc_windows_msvc",
|
|
],
|
|
)
|