Skip to content

ci: Refactor wheel packaging workflows#24

Merged
zhanglei1949 merged 42 commits intomainfrom
ref-wheel-package
Mar 19, 2026
Merged

ci: Refactor wheel packaging workflows#24
zhanglei1949 merged 42 commits intomainfrom
ref-wheel-package

Conversation

@zhanglei1949
Copy link
Collaborator

@zhanglei1949 zhanglei1949 commented Mar 10, 2026

Greptile Summary

This PR refactors the wheel-packaging CI by extracting all build logic into a new reusable workflow (wheels-common.yml) that is called by both the release (build-wheel.yml) and nightly (build-nightly-wheel.yml) callers. It also bumps cibuildwheel to 3.4.0 across all workflows, adds Python 3.14 pre-release support, resolves several macOS arm64 and Python 3.14 build compatibility issues in CMake, and fixes a bash array syntax bug in install_deps.sh.

Key changes:

  • New wheels-common.yml: Single source of truth for wheel builds across all platforms; exposes mode, python_versions, cibuildwheel_version, and platform-enable flags as inputs to callers.
  • cibuildwheel 3.4.0 everywhere: Aligns nightly and release to the same tool version, resolving the previous mismatch noted in earlier review rounds.
  • Python 3.14 support: pyproject.toml adds cp314-* with enable = ["cpython-prerelease"]; BuildBRPCAsThirdParty.cmake patches brpc's HAVE_DLADDR redefinition error; BuildAbseilThirdParty.cmake strips x86-only Abseil flags on Apple Silicon.
  • Security gap: The build_wheels_linux_arm64 nightly job is missing the github.repository == 'alibaba/neug' guard present in the equivalent x86_64 and macOS aarch64 jobs. Because build-nightly-wheel.yml triggers on pull_request with no repository filter, a fork PR will dispatch this job to the self-hosted arm64 Linux runner and execute untrusted code on it.

Confidence Score: 3/5

  • Mostly safe to merge after fixing the self-hosted runner security gap in the arm64 nightly job condition.
  • The refactor is well-structured and addresses several pre-existing issues (version mismatch, macOS deployment target, MACOSX_DEPLOYMENT_TARGET diagnostic). However, the missing github.repository guard in build_wheels_linux_arm64's nightly if condition is a meaningful security issue — fork PRs can trigger untrusted code execution on the self-hosted arm64 runner — and should be fixed before merging.
  • .github/workflows/wheels-common.yml — specifically the build_wheels_linux_arm64 job's nightly if condition (lines ~247–261).

Important Files Changed

Filename Overview
.github/workflows/wheels-common.yml New reusable workflow consolidating all wheel-build logic; has a security gap in the Linux arm64 nightly job — missing github.repository guard allows fork PRs to run on the self-hosted runner.
.github/workflows/build-wheel.yml Refactored to delegate to wheels-common.yml; adds NEUG_VERSION to paths filter and v* tag trigger as requested; bumps cibuildwheel to 3.4.0 and runner Python to 3.14 for x86 and macOS.
.github/workflows/build-nightly-wheel.yml Simplified to a thin caller of wheels-common.yml; adds optional python_versions dispatch input and aligns cibuildwheel version to 3.4.0, resolving the previous nightly/release version mismatch.
.github/workflows/build-extensions.yml Adds rm -rf /__w/_tool/Python pre-cleanup step to the two self-hosted Linux jobs before setup-python; appropriate here since both jobs explicitly target self-hosted runners.
.github/workflows/build-image.yml Updates Docker image references from graphscope/neug to neug/neug; verified correct — docker/Makefile already builds the image as neug/neug:${ARCH}.
cmake/BuildAbseilThirdParty.cmake Adds _neug_strip_absl_randen_flags helper to remove x86-only flags (-msse4.1, -maes, -Xarch_x86_64) from Abseil Randen targets when building for Apple Silicon; fix is correctly scoped to APPLE + arm64 detection.
cmake/BuildBRPCAsThirdParty.cmake Patches third_party/brpc/src/butil/build_config.h at configure time to add a #ifndef guard around HAVE_DLADDR, fixing a macro-redefinition error with Python 3.14+; idempotent thanks to the existing-guard check.
scripts/install_deps.sh Fixes bash array syntax (removes erroneous commas in INTERACTIVE_MACOS) and adds an OPENSSL_ROOT_DIR export for macOS to the generated env config.
tools/python_bind/pyproject.toml Adds cp314-* to the cibuildwheel build target list and enables cpython-prerelease to support building Python 3.14 wheels.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[build-wheel.yml\nmode: release] -->|workflow_call| C
    B[build-nightly-wheel.yml\nmode: nightly] -->|workflow_call| C

    C[wheels-common.yml\nwheels-common.yml]

    C --> D[determine_python_versions\nubuntu-latest]
    D --> E[build_wheels_linux_x86_64\nself-hosted / linux / x64]
    D --> F[build_wheels_linux_arm64\nself-hosted / linux / arm64]
    D --> G[build_wheels_macos_x86_64\nmacos-15-intel\nenable_macos_x86=false]
    D --> H[build_wheels_macos_aarch64\nmacos-15]

    E -->|on tag or dispatch| E1[Upload artifact]
    E -->|on tag or main push| E2[Publish to PyPI]

    F -->|on tag or dispatch| F1[Upload artifact]
    F -->|on tag or main push| F2[Publish to PyPI]

    H --> H1[test_wheels_macos_aarch64\nmacos-15]

    G -.->|disabled| G1[test_wheels_macos_x86_64\nmacos-15-intel]

    style F fill:#ffcccc,stroke:#cc0000
    style G fill:#eeeeee,stroke:#999999,stroke-dasharray:5
Loading

Last reviewed commit: "Merge branch 'main' ..."

Greptile also left 1 inline comment on this PR.

@zhanglei1949
Copy link
Collaborator Author

@greptile review this PR and update the PR summary

@zhanglei1949
Copy link
Collaborator Author

@greptile

@zhanglei1949
Copy link
Collaborator Author

TODO: @zhanglei1949 Remove the if: true condition for all macos runners.

@zhanglei1949 zhanglei1949 requested a review from longbinlai March 10, 2026 08:39
@lnfjpt lnfjpt force-pushed the ref-wheel-package branch 2 times, most recently from bbf0467 to 74765c9 Compare March 13, 2026 07:24
#define nullptr NULL
#endif

+#ifndef HAVE_DLADDR
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhanglei1949 py14 have defined this HAVE_DLADDR macro which blocks building wheel for py14. AI proposes to add this macro guard. Could you help check whether this is okay,

@CLAassistant
Copy link

CLAassistant commented Mar 13, 2026

CLA assistant check
All committers have signed the CLA.

@longbinlai longbinlai force-pushed the ref-wheel-package branch 2 times, most recently from aa0d249 to de003fa Compare March 16, 2026 04:10
@longbinlai
Copy link
Collaborator

Resolve the conflicts. Any, if possible, make this pr-triggered wheel packaging only for linux arm64, and left all platform packaging into nightly trigger.

shirly121 and others added 16 commits March 17, 2026 18:01
* support varchar(max_length) in NeuG type system

Committed-by: Xiaoli Zhou from Dev container

* skip specific regex when comparing physical plans

Committed-by: Xiaoli Zhou from Dev container

* minor fix according to review

Committed-by: Xiaoli Zhou from Dev container

* fix ci

Committed-by: xiaolei.zl from Dev container

Committed-by: xiaolei.zl from Dev container

* avoid change func api

Committed-by: xiaolei.zl from Dev container

* ci strange failure

Committed-by: xiaolei.zl from Dev container

---------

Co-authored-by: xiaolei.zl <xiaolei.zl@alibaba-inc.com>
Committed-by: nengli.ln from Dev container
Committed-by: xiaolei.zl from Dev container

Committed-by: xiaolei.zl from Dev container

minor

Committed-by: xiaolei.zl from Dev container

fix

Committed-by: xiaolei.zl from Dev container

Committed-by: xiaolei.zl from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: xiaolei.zl from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: xiaolei.zl from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: xiaolei.zl from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: xiaolei.zl from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: xiaolei.zl from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: xiaolei.zl from Dev container

Committed-by: xiaolei.zl from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: nengli.ln from Dev container
Committed-by: nengli.ln from Dev container
Committed-by: nengli.ln from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: nengli.ln from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: nengli.ln from Dev container

Committed-by: nengli.ln from Dev container
Committed-by: nengli.ln from Dev container
Committed-by: nengli.ln from Dev container
Committed-by: xiaolei.zl from Dev container
Committed-by: nengli.ln from Dev container
longbinlai
longbinlai previously approved these changes Mar 18, 2026
Copy link
Collaborator

@longbinlai longbinlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhanglei1949
Copy link
Collaborator Author

@greptile

zhanglei1949 and others added 2 commits March 18, 2026 18:41
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
@zhanglei1949
Copy link
Collaborator Author

@greptile

@zhanglei1949
Copy link
Collaborator Author

@greptile

zhanglei1949 and others added 3 commits March 18, 2026 19:13
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
@zhanglei1949
Copy link
Collaborator Author

@greptile

zhanglei1949 and others added 4 commits March 19, 2026 09:49
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
@zhanglei1949
Copy link
Collaborator Author

@greptile

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

zhanglei1949 has reached the 50-review limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@zhanglei1949
Copy link
Collaborator Author

@CodiumAI-Agent Review

Copy link
Collaborator

@longbinlai longbinlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhanglei1949 zhanglei1949 merged commit 41a70b0 into main Mar 19, 2026
26 checks passed
@longbinlai longbinlai deleted the ref-wheel-package branch March 19, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants