feat: unified pypi hub repository#3837
Draft
rickeylev wants to merge 18 commits into
Draft
Conversation
Introduces the finalized implementation blueprint under .agents/plans/ to document our automatic canonical proxy repository and executable transition specifications.
Incorporate disjoint package execution-phase failure targets so cquery/query pass successfully over unrepresented select branches. Adds multi-hub memory minimization guidelines and test specifications.
Incorporate dynamic unionizing of custom extra_hub_aliases into the canonical proxy repository to fully support specialized wheel targets. Adds extra alias test specifications.
Currently, standard `py_library` targets cannot easily depend on third-party PyPI packages that might resolve to different concrete hubs based on the `py_binary` target including them. To enable this, synthesize an automatic canonical `@pypi` proxy repository that exposes package aliases using `select()` expressions over a new `pypi_hub` Starlark build setting. This builds upon executable transitions so `py_binary` automatically configures the appropriate PyPI resolution spoke. Sibling or disjoint packages missing from a specific spoke fail gracefully during the execution phase. * Adds `pypi_hub` build setting and base executable transition labels * Implemented `proxy_hub_repository` and `missing_package_error` * Unified pip wheel modifications and extra spoke aliases in proxy * Author integration test verifying multi-hub fallbacks and CLI overrides
Currently, if a `pip.default` tag is used solely to delete a platform (by passing only `platform`), it fails validation because `config_settings` is enforced. Furthermore, unit test mock structs lacked `default_hub`. To fix, refactor `build_config` to identify platform deletions and skip `config_settings` validation for them. Additionally, update `_default` and `_parse_modules` mock helpers in Starlark unit tests.
- Rename proxy_hub_repository to unified_hub_repository and match _impl naming conventions - Factor JSON configuration encoding out of repository rule attributes into distinct Starlark attributes - Reference public pypi_hub config_setting flag by canonical label instead of string attribute - Update missing package error messages to refer to PyPI packages - Factor unified hub synthesis in extension.bzl into a dedicated helper function
…et setup - Add documentation for pypi_hub build flag to config_settings API docs - Update missing package error execution prefix to 'ERROR:' - Rename unified_hub_repository.bzl to unified_hub_repo.bzl to comply with repository rule file naming guidelines - Factor BUILD file population logic out of unified_hub_repo.bzl into dedicated define_pypi_hub_flag_config_settings and define_pypi_package_targets helper functions in setup_unified_hub.bzl - Remove mandatory enforcement of default_hub to allow seamless error reporting when 0 fallback hubs exist
- Rename setup_unified_hub to unified_hub_setup to align with file naming conventions - Update config_settings docs for pypi_hub flag to clarify target transitions - Implement render.str in text_util.bzl to cleanly handle None values - Refactor unified_hub_repo.bzl to utilize render.str and parenthesized boolean expressions
Contributor
There was a problem hiding this comment.
Code Review
This pull request implements the Canonical Automatic PyPI Proxy Hub feature for rules_python under bzlmod, introducing a @pypi proxy repository that dynamically routes package dependencies to concrete hubs based on the new pypi_hub build setting. It also handles disjoint packages by generating execution-phase action failures and supports unionizing custom extra hub aliases. The feedback highlights three key issues in python/private/pypi/extension.bzl: a bug where multiple pip.default tags trigger false duplicate tag failures, a backward compatibility issue with the mandatory config_settings check on platform overrides, and a lack of validation to ensure default_hub matches a defined concrete hub.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
- Rename setup_unified_hub_bzl target to unified_hub_setup_bzl referencing the correct file - Move target to maintain strict alphabetical sorting in BUILD.bazel - Add missing text_util_bzl dependency to unified_hub_repo_bzl target
- Regenerate tests/integration/bzlmod_lockfile/MODULE.bazel.lock following upstream uv changes
Only enforce duplicate default hub tag check when tags explicitly define default_hub, allowing multiple pip.default tags to co-exist without failures. Also validate default_hub is defined at extension phase. - Update duplicate tag loop in python/private/pypi/extension.bzl - Add default_hub definition validation in python/private/pypi/extension.bzl - Add unit test scenario in tests/pypi/extension/extension_tests.bzl - Add integration test in tests/integration/unified_pypi_test.py
Detail the unified PyPI hub proxy feature and related robustness fixes in the new news fragment.
Update phrasing to match user's requested concise description.
Add section under docs/pypi/download.md to explain the unified @pypi proxy hub repository for multi-hub configurations.
Address review feedback on download.md and update Starlark docstrings in extension.bzl to link to the new unified hub section.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements a pypi hub that is the union of all pypi hubs.
The basic design is:
pipextension always creates a@pypirepo.--pypi_hubflag dispatches to one of the hubs. If not set,then it uses a default one (first, or as configured)
The set of packages and targets the unified hub exposes is a union
of all other hubs. If the unified hub routes to a hub that doesn't
support such a target, then it points to a target that fails at
execution time. This is to allow query and cquery to work even if
some targets don't exist in some hubs.