fix(deps): support multi-host dependency identity

[danielmeppiel]

fix(deps): support multi-host dependency identity

TL;DR

This PR implements the board-approved #773 items 1-4: host-aware lockfile identity, consistent per-dependency token resolution for file downloads, explicit GitLab host signaling, and clearer download-attempt errors. Existing github.com lockfile keys remain byte-stable as bare owner/repo; only non-default hosts add the host segment. Closes #773.

Important

Lockfile migration note: existing github.com keys are preserved as-is. New non-default host entries use host/owner/repo, so github.com/team/skills and gitea.myorg.com/team/skills can coexist without colliding.

Problem (WHY)

• Multi-host dependency support: lockfile identity, token resolution consistency, error clarity #773 reports that get_unique_key() returned the bare repo path, so two hosts with the same path could "collide silently in apm.lock".
• Multi-host dependency support: lockfile identity, token resolution consistency, error clarity #773 also calls out that _download_github_file resolved tokens differently from clone, causing "misleading 401 errors when a GitHub PAT is sent to a Gitea / GitLab server".
• GitLab detection needed an explicit signal because hostname conventions can miss bespoke instances like code.acme.com.
• The download fallback chain swallowed non-404 failures, so users saw "file not found" instead of the failing endpoint, status, or network error.

Approach (WHAT)

Area	Change
Lockfile identity	Shared build_dependency_unique_key() keeps github.com implicit and prefixes only non-default hosts.
Auth consistency	Single-file downloads now resolve through _resolve_dep_token() / _resolve_dep_auth_ctx(), matching clone boundaries.
GitLab signaling	Object-form deps accept type: gitlab; the hint flows through auth, backends, validation, and lockfile replay.
Error clarity	Raw/API attempts emit verbose breadcrumbs and only 404 participates in fallback; non-404 HTTP/network failures surface host + endpoint context.
Docs	User docs and apm-usage guide document type: gitlab and the lockfile key migration rule.

Implementation (HOW)

Files	What changed
src/apm_cli/models/dependency/reference.py, src/apm_cli/deps/lockfile.py	Added shared host-aware identity keys plus host_type parse/round-trip support.
src/apm_cli/core/auth.py, src/apm_cli/deps/host_backends.py	Threaded explicit host type into host classification and backend selection.
src/apm_cli/deps/download_strategies.py, src/apm_cli/deps/github_downloader.py	Reused per-dependency auth resolution for file downloads, routed type: gitlab to GitLab REST, and surfaced endpoint-specific errors.
src/apm_cli/install/validation.py, src/apm_cli/deps/github_downloader_validation.py	Kept validation probes aligned with explicit GitLab classification.
tests/**	Added regression traps for host-key collisions, generic-host token boundaries, bespoke GitLab routing, and non-404 error surfacing.
Docs + CHANGELOG.md	Documented dependency format, authentication behavior, lockfile migration, and user-facing fix.

Diagram

The diagram shows the new dependency flow from manifest identity to host-aware lockfile key and download backend.

flowchart LR
  manifest[Manifest dependency] --> ref[DependencyReference]
  ref --> key{Host value}
  key -->|github.com or unset| ghkey[Lock key owner/repo]
  key -->|non-default host| hostkey[Lock key host/owner/repo]
  ref --> type{Explicit type gitlab}
  type -->|yes| gitlab[GitLab REST and GitLab token chain]
  type -->|no| generic[Generic raw/API attempts without managed PAT]

Loading

Trade-offs

• The lockfile key schema stays additive: no rewrite of existing github.com entries, but non-default hosts now become visible in the in-memory key.
• type: gitlab is explicit instead of heuristic or probe-based, avoiding network capability detection in manifest parsing.
• Generic HTTP file downloads no longer attach git-credential-fill tokens as headers; private generic-host file reads should use a supported backend signal or clone path.
• Optional issue item 5 was not included; this PR focuses on the board-required items 1-4.

Benefits

1. Same owner/repo on two hosts no longer overwrites a lockfile entry.
2. Generic hosts no longer receive managed GitHub/GitLab PAT headers from the file-download path.
3. Bespoke GitLab instances can opt into GitLab REST and token behavior without hostname conventions.
4. Non-404 download failures now name the host, endpoint, and HTTP/network failure instead of degrading to a missing-file message.

Validation

Scenario evidence

Scenario	Test evidence	Principle
Non-default hosts get distinct lock keys while github.com remains implicit	tests/test_lockfile.py	Reproducibility
Generic file downloads follow clone auth boundaries	tests/test_github_downloader.py::TestGiteaRawUrlDownload, tests/unit/deps/test_download_strategies_*	Credential containment
Bespoke GitLab host uses GitLab REST and token chain	tests/test_github_downloader.py::TestGiteaGogsApiVersionNegotiation::test_object_form_type_gitlab_routes_bespoke_host_to_gitlab_api	Correct host routing
Non-404 raw/API failures surface endpoint context	tests/test_github_downloader.py::TestGiteaRawUrlDownload	Diagnosability

Commands run

uv run --extra dev pytest tests/test_lockfile.py tests/unit/test_auth.py::TestClassifyHost tests/unit/test_github_host.py tests/test_github_downloader.py::TestGiteaRawUrlDownload tests/test_github_downloader.py::TestGiteaGogsApiVersionNegotiation tests/test_github_downloader.py::TestGitLabInstallFileDownload tests/test_github_downloader.py::TestDownloaderCredentialFallback tests/test_github_downloader.py::TestErrorHandling::test_download_github_file_403_rate_limit_no_token tests/test_github_downloader.py::TestErrorHandling::test_download_github_file_403_rate_limit_with_token tests/test_github_downloader.py::TestErrorHandling::test_download_github_file_403_non_rate_limit_still_auth_error tests/test_github_downloader.py::TestMixedSourceTokenSelection tests/unit/deps/test_download_strategies_selection.py tests/unit/deps/test_download_strategies_phase3.py -q
410 passed in 1.95s

uv run --extra dev ruff check src/ tests/
All checks passed!

uv run --extra dev ruff format --check src/ tests/
1234 files already formatted

uv run --extra dev python -m pylint --disable=all --enable=R0801 --min-similarity-lines=10 --fail-on=R0801 src/apm_cli/
Your code has been rated at 10.00/10

bash scripts/lint-auth-signals.sh
[+] auth-signal lint clean

python3 CI guard equivalents for yaml dump, src file length, and raw str(relative_to())
passed

How to test

• Create an apm.yml with github.com/team/skills and gitea.myorg.com/team/skills, run apm install, and confirm both lockfile entries remain present.
• Configure a generic host dependency and run apm install --verbose; confirm raw/API attempts are logged and no GitHub PAT header is sent.
• Add an object-form GitLab dependency with type: gitlab for a bespoke hostname and confirm GitLab REST URLs are used.
• Simulate a 500 or network failure from a generic-host raw endpoint and confirm the error names the host and endpoint.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

This PR addresses multi-host dependency correctness and UX by making lockfile identity host-aware (while keeping github.com keys stable), aligning single-file download auth boundaries with clone auth, adding an explicit type: gitlab host signal for bespoke GitLab instances, and improving download error clarity (only 404 falls through; non-404/network errors surface with endpoint context).

Changes:

• Introduces a shared host-aware dependency unique key (build_dependency_unique_key) and threads host_type through dependency parsing, lockfile round-trips, host classification, and backend selection.
• Updates generic-host file download behavior to use per-dependency auth resolution (avoiding managed PAT headers on generic HTTP downloads) and improves verbose/error diagnostics for raw/API attempts.
• Adds regression tests plus documentation updates for type: gitlab and the lockfile key migration rule.

Show a summary per file

File	Description
src/apm_cli/models/dependency/reference.py	Adds shared host-aware unique-key builder and host_type parsing for object-form deps.
src/apm_cli/deps/lockfile.py	Persists host_type and uses shared unique-key logic for lockfile indexing.
src/apm_cli/core/auth.py	Extends host classification + auth resolution cache keys to incorporate explicit host_type.
src/apm_cli/deps/host_backends.py	Threads host_type into backend selection/classification to route GitLab correctly.
src/apm_cli/deps/github_downloader.py	Ensures GitLab routing decisions consider host_type in host classification.
src/apm_cli/deps/github_downloader_validation.py	Aligns validation host classification with host_type.
src/apm_cli/deps/download_strategies.py	Uses per-dependency auth boundary for file downloads; improves fallback/error surfacing; adds helper error builders.
src/apm_cli/install/validation.py	Makes GitLab detection in validation respect explicit host_type.
tests/test_lockfile.py	Adds regression tests for host-aware lockfile keys and host_type round-trip.
tests/test_github_downloader.py	Updates/extends tests for token boundaries, GitLab routing via type: gitlab, and non-404 error surfacing.
tests/unit/deps/test_download_strategies_selection.py	Updates host mock to reflect generic-host token boundary behavior.
tests/unit/deps/test_download_strategies_phase3.py	Same as above for phase3 coverage.
packages/apm-guide/.apm/skills/apm-usage/dependencies.md	Documents type: gitlab and lockfile key migration behavior.
packages/apm-guide/.apm/skills/apm-usage/authentication.md	Documents GitLab host classification options including type: gitlab.
docs/src/content/docs/reference/lockfile-spec.md	Documents host_type and the new host-aware lockfile key rule.
docs/src/content/docs/getting-started/authentication.md	Adds type: gitlab as an explicit GitLab classification mechanism.
docs/src/content/docs/consumer/manage-dependencies.md	Documents object-form type: gitlab and provides examples/usage guidance.
CHANGELOG.md	Adds an Unreleased “Fixed” entry describing the multi-host lockfile + auth + error improvements.

Copilot's findings

• Files reviewed: 18/18 changed files
• Comments generated: 3

[danielmeppiel]

APM Review Panel: ship_with_followups

Host-aware lockfile identity stops same-path deps on different hosts from colliding in apm.lock, and aligns single-file download auth with clone auth -- migration-safe and additive.

cc @danielmeppiel @sergio-sisternes-epam -- a fresh advisory pass is ready for your review.

This PR delivers the board-approved #773 items 1-4 cleanly. The central move is a single shared build_dependency_unique_key() helper consumed by both DependencyReference and LockedDependency, so lockfile identity is host-qualified for non-default hosts (host/owner/repo) while github.com stays implicit and byte-stable. The deliberate split between host-qualified identity (get_unique_key) and the host-blind matching key (get_canonical_dependency_string, used for orphan/source/filesystem matching) is now explicit and documented at both call sites -- the deps/cli.py orphan-matching switch to the host-blind key is the correct companion fix.

Panel signals converge strongly. Supply-chain reads the host-qualified key as closing a real cross-host overwrite vector, and the file-download path now resolving through the same per-dependency token boundary as clone (_resolve_dep_token / _resolve_dep_auth_ctx) is a net secure-by-default tightening: generic hosts no longer receive APM-managed PATs or host-scoped credential-fill tokens over HTTP. Auth confirms host_type is threaded consistently and -- importantly -- folded into the resolve() cache key, so a hint cannot poison a sibling cache entry. Test coverage is TDD-first and traps every claimed behavior (collision, github.com preservation, host lowercasing, host_type round-trip, bespoke GitLab routing, non-404 surfacing, generic token boundary).

Dissent. None. The one posture shift worth a maintainer eye -- credential-fill tokens are no longer forwarded to generic hosts on the file-download path -- is an intentional, documented trade-off, not a contested call.

Aligned with: Multi-harness / multi-host: same owner/repo on two hosts now coexists instead of overwriting. Secure by default: file downloads inherit the clone token boundary; unknown type values fail closed.

Growth signal. Bespoke self-managed GitLab and Gitea/Gogs shops are a concrete adoption segment; type: gitlab is a clean, hostname-heuristic-free opt-in worth calling out in the release note.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	1	1	Shared key helper removes duplication; host-blind vs host-qualified split is clean and documented.
Auth Expert	0	1	0	host_type threaded everywhere and folded into the resolve cache key; GitLab token precedence correct; unknown types fail closed.
Supply Chain Security	0	1	0	Host-qualified identity closes a cross-host overwrite vector; generic-host downloads stop leaking managed/credential-fill tokens.
CLI Logging Expert	0	1	1	Download errors now name host + endpoint + status/exception; verbose breadcrumbs unified across hosts.
DevX UX Expert	0	1	0	Migration-safe and additive; determinism preserved; orphan/install UX unaffected by the new key.
Test Coverage Expert	0	0	1	TDD-first; every behavior trapped. Error-message URL parsing in tests is format-brittle.
Performance Expert	0	0	0	No added round-trips (resolve is cached); fail-fast on hard download errors trims wasted attempts.
OSS Growth Hacker	0	0	0	type: gitlab opens self-managed GitLab/Gitea adoption; amplify in release notes.
Doc Writer	0	0	0	lockfile-spec migration section, type: gitlab, CHANGELOG, and apm-usage guide all updated; no drift.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 4 follow-ups

1. [Supply Chain Security] Keep the credential-fill posture shift visible -- private generic-host (Gitea/Gogs) file reads via the HTTP download path now require a clone path or a host backend signal. -- The trade-off is documented, but it is the one behavior change an existing user could feel; a prominent migration line reduces surprise 401/empty-read reports.
2. [Python Architect] The two get_canonical_dependency_string() bodies (lockfile.py and reference.py) are now near-identical. -- Consider sharing the host-blind branch the same way build_dependency_unique_key() was extracted, before it drifts or trips the R0801 duplication gate later.
3. [Test Coverage / CLI Logging] Download-error tests parse URLs out of the message via split() + startswith("https://") + rstrip("."). -- Asserting on structured context (host, endpoint, status) instead of message scraping would survive future wording tweaks.
4. [Auth Expert] type currently accepts only gitlab and raises on anything else. -- Track the extension path (e.g. future gitea/bitbucket hints) so the fail-closed ValueError is a conscious gate rather than a future foot-gun.

Architecture

classDiagram
    class DependencyReference {
      +str repo_url
      +str host
      +str host_type
      +int port
      +get_unique_key() str
      +get_canonical_dependency_string() str
    }
    class LockedDependency {
      +str repo_url
      +str host
      +str host_type
      +get_unique_key() str
      +get_canonical_dependency_string() str
    }
    class build_dependency_unique_key {
      <<function>>
      +repo_url
      +host
      +source
      +virtual_path
      +registry_prefix
    }
    class AuthResolver {
      +classify_host(host, port, host_type) HostInfo
      +resolve(host, org, port, host_type) AuthContext
      +resolve_for_dep(dep_ref) AuthContext
    }
    DependencyReference --> build_dependency_unique_key : host-qualified key
    LockedDependency --> build_dependency_unique_key : host-qualified key
    AuthResolver ..> DependencyReference : reads host_type
    class DependencyReference:::core
    class build_dependency_unique_key:::core

Loading

flowchart TD
    manifest[apm.yml dependency] --> ref[DependencyReference]
    ref --> typehint{explicit type gitlab?}
    typehint -->|yes| gitlab[GitLab REST + GITLAB_APM_PAT chain]
    typehint -->|no| classify[AuthResolver.classify_host]
    ref --> keyfn[build_dependency_unique_key]
    keyfn --> hostq{host == github.com or unset?}
    hostq -->|yes| barekey[bare owner/repo lock key]
    hostq -->|no| qualkey[host/owner/repo lock key]
    barekey --> lock[apm.lock.yaml]
    qualkey --> lock
    ref --> orphan[get_canonical_dependency_string]
    orphan --> match[host-blind orphan / Source match]
    classify --> token[_resolve_dep_token / _resolve_dep_auth_ctx]
    token --> dl[single-file download = clone auth boundary]

Loading

Recommendation

Merge once you are comfortable with the documented credential-fill posture shift (follow-up 1) -- it is the only change an existing user could feel, and it is intentional and secure-by-default. Everything else is polish: track the shared-helper duplication (follow-up 2) and the test message-scraping brittleness (follow-up 3) as low-priority cleanups. CI is green, tests are TDD-first and comprehensive, and the migration is additive. Note this PR shares lockfile.py with #1738 (#1209); whichever lands second should rebase.

---

Full per-persona findings

Python Architect

• [recommended] Shared build_dependency_unique_key() is consumed by both DependencyReference.get_unique_key() and LockedDependency.get_unique_key() at src/apm_cli/models/dependency/reference.py and src/apm_cli/deps/lockfile.py.
  Good extraction: it removes the prior copy-paste of the key logic and makes the host-qualification rule single-sourced. The registry-prefix carve-out (proxy host is transport, not identity) is correctly documented and preserves the manifest/lockfile key correspondence.
  Suggested: none -- this is the right shape.
• [nit] The two get_canonical_dependency_string() implementations are now near-identical (local_path / virtual_path / repo_url) across lockfile.py and reference.py.
  Below the R0801 10-line threshold today, but a candidate for the same extraction treatment before it drifts.

Auth Expert

• [recommended] host_type is threaded through classify_host, resolve, resolve_for_dep, backend_for, and validation, and is folded into the resolve() cache key tuple at src/apm_cli/core/auth.py.
  Including host_type in the cache key is the correct call -- it prevents a hinted classification from poisoning an unhinted sibling entry for the same (host, port, org). GitLab token precedence (GITLAB_APM_PAT -> GITLAB_TOKEN -> credential fill) is preserved, and an unsupported type raises ValueError (fail closed).

Supply Chain Security

• [recommended] Host-qualified lockfile identity (host/owner/repo for non-default hosts) closes a cross-host key-overwrite vector, and download_github_file now resolves tokens via _resolve_dep_token / _resolve_dep_auth_ctx at src/apm_cli/deps/download_strategies.py.
  Net secure-by-default tightening: generic hosts no longer receive APM-managed PATs, and the previously-forwarded host-scoped credential-fill token is no longer attached on the HTTP file-download path. The trade-off (private generic-host file reads must use clone or a host backend signal) is documented in the PR body and docs. No blocking concern; flagged as a follow-up for user-facing migration visibility.

CLI Logging Expert

• [recommended] New _build_download_http_error / _build_download_network_error helpers surface host + endpoint + status (or exception type), and verbose breadcrumbs now log Contents API attempts uniformly (previously gated on not is_github_host).
  This directly satisfies the download-error-clarity item: a non-404 failure now names the failing endpoint instead of degrading to a generic missing-file message. ASCII-clean.
• [nit] Error strings end with a trailing period that tests strip (rstrip(".")); harmless but couples message punctuation to test parsing.

DevX UX Expert

• [recommended] Migration is additive and deterministic: existing github.com keys stay byte-stable, non-default hosts become visible only in their own key, and host casing is normalized so it cannot mint duplicate keys (tests/test_lockfile.py::test_get_unique_key_lowercases_non_default_host).
  Because orphan/Source matching uses the host-blind key, the install/orphan UX is unchanged for existing users -- the new key surface is invisible unless a non-default host is actually in play.

Test Coverage Expert

• [nit] Coverage is TDD-first and traps every claimed behavior: collision (test_add_dependency_keeps_same_repo_from_different_hosts), github.com preservation, host lowercasing, host_type round-trip, bespoke GitLab routing (test_object_form_type_gitlab_routes_bespoke_host_to_gitlab_api), non-404 surfacing, and the generic token boundary.
  The only soft spot: error-path tests reconstruct URLs by scraping the message string; asserting on structured fields would be more robust. (The ~16 local test_download_* failures are pre-existing network/env artifacts, green in CI -- not attributable to this PR.)

Performance Expert

No findings. The file-download path swaps a direct resolve() call for the cached _resolve_dep_token / _resolve_dep_auth_ctx boundary (no added round-trips), and narrowing raw-URL fallback to 404-only trims wasted attempts on hard failures. classify_host with host_type stays pure/cheap.

OSS Growth Hacker

No findings. Adoption note only: type: gitlab plus host-qualified identity makes APM viable for self-managed GitLab and multi-host Gitea/Gogs estates without hostname heuristics -- a concrete contributor/adopter segment worth amplifying in the release announcement.

Doc Writer

No findings. The lockfile-spec gains a "Lockfile identity keys" migration section, type: gitlab is documented across manage-dependencies.md, authentication.md, and the apm-usage guide, and the CHANGELOG carries a user-facing "Fixed" entry. No code/doc drift detected.

_{This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.}

[danielmeppiel]

Shepherd-driver: conflict resolved + panel follow-ups folded

This PR was CONFLICTING / DIRTY against main after #1732 (closes #1571)
landed on the shared deps/lockfile surface. Driven to mergeable + green CI
with the apm-review-panel follow-ups folded. No merge performed -- the
maintainer review gate stands.

Conflict resolution

Rebased onto current origin/main (1f6d5225, which includes #1732). The
stale merge-commit base (d410382f) that produced the conflict was replaced
with a clean linear history on top of current main. CHANGELOG.md keeps BOTH
the #1732 and the #1735 (closes #773) entries. Pushed with
--force-with-lease=danielmeppiel/773-multi-host-deps:d410382f (never bare
--force). The #773 board items 1-4 and the Phase-6 identity regression fixes
are preserved (host-blind get_canonical_dependency_string; host-qualified
get_unique_key; is_local local-detection; registry_prefix short-circuit).

Folded in this run

• (panel FU2, Python Architect) De-duplicated the near-identical host-blind
  canonical-string body into build_canonical_dependency_string(), consumed by
  both DependencyReference and LockedDependency -- mirrors the
  build_dependency_unique_key() extraction. Each caller passes its own
  is_local signal (is_local property vs source == "local") so the two
  identity models keep their distinct semantics and the Phase-6 fixes are not
  regressed. Resolved in 36511b5.
• (panel FU4, Auth) Documented the deliberate fail-closed gate in
  _parse_host_type so the future gitea/bitbucket extension path is a
  conscious choice, not a foot-gun. Resolved in 36511b5.
• (panel FU3, Test Coverage / CLI Logging) Replaced the brittle
  split()/startswith("https://")/rstrip(".") URL scraping in the
  generic-host download-error tests with a shared _error_url_components()
  helper that asserts on parsed (scheme, hostname, path) tuples, per the
  tests/ urllib.parse convention. Resolved in ac5a4c6.
• (panel FU1, Supply Chain Security) The credential-fill posture shift
  (private generic non-default-host file reads now require a clone/type signal)
  is documented prominently across CHANGELOG.md, manage-dependencies.md,
  authentication.md, and lockfile-spec.md -- preserved through the rebase.

Deferred

None within scope. The lockfile.py overlap with #1209 (#1738) is a cross-PR
MERGE-ORDER concern, not a fold: whichever lands second should rebase.

Lint contract

uv run --extra dev ruff check src/ tests/ and
uv run --extra dev ruff format --check src/ tests/ both silent; pylint R0801
10.00/10; scripts/lint-auth-signals.sh clean.

CI

All required checks green at ac5a4c6 (14 SUCCESS, 1 SKIPPED deploy);
0 CI recovery iterations needed.

Mergeability

gh pr view --json mergeable,mergeStateStatus reports
MERGEABLE / BLOCKED -- the conflict is resolved; BLOCKED is the
branch-protection review gate awaiting the maintainer, not a conflict or a
CI failure. Ready for maintainer review.