UrlChecker can't see private on-prem repos: release notes and version diffs silently dropped

[kubukoz]

Summary

On a forge whose web UI is cookie-authenticated (GitHub Enterprise, private GitLab), UrlChecker.exists can't tell a repo that exists from one that doesn't — both answer 302 → /login. So every candidate URL is judged absent, and a dependency hosted there gets no Release Notes, no Changelog, no Version Diff, and no repo link, even with a valid <scm><url> in its POM.

Proposal: when the URL's host belongs to a configured forge, check existence via that forge's API, while still displaying the web URL.

The problem

HEAD against a GHE web URL, for a repo that exists and one that doesn't:

probe	real repo	nonexistent repo
web, no auth	302 → /login	302 → /login
web, Authorization: Basic … (BasicAuthAlg)	302	302
web, Authorization: Bearer … (GitHubAuthAlg)	302	302
API GET /api/v3/repos/org/repo	200	404

Not an auth gap — the web UI ignores Authorization headers and wants a session cookie. In PAT mode BasicAuthAlg.authenticateApi already attaches credentials to every request, and it changes nothing. The endpoint is wrong, not the header.

Nothing renders at all because two things fail:

1. DependencyMetadata.filterUrls (coursier/DependencyMetadata.scala:40) discards failing URLs before repoUrl is derived from them. With scmUrl gone, forgeRepo is None — candidates are never generated, and artifactIdToUrl (nurture/NurtureAlg.scala:209) loses the plain repo link too.
2. findUpdateInfoUrls ends in .filterA(u => urlChecker.exists(u.url)) (nurture/UpdateInfoUrlFinder.scala:49), covering all 30 candidates per dependency — including the 3 version diffs, which are otherwise pure URL construction and always valid.

exists requires exactly Status.Ok (util/UrlChecker.scala:56) on a disableFollowRedirect client (application/Context.scala:109), so 302 is a definitive no.

Relationship to #2363

Same behaviour, opposite side: there redirects were followed, so 302 → /login became 200 and every changelog URL looked valid. 513f4cf fixed that by not following redirects, turning the false positive into today's silent false negative.

I don't think #2363 should be reverted — following and not following redirects are two horns of one dilemma, and neither works while the probe targets an endpoint that returns 302 either way. Asking the API escapes it.

Proposal

In UrlChecker, when ForgeType.fromRepoUrl classifies the URL as the configured forge, translate to the API equivalent, probe that, report the result for the original URL. Fall back to the existing raw HEAD otherwise, so public github.com and unrecognised hosts are unaffected.

UrlChecker.create already gets Config and an auth function, so the seam exists. All four candidate kinds have an API analogue — verified against a real GHE instance:

candidate (web)	API probe	exists / absent
/org/repo	/repos/org/repo	200 / 404
/org/repo/blob/<branch>/CHANGELOG.md	/repos/org/repo/contents/CHANGELOG.md	200 / 404
/org/repo/releases/tag/v1.2.3	/repos/org/repo/releases/tags/v1.2.3	200 / 404
/org/repo/compare/v1.2.2...v1.2.3	/repos/org/repo/compare/v1.2.2...v1.2.3	200 / 404

Design questions I'd want input on first:

• Where does translation live? A new ForgeType member beside diffs/files, or reuse ForgeApiAlg (which already has a per-forge getRepo)? The latter is better factored, but UrlChecker is built earlier and more cheaply.
• GitHub App tokens. GitHubAuthAlg.authenticateApi (forge/github/GitHubAuthAlg.scala:66-75) only attaches a token for repos the App is installed on — a dependency's repo usually won't match, so the API probe would be unauthenticated. PAT mode is unaffected. Widen it, or leave App mode as-is?
• Cache key. The cache is keyed on url.renderString (the web URL) while the API URL is what's fetched — worth being deliberate.

Happy to implement if the shape sounds right.

Not in scope

ForgeType.GitHub.files hardcodes master (forge/ForgeType.scala:108), so changelog candidates 404 on main-default repos even once this is fixed. That's #3043, already open and independent.

---

🤖 Investigated with Claude Code (Claude Opus 5). The probe tables were verified against a real GHE instance; hostnames and repo paths are scrubbed.

[kubukoz]

Also wondering about possibly having multiple forges in an org. Like, even if you're on a GHE, maybe your dep comes from your corporate gitlab/bitbucket. That seems like an additional level of complexity that could be ignored until later, though (personally I don't have a stake here as I only have GHE+github.com)

[exoego]

PR welcome 🙏