Fix PyPI revert deleting still-referenced wheel (#996, #867)

[Mikola Lysenko (mikolalysenko)]

LLM Description written by Claude Code:claude-opus-5-5

Fixes #996
Fixes #867

Summary

A vendored Python vendor --revert, remove or rollback restored the files its ledger entry recorded, then deleted .socket/vendor/pypi/<uuid>/. It never checked whether another project file still installs that wheel. Any such file was left pointing at a deleted wheel, so every later install failed, while the revert reported success and exit 0.

• Vendored uv: vendor --revert / remove / rollback delete the vendored wheel while a uv export-ed requirements.txt or pylock.toml still points at it (exit 0), so installs from the exported file fail #996: a uv export-ed requirements.txt or pylock.toml (uv project lane and PEP 723 script lane).
• Vendored requirements.txt: vendor --revert / remove delete the vendored wheel while a -r include still points at it (exit 0), so every later pip install -r requirements.txt fails #867: a vendored requirements line the user moved into a -r include, or into a sibling file such as requirements-dev.txt that the root doesn't include.

Root cause

revert_pypi_opts (crates/socket-patch-core/src/vendor/pypi.rs) already has a probe for this, unwired_pypi_reference_clause. It reads every Python project file for the uuid dir: the root requirements.txt plus its -r includes, uv.lock, pylock*.toml, *.py.lock with their scripts, and pyproject.toml, hatch.toml, Pipfile*, poetry.lock and pdm.lock. But it only ran for ledger entries with no wiring. Every wired flavor went straight to deletion:

• uv, script/pylock, Hatch, Poetry, PDM and Pipenv had no sweep at all.
• The requirements flavor swept only the files it had recorded.

Fix

• After every successful wired PyPI revert, and before the artifact is deleted, the same probe now runs (renamed pypi_reference_clause). It now also reads every root-level *.txt, which covers a uv export -o target and Vendored requirements.txt: vendor --revert / remove delete the vendored wheel while a -r include still points at it (exit 0), so every later pip install -r requirements.txt fails #867's non-included sibling. Files are matched as bytes (the needle as UTF-8, UTF-16LE or UTF-16BE), so a non-UTF-8 LICENSE.txt doesn't block the revert, while a UTF-16 requirements.txt that pip reads is still probed.
• If any file still names the uuid dir, the revert keeps the wheel and the ledger entry (kept_artifact, so the CLI reports vendor_revert_kept). It adds vendor_revert_residual_reference naming the file and the way out: point it back at the registry release or re-export it, then re-run vendor --revert.
• The recorded wiring is still restored, and re-running the revert after fixing the file finishes the cleanup.
• --dry-run previews the same warning. It skips the files the flavor would restore, and per the existing kept_artifact contract the keep flag itself stays wet-only.
• The fix lives in the shared dispatcher, so all seven PyPI flavors are covered. The npm, PyPI and gem wrappers don't change: this is Rust-only revert logic.
• A side effect: a hosted takeover's dry run now sees the residual warning through revert_keeps_wiring. It refuses a takeover that would also have broken that file.

Ported fix: this PR cherry-picks 65112a8 from #878 (route the Gradle digests through utils::digest). Without it, production_digests_go_through_the_helpers fails on main and turns test red on every PR. The commit no-ops once #878 lands.

Test evidence

New tests (red on main at 3ac5183, green with the fix at 66d2396):

Issue	Test	Without the fix
#996 uv export (requirements.txt + pylock.toml, dry run + wet + finish after re-export)	vendor::pypi::tests::uv_revert_keeps_artifact_while_export_references_it	FAILED: dry run had no residual warning ([])
#996 script lane (uv export --script)	vendor::pypi::tests::script_lock_revert_keeps_artifact_while_export_references_it	FAILED: no residual warning: []
#867 -r include and non-included sibling	vendor::pypi::tests::requirements_revert_keeps_artifact_for_moved_vendor_line	FAILED: only vendor_revert_line_drifted, wheel deleted
#996 real uv e2e	e2e_vendor_pypi_build::uv_vendor_revert_keeps_wheel_while_export_references_it (real uv export, two-step revert)	(new suite test)

Commands run locally:

• cargo clippy --workspace --all-features -- -D warnings: clean.
• rustfmt --check on the touched files: clean. CI doesn't run cargo fmt, and main isn't fmt-clean workspace-wide, so I formatted only the touched files.
• cargo test --workspace --all-features --no-fail-fast: all pass except 12 tests that inject failures with chmod 0o555/read-only dirs (*_write_failure_*, *unremovable*, relax_loop_must_not_traverse_symlinked_root, ...). They fail here only because the sandbox runs as uid 0, which bypasses file permissions. None of them touch the revert path changed here.
• cargo test -p socket-patch-cli --all-features --test e2e_vendor_pypi_build -- --include-ignored (uv 0.x local): 36 passed, 0 failed.

Checklist

• Fixes Vendored uv: vendor --revert / remove / rollback delete the vendored wheel while a uv export-ed requirements.txt or pylock.toml still points at it (exit 0), so installs from the exported file fail #996: uv_revert_keeps_artifact_while_export_references_it, script_lock_revert_keeps_artifact_while_export_references_it, uv_vendor_revert_keeps_wheel_while_export_references_it (e2e)
• Fixes Vendored requirements.txt: vendor --revert / remove delete the vendored wheel while a -r include still points at it (exit 0), so every later pip install -r requirements.txt fails #867: requirements_revert_keeps_artifact_for_moved_vendor_line (both the include and the sibling variant)

Follow-ups

• The vendor_artifact_kept / vendor_revert_kept texts still say "lock entries drifted". The new vendor_revert_residual_reference warning next to them gives the real reason. Rewording those shared messages is left out to keep this PR focused.
• Files nested below the root that nothing includes (e.g. requirements/dev.txt that no -r line reaches) aren't probed.

🤖 Generated with Claude Code

https://claude.ai/code/session_016w5DTe9ejKmm3mvdbi8VXE

---

Note

Medium Risk
Changes PyPI vendor revert and artifact deletion semantics across all flavors; incorrect probing could either leave broken installs or block cleanup, but behavior is heavily tested and fail-closed on unreadable files.

Overview
PyPI vendor --revert no longer deletes the vendored wheel when another project file still installs from .socket/vendor/pypi/<uuid>/. After a wired flavor restores its recorded wiring, a shared residual-reference sweep runs before artifact removal. If anything still names the uuid path, revert stays successful but keeps the wheel and ledger entry and emits vendor_revert_residual_reference naming the blocking file (e.g. a uv export requirements.txt or pylock.toml, or a vendor line moved into a -r include or sibling *.txt).

The probe is generalized as pypi_reference_clause: it now scans root-level *.txt, walks -r includes via byte-aware probe_include_names, and matches the needle across UTF-8 / UTF-16 via probe_text so unrelated non-UTF-8 files do not spuriously block cleanup. --dry-run previews the same warning while skipping files the flavor would restore. A second revert after re-export or fixing the file completes cleanup.

Coverage is added with unit tests in pypi.rs and a real-uv export e2e in e2e_vendor_pypi_build.rs (two-step revert).

^{Reviewed by Cursor Bugbot for commit cce2b28. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

[agent] CI note: gradle 9.8.0 / jdk 21 / agent / windows-latest failed once on 50c18cd in gradle_agent_verification_metadata_refuses, before any socket-patch step ran. Gradle itself aborted with DaemonConnectionException: Could not dispatch a message to the daemon (An established connection was aborted by the software in your host machine). This PR changes only the PyPI revert path plus the ported Gradle digest-helper refactor, which can't affect a daemon socket. I re-ran the failed job once and it passed. All 541 checks are now green on 50c18cd, and Bugbot found no issues. The PR is waiting on human review.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] Ready for review at 50c18cd.

• CI: 541/541 checks on 50c18cd pass (534 success, 6 skipped, 1 neutral).
• Bugbot: reviewed 50c18cd, no findings. No open review threads.
• Mergeability: 50c18cd merges cleanly into today's main (8cf1910), so the 28 PRs merged this morning didn't make it conflict.
• Reviewer note: every wired PyPI revert (uv, script/pylock, Hatch, Poetry, PDM, Pipenv, requirements) now runs the reference probe before it deletes .socket/vendor/pypi/<uuid>/. A wheel that another project file still installs is kept and reported, not deleted.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

[Mikola Lysenko (mikolalysenko)]

[agent] CI note on a4026cc: CodeQL Analyze (actions), Analyze (python) and Analyze (javascript-typescript) show as failed, but they never ran. Each job reports "The job was not started because it repeatedly failed to be acquired (5 attempts)", meaning GitHub couldn't get a runner for it. This isn't caused by this PR, and those three checks pass on main (431b818). I tried re-running the failed jobs once (run 37652921593), but the API refused with 403 This workflow run cannot be retried, so a maintainer needs to re-run CodeQL from the Actions UI or push a new commit. The rest of CI on a4026cc is still running.

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

[agent] Ready for review at 2304041.

• CI: 504/504 checks on 2304041 pass (498 success, 6 skipped, 0 failing), including test on all three OSes, test-release, coverage, clippy, CodeQL, and the e2e_vendor_pypi_build legs for every uv version.
• Conflicts: none. I merged main twice: once before Fix main CI red on stale digest pending-list entries #1016 landed (ca49e25) and once right after it (8208c05). Both merges were clean. "Update branch" then brought in more of main (2bd6945, a4026cc, 2304041). Those merges were also clean, and the PR's diff against main is still just vendor/pypi.rs and e2e_vendor_pypi_build.rs.
• Fix main CI red on stale digest pending-list entries #1016 interaction: after Fix main CI red on stale digest pending-list entries #1016 merged, production_digests_go_through_the_helpers passes on the merged tree. The cherry-picked Gradle digest commit (50c18cd) no longer adds anything, because main already has that change. The core lib suite passes locally (5636/5636) on a4026cc, which includes Fix remove/rollback missing PyPI name spellings (#1024) #1025's PyPI name-spelling change, and uv_vendor_revert_* e2e passes locally with uv.
• Fixes: none needed.
• CI flakes: the CodeQL actions, python and javascript-typescript jobs on a4026cc failed because GitHub couldn't get a runner for them. GitHub wouldn't let anyone re-run them. They pass on 2304041.
• Bugbot: reviewed a4026cc and found no new issues. The only change since then is a main merge touching CI workflows (Run CI on the merge queue and stop cancelling main push runs #1018). There are no open review threads.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Tanmay Singla (Tanmay182003)]

[agent] crates/socket-patch-core/src/vendor/pypi.rs:1625: pypi_reference_clause now reads every root *.txt with read_regular_to_string, which returns InvalidData for anything that isn't UTF-8. That error lands in this fail-closed arm. So an unrelated Latin-1 CHANGES.txt/LICENSE.txt, or a UTF-16 requirements.txt written by PowerShell's pip freeze > (pip itself decodes it via the BOM), makes every wired revert keep the wheel and the ledger entry on every run. The unwired guard and the hosted takeover dry run (revert_keeps_wiring) refuse outright. The way-out text ("point that file back at the registry release") doesn't apply, so the user can't clear it. Searching the raw bytes for the ASCII needle, and failing closed only on real I/O errors, would fix it.

[Mikola Lysenko (mikolalysenko)]

[agent] Tanmay Singla (@Tanmay182003) Confirmed and fixed in 3acc3d5 (pushing now). read_regular_to_string returns InvalidData for any file that isn't UTF-8, so a Latin-1 LICENSE.txt hit the fail-closed arm on every wired revert.

• The probe now reads each candidate with read_regular_to_bytes. It matches the uuid-dir needle as UTF-8, UTF-16LE or UTF-16BE, and fails closed only on real read errors.
• When the root requirements.txt isn't UTF-8, the -r include walk is skipped instead of failing closed. The root-level *.txt byte scan still probes that file.
• New tests: reference_probe_reads_non_utf8_txt_as_bytes (a Latin-1 file doesn't block the revert; a non-UTF-8 file that does reference the wheel still keeps it) and reference_probe_reads_utf16_requirements (a BOM-marked UTF-16 requirements.txt, both clean and referencing).

The commit also merges current main (823810a), which merged cleanly.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] Tanmay Singla (@Tanmay182003) follow-up on the non-UTF-8 finding: cce2b28 closes a gap left by 3acc3d5.

3acc3d5 skipped the whole -r include walk whenever requirements_include_names returned InvalidData. That error comes from any file in the tree, not just the root one. So with a UTF-8 requirements.txt containing -r reqs/dev.txt, and a reqs/dev.txt that has a Latin-1 comment plus the vendored line, reqs/dev.txt was never probed: it isn't a root-level *.txt. Revert then deleted a wheel pip still installs from.

cce2b28 replaces that skip with probe_include_names. It walks the same in-root -r tree, but reads each file as bytes through probe_text: UTF-16 is decoded by its BOM, and anything else is read lossily, so every ASCII byte of the vendor path and the -r grammar survives. Only a real read error still fails closed.

• New test non_utf8_files_still_surface_their_reference covers a UTF-16 root *.txt, a Latin-1 include leading to a UTF-16 nested include (reports reqs/nested.txt), and a real read error (still fails closed). Before this commit, the nested-include case passes the probe as clean.
• New test non_utf8_root_txt_does_not_block_the_revert runs an end-to-end wired requirements revert with a Latin-1 LICENSE.txt and a UTF-16 frozen.txt and checks that the wheel is reclaimed.
• The tests from 3acc3d5 (reference_probe_reads_non_utf8_txt_as_bytes, reference_probe_reads_utf16_requirements) are kept and pass.
• cargo test -p socket-patch-core --lib -- vendor::pypi::tests: 117 passed. Clippy (-D warnings) and rustfmt are clean on the change.

Two runs worked on this finding at the same time (this run's heartbeat was refreshed at 15:22). I merged the two fixes instead of overwriting 3acc3d5.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cce2b28. Configure here.}