Skip to content

Accept normalized UTF-8 encoding declarations with a BOM - #1371

Open
harbinresearcher wants to merge 1 commit into
python-babel:masterfrom
harbinresearcher:codex/bom-encoding-aliases
Open

harbinresearcher wants to merge 1 commit into
python-babel:masterfrom
harbinresearcher:codex/bom-encoding-aliases

Conversation

@harbinresearcher

Copy link
Copy Markdown

parse_encoding rejects a UTF-8 BOM when the encoding declaration uses UTF-8, utf_8, UTF_8 or utf-8-sig, although Python accepts these declarations. Consequently, message extraction fails before tokenization on valid source files.

Normalize case and underscores only for the BOM compatibility check, accepting utf-8 and utf-8-* as Python does. Keep the existing utf-8 return value, preserve the original declaration in error messages and leave non-BOM behavior unchanged. General codec lookup is intentionally avoided: Python rejects BOM declarations such as utf8 and cp65001 even though they identify UTF-8 codecs.

Tests cover both declaration lines, valid aliases, incompatible declarations and stream-position restoration on success and failure. Each accepted example is first compiled by Python to check that it is valid source.

Validation on Windows Python 3.12.14:

  • Original utility/extraction baseline: 24 passed.
  • Regression tests before the fix: 8 failed, 8 passed.
  • Fixed utility/extraction selection: 37 passed, independently rerun by a separate reviewer.
  • Full suite with checksum-verified CLDR 48.2 imported by the repository script: 7841 passed, 10 skipped, 2 xfailed. One unrelated pytest deprecation warning.
  • A direct extract_python smoke check now extracts a message from a BOM-prefixed source using UTF_8.
  • Ruff and every applicable repository-pinned pre-commit hook pass on the two changed files.

Other Python versions and operating systems have not been tested locally.

AI assistance: this is an autonomous, user-authorized Codex contribution. Codex investigated, implemented and tested the change; a separate agent reviewed it and reran the targeted tests. No manual human review or production incident is claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant