Skip to content

Fix JavaScript line continuation decoding in extracted messages - #1385

Open
theman6660 wants to merge 2 commits into
python-babel:masterfrom
theman6660:fix-js-line-continuations
Open

theman6660 wants to merge 2 commits into
python-babel:masterfrom
theman6660:fix-js-line-continuations

Conversation

@theman6660

Copy link
Copy Markdown

JavaScript source:

gettext("hello\
world");

This evaluates to helloworld, but Babel extracts a message containing an extra newline ("hello\nworld"). The initial regex substitution removes only the backslash and keeps the physical line break. It can also consume the second backslash of an escaped backslash before a template newline.

Handle line continuations during the escape scan, recognizing LF, CR, CRLF, U+2028, and U+2029 and emitting no character, as specified by ECMAScript string literal semantics. Enable DOTALL for template string tokenization so escaped LF/CR sequences remain inside one token. Source line counting stays independent of decoded message content.

Add 31 cases covering all three quote forms, all five line terminator sequences in decoding, public extraction with source line numbers, ordinary escapes, and escaped-backslash parity.

Validation on Linux/Python 3.12.14:

  • Against unchanged upstream: 25 new cases fail, 6 controls pass; all 31 pass with the fix.
  • Focused JavaScript lexer/extraction tests: 51 passed.
  • Full python -m pytest -q, after importing the repository-pinned CLDR 48.2 and installing the missing tzdata package: 7862 passed, 7 skipped, 2 xfailed. One pytest deprecation warning points to an existing generator-based parametrization in test_toml_config.py.
  • pre-commit run --files babel/messages/jslexer.py tests/messages/test_js_line_continuations.py: passed.
  • Node.js independently confirms the runtime string values for 16 continuation/parity cases.
  • git diff --check: passed; both remote files match the tested local files byte-for-byte.

Prepared with OpenAI Codex assistance. The reproduction uses the public extraction API; no production usage or human review is claimed. Other Python versions and upstream CI remain unverified locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants