Make indentation scanning stateful

[moon-jam]

Summary

Refactor the indentation scanner into language-specific rules and carry block-comment state across lines.

The scanner previously treated each line as fresh code. As a result, an opening delimiter inside a continued /* ... */ comment could add an incorrect indentation level.

Scanning now resumes after */, while the existing indentation behavior for comments, single-line quoted strings, preprocessor directives, and matching delimiters remains unchanged.

This refactor also provides a cleaner base for the optional limited editor mode discussed in #73. It does not implement the limited editor mode itself.

Known Limitations

As noted in #61, multiline strings and JavaScript regular expression literals remain outside the scanner's scope. JavaScript template interpolation is also not parsed.

This change carries block-comment state only, which fixes the cross-line indentation problem described in #61 without expanding the scanner into a general-purpose tokenizer.

Test plan

• node --test tests/browser/editor.test.js: covers cross-line block comments, closing transitions, and language-specific behavior in JavaScript, C, C++, and Java
• BROWSER_CHECK_AGENT=offline npm run browser-check: verifies Enter handling in the real textarea, caret placement, highlight-overlay and line-number synchronization, undo/redo, and content retention across language switches

Closes #61.
Related to #73.

---

Summary by cubic

Makes indentation scanning stateful via a new language-aware tokenizer so an opening delimiter inside a continued /* ... */ comment no longer adds an incorrect indentation level. The scanner now carries block-comment and string context across lines, resumes after closing delimiters, and keeps existing behavior for line comments, preprocessor directives, and matching delimiters. Refactors scanning out of the editor into web/tokenizer.js with per-language rules and exposed token ranges. Adds browser and unit tests covering cross-line comments, multiline strings, JavaScript regex literals, and language-specific edge cases. Closes #61; related to #73.

^{Written for commit a1ff7c9. Summary will update on new commits.}

[moon-jam]

Thanks for your careful review. Carrying block-comment state across lines fixes the reported comment case, but it also makes a false /* inside a multiline string, JavaScript regular expression literal, C++ raw string, or Java text block affect later lines. Fixing these cases one by one in editor.js would add more language-specific logic while highlighting maintains its own rules.

I’m considering a shared tokenizer that tracks comments, strings, and state across lines for both indentation and highlighting. It could also provide the comment/string awareness needed for #85, though bracket auto-close would remain a separate change. Do you think that fits the scope of this PR, or would you prefer a narrower fix here and shared logic in a follow-up?

[jserv]

Do you think that fits the scope of this PR, or would you prefer a narrower fix here and shared logic in a follow-up?

Submit commits within this pull request.

[moon-jam]

I've updated the implementation to move comment, string, and regexp classification into a separate tokenizer with language-specific rules. The indentation logic now uses these ranges, with regression coverage for multiline templates, C++ raw strings, Java text blocks, and Python triple-quoted strings.

The tokenizer returns source ranges rather than HTML. This should let highlighting reuse the same classification while keeping its keyword tables and rendering logic. I'd like to keep this PR focused on indentation and handle that integration in a follow-up PR.

JavaScript regexp detection still relies on preceding-code context and has known gaps. For example, division after a private property named #return can be misclassified, and a regexp after a function declaration can be missed. If that regexp contains /*, the mistake can still affect indentation on later lines.

We can continue improving coverage by addressing these cases individually and adding regression tests. However, distinguishing regexp literals from division in more contexts would also make the scanner track more JavaScript syntax, increasing its complexity and maintenance cost.

Would you prefer that I keep extending the current scanner to cover these cases as thoroughly as possible, or evaluate an existing parser such as Lezer? Using a parser would reduce the syntax-handling logic we maintain ourselves, though it would also introduce additional dependencies.

[jserv]

Would you prefer that I keep extending the current scanner to cover these cases as thoroughly as possible, or evaluate an existing parser such as Lezer? Using a parser would reduce the syntax-handling logic we maintain ourselves, though it would also introduce additional dependencies.

PR #71 is already working on tree-sitter-based parsers, which offer a more flexible approach to parsing different language constructs.

[moon-jam]

Thanks for pointing that out. I'll check #71 and follow up on whether its parsers can support indentation.

[jserv]

I'll check #71 and follow up on whether its parsers can support indentation.

Good. Use tree-sitter to parse the specific operations.