Feat/lint cache and stdin

[AlexCannonball]

Summary:

Addresses #443

The majority of time is spent parsing

This PR introduces support for linting from stdin and significantly improves performance for linting by implementing an in-memory AST cache.

Key Improvements:

Performance (~7.2x faster): By caching the parsed AST and raw file content, we eliminated redundant disk I/O and parsing cycles.

Benchmark (googleapis/google/ads, WSL + Ubuntu): Master: 54s → This PR: 7.5s.

Benchmarking script

#!/bin/bash
BINARY=$1
# We use the relative path to the google folder
TARGET_DIR="./ads/"

echo "Benchmarking: $BINARY"

for i in {1..3}; do
  echo -n "Run $i: "
  # We use 'time' and redirect output to /dev/null to see only the speed
  { time $BINARY lint $TARGET_DIR > /dev/null 2>&1; } 2>&1 | grep real
done

Virtual File System (VFS): Implemented a thread-safe, memory-first VFS wrapper (file.Open, file.ReadFile). This allows rules to access stdin content or "shadow" disk files without physical file handles.

LSP Ready: The architecture is now thread-safe (using sync.RWMutex) and supports virtual filenames, making it good for future Language Server Protocol (LSP) integrations.

MCP can potentially benefit from piping file buffers directly into the linter. This enables faster, more secure "lint-as-you-type" or "AI-fix" workflows without the overhead of temporary file management.

WASM Readiness:
This refactoring prepares protolint for high-performance WebAssembly (WASI) execution. By implementing a memory-first VFS and single-pass parsing, we've eliminated the costly I/O overhead typical in WASM runtimes.

I've also tested it in my LSP draft. This optimization brings WASM linting time for a ~600 line file down from 250+ms to ~54ms, making real-time IDE integration viable.

Robust Validation: Implemented fail-fast guards to prevent incompatible flag combinations (like -fix with stdin).

Technical Details:

• Changed ProtoFile to a pointer-based receiver to maintain state across the linter lifecycle.
• Added cachedData and cachedProto fields to avoid re-parsing for each of the rules.

[AlexCannonball]

Hi @yoheimuta! Just gently pinging you on this PR. It's fully ready from my side, the benchmark shows a ~7.2x performance boost along with stdin support.

I currently have limited capacity to work on further changes, so I'd appreciate it if you could review and merge it as is if everything looks good. If any minor tweaks are needed, feel free to apply them directly! Thanks!

[yoheimuta]

Thank you for the ping and the performance improvement!

[yoheimuta]

Could we scope these virtual entries to the stdin input rather than keeping them globally?

For example:

cat other.proto | protolint lint -stdin_filename=foo.proto - foo.proto

Stdin registers its content under "foo.proto". When the real foo.proto is subsequently linted, ProtoFile.Parse reads its AST directly from disk, but MAX_LINE_LENGTH calls file.Open("foo.proto") and receives the stdin content instead. This means different rules can inspect different content for the same file.

The entries also remain after linting, potentially affecting later calls in the same process. Could we prevent this collision and add a regression test for stdin and a physical file sharing the same display path?

[AlexCannonball]

cat other.proto | protolint lint -stdin_filename=foo.proto - foo.proto

Thank you for the catch! You are completely right—allowing mixed inputs creates unnecessary ambiguity. I realized that the actual utility of mixing stdin with physical files is close to zero for users. Restricting this cuts out a whole layer of bugs and complexity.

I have fixed this in a6a49b2. The tests have been updated accordingly, and they now act as the regression test you've mentioned above.

---

Could we scope these virtual entries to the stdin input rather than keeping them globally?

I want to make sure I fully understand your concern here.

The primary role of the VFS layer here is not just disk optimization, but providing a unified file system abstraction. It potentially may be a long-term architectural benefit for protolint:

1. Unified Source Resolution: It allows the AST caching mechanism to seamlessly resolve and parse files regardless of the source.
2. Testability: Much like popular VFS packages in the Go ecosystem (like afero), having an in-memory filesystem abstraction makes unit tests easier. For example, we can test rule behaviors entirely in-memory without the need to generate, maintain, and clean up physical .proto files.

Since protolint currently operates in CLI mode, the process memory is naturally fresh on every run, meaning the global state is guaranteed to be clean between different linting sessions. If the input validation fix above isn't enough, could you please elaborate a bit more on the specific risks or execution environments you'd like to address?

[AlexCannonball]

Hi @yoheimuta

I went ahead with your suggestion and scoped the virtual entries strictly to the stdin input, leveraging the single-input restriction introduced in a6a49b2.

As a major side benefit of this isolation, I was able to easily unlock the -fix functionality for stdin and pipe the mutated result to stdout (which is a massive win for IDE/LSP extensions).

Could you please take a look at the refactored approach in ef88496?

[yoheimuta]

@AlexCannonball
Thanks! Rejecting mixed inputs resolves the CLI example I reported. The remaining concern is the existing lib.Lint() API, which allows multiple lint calls within one process.

Could we address the concern and add a regression test with two sequential lint calls?

[AlexCannonball]

@yoheimuta Thank you! You are completely right about the long-running process risks via the lib.Lint() API.

I have addressed this concern in e9cdb77 by enforcing ResetVFS().

I also added a test that executes two sequential lib.Lint() calls (one for stdin and one for a physical file sharing the exact same relative path) to guarantee that no collisions can occur.

The test fails like this when removing the fix:

Could you please take a look?