query: build models once per release, read snapshots in place

[smypmsa]

A build forked one query worker per query, and each worker copied every snapshot into memory and rebuilt every model before running its one SELECT. On a project with about 4M event rows behind five models and 19 queries, build took 345 s, and on a busy machine single queries ran past the 60 s deadline, so the build failed outright.

What changes

• One worker per build. The worker loads the snapshots, shuts external access and builds the models once, then runs each query in turn and reports one line per query.
• Snapshots are read in place. Each is a view over its parquet file, so a model scans only the columns it uses instead of a full copy held in memory. That copy is what spilled past the 1 GB cap and ran everything at disk speed.
• Deadlines are per step. Loading and models get 60 s, then each query gets its own 60 s. A timeout or failure names the model or query it belongs to; query deadline exceeded used to name neither.
• CHAINPLOT_QUERY_MEMORY_LIMIT works. The worker's environment is stripped to PATH/HOME/LANG, so the override documented in capabilities.md never reached it. The parent now reads and validates it and passes it in the request; a value that is not a size is refused by name. The default stays 1 GB.

runQuery is a batch of one, so query, test and describe behave as before.

Why the sandbox holds

• File access is allowed for exactly the declared snapshot paths (allowed_paths, not their directories), set before external access is shut. DuckDB refuses to widen that list or to re-enable access afterwards. A file beside a snapshot, /etc/passwd, and any other path stay unreadable; there are tests for each.
• Sharing one session gives one query nothing over another: each is still admitted only as a single SELECT, which cannot change the session the next one runs in, each keeps its own row limit, and all of them come from the same recipe.
• docs/security.md and docs/capabilities.md describe the batch, the exact-path allowlist, the per-step deadlines and the memory override.

Testing

• pnpm test: 381 passed; the 3 live tests skipped. The first commit alone passes 377.
• New tests/query/batch.test.ts (10): models built once (a random() model reads the same in two queries), failures and timeouts name the model or query, per-query row limits, external access shut for every query in a batch, a parquet beside a snapshot refused, the memory override reaching the worker and a bad value refused.
• The 4M-row project above: 345 s on main, 12 s on this branch at the default 1 GB, 10 s at 2GB. All 19 result files are identical to main's.

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 2f95f7b6-0301-4c2a-a041-a91a75f5c44a

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}