Pre-flight token estimator wildly overcounts, triggering premature compaction (~150x actual content size) #155

Bug: Pre-flight token estimator wildly overcounts, triggering premature compaction (~150x actual content size)

Version

CyberStrike 1.1.16 (npm), Windows 10

Setup

A custom OpenAI-compatible provider added via cyberstrike provider add --url <endpoint>/v1 --key <key>. The provider correctly advertises context_length: 500000 for the model in use (verified via GET /v1/models), and this is reflected correctly in the session's actual contextLimit:

service=session.prompt input=54860 output=474 cacheRead=1152 cacheWrite=0 total=56486 contextLimit=468000 utilization=12 context usage

What happened

A few turns later, in the same session, a normal turn (reading 3 small PNG mock files via the read tool — each returned a 23-byte placeholder, not inlined image bytes — plus a handful of small text/JSON file reads and one bash command) triggered an unexpected pre-flight overflow and forced auto-compaction:

service=session.prompt estimated=2484526 limit=468000 utilization=531 pre-flight overflow detected, triggering compaction
service=session.compaction original=7 limited=2 budget=374400 estimated=263 compaction input limited

estimated=2484526 is ~531% of the 468,000 token limit.

Actual content size (measured directly from the session DB)

Summing every tool-call output in that turn (message parts table):

• memory_context: 69 chars
• glob (project files): 6,635 chars
• bash (git log/status/diff): 5,052 chars
• glob (.impeccable dir): 1,166 chars
• read PRODUCT.md: 5,685 chars
• read session-*.md (limit 200 lines): 16,521 chars
• glob (images): 568 chars
• read a4dbcfee.answer.json: 304 chars
• read a4dbcfee.state.json: 255 chars
• read decision/payload.json: 8,007 chars
• read a4dbcfee.log: 165 chars
• read SKILL.md (limit 250 lines): 12,478 chars
• glob (desk files): 1,248 chars
• bash (impeccable.cmd context): 85 chars
• read 01-split.png: 23 chars
• read 02-beside.png: 23 chars
• read 03-fullslip.png: 23 chars
• read assigned.prompt.txt: 2,314 chars
• read a4dbcfee.flip.json: 195 chars

Total: ~60,816 characters ≈ 15,000–20,000 tokens at any reasonable chars-per-token ratio.

Additionally, at the moment of the overflow check, the lazy tool registry reported:

service=tool.lazy-registry totalTools=0 loadedTools=0 lazy tool registry initialized

so no tool-schema payload was in flight either.

There is no log output breaking down what the estimated=2484526 figure is actually counting (message history, tool schemas, images, cache, etc.), so I can't pinpoint the exact source from the CLI logs alone — but it's off from the measurable real content by roughly 150x, and the 3 image reads (the only binary/multimodal content in the turn) each produced only a 23-byte tool result, so they can't be the explanation either.

Impact

Sessions using a large-context custom provider (500K) get force-compacted at effectively ~12% real utilization, defeating the purpose of the larger context window and causing unnecessary loss of conversation history/state.

Ask

• Could the pre-flight estimator log a breakdown (history tokens / tool-schema tokens / attachment tokens) when it triggers overflow, to make this diagnosable?
• Is there a known issue with the estimator over-counting for custom (non-catalog) OpenAI-compatible providers specifically?

Happy to provide the full session DB export or additional log excerpts if useful.