fix(mothership): explain withheld code and workflow results and bound workflow block logs by waleedlatif1 · Pull Request #8459 · simstudioai/sim

waleedlatif1 · 2026-09-30T09:15:21Z

Summary

Successful run_code, run_function and run_workflow calls could reach the model as a bare { success: true } with no output, and the model never learned why. This PR fixes what can be fixed without weakening the secret boundary: a resolved secret still never reaches the model.

Keep a tainted mount's verdict across the run_code crossing.
- When a code run mounts a workspace file whose secret provenance is unknown, its result is still withheld. That is correct, because the code can read those bytes.
- What was wrong: the crossing into the Copilot per-call registry recorded only source-provenance-incomplete, a reason in the registry's absence set. The refusal named no guard, and a file written from that registry (outputs.files[].path) was recorded as unrecorded absence instead of taint.
- The crossing now carries the mounted registry's own reasons forward (mounted-file-provenance-unavailable). Mount refusals also report the file id.
Tell the model why a result was withheld.
- A withheld result now carries withheldReason inside the existing resultWithheld disclosure. The text is fixed wording chosen by the guard that tripped:
  - an input with unknown secret provenance;
  - provenance that could not be verified;
  - content that could not be checked.
- No content, reason literal, origin or file name crosses. The key is reserved so an effect id cannot displace it. A call with no registry still carries nothing.
Log the size of a withheld result.
- Withheld-result log lines now report resultBytes and resultValues (numbers only), so a content-refused refusal can be traced to the byte cap or to the traversal cap.
- One shared measureModelContent does the counting. It walks the value by JSON's rules instead of serializing it, and stops at the first value, byte or depth limit, so a huge payload is never serialized just to be measured.
Bound run_workflow block-log outputs, on both the server and browser paths.
- A run_workflow result echoes every block's output. When a call has an active secret, the projection refuses any payload whose walk exceeds its 100k-value traversal cap. So a synthetic run whose block outputs exceed that cap, while staying under the byte cap, was withheld whole.
- A result without an active secret is byte-identical to staging. Without one, the projection passes JSON through under its 16 MB byte cap alone and never walks it, so nothing here changes it: block-log and lifted outputs cross in full, the server handler keeps its previous input preview marker, and the browser-run path leaves its logs untouched in their original position. The check (copilotProjectionWalksContent) mirrors the projection's own matcher on the same per-call registry.
- With an active secret, only a result that would pass a cap is bounded. The assembled model-facing result (envelope, logs, error included) is measured the way the projection will walk it. A result that fits crosses in full, as on staging. One that would pass a cap first has its bulkiest block-log outputs replaced, down to a quarter of each cap (derived from MAX_CONTENT_NODES and the exported default byte cap), and if it still would, its final output too. This applies to the server handler and to the restoration of a browser-run workflow.
- The bulkiest outputs are replaced, largest first, with a pointer: …[output omitted: N values; inspect with logs get <executionId> --trace].
- The pointer carries no byte count. The truncated-input marker keeps nothing of the raw input: no length, and no prefix that could cut through a secret and leave an unredactable fragment. Both are written before secret projection.
- The final output covers a Response block's output and a block output lifted into output by run_block or run_workflow_until_block, on both paths. A result or block output JSON cannot encode (a cycle, a BigInt) is never replaced: it cannot be checked, so the projection refuses it as before. The bound handles size only. A block-log output past the projection's depth limit becomes a pointer instead of voiding the run.
- With an active secret, both paths replace long echoed block inputs with the marker above; without one, inputs are presented exactly as before.
- The executionId, status, final output, error and select values are untouched and keep the remaining headroom.
- Both paths build their log fields through one shared helper, from raw logs and before projection. A select resolves against the full logs and replaces them, so a browser-run workflow with a selector is no longer withheld either. Selected values are still projected.

Testing

Tainted mount through run_code: runs through the real handler, projection and file-classification path. Before the fix, the refusal named no guard and the output file was classified as unrecorded. Exact-secret mounts still redact and clean mounts stay readable.
Withheld-reason tests: the new tests failed before the change. Removing the reserved key or the reason mapping turns them red.
run_workflow budget tests (synthetic trace-shaped runs):
- A result over the traversal cap with an active secret now projects on the server path, and its secret is still redacted in the final output and error.
- The same holds on the browser-run path. This test failed before the fix.
- A large final output keeps its headroom. Setting both budgets to the full caps turns this test red.
- Narrow rows are bounded by value count alone. Setting the value budget to the full cap turns this test red.
- select returns full values and bypasses the budget. Compacting regardless of select turns this test red.
- A browser-run workflow over the cap with a select projects the selected values, with the secret redacted, instead of being withheld. This test failed before the fix.
- Outputs that differ only in a secret's length produce identical pointer text. Adding bytes back to the pointer turns this test red.
- A lifted terminal block output over the cap becomes a pointer, and the run projects. One of about 27.5k values crosses in full while its log copy is bounded. This test failed before the fix.
- A Response block's final output past the cap becomes a pointer instead of voiding the run, on the server and browser-run paths. A lifted output and a log output each just inside their share, whose whole result passes the cap, also end with the final output as a pointer. These three tests failed before the fix.
- A run with a BigInt block output beside a bulky one is still refused rather than the unencodable value being hidden behind a pointer. This test failed on the previous commit.
- With a configured but inactive secret, a large lifted output and its log copy cross in full. This test failed before the fix; forcing the bound on turns it red, and forcing it off turns the seven secret-path budget tests red.
- Block inputs that differ only in a secret's length produce identical truncation markers.
- A secret that straddles the old 200-character cut does not appear, even partially, in the model-facing result.
- A block output nested past the depth limit becomes a pointer.
- A browser run truncates long echoed inputs.
- measureModelContent reports strings past the byte cap, content past the value cap, and content nested past the depth limit as over, and refuses BigInt and cycles.
- Each of these tests failed before its fix, and reverting that fix turns it red again.
- A result that fits the caps crosses in full with an active secret: a lifted output and its log copy of about 27.5k values, and one 4.5 MB block output. A long input keeps the server's preview marker, and a browser run's logs cross untouched, when no secret is active. These three tests failed on the previous commit.
- Staging parity probe (server path, 19 scenarios, SHA-256 of the model-facing result on the merge-base and this head): all 12 no-secret scenarios (no registry, an empty registry, an inactive secret; small, large, long-input and lifted runs) hash identically. With an active secret, the 30k-value, 4.5 MB and 90k-final-output results also hash identically. The four that differ are intended: three runs staging withheld whole now project with pointers, and the long-input marker keeps no prefix or length.
bun run lint, bun run type-check and bun run check:audits pass.
After the last fix (rebased on staging with fix(sandbox): withhold workbench certification after an unprovenanced file mount #8456, fix(mothership): settle chat runs no controller owns #8457, test(mothership): keep the preview turn-budget test fast under load #8458 and fix(sim-cli): report an embedded SimApiError with its own exit code #8462): the lib/mothership, lib/execution, executor/utils, providers and Copilot/function route suites pass (6,177 passed, 23 skipped).
Full apps/sim Vitest on this change: 34,747 passed, 25 skipped, 0 failed.

vercel · 2026-09-30T09:15:27Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 30, 2026 6:49pm UTC

greptile-apps · 2026-09-30T09:19:17Z

Confidence Score: 5/5

[Medium risk] Adds content measurement and bounding for workflow results.

The PR appears safe to merge; no actionable new issue or outstanding previous finding remains.

Summary

The PR explains why Copilot tool results are withheld, preserves mounted-file provenance across run_code, adds bounded withholding diagnostics, and limits workflow block-log output before secret-aware projection.

Server and browser workflow paths share log presentation and result budgeting.
No new actionable issue was established in the PR changes since the previous review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Tool or workflow result] --> B[Per-call secret provenance]
  B --> C[Present and, when needed, bound workflow logs]
  C --> D[Secret-aware result projection]
  D -->|Safe| E[Model-facing result]
  D -->|Refused| F[Structural result with withholding reason]

_{Reviews (6) · Last reviewed commit: "fix(mothership): leave a result that fit..."}

cubic-dev-ai

All reported issues were addressed across 14 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T10:04:39Z

@greptile

waleedlatif1 · 2026-09-30T10:04:40Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T10:04:45Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 17 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 · 2026-09-30T17:22:20Z

@greptile

waleedlatif1 · 2026-09-30T17:22:21Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T17:22:28Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 17 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T17:42:34Z

@greptile

waleedlatif1 · 2026-09-30T17:42:35Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T17:42:42Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 17 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T18:00:36Z

@greptile

waleedlatif1 · 2026-09-30T18:00:38Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T18:00:46Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 17 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic}

…rossing A run_code / run_function that mounted a workspace file whose sidecar is `unknown` latched its per-call registry through the crossing with only `source-provenance-incomplete` and `inherited-incomplete-source`. Both are in the registry's absence set, so the withheld result named no guard, and an output file written from that registry (outputs.files[].path) was recorded as `unrecorded` absence instead of taint. The crossing now inherits the mounted registry's own reasons when it latched, so the refusal names `mounted-file-provenance-unavailable` and writers keep the taint. The mount refusal also reports the file id. No projection is relaxed: the result is still withheld, exact mounts still redact, clean mounts are unchanged.

A withheld result reached the model as a bare `{ success: true }`, so the agent could not tell a tainted input from an oversized payload and retried or guessed. The withheld result now carries `withheldReason`, chosen from code-defined wording by the guard that tripped: an input file, table, or document with unknown secret provenance; provenance that could not be verified; or content that could not be checked. It rides the existing `resultWithheld` disclosure beside any effect ids, and the key is reserved so an id cannot displace it. No content, reason literal, or origin crosses; an absent registry still carries nothing.

A `content-refused` withholding said only that a complete registry refused the payload. It can be refused by its encoded size or by the number of values the projection walks. Row-shaped payloads reach the 100k-value traversal cap well before the 16 MiB byte cap, and only while the call has an active secret. The withheld log lines now report the result's encoded bytes and value count, so a refusal names which cap it hit. Numbers only; no content is logged.

…et projection A run_workflow result echoes every block's output in `logs`. A run with many row-shaped block outputs can exceed the projection's 100k-value traversal cap while staying under its byte cap. Whenever the call had an active secret, the whole result was then withheld, including the final output and error, and the model saw a bare success. Block-log outputs are now bounded to a quarter of each projection cap (`MAX_CONTENT_NODES` values and the default byte cap). Past the budget, the bulkiest outputs are replaced, largest first, with a `logs get <executionId> --trace` pointer, in the same form the oversized-input compaction already uses. The executionId, status, final output, error, and `select` values are untouched, and the other three quarters of each cap stay free for them.

… length-blind run_workflow can complete in the browser, and that restoration projected the raw block logs, so a large browser-run workflow was still withheld. The block-log compaction now lives in the shared workflow-output module and applies to both the server handler and the client restoration when no `select` is given. `select` still reads full values. The omitted-output pointer no longer reports bytes. They were measured before secret projection and so disclosed a secret's length. It reports the value count and says "inspect with logs get" rather than promising the full value. One `measureModelContent` in the projection module now serves both the compaction and the withheld-size logging. It stops counting at the value cap and tolerates values JSON cannot encode. The byte cap is exported beside `MAX_CONTENT_NODES`.

…jection The browser-run restoration projected the full raw block logs and only then applied `select`, so a large run was still withheld whenever a selector was given. The server handler selects first. Both paths now build their log fields through one shared `presentWorkflowLogsForModel`, from raw logs and before projection: a `select` resolves against the full logs and replaces them; otherwise the echoed logs are bounded. Selected values are still projected, so a selected secret is redacted.

The provider tool path projects through the same withheld-result shape, so an omitted model result now carries `resultWithheld` and its fixed-wording `withheldReason`. The expectations still asserted the old empty output.

…d a walk-based measure - run_block and run_workflow_until_block lift the stopping block's output into `output`. That copy is now bounded with the same budget and pointer as a block-log output, so one huge block no longer gets the whole response withheld. - The truncated-input marker no longer carries the input's length. It is written before secret projection, so the length disclosed a secret's length. - `measureModelContent` now walks the value by JSON's rules instead of serializing it. It stops at the first value, byte, or depth limit it passes and reports `exceeded`, measuring a string only while it fits the remaining byte budget. An output past a limit, including one nested past the projection's depth limit, becomes a pointer instead of voiding the run. - The browser-run restoration now truncates echoed block inputs as the server handler does: both paths build their log fields through `presentWorkflowLogsForModel`.

A truncated block input kept its first 200 raw characters, written before secret projection. A secret straddling that cut left a fragment that whole-literal redaction cannot match, so part of the secret reached the model. The marker now keeps nothing of the raw input: `…[input omitted; inspect with logs get <executionId> --trace]`. Inputs over the limit are echoed upstream data the caller already has or can fetch, so the preview carried little.

…m against a secret Without an active secret the model-facing projection passes JSON through under its byte cap alone, so the block-output budget turned a large lifted run_block output into a pointer for no reason. Output compaction now applies only when the call's registry makes the projection walk the result, and a lifted output keeps the final output's share beside the bounded logs.

…its final output A fixed share for the lifted output left no room for the log entries, envelope and error the projection also counts, and a Response block's final output was never bounded, so a run near the caps was still withheld whole. The assembled result is now measured as the projection will walk it, and a final output that would push it past a cap is replaced with a pointer, on the server and browser-run paths alike. An unencodable result is still refused as before.

… to refuse The output bound handles size only. A block output JSON cannot encode cannot be checked, so it is no longer sized past the budget and replaced with a pointer; the projection refuses it as it did before this PR.

…without a secret, as staging returns it Block-log outputs were bounded whenever a secret was active, so a 30k-value or 4.5 MB result that crossed in full before became pointers. Outputs are now bounded only when the whole result would pass a projection cap. Without an active secret the server handler keeps its preview input marker and the browser-run path leaves its logs untouched, in their original position, so a result with no secret is byte-identical to before; only a walked call gets the marker that keeps nothing.

waleedlatif1 · 2026-09-30T18:22:33Z

@greptile

waleedlatif1 · 2026-09-30T18:22:34Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T18:22:39Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 17 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic}

…he whole-result measure The unencodable-output test placed the BigInt in the lifted output, which the whole-result walk reaches first, so it passed without the guard. It now puts a bulky log ahead of the BigInt log. A new test fails if the error is left out of the measure the result is bounded against.

waleedlatif1 · 2026-09-30T18:49:09Z

@greptile

waleedlatif1 · 2026-09-30T18:49:11Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T18:49:19Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 17 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic}

vercel Bot temporarily deployed to Preview September 30, 2026 09:15 Inactive

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/executor/utils/resolved-secret-content-projection.ts Outdated