fix(mothership): re-sync Chat reconnects from the worker log and trim the replay ring by bytes by waleedlatif1 · Pull Request #8469 · simstudioai/sim

waleedlatif1 · 2026-09-30T16:49:36Z

Summary

This PR stops the replay ring from ending long Chat turns.

The bug it fixes: byte-budget refusals seen in production. A long run's replay ring reached its owner byte ceiling, and the refused write ended the turn. The byte-trimmed ring below fixes that.
Enabling work: once the ring trims by bytes, it can lose its head, so reconnects past a trimmed head must be re-synced. The replay_gap re-sync is that enabling work. It is not a fix for a replay_gap observed in production.

Merge order

It builds on fix(mothership): keep Chat stream legs alive without a wall clock #8463 (stream-leg continuity), which is already in staging.
It needs mothership#555's read-only replay endpoint (POST /api/streams/replay) deployed first. Its protocol types are synced here with bun run contracts:sync.
Worker without the endpoint, or a key that may not call it: it answers 404, 401 or 403, and a reconnect keeps today's behaviour: the replay_gap terminal.
Unreachable worker: the route answers 503, and the client retries with backoff.

Byte-trimmed replay ring

The ring now trims by bytes as well as by count. The owner counter tracks exactly the retained bytes, so a long run slides instead of hitting the owner budget and failing.

Re-sync from the worker's log

The ring can't serve a reconnect when:

it lost its head (a fresh tab or reload from cursor 0 included);
the cursor is behind it;
its numbering restarted.

In those cases Sim re-syncs the reader from the worker's durable log through the read-only replay.

Response: frames carry this response's own cursors, starting at 1, and the header x-mothership-stream-replay: log. The client rebuilds the turn from an empty response.
The reader stays on the log: from then on the client sends source=log, and the ring never serves that reader again. The ring and the log share no position to join on.
Endings:
- A terminal or the worker's 5-minute cap ends the response at once.
- A park holds it while the run waits on Sim, and ends it as soon as the run resumes.
- Any held response ends as soon as the run reaches a terminal.
- A stall, a cut connection, or a park Sim has not yet marked holds it at least 10 s, so a reader replays the log at most every 10 s while nothing moves.
Batch reads the ring can't serve return no ring events.
Live tails: a live tail whose ring loses its head or restarts ends without a terminal event, so it re-attaches and is re-synced.
replay_end frames are parsed with a schema tied to the protocol's reasons. An unknown reason ends the replay and is never forwarded as a run event.

Recovery and snapshots on a ring that lost its head

Recovery: a run whose Sim controller died (for example, across a deploy) recovers from an empty context, as it would from an expired ring. The new controller re-attaches with an empty receipt. The worker re-sends the whole response and re-hands its parked calls. The turn then finalizes with its full content persisted.
Usage: usage stays with the worker's per-run settlement. A re-attach under the same message identity is never a second run.
Snapshots: both chat snapshot routes skip the snapshot, through one shared reader, so the client re-syncs through the reconnect route.

Cost

A reconnect on a run whose ring lost its head replays up to the whole log. A tab that stays attached replays it again about every 5 minutes, when the worker's replay connection reaches its cap. It also replays at most every 10 s while a run is stalled. Sharing replays between readers is a worker follow-up.

Known limits

Worker deadline: the worker's run deadline still ends runs until its flag is turned off.
Restarted rings in recovery and snapshots: these check only that the ring's first retained seq is 1. A ring that expired and restarted at 1 still looks complete to them. The heartbeat TTL refresh makes this rare.
Duplicated text in the ring: after an empty-context recovery, the ring holds the re-sent response after its old tail. Readers never see it twice: a trimmed ring always re-syncs from the log, and live tails leave the ring when it loses its head.
Inbox runs: Chat runs started from the inbox still run in a Trigger.dev task with a 90-minute maxDuration.

Next step: one durable read path

Key the ring by the worker's durable seq and serve every reader, live or late, from one read path. Cursors would then name positions in the durable log, and nothing would be re-synced or joined.

Test plan

Integration suites against real Postgres and Redis (bun run test:integration, all suites). The worker is mocked only at the HTTP boundary.
- replay-gap.integration.ts:
  - re-sync from cursor 0, from behind the ring and from a restarted ring;
  - source=log readers stay on the log;
  - parked and stalled holds, and an unknown end reason;
  - live tails leaving a restarted or headless ring;
  - batch reads;
  - the 404, 401 and 403 fallbacks.
- stream-recovery.integration.ts: a controller dies on a run whose ring lost its head. The new controller recovers with the real lifecycle.
  - The turn persists its full content once, after one re-attach with an empty receipt.
  - A replayed tool and a re-handed call the dead controller already ran each appear once. The re-handed call is resumed with its stored result, never run again.
- replay-budget.integration.ts: a stream far past the owner budget; recovery and snapshots on a ring that lost its head.
- buffer-ttl.integration.ts.
Unit and DOM tests, including the client rebuild and source=log.
bun run lint, bun run type-check, bun run check:audits, full apps/sim vitest.

vercel · 2026-09-30T16:49:41Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 30, 2026 8:12pm UTC

cubic-dev-ai

All reported issues were addressed across 37 files

_{Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic}

greptile-apps · 2026-09-30T16:56:25Z

Confidence Score: 5/5

[High risk] Refactors chat stream recovery and replay ring trimming logic.

The PR appears safe to merge based on the reviewed changes and resolved prior findings.

Summary

The PR bounds the replay ring by retained bytes and re-syncs readers from the worker log when the ring cannot serve their cursor. The latest changes reuse the shared SSE idle timeout and track the currently log-sourced stream by ID.

Recovery and snapshots avoid rebuilding a turn from a ring that has lost its head.
Reconnect responses identify log replay so the client can rebuild from its beginning.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Chat reconnect] --> B{Ring can serve cursor?}
  B -->|Yes| C[Read and tail ring]
  B -->|No| D[Open worker log replay]
  D --> E[Mark response as log replay]
  E --> F[Client rebuilds turn]
  F --> G[Later reads use source=log]

_{Reviews (8) · Last reviewed commit: "refactor(mothership): reuse the SSE idle..."}

waleedlatif1 · 2026-09-30T18:15:57Z

@greptile

waleedlatif1 · 2026-09-30T18:15:58Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T18:16:04Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 37 files

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T18:28:08Z

@greptile

waleedlatif1 · 2026-09-30T18:28:10Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T18:28:16Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 38 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T18:36:27Z

@greptile

waleedlatif1 · 2026-09-30T18:36:29Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T18:36:35Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 38 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

Adds StreamReplayRequest and StreamReplayEnd from the worker's contracts (bun run contracts:sync).

…m the worker log A reconnect whose cursor fell behind the ring, a fresh tab reading a ring that lost its head, and a cursor ahead of a ring whose numbering restarted now stream the run from the worker's read-only replay instead of ending the turn with replay_gap or replaying a partial response. - The reconnect route opens POST /api/streams/replay with no receipt and forwards it under its own cursors from 1, with x-mothership-stream-replay: log so the client rebuilds the turn from an empty response. A response on the replay is never handed back to the ring, which shares no position with the log. - A parked replay holds the response open until the run resumes; a capped, stalled, or cut replay ends it without a terminal so the client re-attaches. - A batch read the ring cannot serve returns no ring events. - A live tail whose ring restarts under it ends without a terminal, so the client re-attaches and is re-synced. - An unknown run keeps the replay_gap terminal; an unreachable worker answers 503 so the client retries.

…nstead of refusing The append script pruned the oldest events by count only, so any stream averaging more than ~335 B per event reached the 32 MiB owner budget before 100k events and the refusal ended the turn. The ring now also trims its oldest events until the retained bytes fit three quarters of the owner ceiling, refunding exactly what it drops in the same script. A byte trim never drops a member the write adds, so an inflated counter still refuses rather than silently discarding the new frame.

…its head A byte trim advances the ring past seq 1, and three callers read it from seq 0 assuming the head was there: stream recovery rebuilt a controller's context from the tail, and both chat snapshot routes painted a truncated turn. One predicate, startsAtReplayHead, now guards them and the reconnect route's gap check: - recovery refuses with StreamReplayHeadTrimmedError instead of persisting a truncated turn; - the snapshot routes skip the snapshot, so the client reconnects; - the reconnect route re-syncs the view from the worker's log (stream) or serves no tail events (batch). When recovery was refused, a parked or stalled replay ends the view with recovery_unavailable instead of re-attaching forever. The append script also trims a replayed member that lands below the ring for bytes, keeping the tail contiguous, and caps byte trimming at 4096 members per append so an oversized ring catches up over several appends. The budget docs now say the user counter bounds bytes held, not bytes written per hour.

…ep log readers on the log - Recovery no longer refuses a run whose ring lost its head, which orphaned long runs after a Sim deploy. It treats that ring like an expired one: the new controller starts from an empty context at the ring's latest seq, and re-attaches with an empty receipt. The worker then re-sends the whole response and re-hands its parked calls. Usage stays with the worker's per-run settlement; a re-attach under the same message identity is never a second run. - A client re-synced from the log sends source=log from then on, so the ring never serves its log cursors, even after it restarts and grows past them. - A live tail ends without a terminal as soon as its ring loses its head or restarts, so it re-attaches and re-syncs instead of reading re-sent text. - A replay that ends short of the terminal and cap holds its response at least 10 s (longer while parked), so a stalled run is not replayed every second. - replay_end is parsed with a schema tied to the protocol's reasons; an unknown reason ends the replay instead of passing as a run event. - The replay forwarder moves into session/run-replay.ts, the chat snapshot reader is shared by both chat routes, and checkForReplayGap is removed.

…shes, and check a busy tail's ring less often - A replay held after a park now ends as soon as Sim sees the run leave its park, and any held replay ends as soon as the run reaches a terminal, so an approval no longer freezes the view for up to 10 s. A park Sim has not yet marked, a stall and a cut connection keep the 10 s floor. - A live tail checks that its ring can still serve it only after a quiet poll or every eighth busy one, instead of two Redis reads on every 250 ms poll. - The recovery integration test re-sends a replayed go tool and re-hands a Sim call the dead controller already ran: it is resumed with its stored result, never run again, and the turn keeps one block per tool.

…play's key A deployment whose key may not call the worker's replay (401/403) now falls back to the replay_gap terminal as a missing run (404) does, instead of answering 503 until the client's reconnect budget runs out. A failed buffer TTL refresh during the chat-lock heartbeat is logged as such, not as a lock-extension failure.

…sor, and re-sync an expired ring - The worker replay is bounded like a stream leg: no response headers, or no bytes including keepalives, for the idle timeout ends it so the reader re-attaches. - A ring read that starts after the reader's next event (the ring trimmed its head between the gap check and the read) is never delivered; the reader re-attaches and re-syncs from the log, in both the live tail and batch reads. - An empty ring serves only a reader starting from cursor 0; a live run whose buffer expired under a reader re-syncs from the log, and a finished one answers its terminal since its transcript is persisted. - A leg that delivers events after a failure starts a fresh 30 s reachable window; its three retries still refill only after five minutes of delivered events. - The two new integration suites close their worker server and restore env even when they are skipped.

…tral agent-url mock

…he mid-tail trim race - A 401 or 403 from the worker's replay endpoint is logged with its status, so a rotated or wrong worker key is visible instead of every reader silently falling back to replay_gap. - A live tail whose ring trims past its cursor between polls ends without a terminal and never delivers the events after the gap.

waleedlatif1 · 2026-09-30T19:31:36Z

@greptile

waleedlatif1 · 2026-09-30T19:31:38Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T19:31:44Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 27 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T19:58:14Z

@greptile

waleedlatif1 · 2026-09-30T19:58:15Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T19:58:23Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 27 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

…and track the log re-sync as one stream id - processSSEStream takes an optional idle timeout and passes it to readSSELines, so the run replay uses the shared idle bound instead of its own reader wrapper. Callers that omit it are unchanged; the replay's header wait keeps its own timer. - A chat view re-syncs one stream at a time, so the log re-sync flag is the stream's id rather than a set.

waleedlatif1 · 2026-09-30T20:12:41Z

@greptile

waleedlatif1 · 2026-09-30T20:12:42Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T20:12:49Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 28 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

waleedlatif1 · 2026-09-30T20:25:47Z

@greptile

waleedlatif1 · 2026-09-30T20:25:48Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T20:25:54Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 28 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/session/run-replay.ts

Comment thread apps/sim/app/api/copilot/chat/stream/route.ts Outdated

vercel Bot deployed to Preview September 30, 2026 18:18 View deployment

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/session/abort.ts

vercel Bot temporarily deployed to Preview September 30, 2026 18:27 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/app/api/copilot/chat/stream/route.ts

vercel Bot temporarily deployed to Preview September 30, 2026 18:36 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

waleedlatif1 added 10 commits September 30, 2026 12:24

chore(mothership): sync the worker protocol for the read-only run replay

c4d4d35

Adds StreamReplayRequest and StreamReplayEnd from the worker's contracts (bun run contracts:sync).

test(mothership): drive the run replay liveness tests through the cen…

0b6935f

…tral agent-url mock

waleedlatif1 force-pushed the fix/stream-continuity branch from 02e0ecd to 6a57d07 Compare September 30, 2026 19:31

vercel Bot temporarily deployed to Preview September 30, 2026 19:31 Inactive

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/app/workspace/[workspaceId]/home/hooks/use-chat.ts Outdated

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/session/buffer.ts

fix(mothership): create the log re-sync set outside render

df43c8a

vercel Bot temporarily deployed to Preview September 30, 2026 19:58 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview September 30, 2026 20:12 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/app/workspace/[workspaceId]/home/hooks/use-chat.ts

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

waleedlatif1 merged commit 9ed10e1 into staging Sep 30, 2026
32 checks passed

waleedlatif1 deleted the fix/stream-continuity branch September 30, 2026 21:00

icecrasher321 mentioned this pull request Oct 1, 2026

v0.9.8: search cleanup recovery, new search providers, billing and chat reliability #8494

Merged

waleedlatif1 mentioned this pull request Oct 1, 2026

fix(mothership): number recovered turns past an unreadable ring and floor the replay TTL #8501

Merged

5 of 6 tasks

Conversation

waleedlatif1 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Merge order

Byte-trimmed replay ring

Re-sync from the worker's log

Recovery and snapshots on a ring that lost its head

Cost

Known limits

Next step: one durable read path

Test plan

Uh oh!

vercel Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

waleedlatif1 commented Sep 30, 2026 •

edited

Loading

vercel Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading