Skip to content

Claim only delivery types supported by each worker - #120

Open
endrju19 wants to merge 5 commits into
mainfrom
fix/route-aware-claim-116
Open

endrju19 wants to merge 5 commits into
mainfrom
fix/route-aware-claim-116

Conversation

@endrju19

@endrju19 endrju19 commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Workers now claim only PENDING entries for their registered delivery types. The processor rotates its starting type and uses at most two claim queries per batch, independent of the number of types: one for the starting type and, when needed, one IN query for the rest. A mismatched entry is also rejected before delivery.

PostgreSQL gets a partial (delivery_type, created_at, id) WHERE status = 'PENDING' index; MySQL gets (status, delivery_type, created_at, id) plus an exact, binary type check for case-insensitive collations. The unrestricted OutboxStore.claimPending(limit) remains available to direct callers and keeps its existing index. Custom stores used by Okapi's processor must implement RouteAwareOutboxStore; Spring autoconfiguration rejects an unsafe store at startup.

Verification

  • After resolving conflicts with main, ./gradlew build ktlintCheck passed locally, covering all modules and database, HTTP, Kafka and Spring Boot tests.
  • The HTTP batch concurrency test now uses a server-side request barrier. Every request must arrive before any successful response is sent; intentionally switching delivery to sequential makes the test fail. Both PR Claim only delivery types supported by each worker #120 and chore(deps): bump the production group with 8 updates #121 include the same test fix.
  • Core, Spring Boot and integration tests passed locally on PostgreSQL and MySQL Docker containers, including migrations, concurrent heterogeneous workers, exact type matching, HTTP and Kafka. ktlintCheck passed.
  • On a 1,000,000-row outbox with 900,000 older unsupported rows, a 100-type absent IN claim used the route index and took about 0.16 ms on PostgreSQL and 0.21 ms on MySQL in local measurements. The previous full PostgreSQL index produced a 165 ms scan for that case. Scripts and context are in benchmarks/route-aware-claim/.
  • There is a write cost from the additional index. In exploratory repeated local runs, PostgreSQL insert/update medians were 1.47×/1.07× versus the legacy index alone; MySQL medians were 1.18×/1.29×. These are workload-specific measurements, not throughput guarantees.

Rollout

The branch includes the current main, preserving #119 startup validation and #115 outbox-id headers. For manually managed schemas, the README links both migrations: apply 001 and 002 in order for a new installation, or apply 002 before starting updated workers on an existing schema. Run PostgreSQL migration 002 outside a transaction because it creates the index concurrently. Older workers can still claim unsupported types, so publish a new type only after all old processors are upgraded or stopped. Previously FAILED rows need separate recovery.

Closes #116

Reject duplicate MessageDeliverer types instead of silently replacing one transport.
Fail default processor startup when no deliverer is registered while preserving
publisher-only and custom processor configurations.

Closes #117
Closes #118
@endrju19
endrju19 changed the base branch from fix/deliverer-validation-117-118 to main September 26, 2026 10:10
@endrju19 endrju19 closed this Sep 26, 2026
@endrju19 endrju19 reopened this Sep 26, 2026
Add route-aware claims and a matching index for PostgreSQL and MySQL. Rotate
the first route between batches, reject unsafe custom stores, and prevent a
single deliverer from processing an entry of another type.

Cover heterogeneous workers, concurrent claims, migrations, and a million-row
Docker benchmark. Document the custom-store and rolling-upgrade requirements.

Closes #116
@endrju19
endrju19 force-pushed the fix/route-aware-claim-116 branch from 83c1985 to a4b6894 Compare September 26, 2026 10:18
Resolve conflicts with the merged deliverer validation and outbox-id changes. Link both schema migrations and explain the required MySQL index and PostgreSQL concurrent migration. Replace the timing-based HTTP concurrency test with a request barrier.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker can fail an entry it cannot deliver

1 participant