The Transactional Outbox Pattern: Reliable Event Publishing with AI
Writing to your database and publishing an event are two separate operations — and the gap between them is where events get silently lost. The outbox pattern closes it. How to implement it correctly with AI.
Save an order to the database, then publish an OrderPlaced event to a message broker — that's two separate operations against two separate systems, and there's no way to make them atomic with a normal database transaction. If the process crashes between the two, or the broker call fails, you've either lost an event that should have fired, or fired one for a write that got rolled back. The transactional outbox pattern makes this atomic by keeping both operations inside the same database transaction.
How it works
- In the same database transaction as your domain write (e.g. inserting the Order row), insert a row into an Outbox table containing the event payload.
- Commit. Both the domain write and the outbox row are now atomically persisted — or neither is, if the transaction fails.
- A separate background process (a poller, or a database change-data-capture stream) reads unpublished rows from the Outbox table and publishes them to the message broker.
- Mark the row as published once the broker confirms receipt — or delete it, depending on your retention needs.
The outbox pattern gives you at-least-once delivery, not exactly-once — the publisher might publish a row twice if it crashes after publishing but before marking it published. Consumers of these events must be idempotent. This isn't a flaw to fix; it's a constraint to design for.
Where implementations go wrong
| Mistake | Consequence |
|---|---|
| Outbox insert happens in a separate transaction from the domain write | Defeats the entire purpose — you're back to two non-atomic operations |
| Poller doesn't handle publish failures with retry + backoff | A transient broker outage causes events to be silently stuck rather than eventually delivered |
| No idempotency key on the consumer side | At-least-once delivery causes duplicate processing — e.g. double-charging a customer if the event triggers a payment |
| Outbox table grows unbounded | No cleanup/archival of published rows turns a reliability pattern into a performance problem |
Worked example
Context: .NET 8, EF Core, PostgreSQL. Domain: placing an order should atomically persist the Order and produce an OrderPlaced event for downstream consumers.
Task: Implement the transactional outbox pattern: (1) an OutboxMessage entity, (2) the write path that saves Order and the outbox row in one SaveChangesAsync call, (3) a background poller (IHostedService) that publishes unprocessed rows and marks them published, with retry on transient failure.
Constraints:
- The Order save and OutboxMessage insert MUST be in the same DbContext SaveChanges call — no separate transactions.
- Poller must use row-level locking (SELECT ... FOR UPDATE SKIP LOCKED or EF Core equivalent) so multiple instances of the service don't double-publish the same row.
- Include the idempotency key in the published event payload so consumers can dedupe.
- Do not use a distributed transaction (2PC) across the database and the broker — that's the exact coordination problem this pattern avoids.Outbox vs. saga: different problems
The outbox pattern solves reliable event publishing from a single service. It doesn't, by itself, coordinate a multi-step business process across several services — that's what the saga pattern is for. In practice they're often used together: each step of a saga publishes its completion or failure event reliably via its own outbox, and the saga orchestrator or choreography reacts to those events.
The Reliability module in the Advanced program covers outbox implementation alongside the choreography-based saga pattern for exactly this reason — most real distributed-transaction problems need both, not either.