R4 — Outbox, relay and inbox: making the record of an event survive

The situation

A payment service finishes a piece of work. Two things now have to happen:

Outbox, relay and idempotent consumer

flowchart LR
  SVC["Service handler"] -->|"one transaction"| DB[("Business row + outbox row")]
  DB --> RLY["Relay: SELECT ... FOR UPDATE SKIP LOCKED<br/>or CDC on the WAL"]
  RLY -->|"at-least-once"| K["Broker, keyed by aggregate_id"]
  K --> CON["Consumer"]
  CON -->|"same transaction"| IN[("Inbox: processed_messages<br/>+ the effect")]
  IN -->|"conflict = already seen"| DROP["Acknowledge and drop"]

Its own database records it — payment.state = 'submitted'.
Other services are told — the ledger posts entries, notifications emails the customer, the read model updates.

The first is a database transaction. The database guarantees it either happened or it did not, and you get that for free.

The second is a message to a broker — a different system, on a different machine, reached over a network.

Nothing makes those two things one thing. And in between them, your process can die.

That is the whole problem. Everything below is the fix.

Watch it break

You have to do the two steps in some order. There are only two options, and both have a moment where a crash ruins the day.

Option A — save first, then announce.

09:41:07.100   BEGIN; UPDATE payment SET state='submitted'; COMMIT;   ✓
09:41:07.140   ← process killed here
09:41:07.???   publish("PaymentSubmitted")                             never happens

Your database says the payment was submitted. Nobody else ever hears. The ledger never posts, the customer is never emailed, the payment never reaches the rail.

And here is the part that makes this the worse option: your system looks completely healthy. The row is right there, committed and correct. There is no error, no exception, no failed request in any log. Nothing anywhere knows a message is missing. You find out days later, from a reconciliation break or a phone call.

Option B — announce first, then save.

09:41:07.100   publish("PaymentSubmitted")                             ✓
09:41:07.140   BEGIN; UPDATE payment SET state='submitted'; ROLLBACK;  ✗ deadlock

Now the ledger is posting entries for a payment that does not exist on your side. You have created a phantom, and the services downstream of you are all confidently wrong.

"Just wrap it in a try/catch and retry." This is the first thing everyone says, and it does not work. The failure you need to survive is the process being killed between the two lines — a pod eviction, an OOM kill, someone pulling a node. No catch block runs after that, and an in-memory retry queue dies with the process holding it.

"Use a distributed transaction." Two-phase commit across a database and a broker is exactly what R2 is about, and R2's conclusion is that you do not want it. Kafka cannot participate in one anyway.

So both orders are broken and neither escape hatch works. Something else has to change.

The trick: stop having two writes

Look again at why this is hard. The problem is not the broker. The problem is that the thing you want to be atomic spans two systems.

So do not let it.

Instead of sending the message, write the message into a table in your own database, in the same transaction as the business change.

BEGIN;
  UPDATE payment SET state = 'submitted' WHERE id = 8842;
  INSERT INTO outbox (aggregate_id, event_type, payload, created_at)
       VALUES ('8842', 'PaymentSubmitted', '{...}', now());
COMMIT;

That is it. That is the outbox.

Now there is one write, to one system, protected by the transaction you already had. Either both rows exist or neither does. There is no moment in between where a crash can leave the two disagreeing — because there is no "two".

Think of it as an outgoing mail tray on your own desk. You do not hand the letter to the postman as part of finishing your work — you drop it in your own tray, in the same motion. The letter is now safe, in your building, whether or not the postman is having a bad day.

flowchart LR
  subgraph TXN["ONE database transaction"]
    B["Business row<br/>payment.state = submitted"]
    O["Outbox row<br/>PaymentSubmitted"]
  end
  TXN --> R["Relay<br/>reads committed rows"]
  R -->|"unreliable hop,<br/>retried from durable state"| K["Broker"]
  K --> C["Consumers<br/>each with an inbox"]

The table itself is unremarkable:

CREATE TABLE outbox (
  id             bigserial PRIMARY KEY,
  aggregate_id   text NOT NULL,      -- the account or payment; becomes the message key
  event_type     text NOT NULL,
  payload        jsonb NOT NULL,
  created_at     timestamptz NOT NULL,
  published_at   timestamptz         -- null until the relay has sent it
);

Be precise about what you just bought, because this is the first follow-up question in an interview. You made the record of the message atomic with the state change. You did not make delivery reliable. Getting the row to the broker is still an unreliable network hop, it is still going to be retried, and messages will still sometimes arrive twice. The problem did not vanish — it moved somewhere a cheap fix exists.

A tempting simplification, and why it is not enough

A natural way to describe what just happened is: "business logic and sending a message are two separate concerns, so handle them separately." That sounds right, and it is the sentence most people arrive at on their own — but on its own it will cost you marks, because it names the consequence rather than the reason.

Separation of concerns does not fix this. You can separate the two beautifully and still lose every message. The canonical example is a Spring @TransactionalEventListener(phase = AFTER_COMMIT): business logic in one place, publishing in a clean listener, no coupling anywhere. Textbook separation — and it loses the message whenever the pod dies after the commit, because the listener runs in memory once the transaction has closed. The same bug as Option A, in tidier code.

The problem was never that the two were tangled. Two perfectly separated things that must both happen still need something making them one commit.

The outbox does split something in two, but the split is not business-logic-versus-communication. It is:

Deciding to tell someone — that is business state, and it belongs inside the transaction.
Actually telling them — that is plumbing over an unreliable network, and it belongs outside it.

Said precisely: recording the intent to publish is part of the business transaction; performing the publication is not. That phrasing stays intuitive and stays correct, because it tells you which side of the commit each piece goes on — which is the whole design decision. "They are separate concerns, handle them separately" does not, and an interviewer hearing only that will ask "so you publish after the commit?" and you are back in the failure.

The line to say out loud:

"The decision to tell someone is business state. The telling is plumbing. So I commit the decision with the work, and let the plumbing retry from the durable record."

The postman: the relay

Something has to pick the letters up. That something is a small background process — the relay.

Its whole job is three lines: read rows where published_at IS NULL, publish them, mark them sent.

It is slow, boring and unglamorous, and that is exactly right. The important part already happened when the row committed. If the relay crashes it simply resumes, because the work is sitting in a table rather than in some process's memory.

Publish, then mark, then commit — in that order. Think about the alternative: mark first and then publish, and a crash in between loses the message permanently, which is the thing you just spent all this effort eliminating. With publish-first, a crash in between means you publish it again next time. A duplicate. Which is the failure you already chose.

Running two relays. The obvious query hands both instances the same rows, and both publish. Row-level skip locking gives each instance a different batch instead:

SELECT * FROM outbox
 WHERE published_at IS NULL
 ORDER BY id
 LIMIT 500
 FOR UPDATE SKIP LOCKED;

Order matters per account, not globally. Publish in id order and use aggregate_id as the message key, so one account's events stay in sequence. If you need more throughput, hash aggregate_id into lanes and give each relay instance a lane. Never split the work randomly, or two events for the same account overtake each other.

Watch the table. It is the busiest write table in the service and it grows forever. Partition by day or delete sent rows aggressively, index only the unsent ones (WHERE published_at IS NULL), and alert on the age of the oldest unsent row — that number is your publishing lag, and it is the single best health metric this mechanism has.

The no-polling alternative. Instead of a relay querying the table, a tool like Debezium reads the database's replication log directly and turns new outbox rows into messages. Lower latency, no polling load. But you now own a new failure mode: if that reader falls behind, the database holds on to its log files and can eventually fill the disk. That lag becomes a first-class alert.

"But isn't the relay doing the same dual write?"

Yes — and noticing that is the right instinct, so have the answer ready, because a good interviewer will ask.

The relay writes to two systems with no transaction spanning them:

publish to broker          ← system 1
mark published_at = now()  ← system 2

A crash in between is entirely possible. That gap never goes away, and R0 already explained why: exactly-once delivery does not exist, so somewhere there is always a moment where two systems can disagree.

What changed is not whether the gap exists. It is which failure the gap produces.

	The two writes	A crash in between leaves
Naive publish	commit the payment, then publish	Loss — the intent existed only in a dead process's memory, and nothing on disk says a message is owed
The relay	publish, then mark sent	A duplicate — the row still has `published_at IS NULL`, so the next pass sends it again

So the outbox does not eliminate the dual-write problem. It is a failure-mode converter: it turns "might silently lose this forever" into "might send this twice". That is principle 1 from R0 being cashed in — and duplicates are already handled, by the inbox below.

Three things make the relay's version of the gap benign where the original was not:

The durable record survives the crash. The unsent row is the truth about what is owed, and it lives in the database. Retrying is not a best-effort attempt in memory; it is a query.
The relay cannot create a phantom. It only publishes rows that are already committed, so every message corresponds to state that genuinely exists. The naive publish-then-commit could announce a payment that then rolled back. The relay structurally cannot.
The order is a deliberate choice, not an implementation detail. Mark-then-publish would put you straight back into loss: recorded as sent, crash, never sent. Publish-then-mark keeps the residue on the duplicate side.

Two tests to carry into any version of this question: after a crash, is there a durable record of what still needs doing? And which way does the residue fall — loss or duplication? If the answers are "a row in a table" and "duplication", the design is sound however many hops it has.

The receiving side: the inbox

The outbox guarantees the message is never lost. The price was that it may arrive twice — from a relay retry, a broker redelivery, or a consumer rebalance.

So the receiver needs one thing: a way to recognise a message it has already handled.

That is a list of message IDs it has seen, checked before doing the work:

CREATE TABLE processed_messages (
  message_id   uuid NOT NULL,
  consumer     text NOT NULL,        -- each consumer tracks its own
  processed_at timestamptz NOT NULL,
  PRIMARY KEY (consumer, message_id)
);

The entire rule is one sentence: insert the message ID in the same transaction as the work it triggers.

BEGIN;
  INSERT INTO processed_messages (message_id, consumer, processed_at)
       VALUES ('018f3c1a-...', 'ledger', now());    -- conflict here means: already done
  INSERT INTO ledger_entry (...) VALUES (...);      -- the actual work
COMMIT;

If that first insert conflicts, you have seen this message before: acknowledge it and move on without doing anything. Commit the queue offset after that transaction, never before.

Notice why they must be in the same transaction. Record the ID first and commit separately, and a crash before the work means the message is marked done but never actually processed — you have invented a new way to lose things.

Notice also the composite key. One event goes to five consumers and each handles it independently, so "have I seen this?" is a question per consumer, not a global one.

Inbox versus idempotency: the same thing or different?

This confuses people, so be exact.

They are not competing patterns. Idempotency is the property; the inbox is one way of achieving it.

Idempotency is a property of an operation: doing it twice leaves the world the same as doing it once.
The inbox is a mechanism: a table of message IDs already handled, checked inside the transaction that does the work. Its other name — idempotent consumer — is the clearer one.

So: the inbox is idempotency applied to message consumption, keyed on the message ID. R3's idempotency key is that same idea applied to an API request, keyed on a value the client supplies. Same rule in both places — record what you have handled, in the same transaction as handling it — with the key coming from a different place.

	R3's idempotency key	The inbox
Trigger	An API call from a client	A message from a broker
Who supplies the key	The client, who must reproduce it on retry	The producer, as `event_id`
Scope of uniqueness	Per client — `(client_id, key)`	Per consumer — `(consumer, message_id)`
What it protects	A whole request, including calls to external systems, across phases with a recovery point	One consumer's local transaction
Stored response	Yes — a replay returns the original response	No — nobody is waiting for a reply

And you do not always need the table. Work down this ladder and stop at the first one that fits:

The handler is naturally idempotent. UPDATE payment SET state='settled' WHERE id=42 gives the same result the tenth time. Nothing to store, nothing to check.
Your own data already answers the question. The message says payment 42 settled; you look at payment 42 and it already says settled, so you drop the message. A state machine that rejects repeated or backward transitions is deduplication.
The effect is additive, so you need a record. Posting a ledger entry, sending an email, incrementing a counter — you cannot tell by looking whether an existing row came from this message or an earlier one. Now you store a key.

When you do store a key, think about what it means. The message ID is convenient, but it is only a proxy for the fact. A settlement can reach you twice by different routes: once as a real-time notification, once in the end-of-day file. Two messages, two different IDs, one real-world event. An inbox keyed on message_id cheerfully processes both. A check keyed on the business fact — "is payment 42 already settled?" — does not.

So key on message_id when news can only reach you one way, and on the business identity when it can arrive by more than one. In payments, more than one route is the normal case.

The limit worth knowing. The inbox only protects what commits with it. If your handler inserts the message ID, commits, and then calls a payment rail, the inbox has protected nothing about that call. That is R3's territory, where a durable marker sits around each external step rather than around the handler as a whole.

When you need none of this

Say this before the patterns in an interview, because it shows judgement rather than recall:

One service, one database. A plain transaction already gives you everything this section approximates. Most code is this.
Losing the message costs nothing — a metric, an analytics ping, a cache warm. Fire and forget.
The receiver can pull instead of being pushed. If the ledger can ask "which payments settled since X?", a lost message becomes a late one, and a polling endpoint replaces the whole mechanism. Often the simpler design where latency allows.
The sender can find the work again. If your own rows still say ready_to_submit, a background sweep re-issues the instruction. Nothing was truly lost, because your state still knows what should happen.

That last one is the real test, and it is worth phrasing as a question: can the fact be reconstructed from your own data? If yes, a lost message is an inconvenience. If no — if your database says the work happened and that message was the only thing that would ever have told anyone — then losing it is silent and permanent, and you need an outbox.

(Commands versus events is defined in R0. The short version: an event is the thing that must never be lost, because it is the only record instructing others to act. Undoing work when a later step fails is a different problem — that is R5.)

What to say in the interview

"You cannot remove the dual write — you can only choose which failure it leaves behind, and loss is the one you cannot recover from. So the event and the state change go into one transaction, with the message in an outbox table — one system, one commit, no window. A relay publishes from that table, which makes delivery at-least-once by design, so every consumer is an idempotent consumer: it records what it has handled and commits that record with its own work. Where the same fact can reach a consumer by more than one route, I key that check on the business identity rather than the message ID."