R5 — Sagas and compensation: undoing work that spans services

R2 showed that a distributed transaction is unavailable. R3 made repeated work safe, and R4 made sure the news survives. None of them answers the question this section does: the third step failed permanently, and the first two already happened in other services. Now what?

Saga step types around the pivot

flowchart LR
  S1["Compensatable<br/>reserve funds"] --> S2["Compensatable<br/>screen and validate"] --> P["PIVOT<br/>submit to rail"] --> S3["Retriable<br/>record acknowledgement"] --> S4["Retriable<br/>notify customer"]
  S2 -. "compensate: release" .-> S1
  P -. "failure before pivot:<br/>compensate backwards" .-> S2
  S3 -. "failure after pivot:<br/>retry until success" .-> S3
  S4 --> F{"Still failing?"}
  F -->|yes| NR["needs_repair + alert + ops queue"]

Interactive — pick a scenario

waiting processing complete failed duplicate, dropped compensated being checked never arrived

Scenario 5 is this section in one picture: 3.1 and 3.2 have committed real state when 3.3 rejects permanently, and the work gets walked backwards. Scenario 6 is the follow-up question — what happens when one of those undos fails too.

The situation

A customer sends ₹50,000. Three services are involved, and each owns its own database:

Accounts    reserve ₹50,000 on the customer's account      ✓ committed
Screening   run sanctions and fraud checks, record a pass  ✓ committed
Rail        submit the payment                             ✗ rejected, permanently

The rail says no, and it means it — the beneficiary account is closed. Retrying will not help; you could retry all afternoon and get the same answer.

Now look at what you are holding. ₹50,000 is reserved on a real customer's account, right now, in a database you do not own. A screening record exists saying this payment was checked. Neither service did anything wrong, and neither of them knows the payment is dead.

If you walk away, the customer's money stays locked up until someone notices. That is the situation, and undoing it is your job.

Why undo stopped being free

Inside one database, this is not even a conversation. You call ROLLBACK, the engine replays its undo log, and the half-finished state never existed as far as anyone outside the transaction is concerned. You got that for nothing.

Across services, none of that machinery exists. Each service ran its own transaction and committed it properly — those rows are real, they are visible, other requests are already reading them. There is no engine anywhere that can reach across and take them back.

So if you want undo, you have to write it. Every step needs a partner: the thing that cancels it out.

Step	Service	Forward action	How you undo it
1	Accounts	Reserve ₹50,000	Release the reservation
2	Screening	Record a screening pass	Void the screening record
3	Rail	Submit the payment	Nothing — see below
4	Ledger	Post settlement entries	Post reversing entries

That list is a saga. It is not a clever piece of technology — it is a sequence of ordinary local transactions, plus a written-down answer for each one about how to take it back.

When step 3 fails, you run the undos for steps 2 and 1, in that order. Backwards, strictly. Step 2 happened because step 1 had already succeeded, so unwinding them in the other order would mean releasing funds while a screening record still claims they are committed to something.

And notice what "undo" actually means here. You do not delete the reservation row and pretend it never existed. You release it — a new action, recorded, with its own timestamp. In the ledger the same rule is starker: a debit is undone by posting a credit, and both entries stay there forever. This is why the word is compensation rather than rollback. You are not erasing history, you are adding to it.

That sounds like a technicality until you are the one being audited, at which point it is the whole point.

The step that cannot be undone

Look back at step 3. What is the compensation for "submitted the payment to the rail"?

There isn't one. On an irrevocable scheme the money is gone the moment the rail accepts it. You can ask nicely for a recall and the beneficiary bank may or may not cooperate, but nothing you build can guarantee reversal.

Every saga has a step like this, and it is worth naming the three kinds it creates:

Compensatable — the steps before it. If something later fails, walk these back.
The pivot — the irreversible one. Once it succeeds, the saga can no longer be abandoned.
Retriable — the steps after it. Recording the acknowledgement, updating the read model, emailing the customer. None of these can be undone either, so they simply have to keep trying until they succeed.

Once you see the shape, the design principle falls out on its own: put everything risky before the pivot. Validation, limits, screening, the funds hold — all of it goes first, precisely so that the only work left after the irreversible step is bookkeeping that will eventually succeed on retry.

If you find yourself needing to undo something after the pivot, the sequence is wrong, and that is usually the real feedback in a design review.

Who runs the sequence?

Two ways, and the difference shows up when something breaks.

Choreography — each service listens for the previous one's event and reacts. Accounts publishes FundsReserved, screening hears it and does its bit, publishes ScreeningPassed, and so on. Nobody is in charge.

Orchestration — one component holds the sequence, calls each service, and decides what happens next.

	Choreography	Orchestration
Where the flow lives	Implicitly, spread across everyone's subscriptions	Explicitly, in one state machine
Answering "where is payment 8842?"	Reconstruct it from logs in five services	Query one row
Coupling	Low — nobody knows who comes next	The coordinator knows every participant
Compensating in reverse order	Hard — nobody owns the order	Straightforward — the coordinator has the list
Falls apart when	Past three or four steps	Rarely; the cost is one more component

Choreography feels elegant, and for two steps it is. The trouble starts the first time an operator asks where a payment is stuck, or the first time compensations have to run in a specific order — because both questions need someone who knows the whole sequence, and in choreography nobody does.

For money, orchestration. The extra component pays for itself the first week you are on call.

How an orchestrator actually runs it

Everything above says what has to happen. This is the machinery that makes it happen, and it is less exotic than the name suggests: an orchestrator is a service that owns a table.

The table has one row per saga instance — which payment, which step it is on, how many attempts, when the current step times out — and one row per step with its status. That table is the saga. Everything else is a loop:

Read the saga's current position.
Send the command for the next step, carrying a derived idempotency key.
Wait for the reply, or for the timeout to fire.
Commit the new position, in its own transaction.
Repeat — or, if the reply was a permanent failure, walk the completed steps backwards.

Step 4 is the one that matters. The position is written to a database, not held in memory, which is what lets a fresh instance pick up a half-finished saga after a crash. Everything the interactive panel below shows follows from that single decision.

Interactive — how an orchestrator actually runs a saga drag the bottom-right corner to resize, the middle bar to re-split

Four things worth watching for in those scenarios:

The table on the right is the design. Compensation runs in reverse order because the orchestrator can read which steps succeeded and in what order. Choreography has no such list, which is why it struggles here.
Parallel steps are tracked separately. Screening and the FX quote run at once, and the orchestrator only advances when both have replied.
A crash costs seconds, not a payment — provided the position is durable and every command is idempotent, so a resumed instance can safely re-send one.
Stopping is a decision the orchestrator makes. After a bounded number of failed compensations it parks the saga rather than retrying forever, and that parking is a state in the table like any other.

The part that catches people: there is no isolation

Atomicity you rebuilt with compensation. Isolation you did not rebuild at all — and that is the more dangerous gap, because nothing announces it.

Here is the concrete version. Your saga has reserved ₹50,000 and is waiting for screening. During that window, another request arrives on the same account:

10:04:02  saga reserves ₹50,000        (balance still shows ₹50,000)
10:04:05  card payment for ₹40,000 arrives, reads the balance, approves it
10:04:09  screening fails, saga releases the reservation

Two things just spent the same money, because the second one read a state the first one was in the middle of changing. A database transaction would have made that impossible. Your saga cannot.

The anomalies have names worth knowing — lost updates (someone overwrites a change your saga is relying on), dirty reads (someone reads state that is about to be undone), and fuzzy reads (the same value read twice inside your saga comes back different). But the fix is more useful than the taxonomy:

Make the intermediate state visible in the domain. Do not leave the reservation invisible — record it as a row, mark the balance as held, set the payment's status to PENDING. Then the card payment above reads an available balance of zero and declines, correctly.

That is a semantic lock: isolation rebuilt by hand, in your own data model, because the database can no longer provide it. Three things follow from that, and all three matter:

Nothing enforces it for you. Every code path that touches the balance has to respect the hold, and the one that forgets is the bug.
It is visible to users — which is exactly why your banking app shows "available balance" separately from "current balance". The semantic lock leaked into the product, deliberately.
It needs an expiry. A hold whose saga died is a hold that never releases, so something has to sweep them.

Three other tools help, roughly in this order of preference: reorder the steps so the hardest-to-undo update happens as late as possible; write commutative updates where order stops mattering (balance = balance - 30 rather than balance = 70); and re-read with a version check before acting on a value you fetched earlier.

R8 comes back to isolation properly, including why it is the ACID property you lose first.

What if the undo itself fails?

This is the question every interviewer asks, and the wrong answer is to wave it away.

Picture it: the payment is dead, you are unwinding, and the accounts service will not release the reservation. Maybe it is down. Maybe the hold has already been consumed by something else. Maybe the account closed in the meantime.

Retry with backoff first — most of these are transient and clear on the second or third attempt. But if it does not clear, stop. Park the saga in a terminal state such as needs_repair, raise an alert, and hand a human everything they need: which step, which amount, which reference, what has already been undone and what has not.

What you must not do is swallow it. A silent failure here means money is locked up and nobody knows.

Two consequences worth stating:

Compensations get retried, so they have to be idempotent too. Same derived-key discipline as forward steps (R3). Releasing a reservation twice should release it once.
A compensation can fail for a legitimate business reason, not only a technical one. The funds may already have been swept to another payment. That is a genuine break rather than a bug, and it goes to the same repair queue — which is exactly the kind of case reconciliation (R6) is built to find and settle.

The honest framing: a failed compensation is not a hole in the design, it is a visible, owned, finite queue of work — and the value sitting in that queue is a number you can report.

Implementing one, step by step

A recipe rather than a description. These thirteen steps apply to any saga, choreographed or orchestrated.

1. Check that you need one. One service and one database? Use a transaction. A synchronous call the caller can retry? That is a retry. A saga is for work that spans services where a later step can fail permanently.

2. Write the steps down, with an owner for each. Literally a list: step, which service performs it, what it changes. If two adjacent steps are owned by the same service, consider whether they are really one step.

3. Write each step's compensation next to it. Do this before writing any code, because the answer is sometimes "there isn't one" — and that is the discovery that matters. Reserve is undone by release; a posting is undone by a reversing posting; a submission to a rail is undone by nothing.

4. Find the pivot and re-order around it. The first step with no compensation is your point of no return. Move every risky, reversible step before it — validation, limits, screening, the funds hold — so that everything after it is bookkeeping that will eventually succeed. If you need to undo something after the pivot, the order is wrong.

5. Define the state machine explicitly. The states a saga instance can be in, which transitions are legal, and which states are terminal. Three terminal states, not one: completed, compensated, and needs_repair. A saga with only success and failure has nowhere to put the case a human must resolve.

6. Choose the identity, then derive from it. The saga ID is the business key — the payment ID. Every command it sends carries a key derived from it, such as hash(saga_id + step_name), so a resumed or retried command is recognised by the receiver as a replay (R3).

7. Persist intent before acting, not after. Write "step 3 starting, attempt 1, deadline 30s" and commit; then make the call. A crash after the call is now interpretable — you know an instruction may be out there. A crash before it leaves no ambiguity either.

8. Make every step idempotent — and every compensation too. Both get retried. Releasing a hold twice must release it once.

9. Give every step a timeout, and decide what the timeout does. A step waiting on a reply that never arrives will wait forever while looking healthy. For each step choose: retry, probe for the outcome, compensate, or park. "Wait longer" is not an option.

10. Add the isolation countermeasure. A saga exposes half-finished state, so decide what other requests will see: a hold, a PENDING status, a reserved flag. Give it an expiry and something that sweeps expired ones.

11. Set the failure policy numerically. How many retries, what backoff, how much jitter, and the attempt count at which you stop retrying and start compensating. Then the same again for compensations, ending in needs_repair rather than an infinite loop.

12. Emit an event at every transition, through the outbox. The saga's own state change and the event announcing it commit together (R4). This is what makes the whole thing observable and what lets downstream services react without polling you.

13. Test the ugly paths deliberately. Kill the process between each pair of steps. Deliver the same command twice. Make a compensation fail. Start a saga on one version and finish it on the next. Until those four are tested, the design is a hypothesis.

What changes when an orchestrator drives it

Steps 1 to 13 are the saga. These ten are the coordinator — the machinery that makes one service responsible for the sequence, and the part candidates usually hand-wave.

14. Model the saga as two tables, not one. The instance holds where it is; the step rows hold what happened.

CREATE TABLE saga_instance (
  id             uuid PRIMARY KEY,           -- the business key, e.g. payment id
  type           text NOT NULL,              -- 'outbound_payment'
  definition_ver int  NOT NULL,              -- which version of the flow it started on
  state          text NOT NULL,              -- running | compensating | completed
                                             -- | compensated | needs_repair
  current_step   int  NOT NULL DEFAULT 0,
  attempt        int  NOT NULL DEFAULT 0,
  due_at         timestamptz,                -- when this step times out, or when to retry
  owner          text,                       -- which worker holds it now
  lease_until    timestamptz,                -- so two workers cannot advance it at once
  fence_token    bigint NOT NULL DEFAULT 0,  -- R8
  correlation_id text NOT NULL,
  created_at     timestamptz NOT NULL DEFAULT now(),
  updated_at     timestamptz NOT NULL
);
CREATE INDEX ON saga_instance (state, due_at) WHERE state IN ('running','compensating');

CREATE TABLE saga_step (
  saga_id     uuid NOT NULL REFERENCES saga_instance(id),
  step_no     int  NOT NULL,
  name        text NOT NULL,
  direction   text NOT NULL,          -- forward | compensating
  status      text NOT NULL,          -- pending | sent | succeeded | failed | skipped
  idem_key    text NOT NULL,          -- derived; the receiver dedupes on it
  request     jsonb, response jsonb,
  sent_at     timestamptz, settled_at timestamptz,
  PRIMARY KEY (saga_id, step_no, direction)
);

The partial index on (state, due_at) is the one the dispatcher runs constantly, so it is worth having on purpose rather than by accident.

15. Build the dispatcher as a loop over due work. The orchestrator is not driven by the request thread; it is a loop that claims sagas whose due_at has passed:

SELECT * FROM saga_instance
 WHERE state IN ('running','compensating') AND due_at <= now()
 ORDER BY due_at
 LIMIT 100
 FOR UPDATE SKIP LOCKED;          -- each worker takes a disjoint batch

16. Take a lease on the instance before advancing it. SKIP LOCKED stops two workers in the same poll, but the lease is what stops a worker that has stalled from advancing the saga later. Stamp owner and lease_until, and carry the fence_token into any write the step performs (R8).

17. Dispatch commands through the outbox, never directly. The step row transitions to sent and the command lands in the outbox in one transaction. Otherwise the orchestrator can record "step 3 started" and never actually start it, which is the dual-write problem in its purest form (R4).

18. Correlate replies, and expect them late, duplicated and out of order. The reply carries the saga ID and step number. Handling it is an inbox operation: dedupe, match to the step row, ignore anything for a step already settled. A late reply to a step you already compensated must be discarded, not acted on — that is the case people miss.

19. Make timeouts durable, not in-memory. due_at is the timer. A separate sweeper claims overdue sagas — under a lease, because it is a singleton job — and applies each step's timeout policy. A timer that lives in a scheduled executor dies with the pod that owns it.

20. Recover on startup by loading, not by remembering. A fresh instance queries for unfinished sagas and resumes from current_step. There is no in-memory registry to rebuild, which is the whole reason the position is a database row.

21. Version the definition and pin each instance to one. definition_ver means a saga started under version 1 keeps running under version 1 even after you deploy version 3. Without it, a deploy mid-flight can leave a saga executing step 4 of a sequence it never started. This is the single strongest argument for a workflow engine over a hand-rolled coordinator.

22. Give operations real controls. Look a saga up by business key; force a compensation; force completion; retry one step; park. Every one of them audited and attributable, because these are the buttons someone reaches for at 3am when money is stuck.

23. Protect fairness and the downstream. One tenant's 50,000 sagas must not starve everyone else's, so the dispatcher batches by tenant or shard. And concurrency limits per participant service matter — the orchestrator can trivially generate more load than a downstream can absorb (R14).

What an interviewer is checking with all of this: that you know the orchestrator is a state machine with a durable position and a lease, not a method that calls four services in a row. The giveaway question is usually "what happens if your orchestrator restarts halfway?" — and the answer is only convincing if the position was already in a database before the restart.

Build or buy

Hand-rolling is fine for three or four steps. Past that, a workflow engine — Temporal, Camunda — earns its keep, and step 21 above is the reason: a saga started on version 1 may still be running when you deploy version 3, and engines handle in-flight versioning deliberately where hand-rolled coordinators usually do not. You also get durable timers, retries and a visual audit trail without building them.

TCC is a saga with a better first step. Try-confirm-cancel (R2) reserves explicitly, so the undo is guaranteed to work rather than invented afterwards. Where the domain has a natural hold — funds, inventory, seats — prefer it.

When it is not a saga

One service, one database. Use a transaction. This is most code.
A synchronous call the caller can just retry. One service asking another for something, failing, and retrying is a retry, not a saga.
Nothing to undo. If the steps are independent notifications, what you need is reliable delivery (R4), not compensation.

What to say in the interview

"Across services I use a saga: a sequence of local transactions, each with an explicit compensation, driven by an orchestrator so the flow is queryable when something is stuck. I sequence it so everything undoable — screening, limits, the funds hold — sits before the irreversible step, and everything after the pivot is retry-until-success. The part I'd call out is isolation: a saga has none, so intermediate state is visible to everyone, and I make that safe with a semantic lock — a hold or a pending status, with an expiry. And if a compensation itself fails, it parks in a repair state with an alert and a human picks it up; it never fails quietly."