R26 — Cutoffs, kick-off and warehousing: scheduled state transitions and deferred work

This is the part of payments that has no analogue in a normal web system, and it is where domain candidates separate themselves. A payment engine does not run on wall-clock time; it runs on a business date per rail, opened and closed by scheduled processes, with instructions parked until their moment arrives.

flowchart LR
  KO["Kick-off<br/>roll business date, open session,<br/>refresh FX and limits"] --> REL["Release warehoused payments<br/>whose value date is now"] --> REV["Revalidate each one<br/>account, sanctions, mandate, limits"] --> OPEN["Session open — accept and submit"]
  OPEN --> SC["Soft cutoff<br/>stop accepting, prioritise high value"] --> DR["Drain buffer<br/>submit worst-case backlog + one retry"] --> HC["Hard cutoff — rail enforces"] --> EOD["End-of-day: final batch, acks,<br/>statement to reconciliation"] --> CLS["Date closed, then rolled"]
  SC -.->|"missed the window"| NEXT["Rolls to next window<br/>value date may change — tell the customer"]

flowchart TD
  CAP["Captured<br/>captured_at = UTC audit fact"] --> W["warehoused<br/>value_date in the future<br/>or release_after set"]
  W -->|"cancel or amend<br/>races the release job"| CAN["cancelled"]
  W -->|"kick-off on value date"| RV["revalidate at release"]
  RV -->|"fails: sanctioned, closed, expired"| PARK["parked for investigation"]
  RV -->|passes| REL["released<br/>business_date assigned"]
  REL --> SUB["submitted"] --> ACK["acknowledged<br/>rail decides the business date"] --> SET["settled<br/>settlement_date known only now"]
  W -.->|"past value date and still here"| ALERT["ALERT — independent of the job's health check"]

The four dates, kept distinct

Confusing these is the most common modelling error in payment systems.

Store every timestamp in UTC with the originating zone alongside it, and store business date as a plain date with a rail attached. A payment submitted at 23:50 IST may belong to tomorrow's business date on a US rail and today's on a domestic one, simultaneously.

Cutoffs and clearing windows

Each rail has one or more daily windows with a hard cutoff, expressed in the rail's local time and therefore moving twice a year with daylight saving. ACH has several same-day windows and a standard window; high-value rails (RTGS) have a single long session with a late cutoff for bank-to-bank traffic; instant rails run continuously but still roll a business date for accounting.

Operationally you need three times per window, not one: the soft cutoff at which you stop accepting new instructions for that window, the hard cutoff the rail enforces, and the buffer between them, which exists so you can drain the queue. Sizing that buffer is a real engineering decision — it must exceed the time to submit the worst-case backlog plus one retry cycle.

As the soft cutoff approaches, behaviour changes: prioritise high-value and time-critical payments, stop accepting low-priority batch work, and warn callers that new instructions will land in the next window. Missing a cutoff is not an error to retry blindly; the payment rolls to the next window, the value date may change, and the customer must be told.

Kick-off and end-of-day

Kick-off (start-of-day) opens a new business date for a rail. It typically: rolls the business date forward using the rail's holiday calendar, opens the clearing session, releases warehoused payments whose value date is now current, refreshes FX rates and limits, and re-runs validation on everything it releases.

End-of-day closes the window: stop accepting, drain, submit the final batch, receive acknowledgements, generate the statement and hand over to reconciliation, then mark the date closed.

Both are jobs that will fail at some point, so design them accordingly: idempotent (running kick-off twice must not release a payment twice), resumable from a step marker, observable (a kick-off that did not run by its scheduled time is a page, not a dashboard item), and with a manual override, because at some point an operator will need to open a session by hand.

Date rollover

Rollover is where bugs live. Handle these explicitly:

• Non-business days. Weekends and rail holidays are per-rail and per-currency — a payment in a currency pair crosses two holiday calendars. A calendar service, versioned and auditable, is a real component, not a config file someone edits.
• Rolling conventions. When a value date lands on a non-business day, a convention decides the outcome: following moves to the next business day, modified following does the same unless it crosses the month end, in which case it moves backwards. Name these; corporate clients specify them contractually.
• In-flight across the boundary. A payment submitted at 16:59:59 against a 17:00 cutoff may be acknowledged after the session closed. Its business date is decided by the rail's acknowledgement, not your submission clock, so the state machine must be able to re-date a payment on acknowledgement.
• Clock authority. One clock and one business-date service. Never let each service compute "today" from its own container, and never derive business date from a timestamp with a timezone conversion scattered through the codebase.

Warehousing: holding payments until their moment

Date warehousing is holding a future-dated instruction until its value date. A corporate submits payroll on the 25th for value date the 30th; the instruction is accepted, validated and stored, then released at kick-off on the 30th. Warehouses routinely hold instructions for months.

Time warehousing is the intraday version: holding until a specific window or time — until the next same-day ACH window, until funding arrives, until an FX rate is fixed, until a limit frees up.

The critical design rule: validate at intake, revalidate at release. Between the two, the account may have closed, the beneficiary may have been sanctioned, the mandate may have expired, limits may have changed and the FX rate certainly has. A system that trusts its intake-time checks will eventually release a payment to a sanctioned party, which is a regulatory incident rather than a bug.

CREATE TABLE payment_instruction (
  id              bigserial PRIMARY KEY,
  captured_at     timestamptz NOT NULL,   -- UTC audit fact
  rail            text NOT NULL,
  value_date      date NOT NULL,
  release_after   timestamptz,            -- time warehousing
  business_date   date,                   -- assigned at release
  state           text NOT NULL,          -- warehoused | released | submitted
                                          -- | acknowledged | settled | returned
  priority        int NOT NULL DEFAULT 5,
  revalidated_at  timestamptz
);
CREATE INDEX ON payment_instruction (rail, value_date)
  WHERE state = 'warehoused';

CREATE TABLE rail_session (
  rail          text NOT NULL,
  business_date date NOT NULL,
  window_name   text NOT NULL,
  soft_cutoff   timestamptz NOT NULL,
  hard_cutoff   timestamptz NOT NULL,
  state         text NOT NULL,   -- pending | open | draining | closed | rolled
  PRIMARY KEY (rail, business_date, window_name)
);

The release job selects warehoused instructions for the rail and date, revalidates, assigns the business date, and transitions state — all keyed so a rerun is a no-op. Release in a deterministic order (priority, then capture order) so behaviour is reproducible, and watch for the thundering herd: a month-end kick-off may release hundreds of thousands of payroll payments at once, which is a capacity event you plan for rather than discover.

Amendment and cancellation. A warehoused payment can be cancelled or amended right up to release — a whole API surface that only exists because of warehousing, with its own race against the release job. Take the row lock in the release job and let cancellation lose or win cleanly; never both.

What interviewers probe here

What happens if kick-off does not run — do payments silently miss their value date, and would you know? Answer: a payment past its value date and still warehoused is an alert, independent of the job's own health check. What happens to a payment submitted seconds before cutoff and acknowledged after? What if the rail's file arrives late and the window cannot close? How do you handle a holiday calendar change announced mid-month? And the operational one: who can open a session manually at 2am, and what stops that from becoming routine?

The framing to carry through all of it — in payments, time is business state, not infrastructure. It is modelled, persisted, audited and alerted on, exactly like money.

What to say in the interview

"In payments, time is business state rather than infrastructure. I keep four dates distinct — capture timestamp, business date, value date, settlement date — and derive the business date from one clock service rather than from each container's idea of today. Warehoused instructions are validated at intake and revalidated at release, because an account can close or a beneficiary can be sanctioned in between. Kick-off and end-of-day are idempotent, resumable jobs with their own alerting: a payment past its value date and still warehoused pages someone, independent of whether the job reported success."