R15 — Observability, SLOs and incidents: detecting the failures a healthy-looking system hides

R13 and R14 made the system bigger and harder to knock over; this section is how you find out when it is broken anyway. Every earlier pattern — outbox, saga, reconciliation, circuit breaker — fails quietly unless something is watching for it, so this is where the signals, the targets and the incident discipline live.

Monitoring versus observability

Monitoring answers questions you thought of in advance: is CPU high, is the error rate above 1%, is the queue growing. You decided what to watch, and you watch it.

flowchart TD
  subgraph SYS["System signals"]
    G["Latency, traffic, errors, saturation<br/>RED per service, USE per resource"]
  end
  subgraph BUS["Business signals — outrank the above"]
    B1["Payment success rate by rail"]
    B2["Stuck payments by state and age"]
    B3["Reconciliation break age<br/>value unreconciled"]
    B4["Settlement lag"]
  end
  G --> AL{"Symptom users feel?"}
  B1 --> AL
  B2 --> AL
  B3 --> AL
  B4 --> AL
  AL -->|yes| PAGE["Page — must be actionable"]
  AL -->|"cause, not symptom"| DASH["Dashboard + traces"]
  PAGE --> EB["Error budget burn<br/>exhausted: reliability outranks features"]

Observability is whether you can answer questions you did not think of in advance — "why are payments to this one corridor slow, but only for one client, and only since Tuesday?" — without shipping new code to find out.

The difference in practice is dimensions and correlation: enough context attached to what you record that you can slice it afterwards, and enough linkage that you can follow one payment across twelve services. Everything below is in service of those two words.

The three signals, and what each is for

Metric types worth knowing by name, because they behave differently:

• Counter — only goes up. Requests served, payments submitted. You almost always graph its rate, not its value.
• Gauge — goes up and down. Queue depth, connections in use, pending payments.
• Histogram / summary — a distribution, which is what gives you percentiles. Latency is always a histogram; an average latency metric is close to useless.

Cardinality is the thing that will surprise you. A metric's cost is the number of unique label combinations. payments_total{rail, currency, status} with 6 rails, 20 currencies and 5 statuses is 600 series — fine. Add payment_id and it is one series per payment, which will break your metrics backend and your budget in the same afternoon. Identifiers go in logs and traces. Never in metric labels.

Tracing: traces, spans, context and the aggregator

A span is one unit of work with a start and end — an HTTP handler, a database call, a Kafka publish. It carries a name, a duration, attributes, and its parent's ID.

A trace is the tree of all spans belonging to one request, tied together by a shared trace ID.

trace 4bf92f…                                     total 412ms
├─ POST /payments                    api        412ms
│  ├─ validate                       api          8ms
│  ├─ SELECT account                 postgres    11ms
│  ├─ POST /screening                screening  180ms   ← the answer
│  │   └─ SELECT sanctions_list      postgres   174ms
│  └─ publish PaymentSubmitted       kafka        6ms

The value is in that one line: without a trace you know the request took 412ms; with it you know 174ms of it was one unindexed query in someone else's service.

Context propagation is what makes the tree possible. The trace ID and the current span ID travel with the request in a header — the W3C standard is traceparent — and every service reads it, starts a child span, and passes it on. Across async hops this must be done deliberately: put traceparent in the Kafka message headers or the SQS message attributes, or your trace stops at the broker. This is the detail interviewers probe, because most people only ever instrument HTTP.

Sampling. Tracing everything at high volume is expensive, so you keep a subset. Head-based sampling decides at the start (keep 1%), which is cheap and blind — it throws away the slow request you actually needed. Tail-based sampling buffers a whole trace and decides at the end, keeping everything that was slow or errored plus a small sample of the healthy ones. Tail-based is what you want for payments; it costs a collector that can hold traces in flight.

The aggregator (collector). Your services do not talk to the tracing backend directly. They export to a collector — the OpenTelemetry Collector, or an agent like the X-Ray or Datadog daemon — running as a sidecar or a DaemonSet. The collector batches, enriches, samples and forwards. Two reasons this matters: your application never blocks on the observability backend, and you can change backends (X-Ray → Jaeger → Tempo → a vendor) by editing collector config rather than redeploying every service.

Spring Boot: what you actually switch on

Spring Boot gives you most of this with dependencies and configuration rather than code.

Actuator exposes operational endpoints: /actuator/health (with liveness and readiness groups, which is what ECS and Kubernetes probe), /actuator/metrics, /actuator/prometheus, /actuator/loggers for changing log levels at runtime without a restart.

Micrometer is the metrics facade — SLF4J, but for metrics. You write to one API and swap the backend by dependency: micrometer-registry-prometheus, micrometer-registry-cloudwatch2, micrometer-registry-otlp.

@Service
class PaymentService {
    private final Counter submitted;
    private final Timer   railLatency;

    PaymentService(MeterRegistry reg) {
        this.submitted   = Counter.builder("payments.submitted")
                                  .tag("rail", "neft")          // low cardinality only
                                  .register(reg);
        this.railLatency = Timer.builder("payments.rail.latency")
                                .publishPercentileHistogram()   // needed for p99
                                .register(reg);
    }

    void submit(Payment p) {
        railLatency.record(() -> rail.send(p));
        submitted.increment();
    }
}

Out of the box you already get JVM, HikariCP, HTTP server and client, and Kafka consumer metrics — http.server.requests alone gives you RED per endpoint. @Timed and @Observed annotate a method; a Gauge registered against a supplier reports things like pending-payment counts.

Micrometer Tracing (the successor to Spring Cloud Sleuth) handles spans and propagation, bridging to OpenTelemetry or Brave. Add the bridge plus an exporter and you get trace and span IDs injected into the logging MDC automatically, so every log line carries them:

management.tracing.sampling.probability=0.1
management.otlp.tracing.endpoint=http://otel-collector:4318/v1/traces
logging.pattern.level=%5p [${spring.application.name},%X{traceId:-},%X{spanId:-}]

Structured logging. Log JSON, not prose, and put the identifiers in fields: payment_id, trace_id, client_id, rail. One MDC.put("payment_id", id) at the entry point means every downstream log line carries it, and a query like payment_id = "8842" returns the whole story. Prose logs are grep-able; structured logs are queryable, and at 3am that is the difference.

CloudWatch: metrics, custom metrics, alarms

The model. A metric lives in a namespace (AWS/ECS, or your own Payments/Api), has a name, and carries up to 30 dimensions — key-value pairs like rail=neft. Every unique dimension combination is a separate metric, billed separately, which is cardinality discipline again, now with an invoice attached.

Getting custom metrics in, three ways, cheapest last:

1. Embedded Metric Format (EMF) — write a specially structured JSON log line, and CloudWatch extracts metrics from it automatically. No extra API call, no added latency, and you get the log and the metric from one write. This is the default choice on ECS and Lambda.
2. Micrometer's CloudWatch registry — batches and calls PutMetricData in the background. Natural if you already use Micrometer.
3. PutMetricData directly — fine for a one-off, but it is a network call on your path if you do it synchronously. Batch it.

You can also create metrics from logs you already have: a metric filter counts matching log lines (ERROR, sanctions_hit), and Logs Insights queries them ad hoc:

fields @timestamp, payment_id, rail, latency_ms
| filter status = "STUCK" and latency_ms > 5000
| stats count(*) by rail

Alarms. An alarm watches one metric (or a Metrics Insights query) and moves between OK, ALARM and INSUFFICIENT_DATA.

• Static threshold — the default. p99 latency > 800ms for 3 of 5 periods. The "M of N" evaluation is what stops a single spike paging someone.
• Anomaly detection — CloudWatch learns the normal band, which suits metrics with a daily shape, like payment volume that always dips at 2am.
• Composite alarms — combine several: ALARM(high_latency) AND ALARM(error_rate). This is how you cut alert noise, because one real incident stops firing five pagers.
• Missing data matters. Choose treatMissingData deliberately: for a metric that only exists when something is wrong, notBreaching is right; for a heartbeat metric, missing data is the alarm — a kick-off job that did not run emits nothing at all, and breaching is what catches it.

Custom alerts worth having in payments, beyond the infrastructure defaults:

stuck_payments_over_15m  > 0        for 2 periods   → page
reconciliation_break_age > 24h                      → page
unreconciled_value       > ₹10,00,000               → page
dlq_depth                > 0        for 3 periods   → ticket
kafka_consumer_lag       > 10,000   for 5 periods   → page
kickoff_job_heartbeat    missing after 09:05        → page
payment_success_rate     < 99% by rail              → page

Notice how few of those are system metrics. Every one is a symptom a customer would recognise.

Wiring it to PagerDuty. Alarm → SNS topic → PagerDuty integration endpoint. Concretely:

1. Create an SNS topic per severity — alerts-page, alerts-ticket.
2. In PagerDuty, add a CloudWatch integration on the service and copy its endpoint URL.
3. Subscribe that URL to the SNS topic as an HTTPS subscription; PagerDuty parses the alarm payload and opens an incident, resolving it automatically when the alarm returns to OK.
4. Route by severity in PagerDuty, not in your alarm names: page for anything customer-visible, ticket for the rest.

EventBridge is the more flexible alternative when you want to enrich or route on alarm content, and the same SNS topic can also feed a Slack channel for awareness without paging.

Dashboards. Aim for one page that answers "is money moving?" in five seconds, and resist the urge to graph everything. A workable layout for a payments platform:

• Top row — business: payments per minute by rail, success rate, stuck payments by age, unreconciled value.
• Second row — experience: p50/p99 latency per endpoint, error rate, rail response times.
• Third row — plumbing: consumer lag, DLQ depth, outbox oldest-unpublished age, connection pool saturation.
• Side panel: current alarm states, and deploy markers so you can see whether a change caused the shape.

X-Ray and ServiceLens tie the three signals together on AWS: a service map from trace data, with the ability to jump from a slow trace to the logs of that exact request. The ADOT (AWS Distro for OpenTelemetry) collector is the standard way to feed it from ECS.

What to actually watch: a starting list

Alert on the top two blocks; dashboard the rest. That ordering is the whole philosophy in one sentence.

Golden signals, SLOs and alert design

The four golden signals are latency, traffic, errors and saturation — the minimum set for any user-facing service. Two refinements worth quoting: track the latency of failed requests separately, because a fast error and a slow error are different problems and mixing them distorts your percentiles; and treat saturation as "how full", since most systems degrade well before 100% utilisation. RED (rate, errors, duration) is the service-level restatement; USE (utilisation, saturation, errors) is the resource-level one.

Symptoms, not causes. Your monitoring answers two questions: what is broken, and why. Page on the what — the symptom users feel. Keep the why in dashboards and traces. Every page must be actionable and represent a clear failure; anything that fires because "something seems off" trains people to ignore the pager.

Correlation. One ID per payment, propagated through every hop including async ones — put it in Kafka headers, not just HTTP baggage. Most candidates describe tracing only across synchronous calls, and interviewers probe exactly there. Keep payment IDs in traces and logs, never in metric labels, or cardinality explodes. Use tail-based sampling so you retain the slow and failed traces instead of a random sample of healthy ones.

Payments metrics that outrank system metrics.

• Payment success rate, split by rail and corridor.
• Stuck payments by state and age — count and total value.
• Reconciliation break count, break age, and value at risk unreconciled.
• Settlement lag against expected cycle.
• Duplicate detection rate and idempotency replay rate.

The failure mode to name: every system metric green, 200 payments stuck in submitted, and nobody paged. Design alerting to catch that state first.

SLOs and error budgets. Define a small number of user-facing SLOs (payment accepted within X ms at the 99th percentile; 99.9% of payments settled within one cycle). The error budget is the allowance of failure, and its function is organisational: when it is exhausted, reliability work outranks features. That is the mechanism a manager uses to arbitrate, and saying it that way is worth more than naming tools. The SLI, SLO and error-budget arithmetic behind that sentence, including burn-rate alerting, follows after this section.

Incidents. Severity defined by money and customer impact, not by component. Separate roles — commander, communications, scribe — so the person fixing is not the person updating stakeholders. Blameless postmortems with owned, dated actions, and a check that those actions actually close. Measure detection and recovery separately; detection is usually the bigger win, because most long incidents were invisible for the first forty minutes.

The 90-day answer to an alert-fatigued team: measure what is paging, delete the non-actionable alerts, fix the top three causes, then negotiate an error budget policy. Sequenced, with numbers.

The golden signals tell you what to watch; the next three sections turn that into numbers you can commit to — what an SLO actually measures, how availability composes across dependencies, and how to measure latency without fooling yourself.

Measuring it: SLI, SLO, SLA and the error budget

Three words that get used interchangeably and should not be.

• SLI — service level indicator. The measurement itself. "The proportion of payment submissions that returned a valid response in under 500ms."
• SLO — service level objective. Your internal target for that indicator. "99.9% over a rolling 30 days." Chosen by you, and deliberately tighter than anything you promised anyone.
• SLA — service level agreement. The contract with a customer, with consequences attached — service credits, penalties, an exit clause. Always looser than the SLO, because you want to be failing your own target long before you are failing a customer's contract.

If an interviewer asks you to "define an SLA for this system", the strong move is to separate the three and then talk mostly about SLOs, because that is the one engineering actually operates against.

A good SLI is a ratio of good events to valid events, not a gauge:

availability SLI = successful requests / valid requests
latency SLI      = requests served under 500ms / valid requests
freshness SLI    = payments reconciled within one cycle / payments settled

"Valid" is doing real work in those formulas. A request rejected with 400 because the client sent nonsense is not your failure and should not count against you; a 500 is. Health-check traffic and load tests are excluded. Decide what counts as a valid event before you publish the number, or you will be renegotiating it during an incident.

Payments-specific SLIs worth proposing, because they are the ones a business recognises:

The error budget

If the SLO is 99.9%, then 0.1% is not a failure — it is a budget you are allowed to spend.

30-day error budget at 99.9%  = 0.001 × 30 × 24 × 60  = 43.2 minutes
at 99.95%                                              = 21.6 minutes
at 99.99%                                              =  4.3 minutes

The budget turns reliability from an argument into arithmetic. Budget remaining means you can ship; budget exhausted means reliability work takes priority until it recovers. That is a policy the whole organisation can agree to in advance, rather than a debate during every incident.

Burn rate is how you alert on it. Burning budget at 1× means you will exactly exhaust it at the end of the window; at 14.4× you will exhaust a 30-day budget in about two days. The standard approach is multi-window, multi-burn-rate: page on a fast burn (14×, measured over an hour, confirmed over five minutes) and raise a ticket on a slow burn (around 6× over six hours). The slow burn catches the steady 0.5% error rate that no single spike would ever trigger.

Availability, and how to calculate it

Do not measure availability as host uptime. A service whose every instance is running and returning 500s is up by that definition, and useless. Measure successful units of work:

availability = successful requests / valid requests, over the window

The nines, per 30-day month, so you can quote them without doing arithmetic in the room:

Composing dependencies — the part people get wrong

In series, availabilities multiply. If a request must pass through three services, each at 99.9%:

0.999 × 0.999 × 0.999 = 0.997  →  99.7%, about 2 hours of downtime a month

Three "three nines" services produce a two-nines experience. This is the single strongest argument against long synchronous call chains, and it is why R4's asynchronous events are not just a decoupling preference — removing a service from the synchronous path removes its failures from your availability.

In parallel, unavailabilities multiply. With N independent redundant instances each at availability A:

availability = 1 − (1 − A)^N
two instances at 99.9%  →  1 − 0.001²  = 99.9999%

That number is a fantasy unless the failures are genuinely independent — and they rarely are. A shared database, a shared config push, a shared AZ or a shared bad deploy correlates them instantly. Quote the parallel formula, then immediately name the correlated failure modes, because that is what separates someone who has run systems from someone who has read about them.

MTBF, MTTR, and why detection dominates

availability = MTBF / (MTBF + MTTR)

Mean time between failures over mean time between failures plus mean time to recovery. Two ways to raise availability: fail less often, or recover faster. Recovering faster is almost always the cheaper lever, and within it, detection is usually the bigger half — most long incidents are long because nobody noticed for forty minutes. So split MTTR into MTTD (detect), time to diagnose, and time to fix, and measure them separately. Halving detection time halves a large chunk of your downtime without touching the code.

Measure from where the customer is

Server-side success rate has a blind spot: it cannot see the requests that never arrived. A failed load balancer, a DNS problem or a saturated network produces a beautiful 100% server-side availability while customers see nothing.

So measure at three points and compare: synthetic probes from outside the region, client-side telemetry from the app, and server-side metrics. When the three disagree, the disagreement is the finding.

Latency, measured honestly

Percentiles, never averages. Ninety-nine requests at 10ms and one at 3 seconds averages to a comfortable 40ms while one customer is furious. Quote p50, p95, p99, and for payments often p99.9, because at scale the tail is somebody's whole experience.

Three measurement traps worth knowing by name:

• You cannot average percentiles. The mean of each instance's p99 is not the fleet p99 and can be wildly wrong. Aggregate the underlying histograms — Prometheus histogram_quantile, or a t-digest — not the summary values. In Micrometer this is publishPercentileHistogram() rather than publishPercentiles(), which is exactly the setting the Spring Boot section above calls out.
• Coordinated omission. A load generator that waits for a response before sending the next request stops sending during a stall, so the stall never appears in the results. Open-model load tools, or a corrected generator, avoid flattering you.
• Fan-out amplifies the tail. If a request calls ten services each with a p99 of 100ms, the chance that at least one is slow is 1 − 0.99¹⁰ ≈ 9.6%. Your p99 is roughly their p90. The fix is fewer synchronous hops, hedged requests, or accepting partial results — not chasing each dependency's p99.

How a monitoring platform is itself built — ingesting metrics at millions of points per second, storing time series, evaluating alerts at scale — is a design question in its own right. R21's Datadog walkthrough covers it; this section stays on the consumer side of that platform.

What to say in the interview

"System green and money stuck is the state I design alerting to catch, so my top-line alerts are business signals — stuck payments by age, reconciliation break age, value unreconciled — not CPU. Every payment carries one correlation ID propagated across async hops in message headers, identifiers live in logs and traces rather than metric labels, and I use tail-based sampling so the slow and failed traces are the ones I keep. Alarms are symptom-based and composite so one incident doesn't page five people, and they route through SNS into PagerDuty with severity deciding page versus ticket."