Reliability

Align connect, header, body, idle, and upstream deadlines so one layer does not outlive another.

Foreground repair improves observed keys while background comparison closes the long tail.

A reproducible multi-stage build and minimal runtime reduce drift, size, and attack surface.

A leader commits only entries known to be durable on a quorum, preserving one ordered history.

A saga is a state machine of forward actions, durable decisions, and explicit compensations.

Choose buckets around decisions and inspect distributions; averages conceal the users waiting longest.

Version vectors distinguish causality from concurrency when one scalar version cannot.

Instrument bounded dimensions and user outcomes; labels are a data model with a capacity cost.

A breaker protects a dependency and caller capacity; ordinary failures still require explicit policy.

Keep instrumentation semantic and portable while isolating exporter and sampling policy.

A rollout is safe when new instances prove dependencies, warmup, and service health before receiving load.

Start from operator questions and put traffic, errors, latency, saturation, and deploy context together.

Failure detectors produce suspicions from imperfect timing; the application decides how costly suspicion may be.

Commit state and an event record together, then publish asynchronously with idempotent delivery.

Requests drive placement; limits change runtime behavior; both need measurements from realistic load.

Jitter breaks synchronized retry waves while caps and deadlines bound recovery cost.

Timeout, retry, fallback, and reconciliation paths are first-class behavior, not exceptional code.

A lease bounds stale ownership only when fencing tokens protect the resource from expired holders.

Rejecting excess work early preserves useful throughput and bounded latency for admitted requests.

Retries spend remaining deadline and capacity; they do not create either.

Randomized election timeouts and term monotonicity turn competing candidates into one current leader.

Concurrency equals arrival rate times time in system, giving a fast check for queues and resource demand.

Logs need stable event names, correlation, severity discipline, and deliberate data minimization.

Persist the business effect and message identity in one atomic boundary whenever possible.

A hash ring limits key movement, but load balance still depends on virtual nodes and key distribution.

End-to-end exactly-once claims decompose into deduplication, atomicity, and replay-safe effects.

Choose the weakest ordering guarantee the business invariant needs and encode it per entity.

Intersecting read and write quorums provide a reasoning tool, not automatic availability or freshness.

Random peer exchange trades immediate agreement for scalable, failure-tolerant convergence.

Detect skew, isolate the key, coalesce work, and change partitioning only with evidence.

Propagate context, name spans by operation, and attach identifiers without recording sensitive payloads.

Stabilize impact, preserve a timeline, test hypotheses with signals, and separate recovery from diagnosis.

Use short-lived identity-bound credentials and design renewal and revocation as runtime paths.

An SLO connects user-visible success to an error budget and concrete release decisions.

Startup, readiness, and liveness answer different questions and should trigger different actions.