https://escalopa.com/en/tags/distributed-systems/Recent content in Distributed-Systems on EscalopaHugo -- gohugo.ioenahmed.helaly.dev@google.com (Ahmed Helaly)ahmed.helaly.dev@google.com (Ahmed Helaly)© 2026 Ahmed HelalyFri, 10 Apr 2026 00:00:00 +0000https://escalopa.com/en/posts/raft-leader-election-from-first-principles/Thu, 27 Mar 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/raft-leader-election-from-first-principles/Randomized election timeouts and term monotonicity turn competing candidates into one current leader.https://escalopa.com/en/posts/log-replication-and-the-raft-safety-story/Sat, 28 Mar 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/log-replication-and-the-raft-safety-story/A leader commits only entries known to be durable on a quorum, preserving one ordered history.https://escalopa.com/en/posts/designing-idempotent-consumers/Tue, 25 Feb 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/designing-idempotent-consumers/Persist the business effect and message identity in one atomic boundary whenever possible.https://escalopa.com/en/posts/exactly-once-is-usually-a-local-property/Sat, 25 Jan 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/exactly-once-is-usually-a-local-property/End-to-end exactly-once claims decompose into deduplication, atomicity, and replay-safe effects.https://escalopa.com/en/posts/retries-timeouts-and-the-latency-budget/Fri, 28 Mar 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/retries-timeouts-and-the-latency-budget/Retries spend remaining deadline and capacity; they do not create either.https://escalopa.com/en/posts/exponential-backoff-with-jitter/Thu, 19 Jun 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/exponential-backoff-with-jitter/Jitter breaks synchronized retry waves while caps and deadlines bound recovery cost.https://escalopa.com/en/posts/circuit-breakers-are-not-error-handling/Thu, 05 Feb 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/circuit-breakers-are-not-error-handling/A breaker protects a dependency and caller capacity; ordinary failures still require explicit policy.https://escalopa.com/en/posts/consistent-hashing-for-practical-sharding/Sat, 22 Feb 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/consistent-hashing-for-practical-sharding/A hash ring limits key movement, but load balance still depends on virtual nodes and key distribution.https://escalopa.com/en/posts/quorums-without-the-hand-waving/Sat, 26 Oct 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/quorums-without-the-hand-waving/Intersecting read and write quorums provide a reasoning tool, not automatic availability or freshness.https://escalopa.com/en/posts/ordering-events-in-distributed-systems/Tue, 31 Dec 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/ordering-events-in-distributed-systems/Choose the weakest ordering guarantee the business invariant needs and encode it per entity.https://escalopa.com/en/posts/outbox-pattern-for-reliable-event-publishing/Fri, 03 Oct 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/outbox-pattern-for-reliable-event-publishing/Commit state and an event record together, then publish asynchronously with idempotent delivery.https://escalopa.com/en/posts/saga-coordination-without-mystery/Tue, 17 Mar 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/saga-coordination-without-mystery/A saga is a state machine of forward actions, durable decisions, and explicit compensations.https://escalopa.com/en/posts/lease-based-distributed-locks/Mon, 14 Apr 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/lease-based-distributed-locks/A lease bounds stale ownership only when fencing tokens protect the resource from expired holders.https://escalopa.com/en/posts/read-repair-and-anti-entropy/Fri, 10 Apr 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/read-repair-and-anti-entropy/Foreground repair improves observed keys while background comparison closes the long tail.https://escalopa.com/en/posts/handling-hot-keys-before-they-melt-a-service/Fri, 20 Sep 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/handling-hot-keys-before-they-melt-a-service/Detect skew, isolate the key, coalesce work, and change partitioning only with evidence.https://escalopa.com/en/posts/load-shedding-as-a-correctness-feature/Sun, 13 Apr 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/load-shedding-as-a-correctness-feature/Rejecting excess work early preserves useful throughput and bounded latency for admitted requests.https://escalopa.com/en/posts/designing-for-partial-failure/Tue, 13 May 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/designing-for-partial-failure/Timeout, retry, fallback, and reconciliation paths are first-class behavior, not exceptional code.https://escalopa.com/en/posts/version-vectors-in-plain-language/Thu, 26 Feb 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/version-vectors-in-plain-language/Version vectors distinguish causality from concurrency when one scalar version cannot.https://escalopa.com/en/posts/gossip-protocols-and-eventual-membership/Thu, 10 Oct 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/gossip-protocols-and-eventual-membership/Random peer exchange trades immediate agreement for scalable, failure-tolerant convergence.https://escalopa.com/en/posts/a-practical-failure-detector/Thu, 09 Oct 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/a-practical-failure-detector/Failure detectors produce suspicions from imperfect timing; the application decides how costly suspicion may be.