https://escalopa.com/en/tags/system-design/Recent content in System-Design on EscalopaHugo -- gohugo.ioenahmed.helaly.dev@google.com (Ahmed Helaly)ahmed.helaly.dev@google.com (Ahmed Helaly)© 2026 Ahmed HelalyWed, 17 Jun 2026 00:00:00 +0000https://escalopa.com/en/posts/raft-leader-election-from-first-principles/Thu, 27 Mar 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/raft-leader-election-from-first-principles/Randomized election timeouts and term monotonicity turn competing candidates into one current leader.https://escalopa.com/en/posts/log-replication-and-the-raft-safety-story/Sat, 28 Mar 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/log-replication-and-the-raft-safety-story/A leader commits only entries known to be durable on a quorum, preserving one ordered history.https://escalopa.com/en/posts/designing-idempotent-consumers/Tue, 25 Feb 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/designing-idempotent-consumers/Persist the business effect and message identity in one atomic boundary whenever possible.https://escalopa.com/en/posts/exactly-once-is-usually-a-local-property/Sat, 25 Jan 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/exactly-once-is-usually-a-local-property/End-to-end exactly-once claims decompose into deduplication, atomicity, and replay-safe effects.https://escalopa.com/en/posts/retries-timeouts-and-the-latency-budget/Fri, 28 Mar 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/retries-timeouts-and-the-latency-budget/Retries spend remaining deadline and capacity; they do not create either.https://escalopa.com/en/posts/exponential-backoff-with-jitter/Thu, 19 Jun 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/exponential-backoff-with-jitter/Jitter breaks synchronized retry waves while caps and deadlines bound recovery cost.https://escalopa.com/en/posts/circuit-breakers-are-not-error-handling/Thu, 05 Feb 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/circuit-breakers-are-not-error-handling/A breaker protects a dependency and caller capacity; ordinary failures still require explicit policy.https://escalopa.com/en/posts/consistent-hashing-for-practical-sharding/Sat, 22 Feb 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/consistent-hashing-for-practical-sharding/A hash ring limits key movement, but load balance still depends on virtual nodes and key distribution.https://escalopa.com/en/posts/quorums-without-the-hand-waving/Sat, 26 Oct 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/quorums-without-the-hand-waving/Intersecting read and write quorums provide a reasoning tool, not automatic availability or freshness.https://escalopa.com/en/posts/ordering-events-in-distributed-systems/Tue, 31 Dec 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/ordering-events-in-distributed-systems/Choose the weakest ordering guarantee the business invariant needs and encode it per entity.https://escalopa.com/en/posts/outbox-pattern-for-reliable-event-publishing/Fri, 03 Oct 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/outbox-pattern-for-reliable-event-publishing/Commit state and an event record together, then publish asynchronously with idempotent delivery.https://escalopa.com/en/posts/saga-coordination-without-mystery/Tue, 17 Mar 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/saga-coordination-without-mystery/A saga is a state machine of forward actions, durable decisions, and explicit compensations.https://escalopa.com/en/posts/lease-based-distributed-locks/Mon, 14 Apr 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/lease-based-distributed-locks/A lease bounds stale ownership only when fencing tokens protect the resource from expired holders.https://escalopa.com/en/posts/read-repair-and-anti-entropy/Fri, 10 Apr 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/read-repair-and-anti-entropy/Foreground repair improves observed keys while background comparison closes the long tail.https://escalopa.com/en/posts/handling-hot-keys-before-they-melt-a-service/Fri, 20 Sep 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/handling-hot-keys-before-they-melt-a-service/Detect skew, isolate the key, coalesce work, and change partitioning only with evidence.https://escalopa.com/en/posts/load-shedding-as-a-correctness-feature/Sun, 13 Apr 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/load-shedding-as-a-correctness-feature/Rejecting excess work early preserves useful throughput and bounded latency for admitted requests.https://escalopa.com/en/posts/designing-for-partial-failure/Tue, 13 May 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/designing-for-partial-failure/Timeout, retry, fallback, and reconciliation paths are first-class behavior, not exceptional code.https://escalopa.com/en/posts/version-vectors-in-plain-language/Thu, 26 Feb 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/version-vectors-in-plain-language/Version vectors distinguish causality from concurrency when one scalar version cannot.https://escalopa.com/en/posts/gossip-protocols-and-eventual-membership/Thu, 10 Oct 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/gossip-protocols-and-eventual-membership/Random peer exchange trades immediate agreement for scalable, failure-tolerant convergence.https://escalopa.com/en/posts/a-practical-failure-detector/Thu, 09 Oct 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/a-practical-failure-detector/Failure detectors produce suspicions from imperfect timing; the application decides how costly suspicion may be.https://escalopa.com/en/posts/writing-architecture-decision-records/Wed, 21 May 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/writing-architecture-decision-records/Record context, forces, decision, consequences, and revisit triggers while the reasoning is fresh.https://escalopa.com/en/posts/modular-monolith-before-microservices/Sun, 21 Dec 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/modular-monolith-before-microservices/Strong internal boundaries earn optional distribution without paying network and operations costs early.https://escalopa.com/en/posts/finding-service-boundaries-with-invariants/Tue, 17 Feb 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/finding-service-boundaries-with-invariants/Place data and behavior that change under one invariant behind the same consistency boundary.https://escalopa.com/en/posts/the-real-cost-of-a-microservice/Mon, 09 Dec 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/the-real-cost-of-a-microservice/Each service adds deployment, ownership, observability, compatibility, and failure-management obligations.https://escalopa.com/en/posts/hexagonal-architecture-in-practical-go/Mon, 19 Jan 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/hexagonal-architecture-in-practical-go/Keep domain decisions behind ports and push transport and storage details to adapters.https://escalopa.com/en/posts/evolutionary-architecture-with-fitness-functions/Wed, 18 Sep 2024 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/evolutionary-architecture-with-fitness-functions/Automate checks for the architectural properties that must remain true as teams and systems change.https://escalopa.com/en/posts/design-reviews-that-produce-decisions/Thu, 25 Sep 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/design-reviews-that-produce-decisions/Share constraints and alternatives early, assign decision ownership, and record unresolved risks.https://escalopa.com/en/posts/turning-product-requirements-into-system-constraints/Fri, 14 Feb 2025 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/turning-product-requirements-into-system-constraints/Translate user journeys into load, latency, consistency, privacy, availability, and cost envelopes.https://escalopa.com/en/posts/planning-a-safe-service-rollout/Sat, 18 Apr 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/planning-a-safe-service-rollout/Stage exposure, define observability and rollback signals, and test compatibility before scaling traffic.https://escalopa.com/en/posts/cross-team-api-ownership/Wed, 17 Jun 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/cross-team-api-ownership/One team owns the contract while consumers participate through compatibility tests and change visibility.https://escalopa.com/en/posts/technical-onboarding-as-a-system/Thu, 16 Apr 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/technical-onboarding-as-a-system/Give newcomers a map, a runnable path, progressive tasks, and feedback loops instead of a document dump.https://escalopa.com/en/posts/teaching-distributed-systems-with-failure-scenarios/Wed, 03 Jun 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/teaching-distributed-systems-with-failure-scenarios/Start from observable failures and invariants, then introduce algorithms as tools that preserve them.https://escalopa.com/en/posts/code-review-for-high-load-services/Wed, 07 Jan 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/code-review-for-high-load-services/Review correctness under concurrency, bounds, failure behavior, observability, and rollout—not only syntax.https://escalopa.com/en/posts/performance-optimization-as-an-experiment/Sat, 10 Jan 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/performance-optimization-as-an-experiment/State a measurable hypothesis, control the workload, compare distributions, and retain a rollback path.https://escalopa.com/en/posts/building-a-latency-budget-across-services/Tue, 02 Jun 2026 00:00:00 +0000ahmed.helaly.dev@google.com (Ahmed Helaly)https://escalopa.com/en/posts/building-a-latency-budget-across-services/Allocate a percentile-aware deadline across work, dependencies, queues, and limited retries.