Routing Asia Pacific Video Traffic With etcd

A video-streaming platform serving eight Asia-Pacific markets stopped a recurring routing mistake by moving region-specific configuration into etcd. Propagation time for routing changes dropped to about a second. Editors could now boost a music pool for a few hours without touching code, and the platform stopped sending South-Korean viewers the Tokyo feed.

Why the old approach broke

Each router read three values for every request: the trending pool, the language-specific tokenizer, and a fallback chain. Business decisions—promoting a new artist, reacting to a regional outage, testing a recommendation algorithm—driven those values, not code changes.

Initially every deployment bundled a JSON file with the routing table. During an incident an on-call engineer edited the file on one node to point traffic to a backup pool but never updated the other seven nodes. Config drift emerged: eight countries ran divergent tables, and no single source verified the “truth”. The bug that sent Seoul viewers to Tokyo persisted because the code stayed the same; only hidden configuration differed.

Choosing etcd as the single source of truth

The team compared three options:

  • SQLite/MySQL – would force each router to poll a database, adding latency or flooding queries.
  • Consul – a solid service-discovery tool, but the platform didn’t need its full mesh features.
  • etcd – a strongly consistent key-value store with a watch primitive that notifies clients the moment a key changes.

The watch feature tipped the scales. Instead of each router repeatedly asking “has anything changed?”, routers sat idle until etcd pushed an update. Unnecessary network traffic vanished and every instance learned about a change simultaneously.

Patterns that keep the system safe

Etcd alone didn’t solve all risks. Engineers added three complementary patterns:

  1. Leases – an editor can set a temporary boost (e.g., increase the weight of a Korean-pop pool for six hours). The lease expires automatically, so the boost disappears without manual rollback.
  2. Compare-and-swap (CAS) – when two people edit the same setting concurrently, CAS fails loudly for one, preventing silent overwrites.
  3. Sidecar process – PHP struggles with long-lived connections. A tiny Go sidecar on each box watches etcd and writes a snapshot of the routing table to a shared-memory file (/dev/shm). PHP reads that local file, avoiding any network round-trip during request handling.

Resilience built into the architecture

The new design adds several safety nets:

  • Zero-latency reads – the PHP hot path reads from local memory, so requests never stall waiting for a remote store.
  • Graceful degradation – if etcd goes down, routers keep serving the last known good configuration, preventing a sudden outage.
  • Reliable updates – the sidecar handles reconnection logic and guarantees no change is missed, even if the etcd connection drops temporarily.

What changed on the ground

After migration the team saw a stark drop in incidents caused by stale or mismatched routing data. A single console now shows the current configuration, and any edit propagates to all eight regions within a second. Temporary boosts clean themselves up when their lease expires, removing the manual clean-up steps that previously led to human error.

Counter-point: the cost of a sidecar

Adding a sidecar means a second process per server and a Go runtime in a PHP-centric stack. Some operators worry about extra memory use and the need to monitor another binary. In practice the sidecar’s footprint stays modest, and the reliability gains—especially the guarantee that PHP never blocks on a network call—outweigh the operational overhead.

What to watch next

Teams managing multi-region services should monitor:

  • etcd health metrics – the routing layer hinges on a single store; keep an eye on quorum status and latency.
  • Lease expiration handling – match lease times to business windows; overly long leases leave stale boosts in place.
  • Scaling the watch load – as routers grow, watch connections increase; plan capacity for etcd servers accordingly.

Takeaway

For any service that needs fast, coordinated configuration changes across many regions, etcd’s watch, lease, and transaction primitives offer a lightweight, strongly consistent alternative to file-based configs or heavyweight meshes. Turning configuration into a push-driven, self-cleaning store eliminated a whole class of incidents and gave the platform real-time control over its routing logic.