๐ฅ๐ฎ๐๐ฒ-๐๐ถ๐บ๐ถ๐๐ฒ๐ฑ ๐ช๐ฒ๐ฏ๐ต๐ผ๐ผ๐ธ๐ ๐๐ถ๐๐ต ๐ฃ๐๐ฃ ๐ฎ๐ป๐ฑ ๐ฆ๐ค๐๐ถ๐๐ฒ
A viral video crashed our system. We hit 14,000 webhook deliveries in one minute. Partner sites went down. The queue grew to 92,000 rows.
We fixed this with a boring stack. We use PHP 8.4 and SQLite in WAL mode. Cloudflare Workers handle the edge.
Here is the design:
- Per-subscriber limits. One slow partner does not slow others.
- Burst tolerance. Partners handle small spikes.
- Smart backoff. We honor Retry-After headers.
- No duplication. Every retry sends the same bytes.
- Bounded queues. We stop retrying after 24 hours.
We put token buckets in SQLite. We avoid Redis to keep it simple.
We use short leases for workers. This stops two workers from sending the same event. If a worker dies, the lease expires. Another worker picks it up.
We protect privacy for GDPR. We strip usernames from payloads. We use hashed tokens.
Boring choices keep systems honest. Flashy tools break at 3am.