𝗖𝗼𝗺𝗽𝗮𝗰𝘁 𝗩𝗶𝗱𝗲𝗼 𝗠𝗲𝘁𝗮𝗱𝗮𝘁𝗮 𝗪𝗶𝘁𝗵 𝗣𝗿𝗼𝘁𝗼𝗯𝘂𝗳
Our ingest worker pulls thousands of trending videos every two hours. We send this data to three services: a ranking service, a cache warmer, and an analytics job.
For a long time, we used JSON to move this data. It worked until it failed.
A single video record was 2.4 KB in JSON. Moving 40,000 records to three different services meant moving 280 MB of redundant text every two hours. This wasted bandwidth and slowed down our PHP workers.
We switched to Protocol Buffers (Protobuf). Here is why and how it changed our system.
The Three Main Problems with JSON:
- Size: JSON repeats field names in every record. A field name like "viral_score_24h" uses 15 bytes just to carry an 8-byte number.
- Parse Cost: PHP must parse the whole JSON blob even if a service only needs two fields.
- Schema Drift: JSON has no contract. If one service changes a field, others break silently at runtime.
How Protobuf Solved This:
Protobuf uses field numbers instead of names. It uses a .proto file as a single source of truth. This file generates code for PHP, Python, and Go. If you change the schema, the code fails to compile. This turns runtime bugs into compile-time errors.
The Results:
We measured these changes over one week:
- Payload size: Dropped from 2.4 KB to 720 bytes (70% reduction).
- PHP Encode time: Dropped from 310 ms to 95 ms.
- Go Decode time: Dropped from 140 ms to 38 ms.
- Bandwidth: Dropped from 280 MB to 84 MB per cycle.
When to use Protobuf:
Do not use it for everything. If your payloads are small or you need humans to read them easily, stick with JSON. Use Protobuf for high-volume internal traffic where speed and strict contracts matter most.
Our public APIs still use JSON. Our internal services use Protobuf.