𝗠𝘆 𝟰,𝟮𝟬𝟬 𝗗𝗼𝗹𝗹𝗮𝗿 𝗔𝗣𝗜 𝗠𝗶𝘀𝘁𝗮𝗸𝗲

𝗠𝘆 𝟰,𝟮𝟬𝟬 𝗗𝗼𝗹𝗹𝗮𝗿 𝗔𝗣𝗜 𝗠𝗶𝘀𝘁𝗮𝗸𝗲

I thought my API was safe. I installed rate limiting. My tests passed.

Then someone scraped 2 million requests in 4 hours. It cost me 4,200 dollars.

Here is why it failed.

I ran 4 server instances. Each instance had its own memory. This meant the limit was 4 times higher than I thought.

The attacker used 2,000 proxy IPs. This bypassed my IP limits.

The result:

• Traffic spiked 800 percent.
• Database queries timed out.
• My bill hit 4,200 dollars.

I rebuilt the system with three layers:

1. Distributed limits I used Redis. Now all servers share one limit. 100 requests means 100 requests total.

2. Pattern detection IPs are useless against proxy pools. I now track:

• Timing: Bots send requests at exact intervals.
• Headers: Bots use identical User-Agents.
• Entropy: Scrapers hit endpoints in alphabetical order.

3. Cost circuit breakers I assigned a dollar cost to every endpoint. I set a limit of 50 dollars per hour. If costs spike, the system throttles expensive endpoints.

Lessons learned:

• In-memory limits fail in distributed systems.
• IP addresses do not stop professional scrapers.
• Know your dollar cost per endpoint.

Security theater is dangerous. It gives you false confidence.

How do you handle rate limiting?

Source: https://dev.to/kollittle/i-thought-my-api-was-rate-limited-until-someone-scraped-2-million-requests-in-4-hours-509d

Optional learning community: https://t.me/GyaanSetuAi