𝗧𝗵𝗲 𝗚𝗮𝘁𝗲 𝗙𝗶𝗿𝗲𝗱 𝟭𝟵𝟴 𝗧𝗶𝗺𝗲𝘀. 𝗜 𝗖𝗮𝗹𝗹𝗲𝗱 𝗜𝘁 "𝗪𝗼𝗿𝗸𝗶𝗻𝗴."
I built a gate to block bad code. It blocked 198 pieces of code. I thought this meant the gate worked well. I saw a high block count and felt successful.
Then I looked at the actual cases. I realized many of those blocks were mistakes. The gate was rejecting good code. It rejected work that met all requirements just because the structure looked unusual.
I made a common mistake. I confused activity with correctness.
A gate can be very active and very wrong at the same time. A high block count does not prove value. It only proves the gate is firing.
Here is how I learned to tell if a gate is actually doing its job:
- Look at the reasons for blocks. Do they catch real defects? Or do they just trip on the same surface patterns?
- Watch the retries. Does a retry fix the actual problem? Or does the code just change shape to please the gate? If the code just changes shape without improving, the gate is a problem.
- Check final convergence. Does the work eventually pass on its own merits? If you have to weaken the gate to make work pass, the gate was wrong.
A gate that works well makes the system better. A bad gate just makes the system adapt to the gate. You end up with code that is shaped to please the validator rather than code that is actually good.
Stop looking at the total count. The total count shows you activity. The sample shows you truth.
Audit a sample of what your gates, linters, or filters block. If you only test your guards with bad input, you are only asking flattering questions. You must also test if they let good, unusual input pass through.
Do you audit your block counts? How do you decide if a rejection was justified?
Source: https://dev.to/josephyeo/the-gate-fired-198-times-i-called-it-working-45fk
Optional learning community: https://t.me/GyaanSetuAi