Your AI Writes Tests That Can Never Fail

Your AI Writes Tests That Can Never Fail

You ask an AI for tests. It gives you twelve green tests. Your CI passes. You merge the code. Three days later, a bug reaches production. You look at the tests. They passed, but they tested nothing.

A green test is not proof. It is a hypothesis. AI writes hypotheses that never fail.

Look at this function:

func Discount(total int) int { if total > 100 { return total - 10 } return total }

An AI might write this test:

func TestDiscount(t *testing.T) { got := Discount(150) if got < 0 { t.Errorf("result should not be negative") } }

This test is green. It runs the code, so your coverage looks good. But the test only checks if the result is less than zero. If you change the discount to a math error, the test stays green. It does not check behavior. It only checks if the lights are on.

This is a trap. Coverage counts lines you touch. It does not count assertions that matter. A 90% coverage report can hide broken code.

AI wants the shortest path to a green light. It chooses soft assertions and mocks that test nothing.

To fix this, use one rule: check that a test knows how to fail.

Change your code on purpose. If the test stays green, the test is useless. Throw it away.

A real test looks like this:

func TestDiscount(t *testing.T) { if got := Discount(150); got != 140 { t.Errorf("Discount(150) = %d, want 140", got) } }

If you break the logic, this test goes red immediately. It bites.

You cannot do this by hand for every test. Use mutation testing instead. Tools like Gremlins in Go apply small changes to your code. They check if your tests catch the changes.

If a change does not make a test fail, you have a hole in your suite.

I use this for AI code. I do not let an AI declare its work done. I run a gate. The tool mutates the code. If the test stays green, the AI must rewrite it. The AI does not decide if a test is good. The mutation decides.

No AI test enters my codebase without proving it can fail. A test that cannot fail is worse than no test. A missing test is obvious. A fake test is dangerous.

Stop trusting green lights. A test is only worth the red it can produce.

Source: https://dev.to/ohugonnot/your-ai-writes-tests-that-can-never-fail-3i57

Optional learning community: https://t.me/GyaanSetuAi