Your Harness Will Lie To You Before Your Model Does

Your evaluation scoreboard said both engines failed.

Llama3.2 failed 5 out of 6 cases. Anthropic Sonnet failed 6 out of 6 cases.

The label was "malformed." But the causes were different.

One failure was an API error because credits ran out. One failure was terminal control bytes from a CLI command corrupting the text. One failure was valid JSON wrapped in markdown fences that the parser could not read.

If I published that first summary, I would have lied. I would have blamed the models for failures in my own code.

I found these bugs by looking at raw records instead of trusting the summary.

The first bug happened because I used a CLI subprocess to call Ollama. The command produced terminal animations like spinners and cursor movements. These ANSI control bytes landed in my data. The parser saw invisible characters and crashed.

The fix: Switch from CLI subprocess to a direct HTTP API.

The second bug happened because LLMs often wrap JSON in markdown fences. My parser used json.loads() on the raw string. It saw the backticks and failed.

The fix: Add a function to strip code fences before parsing.

Once I fixed the pipe, the real results appeared.

The models were not "dead." They were just being garbled by the harness. After the fix, the gap in quality between the two models became visible and measurable.

Lessons for your AI evaluation pipeline:

  • Keep the raw output. If the summary says malformed, the raw output is your only source of truth.
  • Record the reason for failure. Do not just say "malformed." Say "API error" or "parse error."
  • Freeze your testing standards. Do not change your rules to make results look better.
  • Treat the harness as part of the system under test.

The model is not the only thing on trial. Your code is too.

Source: https://dev.to/kenielzep97/your-harness-will-lie-to-you-before-your-model-does-662

Optional learning community: https://t.me/GyaanSetuAi