𝗦𝗰𝗼𝗿𝗶𝗻𝗴 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀: 𝗗𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 + 𝗮𝗻 𝗟𝗟𝗠 𝗝𝘂𝗱𝗴𝗲

You run many small AI agents. You have agents for backend, frontend, mobile, and devops. Each agent has one job.

When you have many agents, you face a problem. You do not know if they are good. You do not know if a prompt edit makes them better or worse. Saying "it looks fine" does not work at scale.

I built a framework to solve this. It uses numbers to measure performance and improves prompts automatically.

The Strategy

Measure what you can measure with math first. Use an LLM judge only when you must. Deterministic metrics are fast and free. An LLM judge is slow and costs money.

How the system works:

• The harness runs each agent as a separate process. • It feeds a task to the agent. • It captures the output. • It scores the result against expected data.

The agent only needs to read from stdin and write to stdout. It can be Python or a shell script. The harness does not care.

Five core metrics to track:

  • Accuracy: Does the output match the goal?
  • Fuzzy score: How similar is the text to the target?
  • Timeout rate: How often does the agent fail to finish?
  • Safety violations: Does the output match unsafe patterns?
  • Reproducibility variance: Does the agent give the same answer every time?

If an agent is correct but inconsistent, it is a bug.

The LLM Judge

Some things are hard to measure with math. You need to know if an agent stayed in its role or followed constraints.

For these cases, an LLM judge reviews the work. It receives a rubric and the agent output. It returns a structured verdict. I validate this verdict against a JSON schema so it does not break the report.

The judge does more than just grade. It must suggest fixes. A critique like "this is weak" is useless. A critique like "add a JSON block to the prompt" is actionable.

The Improvement Loop

Failures go into a file. This file feeds an automated loop. The system looks at the weakest part of a prompt and tries to fix it. It keeps a pool of good candidates. It writes the best versions back to the code.

A single score is a snapshot. Use history to track trends. This tells you if you are getting better over time.

Build your foundation on deterministic metrics. Use the judge as a scalpel, not a hammer.

Fuente: https://dev.to/pponali/scoring-ai-agents-deterministic-metrics-an-llm-judge-poj

Comunidad de aprendizaje opcional: https://t.me/GyaanSetuAi