在 GitHub Actions 合并队列中度过的八个月，会教会你功能对比矩阵永远无法传授的东西。一个框架可以提供五十种指标、华丽的仪表板以及来自权威研究实验室的引用。但如果它仅仅因为“感觉检查”（vibe check）分数在面对相同代码时从 0.72 漂移到了 0.68 就阻塞了你的部署，那它不仅没用，反而有害。它会变成你交付速度的直接威胁。

这是大多数 LLM 评估汇总所忽略的筛选标准。它们统计的是功能。但它们很少问出合并队列中唯一重要的问题：这个检查在每次运行时，通过和失败的方式是否完全一致？

我通过做一些“不舒服”的工作学到了这一点。我将六个开源的 LLM 评估框架接入了一个真实的 CI 流水线。它们针对真实的生产环境 Pull Request 运行了八个月。其中两个赢得了继续担任“守门员”的资格。其余的都被降级为咨询性质的仪表板，移至夜间作业，或者被彻底移除。教训是深刻且昂贵的：当你守护主分支时，确定性结构胜过概率性质量。

合并门禁的真实职责

CI 门禁不是研究环境，它是一个保安。它的全部目的就是观察特定的变更并回答“是”或“否”。是，这个 PR 可以合并到主分支。否，它不可以。这个答案需要在几秒钟内给出，成本只需几分钱，并且绝不能发生追溯性的反转。如果你在平静的周二和忙乱的周五对同一个提交重新运行相同的流水线，结果必须完全一致。

这正是大多数 LLM 评估框架栽跟头的地方。它们是由数据科学家为数据科学家构建的。它们针对洞察力、探索性和细微的评分进行优化。而合并队列针对的是二元决策、速度和零不稳定性（flakiness）。这两者的目标只有部分重叠。

为什么 LLM-as-Judge 会破坏队列

在我测试中失败的那些工具都有一个共同的设计缺陷：它们过度依赖 LLM-as-judge 调用作为主要的门禁机制。

LLM-as-judge 的提示词要求模型按 1 到 10 分进行评分，或者在两个回答中选出更好的一个，或者评估事实准确性。这种方法对于理解质量趋势非常强大，但对于阻塞式的 CI 检查来说却是毒药。由于温度（temperature）、模型版本和提示词格式都会引入噪声，相同的输入在不同日期可能会产生不同的分数。当该分数与硬阈值和硬退出码（exit code）挂钩时，你的队列就会因为虚无缥缈的原因而阻塞。

失败会迅速级联。非确定性的检查会导致队列积压。工程师们学会了不断重试直到分数达到理想值，这训练了团队去忽视红色的构建失败。Token 成本也会不断堆积，因为每一次重试都会消耗更多的 API 额度。最糟糕的是，信号变得毫无意义。一个红色的构建应该意味着“你引入了一个 Bug”。如果它意味着“裁判模型今天心情不好”，那么信任就会瓦解。

幸存者的不同之处

Promptfoo 和 DeepEval 之所以能存活下来，是因为它们将确定性检查视为一等公民，而将 LLM 裁判分数视为次要的、非阻塞的信号。它们明白，门禁需要的是一个退出码，而不是一个带有主观意见的浮点数。

Promptfoo 以 MIT 协议发布，专为命令行设计。它运行诸如正则表达式匹配、JSON schema 验证、包含检查和精确字符串比较之类的断言。这些并不花哨，它们更像是强化版的 grep 和 jq 命令。但这恰恰是它们能在 CI 中发挥作用的原因。正则表达式要么匹配，要么不匹配；JSON schema 要么通过验证，要么抛出错误。Promptfoo 返回标准的 Unix 退出码，因此 GitHub Actions 可以原生理解何时停止合并。它是语言无关的，因为它作为一个 CLI 工具运行。你不需要为了验证输出而在 Node.js 服务仓库中安装整个 Python 生态系统。

DeepEval 基于 Apache 2.0 协议，是 Python 团队的首选。它的集成方式类似于 pytest。你使用熟悉的语法编写测试，失败时自然会阻塞测试套件。DeepEval 提供了一个庞大的指标库，但关键细节在于你必须谨慎使用它们。对于门禁，请依赖确定性或启发式指标。如果你引入了 G-Eval 或其他基于裁判的评分器，请将它们封装在非阻塞的报告生成器中，而不是硬断言（hard asserts）。通过这种方式使用时，DeepEval 能为你提供测试框架的开发体验，而不会带有研究笔记本那种不稳定性。

其他四个框架的定位

那四个未能作为门禁存活下来的框架仍然具有价值。它们只是应该属于你工具链中的其他环节。

Future AGI (Apache 2.0) ships over fifty metrics and targets teams building custom SDKs. The metrics are thorough. The problem is that the tool expects you to write your own harness to drive it in a CI queue. In a research context, that is a reasonable trade. In a merge queue, every layer of custom wiring is a new source of instability. It is a capable evaluation engine, but not a ready gatekeeper.

RAGAS (Apache 2.0) excels at measuring retrieval-augmented generation quality. Its faithfulness and answer relevance metrics are genuinely useful for understanding how a knowledge base performs over time. Unfortunately, those metrics lean heavily on LLM judges. They are excellent for a nightly quality job that posts trends to Slack. They are poor bouncers for a pull request. Move RAGAS to your scheduled analysis pipeline, not your merge blockers.

Arize Phoenix carries the Elastic License 2.0 and sits at a different intersection entirely. It connects distributed tracing with evaluation, giving you observability into why a model behaved a certain way. You want this when you are debugging a production incident or tracing a hallucination back to a bad retrieval chunk. You do not want a tracing tool deciding whether a junior developer’s feature branch can ship. Its architecture is built for insight, not binary gates.

MLflow Evaluate (Apache 2.0) inherits its pedigree from experiment tracking. It is heavy. Pulling it into a lean CI image adds startup time and dependencies that slow down every single job. If you absolutely must use it inside a pipeline, stick to its heuristic metrics for structural checks. Even then, you are fighting the framework’s fundamental design. MLflow wants to log runs and compare experiments across weeks. A merge queue wants a verdict in under a minute.

Practical Rules for Gating

If you take nothing else from this experiment, take these three rules.

First, gate structure, not vibe. You can enforce that an output is valid JSON. You can enforce that it contains required keys. You can enforce that a classification label belongs to an allowed enum. These checks are fast, cheap, and deterministic. You cannot reliably enforce that a summary is "friendly" or that a rewrite is "creative." Those qualities belong in human review or periodic batch evaluation, not in automated gates.

Second, if a score moves on unchanged input, demote it immediately. Run your evaluation suite twice against the exact same artifact. If any metric flips from pass to fail, it has lost its right to block a merge. Promote it to an advisory dashboard where variance is expected and tolerable.

Third, respect the exit code. A pretty HTML report with a red banner does not stop a merge. A nonzero exit code does. Your evaluation tool must speak the native language of your CI platform. Standard out is for humans. Exit codes are for machines.

The Takeaway

We are still early in figuring out how to test LLM-powered applications. The temptation is to treat evaluation like a human grading rubric: nuanced, contextual, and slightly subjective. That works in a research paper. It collapses in a merge queue.

After eight months of production traffic, my pipeline now runs Promptfoo for structural and schema assertions across services, and DeepEval for Python-side behavioral checks that map cleanly to pass-fail conditions. Everything else reports to nightly dashboards. The queue is stable. The signal is clean. The team trusts a red build again.

You do not need more metrics at your gate. You need fewer metrics that tell the truth every single time.

Based on original testing and write-up shared on Dev.to. For more discussions on building reliable AI systems, join the GyaanSetu community on Telegram.