Most performance claims in the JavaScript tooling space have the shelf life of milk. Someone installs a few packages on a quiet Tuesday morning, captures the terminal output, and publishes a dramatic bar chart. By the next sprint, one of the tools has shipped a patch that invalidates the whole thing. The post remains indexed by search engines. The chart keeps getting shared. The numbers, however, are already lying to you.

This is the rot that infects nearly all package manager benchmarks. They are event photography when what we need is a live feed.

A project called depjs/canary treats this problem as a machine responsibility rather than a content calendar task. It is a living benchmark that watches npm, pnpm, Yarn, and dep, then re-runs its entire suite the moment any of them publishes a new version. The results are public, continuous, and unavoidable. When something breaks, the repository stays red until it heals. There is no cherry-picking, no hiding behind an old blog post, and no assumption that last month’s winner still holds the crown.

Why Speed Claims Need an Expiration Date

JavaScript package managers do not stand still. The gap between minor versions can include rewritten resolution algorithms, altered hoisting strategies, or changes to how the global cache is keyed. A benchmark that captures npm 10.2.1 and pnpm 8.11.0 tells you almost nothing about how those same tools behave two releases later. Yet the web is full of definitive statements like “Tool X is three times faster” based on exactly that kind of frozen snapshot.

Worse, many tests ignore the conditions that define real developer pain. A package manager might blaze through an install in a warm, well-rehearsed environment and then crawl on a CI runner that starts with an empty disk. Without testing both extremes, the benchmark becomes a press release rather than usable engineering data.

How the Canary Automates Comparison

Every two hours, a job polls the npm registry. If a new version of npm, pnpm, Yarn, or dep appears, the canary wakes up. It does not wait for a human to notice the changelog. It immediately executes a full test matrix that pits all four managers against five popular, real-world packages. The selection includes heavy hitters like React, Next.js, and Vite, codebases that actual developers install every day. These are not synthetic micro-projects designed to flatter one particular tool.

This release-triggered approach matters because it ties measurement directly to change. If the benchmark only ran on a nightly schedule, it might miss a mid-day hotfix or swallow a regression for hours. By running specifically on new versions, the canary asks a direct question each time: did this release make things better or worse?

The Four Scenarios That Stress Different Muscles

The test matrix is built around four distinct setups that map directly to workflows you will recognize.

  • Cold cache, no lockfile. This is the fresh clone on a new laptop, or the first install after deleting node_modules. Nothing is cached. Nothing is pinned. The package manager has to resolve, fetch, and write everything from scratch.
  • Warm cache, with lockfile. This is the happy path for continuous integration when things go right. The lockfile exists locally, and the cache still holds the tarballs from a previous run. The tool should move fast because most decisions are already made.
  • Cold cache, with lockfile. Here the lockfile is present, but the cache has been wiped. The manager can skip dependency resolution, yet it still has to download every byte across the network. This isolates network speed from resolution speed.
  • Warm cache, no lockfile. The cache is hot, but the lockfile is gone. The package manager must re-resolve the dependency tree before it can even begin extracting files. This tests the efficiency of the solver and the metadata parser under ideal network conditions.

Each scenario is executed five times, and the canary keeps the median result. That single choice kills a lot of noise. One transient network hiccup or a brief spike in registry latency cannot hijack the narrative. The outlier gets ignored; the typical experience gets recorded.

Smoke Tests Beat Empty Timers

如果不验证结果，单纯的速度很容易造假。包管理器可能会跳过 postinstall 步骤、损坏一些符号链接或安装错误的版本，但仍然能跑出一个令人印象深刻的时间戳。canary 不止于计时。在安装完成后，它会实际运行已安装的代码。

例如，它会启动一个 Express 应用程序，并确认服务器是否在预期的端口上开始监听。如果代码无法运行，基准测试会直接失败。冒烟测试将该套件从一场“竞赛”转变为一次“审计”。它回答了仅凭速度无法回答的问题：安装是否真的成功了？

将“极端诚实”作为一项特性

canary 的作者在流程中写入了三条规则，而大多数基准测试作者通常将其视为可选。

公平的竞技场。 使用标志（Flags）来规范不同工具的行为。如果某个包管理器利用默认设置掩盖了性能缺陷，基准测试会将其暴露出来，而不是让该工具因偶然因素显得表现良好。

真正的冷启动。 在每一次重复测试之前（而不仅仅是第一次），npm 缓存和 pnpm 存储都会被清空。这里的“每一次”起到了至关重要的作用。许多基准测试只清空一次缓存，然后连续运行五次安装。从第二次到第五次运行并非真正的冷启动，数据也会随之虚高。canary 每次都从零开始。

公开的失败状态。 当新版本导致某些功能损坏时，仓库会保持在红色的失败状态。它会一直显示在首页，显得难看且未解决，直到修复补丁发布。这里没有为了让仪表盘保持绿色而进行的静默压制。这种政策强制实现了透明度。评估工具的用户不仅可以看到哪个工具最快，还可以看到哪个工具在长期运行中保持了可靠性。

可检查性是不容谈判的

无法复现的基准测试不过是竞选口号。canary 通过一个简单的 bash 脚本解决了这个问题，让任何人都能在本地运行该套件的任何部分。你不需要信任云提供商的网络，也不需要依赖维护者手动调整的环境。如果你怀疑数据有问题，可以生成自己的数据。

这种透明度也让该项目对维护者非常有用。当出现回归时，下游开发人员可以拉取脚本，对工具的发布版本进行二分查找 (bisect)，并将...交给上游团队一个