Why the test matters

AI-driven code assistants often let teams drop a “rules” file in the repo and expect the model to obey its directives on every request. In practice the model may never see the file, or it may see it but ignore the contents. A recent experiment with Claude Code showed both problems. The tool silently skipped a 72 KB AGENTS.md file; when the same file was renamed to CLAUDE.md the assistant loaded it and increased the token count for each request. That extra token budget inflates latency, cost, and can push a request over a model’s limit.

Developers who assume “the file exists” equals “the model follows the rules” risk hidden inefficiencies and unpredictable output. The three-step test forces concrete evidence at each stage: configuration, loading, and usefulness.

The three questions to ask

  1. Configured – Is the file placed where the assistant looks for it? Different tools hard-code paths or filename conventions; a mismatch means the file never enters the prompt pipeline.
  2. Loaded – Does the assistant expose any proof it received the file? A hash can confirm the file’s identity on disk, but only a delivery trace (e.g., a log line or token count) proves the model actually observed it.
  3. Useful – Does the presence of the file improve the task outcome? A loaded file that adds tokens but leaves the result unchanged is a net loss.

Running the test

The procedure is deliberately minimal so it can be repeated on any platform.

  1. Create a visible rule – Write a simple, observable instruction. For example: “List exactly two files before editing.” The rule’s effect can be checked in the assistant’s response.

  2. Check tool version and model – Open a fresh session, note the version string and the model identifier. Different versions may change the filename they recognize.

  3. Execute two runs Run A: Use a filename the tool does not recognize (e.g., AGENTS.md). Run B: Use the tool’s native filename (e.g., CLAUDE.md).

    Record:

    • The source hash of the file (to prove the on-disk content didn’t change).
    • The exact path used.
    • Any evidence the assistant logged about loading the file (token count bump, explicit “loaded X.md” message, etc.).
    • The token count for each request.
    • The task result (did the assistant list exactly two files?).

If Run B shows the rule being obeyed and the token count rises by the expected amount, the file is both loaded and useful. If the rule is ignored despite the token bump, the file is being read but the model’s prompt parsing discards the instruction. In that case, adding more text to the file won’t help; move the rule to a hard-coded policy gate or a test harness instead.

What the data reveal

The Claude Code case demonstrated a stark gap between configuration and loading. The 72 KB file existed, had the correct hash, and was synced to the repository, yet the assistant never referenced it. Renaming the file to the native CLAUDE.md triggered loading, but also added a substantial token overhead. Each extra token consumes compute cycles and can push the request over rate limits.

The three-step test surfaces such hidden costs before they become production blockers. By capturing the token delta, teams can decide whether the rule’s benefit outweighs its price.

Takeaway

Never assume a rule file is doing work just because it lives in the repo. Use the three-step test—configure, load, prove usefulness—to turn that assumption into measurable evidence. When the proof shows a file is merely a token sink, move the logic out of the prompt and into a deterministic gate. The result is a leaner, faster, and more predictable AI coding workflow.