An AI-driven coding assistant on a free-tier server queue wrote a schema-migration file after the system’s “freeze migrations” rule vanished during automatic context compaction. The incident shows that relying on prompt-based instructions to enforce safety can backfire when long chat sessions are summarized.

What happened

A shared coding agent ran with a system prompt that contained a single rule: do not create or modify migration files. The rule kept schema migrations frozen on the release branch while developers used the agent for routine edits.

During a prolonged session the conversation exceeded the model’s token limit. The server invoked context compaction, replacing the full chat history with a short summary to stay within the token budget. The summary kept file-path mentions but stripped out the negative constraint (“do not write to migrations/**”). When the model generated its next response, the freeze rule was no longer in its working memory.

The agent then suggested a SQL tweak to improve query speed. The accompanying tool saved the change as a migration file on disk. Unit tests passed because they never execute raw SQL migrations, and the alteration was only spotted later in a code-review diff.

Why it matters

  • Guardrails disappeared: Prompt-level policies are volatile. When the model’s context is compressed, any “do not” clause can be lost, leaving the system free to perform prohibited actions.
  • Unit tests aren’t enough: Tests that don’t run migration scripts can give a false sense of security. The change slipped through because the test suite never exercised the new SQL.
  • Human review lagged: By the time a reviewer saw the diff, the migration file had already been committed, requiring a rollback or additional cleanup.

The error didn’t cause an outage, but it forced extra work to revert an unnecessary migration and raised the risk of a similar slip affecting live databases.

How the guardrail failed

  1. Prompt-only rule – The freeze lived as plain text in the system prompt.
  2. Long session – The chat grew beyond the model’s context window.
  3. Compaction – The server swapped the full transcript for a noun-heavy summary.
  4. Negative constraint loss – The summary omitted the “do not” instruction, erasing the rule.
  5. Model output – Without the rule, the model generated a migration-creating response.

Summarizers tend to keep nouns (file paths, function names) and drop negations, which is why the rule vanished silently.

The fix: move guardrails to the write path

Instead of embedding prohibitions in the prompt, enforce them where code touches disk.

  1. Create a policy file (e.g., agent_policy.txt) that lists deny-listed paths such as schema/** or migrations/**.
  2. Add a pre-apply script that reads the policy file and scans every diff before the apply tool runs.
  3. Fail closed if the policy file is missing, empty, or if a diff touches a forbidden path.
  4. Don’t ask the model to confirm the deny list; confirmation is only a signal, not enforcement.

Prompt-only vs. disk-based guardrails

Aspect Prompt-only freeze Disk-based gate
Compaction risk Rule can disappear Policy file stays intact
Unit-test coverage May miss prohibited writes Path check blocks write regardless of tests
Human review timing Damage may already be committed Write never starts, reviewer sees no change
Implementation effort Minimal (just a line in prompt) Small script + policy file

Counter-point

What to watch next

Takeaway

If you need a rule that must never be ignored—such as “no migrations during a freeze”—store it on disk and enforce it at the point of write. Prompt text alone cannot guarantee durability once the model’s context is compressed.