AI has changed how we build software, but it has not changed a basic truth about machines. They drown in noise just like we do. When engineers first experiment with AI-assisted debugging, the instinct is simple: feed the model everything. Raw logs, traces, and metrics all get dumped into the context window. The result is not insight, but failure. The volume is too high. The signal collapses. Metrics sit in one tool, traces in another, and the model cannot stitch them together into a coherent story. Before AI can help you observe your systems, you have to observe them yourself. You must shape the data first.

Why Raw Logs Break AI Pipelines

Modern systems generate telemetry at a rate no human can read. That should make them perfect for artificial intelligence. It does not. A large language model’s context window, while growing, is still a finite pipe. Stuff it with unfiltered production logs and you waste tokens on cron job heartbeats and health-check noise while burying the actual outage. Worse, raw logs lack relationships. A spike in latency at 2:00 p.m. and a database connection error in a log at the same timestamp are clearly related, but unless someone has structured that relationship beforehand, the AI has to guess. Guessing is expensive, slow, and often wrong.

The fix is architectural, not algorithmic. You need to decide what gets collected, how it gets shaped, and which backend answers which question before you ever prompt a model.

Four Axes of Monitoring

At airCloset, the engineering team stopped treating observability as a single firehose. They split monitoring into four distinct axes. Each one owns a specific shape and answers a specific question.

  • Application: Logs and traces answer "What is happening right now?"
  • Infrastructure: Metrics answer "Do we have enough resources?"
  • CI: Logs and alerts answer "What broke and when?"
  • LLM: Metrics and structured records answer "How much are we spending?"

This separation matters because the right shape for a real-time latency graph is useless for a post-hoc cost analysis. Forcing one schema across all four domains creates exactly the kind of noise that makes AI assistance useless.

CI Observability: Pull, Don’t Push

Continuous integration is where code meets reality. When a build fails, developers need the story fast. The naive approach is to have the CI runner push logs directly to your observability backend as it runs. It feels efficient. It is actually dangerous.

At airCloset, they flipped the model. The CI runner does not touch the observability stack. After the GitHub Actions workflow finishes, they pull the logs from the GitHub API and ingest them into Loki.

This pull architecture delivers three concrete wins.

Decoupling. If the ingestion pipeline hiccups or Grafana is unreachable, the test run itself is untouched. The build passes or fails on its own merits. Shipping an observability failure should never kill a deployment.

Security. The CI workflow never needs a Grafana API key. Test code is notorious for touching secrets it should not, and removing that exposure shrinks the blast radius if a dependency gets compromised.

Cross-querying. Once the CI