Two autonomous AI SRE agents running on SigNoz’s Managed Cloud Platform (MCP) pinpoint a root-cause, post a summary to Slack, and finish the whole investigation in under 4 seconds, costing just $0.0013 per incident. The result is a shift from reactive chatbot queries to a self-driving observability workforce that slashes investigation spend while surfacing wasted metrics.
Why this matters
Traditional AI-observability demos still require a human to type a question about traces or logs. The new approach cuts that step out: when an alert fires, the system automatically runs a proven SRE playbook and delivers the answer.
The hackathon that birthed the agents
The agents emerged from the WeMakeDevs × Agents of SigNoz hackathon under the project name MIB (Men in Backend). Instead of a conversational chatbot, the team built an “autonomous observability workforce” of two agents:
- Agent J – The Incident Responder – Listens for alerts, then follows a five-step playbook: confirm the alert, locate offending traces, drill into related logs, compute the delta, and post a root-cause card to Slack. The sequence averages 3.7 seconds.
- Agent K – The Auditor – Runs scheduled audits of metric usage, flags metrics that incur cost but see no consumption, and drafts dashboard changes for a human to approve.
During the hackathon, Agent K reported that 38 % of the top metrics were never read, exposing hidden waste in the observability stack.
How the system works
Deterministic playbooks, not open-ended loops
The agents follow fixed, deterministic playbooks instead of free-form “agentic” reasoning. Each run executes a known SRE workflow—confirm, trace, log, delta—producing consistent output and avoiding the unpredictability of unconstrained LLM text.
Three layers of observability
- Application layer – The monitored app streams traces, logs and metrics to SigNoz.
- Agent layer – Each agent emits its own GenAI-generated semantic spans back to SigNoz, making the agents themselves observable.
- MCP telemetry layer – The MCP server records tool-call telemetry, completing a feedback loop where SigNoz watches the agents that watch SigNoz.
Predictive engine for pre-alert insight
A background process scores trends every 25 seconds. When latency or error rates spike, the engine forecasts an incident before the alert threshold is breached, giving the agents a head start.
Tangible outcomes
| Metric | Result |
|---|---|
| Investigation cost per run | $0.0013 |
| Time to full root-cause analysis | < 4 seconds |
| Unused top metrics identified | 38 % |
Lessons learned on the front lines
- Manual instrumentation for GenAI spans – Adding explicit instrumentation to capture the AI’s semantic output keeps the data accurate and searchable.
- Turn off “thinking” modes – LLMs that try to be conversational often truncate JSON. Disabling those modes forces concise, machine-readable output.
- Patch with RFC 6902 – Applying JSON-Patch (RFC 6902) against the Foundry platform let the team fix container health-checks and latency parameters without redeploying the whole service.
Who stands to gain, and who might be wary
Enterprises that already pay for observability platforms can see savings immediately: reduced analyst time and elimination of idle metric collection.
What to watch next
The open-source code for MIB is on GitHub, and a demo video walks through a full incident run.
Takeaway
Embedding two specialized AI agents directly into SigNoz’s MCP shows that autonomous root-cause analysis can be ultra-fast and ultra-cheap. The approach turns observability from a passive reporting tool into an active participant that acts the moment an incident appears, saving time, money and the inevitable human frustration of digging through logs after the fact.