SRE Agent Swarm — Self-Healing Infrastructure

• •Observer — Detects anomalies from metrics, logs, health checks, and synthetic probes, with deduplication and predictive detection to catch issues early.
• •Diagnoser — Pulls context from logs, traces, and deployments, then uses LLM reasoning to build a causal hypothesis for the root cause.
• •Remediator — Maps a diagnosis onto a runbook knowledge base, then selects, parameterizes, and executes actions such as restarts, rollbacks, or scaling.
• •Safety — Enforces policy, computes blast radius, rate-limits actions, and opens human approval gates for high-risk remediations.
• •Orchestrator — Owns the incident FSM, routes work between agents, manages timeouts and escalation, and builds the incident timeline.
• •Learner — Vectorizes past incidents into a store and uses RAG plus pattern recognition to recommend runbooks that previously worked on similar failures.

• •Messaging Layer: NATS JetStream carries a structured AgentMessage envelope (message ID, correlation ID keyed to the incident, source agent, payload, context) across four streams — AGENTS, INCIDENTS, HUMAN, and BUSINESS.
• •Observability Layer: Prometheus for metrics, Loki for logs, Tempo for traces, and AlertManager — the signal sources the Observer draws on.
• •Persistence Layer: PostgreSQL for incident state, Redis for caching and sessions, Elasticsearch for search, and ChromaDB as the Learner's vector store.
• •Application Layer: A ten-service microservices playground behind an Nginx API gateway, spanning FastAPI, Go/Gin, Node.js, and Django.
• •Human Interface: A React dashboard backed by FastAPI for reviewing incidents, inspecting timelines, and approving gated actions.

Incidents move through a formal finite state machine: detecting → diagnosing → proposing_remediation → safety_review → executing → verifying → resolved → closed. The FSM includes retry loops for failed remediation and verification, a pending_human_approval state the Safety agent can force, and escalation paths when diagnosis or repair exceeds its time budget.

• •Designing a message envelope and correlation scheme that keeps six independent agents coherent about a single incident.
• •Encoding "is this action safe to take automatically?" as something computable — blast radius, policy rules, and rate limits rather than a judgment call.
• •Making an autonomous remediation loop fail safe: every path through the FSM has to terminate, escalate, or hand back to a human.
• •Constraining LLM reasoning to produce hypotheses that map onto concrete, executable runbook actions.
• •Building failure injection realistic enough that the swarm's detection and diagnosis could actually be evaluated.

• •Multi-agent system design — role decomposition, message-driven coordination, and avoiding a single orchestrating bottleneck.
• •Event-driven architecture with NATS JetStream, including stream design and durable consumers.
• •The full observability stack and how metrics, logs, and traces complement each other during diagnosis.
• •Modeling long-running workflows as explicit state machines instead of implicit control flow.
• •Retrieval-augmented generation applied to operational history rather than documents.
• •Human-in-the-loop design: where autonomy should stop and approval should begin.

• •Built an end-to-end autonomous incident response pipeline spanning detection, diagnosis, remediation, safety review, and learning across six agents.
• •Delivered a complete testbed — microservices playground, observability stack, and chaos scenarios for CPU spikes, memory leaks, network partitions, and database overload.
• •Backed the system with 63 test suites across the agent and service layers.