Mike Mwita

My AIOps Runbook for Self-Healing Services

#aiops #self-healing #kubernetes #platform-engineering #sre

G ood code already heals itself. It retries with backoff when a call fails, opens a circuit breaker when a dependency keeps failing, and lets a supervisor restart a process that crashed. Nobody pages an engineer because one HTTP request timed out.

Services deserve the same treatment one level up. That is how I think about AIOps: not as an autonomous agent running production, but as a runbook the platform executes for you. It heals the failures that are safe to heal, and it escalates the rest to a human with the investigation already done.

Self-Healing Patterns, from Code to Platform #

Every self-healing pattern we trust in code has a direct equivalent at the service level:

Pattern in code	Same idea on the platform
Retry with backoff	Kubernetes restarting a crashed container, with `CrashLoopBackOff` as the backoff
Supervisor / let it crash	Restart a hung pod, roll back a bad release
Fallback	Return to the last known good version
Circuit breaker	Stop auto-healing a service after a failed attempt and hand it to a human
Dead-letter queue	Escalate to on-call with the full context, instead of dropping it

Kubernetes already gives us the first row for free. What is missing is everything after it: knowing why the container keeps crashing, and deciding whether the platform can fix it or a person must.

The Architecture #

Here is how I would deploy it. A scanner pulls signals out of every cluster, logs are grouped into patterns before anything tries to reason about them, and a runbook engine decides between healing the cluster itself and paging a human.

flowchart TB subgraph fleet["Kubernetes clusters"] direction LR C1["cluster: prod-eu"] C2["cluster: prod-us"] end SCAN["Cluster scanner\npod status, events,\nmetrics, logs"] subgraph store["Telemetry store"] direction LR MET[("Metrics")] LOG[("Logs")] end subgraph brain["Intelligence"] direction LR GRP["Log grouping\nlines to patterns"] DET["Detect + classify\nmatch runbook rule"] SUM["LLM\nsummary"] end ENG["Runbook engine\nn8n"] PD["PagerDuty"] ONC["On-call\nengineer"] AUD[("Audit trail")] C1 --> SCAN C2 --> SCAN SCAN --> MET SCAN --> LOG LOG --> GRP --> DET MET --> DET DET --> SUM --> ENG ENG -->|"safe fix"| fleet ENG -->|"no safe fix"| PD --> ONC ENG --> AUD

Each component does one thing:

Cluster scanner — runs on a schedule and on alerts, and pulls pod status, events, restart counts, resource usage and recent logs from every cluster it can reach.
Telemetry store — metrics and logs land somewhere queryable, such as Prometheus or Mimir for metrics and Loki for logs.
Log grouping — collapses thousands of near-identical lines into a few patterns, so detection works on kinds of errors rather than individual lines.
Detect and classify — combines metric anomalies with log patterns and matches them against the runbook rules.
Runbook engine — n8n executes the matched rule: heal the cluster directly, or open a PagerDuty incident with the evidence attached.
Audit trail — every decision, action and result is recorded, whichever path was taken.

Why Log Grouping Comes First #

A crash-looping pod can write the same error hundreds of times with only the numbers changing. Grouping replaces the variable parts with placeholders and counts the occurrences:⊕This is usually called log template mining. Algorithms such as Drain build the templates on the fly by comparing the structure of incoming lines.

log grouping · last 15 minutes
1,284  IO Error: Connection reset, connect lapse <N> ms
  312  Retrying connection to <HOST>:<PORT> attempt <N>
   47  Request <ID> completed in <N> ms
    1  Shutting down worker pool

The rule engine now sees one dominant pattern instead of 1,600 lines, and the LLM summarises a handful of patterns instead of a wall of text. That is cheaper, faster and far less likely to send the investigation down the wrong path.

The Runbook Loop #

Every alert goes through the same loop. The key decision is in the middle: is there a known, safe fix for this class of failure?

flowchart TD A["Alert:\npod unhealthy"] --> B["Investigate:\nlogs, events,\nlast rollout"] B --> C{"Safe fix\nknown?"} C -->|"yes"| D["Heal:\nrestart or\nrollback"] D --> E{"Healthy?"} E -->|"yes"| F["Resolve\n+ audit"] E -->|"no"| G["Escalate:\nPagerDuty"] C -->|"no"| G G --> H["On-call fixes\nwith evidence"] H --> I["Update\nthe runbook"]

Two rules make this loop safe:

One healing attempt, then break the circuit. If the fix does not work, the platform stops and escalates. It never tries a second guess on production.
The investigation always runs. Whether the outcome is “healed” or “escalated”, the evidence is gathered and summarised first, so a human never starts from zero.

The Runbook Itself #

This is the table I would start with. Each row is a failure class, the evidence that identifies it, and what the platform is allowed to do about it.

Failure class	How it is recognised	Response
Out of memory	Last state `OOMKilled`	Heal — raise the memory limit one step, within a cap
Bad release	Crash loop started within 30 minutes of a rollout	Heal — roll back, with approval
Hung worker	Pod running but no progress or failing liveness	Heal — restart the pod
Credentials or access	Auth failures, `access denied`, connection reset during authentication	Escalate — PagerDuty to the owning team
Database unreachable	Connection refused, pool exhausted	Escalate — PagerDuty, with pool and DB metrics attached
Missing config	Missing secret, ConfigMap or file	Escalate — owning team
Anything else	No rule matched	Escalate — on-call, with the evidence

Credentials are the clearest example of where healing must stop. A platform should never guess a password or rotate a secret on its own. The right response is to tell the right person exactly what failed, and get out of the way.

Restarting a pod that cannot authenticate does not heal it. It just adds more failed logins, and on some databases that locks the account. Credential errors should skip healing entirely.

Runbook as Code #

The table should not live in a wiki. It should be a file in Git that the workflow engine (I would use n8n) reads, reviewed like any other change:

runbook.yamlyaml
rules:
  - name: out-of-memory
    match:
      terminatedReason: OOMKilled
    action: raise-memory
    params: { step: 25%, max: 2Gi }
    approval: auto

  - name: bad-release
    match:
      exitCode: nonzero
      minutesSinceRollout: { lt: 30 }
    action: rollback
    approval: required

  - name: credentials-or-access
    match:
      logPattern: "(?i)auth|access denied|password"
    action: escalate
    route: pagerduty
    team: owning-team

  - name: fallback
    match: {}
    action: escalate
    route: pagerduty
    team: platform-oncall

Rules decide what happens. An LLM only reads the logs and writes the human-readable summary. A rule is testable and auditable; a model’s judgement on a production action is neither.

Two Investigations #

Here is what the runbook looks like when it runs. Both sessions are illustrative, from a fictional shop namespace.

Healed: a memory limit that was too tight #

aiops-agent · investigation session
02:11 alert     PodCrashLooping shop/cart-api-k2p4x
02:12 inspect   status, events, logs, rollout history
02:14 evidence  terminated: OOMKilled (exit 137)
02:14 evidence  limit 512Mi, peak 509Mi in last hour
02:14 evidence  no rollout in the last 24h
02:15 match     rule out-of-memory (approval: auto)
02:16 heal      limit 512Mi -> 640Mi, PR #482 merged
04:40 verify    rollout healthy, 0 restarts in 2m
04:41 resolve   healed, audit record written

Escalated: credentials the platform cannot fix #

aiops-agent · investigation session
09:17:03 alert     PodCrashLooping shop/billing-worker-x8m2q
09:17:04 inspect   status, events, logs, rollout history
09:17:06 evidence  exit code 1, 6 restarts in 9m
09:17:06 evidence  SQLRecoverableException: IO Error:
                   Connection reset, auth lapse 4812 ms
09:17:06 evidence  no rollout in 3 days
09:17:06 evidence  db-credentials secret changed 40m ago
09:17:07 match     rule credentials-or-access (no auto-fix)
09:17:08 summary   billing-worker cannot authenticate to
                   the billing DB. Failures began after the
                   credentials secret changed; the new
                   password is likely not applied to the
                   database user.
09:17:09 escalate  PagerDuty incident for team billing
09:17:09 circuit   auto-healing paused for billing-worker

The second session is the one I care about most. The engineer who gets paged does not start by running kubectl logs. They start with a likely cause, the timing that supports it, and the one change that probably caused it.

Escalating Through PagerDuty #

When the runbook escalates, it should send everything the investigation found, not just “billing-worker is down”.

sequenceDiagram participant A as AIOps workflow participant P as PagerDuty participant E as On-call engineer participant R as Runbook repo A->>P: trigger incident with summary + evidence P->>E: page the owning team E->>E: read evidence, fix the secret E->>P: resolve P->>A: incident resolved A->>A: resume auto-healing for the service E->>R: add a rule if this will recur

The PagerDuty Events API takes a structured payload, so the evidence travels with the page:

pagerduty-event.jsonjson
{
  "routing_key": "<billing-team-integration-key>",
  "event_action": "trigger",
  "dedup_key": "shop/billing-worker/credentials-or-access",
  "payload": {
    "summary": "billing-worker cannot authenticate to the billing database",
    "source": "shop/billing-worker-7d4b9f6c5-x8m2q",
    "severity": "error",
    "component": "billing-worker",
    "class": "credentials-or-access",
    "custom_details": {
      "evidence": "SQLRecoverableException: Connection reset during authentication",
      "restarts": 6,
      "recent_change": "db-credentials secret updated 40m before first failure",
      "auto_healing": "paused"
    }
  },
  "links": [
    { "href": "https://grafana.internal/d/billing-worker", "text": "Service dashboard" }
  ]
}

Use a stable dedup_key per service and failure class. A crash-looping pod fires the same alert every few minutes, and the dedup key keeps that as one incident instead of a pager storm.

Guardrails #

Self-healing code has limits built in, such as maximum retries and breaker thresholds. The platform needs the same:

An allowlist of actions — restart, rollback, raise memory within a cap. Nothing else.
One attempt per incident — a failed fix escalates; it never retries.
No stateful workloads — databases and StatefulSets always go to a person.
GitOps-aware fixes — in an Argo CD or Flux cluster, heal by opening a pull request, never by patching with kubectl.
A kill switch — one flag turns every automatic action off.

Earning Autonomy #

Every rule starts with approval: required. The platform proposes, a human approves, and the runbook records how often the proposal was accepted unchanged.⊕Human in the loop (HITL) means a person makes the final call on a production action. Removing that step is a per-rule decision backed by data, not a global setting.

When a rule is approved more than nine times out of ten, it graduates to approval: auto, and the human gets a notification instead of a question. Credential and data rules never graduate.

Closing #

We stopped paging people for a single failed request years ago, because our code learned to retry, back off and break the circuit. The same discipline belongs at the service level: heal what is safe, escalate what is not, and make every escalation arrive with the investigation already done.