<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[IRAS]]></title><description><![CDATA[IRAS]]></description><link>https://iras.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sat, 21 Nov 2026 14:23:37 GMT</lastBuildDate><atom:link href="https://iras.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[How I Built IRAS: An AI Incident Response Agent That Pauses Before Touching Production]]></title><description><![CDATA[Build IRAS: an open-source AI incident response agent for triage, RCA, remediation planning, and human-approved fixes.]]></description><link>https://iras.hashnode.dev/how-i-built-iras-an-ai-incident-response-agent-that-pauses-before-touching-production</link><guid isPermaLink="true">https://iras.hashnode.dev/how-i-built-iras-an-ai-incident-response-agent-that-pauses-before-touching-production</guid><category><![CDATA[ai agents]]></category><category><![CDATA[Devops]]></category><category><![CDATA[pydantic-ai]]></category><category><![CDATA[Open Source]]></category><category><![CDATA[langgraph]]></category><dc:creator><![CDATA[Krishna kittu]]></dc:creator><pubDate>Sun, 10 May 2026 20:30:01 GMT</pubDate><content:encoded><![CDATA[<p>Incident response is one of those engineering workflows where the painful part is not always the fix.</p>
<p>The painful part is the investigation.</p>
<p>An alert fires. Someone opens the dashboard. Then the same loop begins:</p>
<pre><code class="language-text">Check severity
Check affected service
Check logs
Check metrics
Check recent deployments
Form a root-cause hypothesis
Write an update
Create a remediation plan
Wait for approval
Apply the fix
Write the post-mortem
</code></pre>
<p>That workflow has a lot of repeated structure.</p>
<p>So I built <strong>IRAS</strong> — an open-source AI incident response agent designed to automate the investigation and planning layers of incident response while keeping a human in control before any production remediation runs.</p>
<p>The goal is not:</p>
<blockquote>
<p>“Let AI blindly fix production.”</p>
</blockquote>
<p>The goal is:</p>
<blockquote>
<p>“Let AI gather context, reason through the incident, propose a safe plan, and pause for human approval.”</p>
</blockquote>
<p>That distinction matters.</p>
<hr />
<h2>What IRAS Does</h2>
<p>IRAS is an autonomous incident response workflow built around a graph-based architecture.</p>
<p>At a high level, the workflow looks like this:</p>
<pre><code class="language-text">Alert
→ Triage
→ Context Gathering
→ Root Cause Analysis
→ Remediation Plan
→ Human Approval
→ Apply or Escalate
→ Post-mortem
</code></pre>
<p>The implementation uses:</p>
<ul>
<li><p><strong>FastAPI</strong> for alert ingestion and approval APIs</p>
</li>
<li><p><strong>LangGraph</strong> for workflow orchestration</p>
</li>
<li><p><strong>Pydantic AI</strong> for structured agent outputs</p>
</li>
<li><p><strong>Claude</strong> for triage, RCA, and remediation reasoning</p>
</li>
<li><p><strong>PostgreSQL</strong> for durable checkpointing</p>
</li>
<li><p><strong>Slack / PagerDuty / Prometheus / Loki-style integrations</strong></p>
</li>
<li><p><strong>Pytest</strong> for unit, integration, e2e, and stress tests</p>
</li>
</ul>
<p>The repo currently includes source code, tests, Docker files, compose config, and environment examples. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<hr />
<h2>Why I Did Not Build This as One Big Prompt</h2>
<p>A lot of AI automation projects start as:</p>
<pre><code class="language-text">“Here is an alert. Find the issue and fix it.”
</code></pre>
<p>That is not enough for incident response.</p>
<p>Incident response needs state.</p>
<p>It needs branching.</p>
<p>It needs retries.</p>
<p>It needs escalation.</p>
<p>It needs human approval.</p>
<p>It needs rollback behavior.</p>
<p>That is why IRAS is implemented as a graph instead of a single prompt chain.</p>
<p>A simplified version of the workflow is:</p>
<pre><code class="language-text">Alert → Triage → Context → RCA → Plan → [Human Approval] → Apply → Post-mortem
                          ↑
                          └── retry if confidence is too low
</code></pre>
<p>The actual project describes the system as a 9-node LangGraph state machine where every stage produces a typed Pydantic model instead of raw unstructured text. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<p>That makes the system easier to test, debug, and control.</p>
<hr />
<h2>The Safety Rule: Do Not Trust the Model</h2>
<p>The most important design decision in IRAS is simple:</p>
<p><strong>The model can recommend. The system decides what is allowed.</strong></p>
<p>For example, an AI model may produce a remediation plan, but the application still needs hard rules like:</p>
<pre><code class="language-python">if step.risk_level == "high":
    plan.requires_human_approval = True

if not step.rollback_command:
    plan.requires_human_approval = True
    plan.reversible = False
</code></pre>
<p>This is the difference between prompt-level safety and system-level safety.</p>
<p>Prompt-level safety says:</p>
<pre><code class="language-text">“Please do not do anything dangerous.”
</code></pre>
<p>System-level safety says:</p>
<pre><code class="language-text">“You cannot execute this class of action without approval.”
</code></pre>
<p>For production automation, I trust the second one much more.</p>
<hr />
<h2>Human Approval Is a First-Class Node</h2>
<p>The approval step is not just a message in Slack.</p>
<p>It is part of the workflow.</p>
<p>IRAS uses LangGraph’s interrupt pattern so the graph can pause, save state, and resume after approval.</p>
<p>Conceptually:</p>
<pre><code class="language-python">human_decision = interrupt({
    "message": "Approve remediation plan?"
})

if human_decision["approved"]:
    return apply_remediation(state)

return escalate(state)
</code></pre>
<p>The important part is durability.</p>
<p>The graph pauses. The incident state is stored. The server can restart. The approval can still resume from the same point.</p>
<p>The IRAS README describes this as a durable approval checkpoint backed by PostgreSQL, where the workflow can resume after a server restart or process crash. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<p>That pattern is extremely useful for any production-grade AI agent.</p>
<hr />
<h2>Alert Ingestion</h2>
<p>IRAS accepts alert payloads through a FastAPI webhook.</p>
<p>Example:</p>
<pre><code class="language-bash">curl -X POST http://localhost:8000/webhook/alert \
  -H "Content-Type: application/json" \
  -d '{
    "title": "High error rate on payment-service",
    "timestamp": "2026-05-03T10:30:00Z",
    "service": "payment-service",
    "error_rate": 0.45
  }'
</code></pre>
<p>The response returns an incident ID:</p>
<pre><code class="language-json">{
  "incident_id": "550e8400-...",
  "status": "processing"
}
</code></pre>
<p>From there, the graph begins running.</p>
<p>IRAS supports alert sources such as PagerDuty, Prometheus AlertManager, Datadog/Grafana-style sources, or any JSON webhook. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<hr />
<h2>Context Gathering</h2>
<p>The context-gathering stage is where the agent starts behaving like a real incident assistant.</p>
<p>It does not just guess.</p>
<p>It pulls evidence from multiple places:</p>
<pre><code class="language-text">Logs
Metrics
Recent deployments
Service metadata
Previous incident state
</code></pre>
<p>The repo describes tool calls for logs from Elasticsearch or Loki, metrics from Prometheus, and deployment data from GitHub Deployments. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<p>That evidence becomes the input for root-cause analysis.</p>
<p>This is important because RCA without evidence is just storytelling.</p>
<hr />
<h2>Root Cause Analysis With Confidence Gates</h2>
<p>The RCA node produces a structured hypothesis.</p>
<p>A useful RCA result should include:</p>
<pre><code class="language-text">Primary cause
Contributing factors
Evidence
Affected services
Confidence score
</code></pre>
<p>If the confidence is too low, the workflow should not pretend to know the answer.</p>
<p>It should gather more context.</p>
<p>That is why IRAS uses a confidence gate:</p>
<pre><code class="language-text">If RCA confidence &lt; threshold:
    gather more context

If max RCA attempts reached:
    escalate
</code></pre>
<p>The project uses a retry path when confidence is below <code>0.7</code>, and escalates after the configured maximum attempts. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<p>This is one of the most important production patterns for agentic systems:</p>
<blockquote>
<p>Uncertainty should create control flow, not fake confidence.</p>
</blockquote>
<hr />
<h2>Remediation Planning</h2>
<p>Once the system has a root-cause hypothesis, it generates a remediation plan.</p>
<p>A good remediation plan should include:</p>
<pre><code class="language-text">Step-by-step action
Expected impact
Risk level
Rollback command
Estimated duration
Required approval
</code></pre>
<p>The rollback command is especially important.</p>
<p>If an AI system proposes a production action and cannot explain how to reverse it, that action should be treated as unsafe.</p>
<p>IRAS models remediation as ordered steps with risk levels, estimated durations, and rollback commands. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<p>This gives the human reviewer something concrete to approve or reject.</p>
<hr />
<h2>Escalation Logic</h2>
<p>Not every incident should be handled by the agent.</p>
<p>Some incidents should escalate.</p>
<p>IRAS escalates when:</p>
<pre><code class="language-text">RCA confidence never reaches the threshold
Human rejects the plan
Approval timeout expires
</code></pre>
<p>The README defines severity classes from P0 to P3 and includes escalation behavior through PagerDuty and Slack-style notifications. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<p>This is another key production principle:</p>
<blockquote>
<p>A good AI agent must know when to stop being autonomous.</p>
</blockquote>
<hr />
<h2>Post-Mortem Generation</h2>
<p>After the workflow finishes, IRAS generates a post-mortem.</p>
<p>This is not just a nice extra feature.</p>
<p>Post-mortems are usually the part engineers delay because the incident is already over and everyone wants to move on.</p>
<p>But the system already has the structured timeline:</p>
<pre><code class="language-text">When the alert started
What service was affected
What evidence was collected
What RCA hypothesis was generated
What plan was proposed
Whether it was approved or rejected
What action was taken
Whether remediation succeeded
</code></pre>
<p>That makes post-mortem generation a natural final node in the graph.</p>
<p>Even if an incident escalates, IRAS still generates a post-mortem record. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<hr />
<h2>Project Structure</h2>
<p>The repo is organized around graph nodes, agents, models, tools, and tests.</p>
<p>A simplified structure:</p>
<pre><code class="language-text">src/iras/
  graph/
    nodes/
      ingestion.py
      triage.py
      context_gathering.py
      rca.py
      generate_plan.py
      approval.py
      apply_remediation.py
      escalation.py
      postmortem.py

  agents/
    one Pydantic AI agent per stage

  models/
    TriageResult
    ContextBundle
    RootCauseHypothesis
    RemediationPlan
    RemediationStep
    PostMortem

  tools/
    Elasticsearch
    Loki
    Prometheus
    GitHub
    Slack
    PagerDuty

tests/
  unit/
  integration/
  e2e/
  stress/
</code></pre>
<p>The repository documents these major directories and includes stress tests for adversarial and real-world scenarios. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<hr />
<h2>What I Learned Building It</h2>
<h3>1. Agent workflows need hard boundaries</h3>
<p>The model should not own the entire workflow.</p>
<p>The system should define what happens when:</p>
<pre><code class="language-text">confidence is low
tools fail
approval is rejected
rollback is missing
execution fails
</code></pre>
<h3>2. Human-in-the-loop should be durable</h3>
<p>A human approval gate is not useful if the workflow loses state during a restart.</p>
<p>For production-grade systems, approval should be part of the state machine.</p>
<h3>3. Structured outputs matter</h3>
<p>Raw LLM text is fragile.</p>
<p>Typed outputs make the graph easier to test and safer to automate.</p>
<h3>4. Escalation is a feature, not a failure</h3>
<p>An agent that escalates correctly is better than an agent that confidently guesses.</p>
<h3>5. The post-mortem should be generated from state</h3>
<p>If the graph stores every decision and transition, the post-mortem becomes much easier to generate accurately.</p>
<hr />
<h2>What IRAS Is Not</h2>
<p>IRAS is not a magic production fixer.</p>
<p>It does not guarantee that every incident will be resolved automatically.</p>
<p>It does not remove the need for on-call engineers.</p>
<p>It is better understood as an autonomous incident assistant that handles the repetitive investigation and documentation layers while preserving human control over production changes.</p>
<p>Current limitations include:</p>
<ul>
<li><p>RCA quality depends on the available logs and metrics.</p>
</li>
<li><p>Alert payloads still need enough useful context.</p>
</li>
<li><p>Remediation plans should be reviewed carefully.</p>
</li>
<li><p>Real production deployment needs strong auth, secrets management, network security, and approval auditing.</p>
</li>
<li><p>Every organization needs custom runbooks and guardrails.</p>
</li>
</ul>
<p>That is intentional.</p>
<p>For incident response, safety is more important than speed.</p>
<hr />
<h2>Quick Start</h2>
<p>The repo can be run locally with Docker and Python.</p>
<p>Basic flow:</p>
<pre><code class="language-bash">git clone &lt;IRAS_REPO_URL&gt;
cd IRAS

docker run -d --name iras-postgres \
  -e POSTGRES_USER=iras \
  -e POSTGRES_PASSWORD=secret \
  -e POSTGRES_DB=iras \
  -p 5432:5432 postgres:16

python -m venv .venv
source .venv/bin/activate

pip install -e ".[dev]"
</code></pre>
<p>Then fire a test alert:</p>
<pre><code class="language-bash">curl -X POST http://localhost:8000/webhook/alert \
  -H "Content-Type: application/json" \
  -d '{
    "title": "High error rate on payment-service",
    "timestamp": "2026-05-03T10:30:00Z",
    "service": "payment-service",
    "error_rate": 0.45
  }'
</code></pre>
<p>The project README includes a quick-start example with Postgres, local install steps, and a sample alert request. (<a href="https://github.com/krishnashakula/IRAS">GitHub</a>)</p>
<hr />
<h2>Final Thought</h2>
<p>The future of AI in DevOps is not fully autonomous agents doing whatever they want in production.</p>
<p>The future is controlled autonomy.</p>
<p>AI should help engineers move faster by doing the repetitive work:</p>
<pre><code class="language-text">triage
context gathering
RCA
planning
documentation
</code></pre>
<p>But production changes still need guardrails.</p>
<p>That is the philosophy behind IRAS:</p>
<blockquote>
<p>Let the agent investigate. Let the system enforce safety. Let the human approve the action.</p>
</blockquote>
<p>I open-sourced IRAS so other engineers can inspect the architecture, break it, improve it, and adapt it to their own incident response workflows.</p>
<p><strong>Repo:</strong> Add your IRAS GitHub repo link here.</p>
<hr />
<p>If you want feedback, contributions, or architecture suggestions, feel free to open an issue or PR. I’m especially interested in ideas around Slack approvals, Datadog integrations, production auth, stronger rollback handling, and long-term incident memory.</p>
]]></content:encoded></item></channel></rss>