// Artifact · AI Agent

Triage Copilot — On-Call AI Assistant

An AI agent that advises but never executes — built for a sleep-deprived engineer at 3am, and designed around what it refuses to do

AI AgentResponsible AIAWS OperationsKnowledge Grounding

// 01 · Introduction

Introduction

Triage Copilot is a Socratic triage guide for a failed AWS data pipeline: it asks one question at a time, sequences an on-call engineer toward a testable hypothesis, and states a confidence level and a falsification condition for every conclusion. It never runs a command and never declares an incident resolved. It's embedded live below — ask it about a Redshift or Lambda failure and see it work the way it was designed to.

// 02 · Description

Description

This bot came out of three earlier AI Lab exercises where I tested an LLM comparison, a custom tutoring GPT, and a research assistant against questions I already knew the answers to — and every one of them failed the same way: not by being wrong, but by being confidently incomplete in ways invisible from inside the output. Triage Copilot is a direct response to that finding. Because the model cannot see live infrastructure state, every design decision follows from one constraint: it advises, it never acts. It won't provide state-changing commands, won't invent a metric or system-table name it isn't sure exists, won't declare an incident over, and says "that is not in the runbook I have" rather than reaching for a plausible-sounding answer from general AWS knowledge. Most of its system prompt is dedicated to constraining the model, not empowering it.

// 03 · Objective

Objective

To design and ship an assistant for a real failure mode I'd already identified in my own domain — a sleep-deprived on-call engineer who needs help sequencing an investigation, not a tool that acts on a half-formed theory — and to prove the design by breaking it on purpose before calling it done.

// 04 · Process

Process

01Ran three earlier AI Lab exercises (LLM comparison, custom GPT, research assistant) against questions I knew the answers to, and found a consistent failure pattern: fluent, plausible output with invisible gaps.

02Applied a design-thinking pass: defined the user as "me, on call, at 3am," and rejected two tempting options — an auto-remediation bot and a log-ingesting bot — because both required trusting the model with live state it can't actually see.

03Chose a Socratic triage guide instead: it asks, it advises, it never acts. The limitation became the design.

04Wrote a diagnostic runbook (Lambda → Redshift failure modes) and a system prompt where every behavioral rule states its own reason, so the model can't reinterpret an unexplained instruction.

05Set the required conclusion format — HYPOTHESIS / CONFIDENCE / FALSIFY — so a confidence level is always visible and every hypothesis names the one check that would disprove it.

06Ran five adversarial test scenarios, including a direct request for a restart command and a question deliberately outside the runbook's scope, to see whether the constraints held under pressure.

07Found a real bug: Chatbase's own "Initial Message" setting silently overrode my no-greeting instruction. Fixed it in the platform config and re-tested — the prompt alone wasn't the whole system.

// 05 · Tools & Technologies

Tools & Technologies

Chatbase (LLM agent platform)Hand-written diagnostic runbook (Lambda ↔ Redshift failure modes) plus AWS documentation as grounding sourcesSystem-prompt design: behavioral constraints, refusal conditions, required output structureDesign thinking (empathy → define → ideate → prototype → test)Adversarial testing against five scenarios, including a jailbreak attempt for a restart command

// 06 · Value Proposition

Value Proposition

Unique Value

Most AI-lab bots are graded on what they can do. This one is graded on what it refuses to do under pressure — it turned down a direct request for a restart command, admitted a question was outside its runbook rather than guessing from general AWS knowledge, and pushed back when I asserted a wrong conclusion with confidence. Constraining a capable model is a harder design problem than making it more capable, and the five test scenarios exist to prove the constraints actually hold rather than just read well in the prompt.

Relevance to My Audience

My day job is building the RAG ingestion infrastructure that grounds enterprise AI in real documents; this artifact is that same judgment applied to a much higher-stakes question — what happens when the model is confidently wrong and the user is too tired to catch it. For hiring managers it's evidence of responsible-AI design instincts, not just prompt-writing; for on-call engineers it's a tool built by someone who has actually been paged.

// 07 · References