A non-engineer tried to systematize operational error response using Claude's Agent Skills

Hello, I'm Harada, a non-engineer.

The other day, a minor operational error occurred within my team.
That became the trigger for me to think, "Could we systematize this with AI?" and I used Claude's Agent Skills to create a skill for handling operational errors.

※ In this article, I will refer to Claude's Agent Skills as "Claude Skills" or simply "skills" for convenience.

The Trigger: The follow-up work after an operational error was always quite a hassle

One of my team's responsibilities is provisioning internal testing environments.
Recently, during that work, some configuration steps were missed, temporarily leaving users unable to access the target environment.
Since it was for internal testing purposes and we were able to recover quickly, the impact was limited.

Those who have had similar experiences may relate, but what comes after is the hard part…
When an operational error occurs, various investigations and confirmations are needed in preparation for writing reports and holding review meetings.
This is always a surprisingly tedious process.

Specifically…

• Checking the extent of the impact (who was unable to access what)
• Reporting to managers and stakeholders what happened, for how long, and how it was handled
• Writing an incident report in a fixed format
• Thinking of measures to prevent recurrence in preparation for the review meeting

None of the items themselves are particularly difficult, but I always felt it was work with a significant psychological and workload burden.
On top of that, review meetings tended to drift toward procedure-based permanent fixes (recurrence prevention measures to avoid repeating the same mistakes), such as "Step XX was hard to read, that's why the mistake happened" or "Let's add XX to the procedure."
Of course, improving procedures is important, but I thought we could also organize our verification perspectives and build more systematic approaches on top of that.

The Idea: Could AI handle all of this?

At our company, there is a system where AI automatically provides review comments on technical blog posts.
Through receiving feedback on technical blog posts myself multiple times, I had come to feel that "AI seems well-suited for use cases involving checking from fixed perspectives."

I thought operational error handling might fall into the same category.

1. First, what to stop or revert (emergency response)
2. How to check the impact (how to read audit logs)
3. How to prevent recurrence (permanent countermeasures)
4. How to report it (incident report)

If we provide the above information to AI in advance, when a mistake happens, just by saying "this is what occurred,"
it could output the response sequence and a draft of the report, leading to workload reduction — that was my thinking.

What are "Skills"?

I normally use Claude's Projects feature frequently, and this was my first time creating a skill.
Since I wasn't sure about the difference between Projects and Skills, here is what I found after researching:

Once you create a skill, when a relevant topic comes up, the AI refers to that approach while responding,
and you no longer need to type out long instructions every time — that's how I picture it,
which is why I thought skills were more appropriate than projects in this case.

References:

• Agent Skills overview

Actually Building It

I also started building it by consulting the AI: "I want to create a skill for handling operational errors."
Through conversation, we settled on the following structure:

• Entry memo: When an operational error is mentioned, in what order to handle it (fact-gathering → type classification → temporary response → permanent countermeasures → audit log review → report creation)
• Per-task procedure memos: For each type of cloud service or task, summaries of "the correct state that should exist," "error-prone points and their emergency responses," and "how to check audit logs," along with specs and best practices published in official documentation for each cloud service
• Report template: Templated based on the report format used internally

As source material, I had the AI read through the work procedure documents and ledgers we normally use (PDFs and documents).
(※ Documents used were organized in accordance with internal rules to exclude confidential information, personal information, account information, etc. before use.)
The AI then picked out the "easy-to-make-mistakes" points from the procedures and compiled them together with the corresponding emergency response methods.

As a key point, for permanent countermeasures (recurrence prevention), I requested that the AI not write based on its own assumptions, but always reference official cloud documentation and include the basis for its suggestions. Since cloud specifications change, this is an important point.

The structural image is as follows:

incident-response-skill/
├── SKILL.md
├── references/
│   ├── cloud-project-setup.md
│   └── access-control-checklist.md
└── templates/
    └── incident-report-template.md

※ File names and content have been generalized for public disclosure.

Trying It Out

I tried talking to the completed skill about the details of an actual error. It then:

1. Classified the type of error
2. Presented the temporary response
3. Referenced official documentation and proposed permanent countermeasures (recurrence prevention measures)
4. Created a draft incident report following the template prepared in advance

and supported me through the entire flow.
Just having the report — which I used to write from scratch — come out as a draft already makes a noticeable difference in how it feels!

Of course, operations that have actual impact — like revoking permissions or sending emails — are not delegated to the AI; I follow the sequence of "proposal → human review → execution." Since it would defeat the purpose if an operation meant to stop the bleeding caused another mistake, I made sure to keep human hands in the loop here.

The review meeting hasn't happened yet, but in the past, discussions about causes and recurrence prevention measures would sometimes run long since they were built from scratch.
However, if we can have AI create a draft for permanent countermeasures and then use the review meeting to verify the content while discussing additional ideas and feasibility,
I believe it will lead to workload reduction while also producing richer content than before.

Sharing with the Team

The created skill could be exported compiled into a single file.
In my environment, I was able to share the created skill as a file, and members who received it could use it by saving it.

Being able to roll out something made individually just as it is — that's convenient!
My plan is to start using it within a small scope first, gather feedback on usability, and then continue developing it going forward.

Reflections After Trying It

Even for non-engineers, if you can write down your own work procedures in text, that itself can directly become a skill!
For me, having been on guard with the thought "I can't write code," this is a major step forward!
The skill I created this time is still being refined, but I recommend starting by trying it out on the routine tasks you do most often.