AWS DevOps Agent Remembered Our Operational Mistake When the Same Incident Happened Twice

AWS DevOps Agent Remembered Our Operational Mistake When the Same Incident Happened Twice

I made two operational mistakes that caused incidents while testing the AWS DevOps Agent memory feature, and used them to examine how memories are formed. Here are the findings and the issues they reveal for preventing recurrence.
2026.11.16

This page has been translated by machine translation. View original

Hello. This is Ogata from the Operations Innovation Department.

Last time, I had AWS DevOps Agent investigate an incident caused by an operational mistake, and verified that it could even come up with hypotheses about why the mistake was made.

https://dev.classmethod.jp/articles/devops-agent-human-error-root-cause-hypothesis/

This article is the follow-up. DevOps Agent has a feature called "memory," which learns from past investigations and remembers them. I looked at whether DevOps Agent would remember the incident caused by the operational mistake from last time.

As a result, it did not create a memory after the first incident. It was only when I caused the same incident again that it finally created one.

Conclusion

  • Running the learning after the first incident did not create a memory
  • The learning agent explained the reason as "a new monitor is memorized only when there are 2 or more investigations"
  • When I caused the same incident from the same operational mistake again and had it investigated, a memory was created

DevOps Agent Memory

DevOps Agent memory comes in several types. What I tested this time is monitors. The official documentation describes it as follows:

  • Records a history of recurring root causes for each alarm
  • The learning agent analyzes the past two weeks of investigations and creates it once a day
  • It is read before an investigation when an alarm fires

In other words, it is a memory that lets the agent recall "this was the cause last time too" when the same alarm fires.

In addition, "Takayama" explains the DevOps Agent memory feature in detail in the blog post below.

https://dev.classmethod.jp/articles/devops-agent-memories/

Details of the Verification

The setup is the same as last time. While disk space was running short, an operator deleted the app's index file (tmp_index), and the health check alarm fired.

The flow of the verification is as follows:

  1. Cause the incident from the operational mistake and have DevOps Agent investigate it (1st time. One investigation each for the disk alarm and the health check alarm)
  2. Run the learning and check whether a memory was created
  3. Cause the same incident again and have it investigated (2nd time)
  4. Run the learning again and check whether a memory was created

The automatic execution of the learning is limited to once a day. To see the results without waiting until the next day, I ran it manually using "Run now" on the learning task screen.

004_devops-agent-memory_018.png

Running the Learning After the First Incident

After the investigation of the previous incident (the 1st time) finished, I ran the learning. It finished in about a minute, but not a single memory was created.

No memories were uploaded since no monitors qualified for memory creation.

004_devops-agent-memory_019.png

The learning agent explained as follows:

  • The monitors store was still empty (this was the first run)
  • Therefore, both alarms were treated as "new monitors"
  • A new monitor becomes a memory only when there are 2 or more investigations, because a single firing cannot be said to be a recurring pattern
  • Both alarms had only one investigation each, so they were skipped

Interestingly, the learning agent had first created working files summarizing the cause of each investigation. The content was a summary: "manual deletion of tmp_index via SSM Session Manager." However, it seems that turning it into a memory happens at a later stage, and only for monitors that meet the conditions.

Causing the Same Incident Again

Using the same command as last time, I deleted tmp_index again. The health check alarm fired, so I had it investigate in the same way.

After the investigation finished, I ran the learning again. This time, one memory was created.

Monitors captured: 1 monitor memory created
  dam-exp-health-check-failed (2 investigations)
Monitors skipped: 1 monitor
  dam-exp-disk-used-high (1 investigation) - skipped because new monitors require 2+ investigations to establish a recurring pattern

004_devops-agent-memory_025.png

The health check alarm became a memory because it now had two investigations. The disk alarm still had only one investigation, so it was skipped again this time.

Here are the contents of the memory that was created.

004_devops-agent-memory_026.png

The memory contained the following:

  • /health fails if the tmp_index index file is missing
  • The root cause is that tmp_index was deleted during manual disk cleanup (two incidents)
  • The conditions for using this memory (for example, when FileNotFoundError appears in the app logs)

Why a Single Incident Doesn't Become a Memory

The official documentation also describes monitors as a history of "recurring root causes." The description of the learning task also says "learns recurring root causes." With only one occurrence, it cannot distinguish whether it happened by chance or is a recurring problem. I think the idea is that only when it happens twice can it be called "recurring."

However, I could not find the specific condition of "2 or more" in the official documentation. In this article, I am writing it as what the learning agent explained during execution. It may change in the future.

From an operations perspective, this means that a memory meant to prevent recurrence is created only after the recurrence. If you want to make use of the first incident before the second one, it looks like you will need a means other than memory.

Summary

  • No memory was created after the first incident
  • According to the learning agent, a new monitor becomes a memory with 2 or more investigations
  • The memory was created only after the same incident occurred twice

References

Share this article