I Looked into Why Kiro CLI Failed After Answering with Steps That Could Be Run in aws-bench's CloudFormation Recovery Task

Introduction

I have written four articles so far about aws-bench, the AI agent benchmark published by AWS. The first covered how the benchmark works, and the second covered building a mini test environment that runs in a single account. The third and fourth covered the results of the Step Functions and VPC Flow Logs diagnosis tasks measured in that mini environment.

https://dev.classmethod.jp/articles/aws-bench-overview/

https://dev.classmethod.jp/articles/aws-bench-mini-tasks-local-runner/

https://dev.classmethod.jp/articles/aws-bench-diagnose-sfn-kiro-cli-17models/

https://dev.classmethod.jp/articles/aws-bench-check-vpc-kiro-cli-17models/

These are again results measured in the same mini test environment. This article covers one task, fix-cloudformation-update-failed-stack, in which the agent must answer the cause and recovery procedure for a stack stuck in UPDATE_FAILED.

The correct answers for the two earlier tasks were 51 out of 51 for Step Functions and 22 out of 51 for VPC Flow Logs. This time it is 12. The reason for the gap is not the difficulty of the task but how little material is available for the judgment.

From here on, one cell means one run of one model, and the denominator of 51 represents 17 models × 3 runs.

The task definition refers to dataset commit 2daf77d2d41c21bae00bf8227fc463be51f721d0. The benchmark itself is edfab47710a505b7e73de66584a84f8b9bb6e26c. Note that claude-sonnet-5, used for judging, is also among the 17 evaluated models. This article does not verify how that overlap affects the judgment.

Identifiers in the article are masked. Stack names, account IDs, and logical IDs have been replaced with dummies.

Comparisons are made only within this task. No totals with the results of other tasks and no overall model ranking are given.

Task Details

The task definition is in the public dataset. Below are the key points; the original text of the instructions and the reference answer can be checked at the link.

https://github.com/aws-bench/aws-bench-datasets/tree/2daf77d2d41c21bae00bf8227fc463be51f721d0/tasks/troubleshooting-multiservice/fix-cloudformation-update-failed-stack

Resources deployed in advance

The resources to be diagnosed are created by a single CDK stack bundled with the scenario.

The anomaly created by the setup script

After deployment, the setup script runs and puts the stack into UPDATE_FAILED.

1. Retrieve the original template
2. Replace the FunctionVersion of AWS::Lambda::Alias with a nonexistent version number
3. Update with DisableRollback=True specified
4. Confirm that the stack has become UPDATE_FAILED

How the anomaly is created is shown with code in the second article. The key to reading the results is DisableRollback=True in step 3. Because no rollback occurs, the stack stops at UPDATE_FAILED rather than UPDATE_ROLLBACK_FAILED.

Request to the agent

The gist of the instruction is: "The specified stack is in UPDATE_FAILED. How should I fix it and continue the deployment?" The state is stated explicitly here. After replacing the placeholder with the actual stack name, it is handed to the agent in the isolated container. The agent investigates AWS with read-only temporary credentials and writes its final answer to /logs/agent/agent-output.txt. The reference answer and rubric are not passed to the container.

How answers are judged

Judging is a binary decision on whether the answer is equivalent to the reference answer. There is no partial credit.

The reference answer lists two causes and one recovery step. The causes are that nonexistent function versions are hardcoded in the two Lambda aliases, and that !GetAtt Function.Version should be used instead of hardcoding. The recovery step is to run continue-update-rollback with the logical IDs of the two aliases passed via --resources-to-skip.

Of the failure conditions listed in the rubric, the following two are relevant to this task.

• Contradicts the reference answer regarding the actual state of the infrastructure or the resource configuration
• The proposed fix would not actually work (missing parameters, wrong target)

There are four others: the judgment is reversed, the cause is different, a major cause is missing, and a false assertion misleads the practitioner.

Because this task asks for a fix to be presented, the second condition, "would not actually work," applies. The relationship between this condition and the recovery step in the reference answer determines the results.

Official Judging Results

There were 12 correct answers out of 51. There were 0 timeouts.

The dividing line between pass and fail was whether --resources-to-skip was stated explicitly. All 12 correct answers mention continue-update-rollback, and 11 of them go as far as writing --resources-to-skip. Among the 39 failed answers, not a single one wrote --resources-to-skip. Meanwhile, 26 of the 39 failed answers do mention continue-update-rollback itself. The presence or absence of that option alone explains the pass/fail outcome of 50 of the 51 cells.

Reading the Contents of the Answers

The actual state of the target stack is UPDATE_FAILED, with DisableRollback set to true. The continue-update-rollback that the reference answer calls for is an API for UPDATE_ROLLBACK_FAILED and cannot be used on a stack in this state.

https://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_ContinueUpdateRollback.html

The options available in UPDATE_FAILED are retrying the update, update-stack with --disable-rollback, and rollback-stack.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stack-failure-options.html

Most of the judgment reasons for the 39 failures take the same form: the diagnosis of the cause matches the reference answer, but the recovery procedure does not reach continue-update-rollback --resources-to-skip. Answers that explicitly said this API cannot be used also failed. Run 2 of claude-opus-5 even gives the reason it cannot be used.

Step 2 - Resume the deployment. The stack is in UPDATE_FAILED with DisableRollback=true,
which means it is directly updatable; you do NOT need continue-update-rollback (that API
only applies to UPDATE_ROLLBACK_FAILED, which is not the case here).

Instead, this answer offers a direct update with the corrected template (Option A) and rollback-stack (Option B). According to the documentation, both are operations that can be executed on a stack in UPDATE_FAILED.

Option B - return to the last known good state first, then deploy the fix:

aws cloudformation rollback-stack \
  --stack-name troubleshooting-multiservice-cloudformation-example-us-east-1 \
  --role-arn arn:aws:iam::111122223333:role/cdk-hnb659fds-cfn-exec-role-...

The judge failed this as "different from the operation the reference answer calls for."

Here is an example from the correct side. Run 1 of claude-opus-4.5 presents exactly the same command as the reference answer.

2. **Continue the stack update** with the corrected template:

   aws cloudformation continue-update-rollback \
     --stack-name troubleshooting-multiservice-cloudformation-example-us-east-1 \
     --resources-to-skip <2つのエイリアスの論理ID> \
     --region us-east-1

Only the answer and the reference answer are available to the judge; the actual state of the stack is not included. Therefore, an answer that writes continue-update-rollback exactly as in the reference answer passes even if the API would not accept that procedure. The earlier condition, "the proposed fix would not actually work," is likewise applied without any material for checking whether it works.

There are also inconsistencies within the dataset. A comment in the CDK stack says the setup script puts the stack into UPDATE_ROLLBACK_FAILED. However, what the setup script actually creates, and verifies itself, is UPDATE_FAILED. It would make sense if the reference answer was written assuming UPDATE_ROLLBACK_FAILED, but this is speculation.

Circumstances specific to the mini environment did not affect this task's results. Of the 51 answers, only run 2 and run 3 of claude-opus-5 touched resources other than the target stack. Both mention another stack included in the same scenario, and in the official procedure that stack is placed in the same account. The judgment reasons for these two also rest only on continue-update-rollback, and the mention of the other stack is not a reason for failure.

Recounting with a Different Criterion

For this task, a correction that merely adds back the misses on the failing side is not enough, because the side that wrote exactly what the reference answer says has the same problem of a non-executable procedure. So I set aside the official pass/fail distinction and applied the same criterion afresh to all 51 cells.

The criterion is whether the answer includes a recovery operation that can be executed on a stack that is in UPDATE_FAILED with DisableRollback set to true. Specifically, I count the presentation of rollback-stack or of update-stack with --disable-rollback. continue-update-rollback does not count, even with --resources-to-skip. The determination was made by string matching on the answer text; I did not actually run the APIs to confirm.

There were 10 matches out of 51. All 10 include rollback-stack, and none matched through update-stack alone. Only 2 overlap with the 12 official passes.

The 8 models not shown in the table are 0/3 under both the official judgment and the other criterion.

Of the 12 answers judged correct officially, 10 contain no operation executable in the actual state at all, and use only continue-update-rollback as the recovery procedure. Conversely, claude-opus-5 and claude-opus-4.7 presented executable procedures in all three runs, yet their official judgments are 1/3 each.

Credits and Duration per Answer

I calculated credits and duration per answer, covering only the 10 answers that matched the other criterion. Answers that did not match are excluded. The counts in the table are separate from the official number of correct answers and represent the number of times an executable procedure was included.

claude-opus-4.5, which scored 3/3 in the official judgment, does not appear in this table. In all three runs it included no executable operation and mentioned only continue-update-rollback. In its place, claude-opus-4.8 and deepseek-3.2, which scored 0/3 officially, appear. Even for the same set of answers, changing the criterion swaps which models appear in the table.

Summary

The results of this task reflected the mechanics of the judgment more than the capability of the models. When the only judging material is the answer and the reference answer, a non-executable procedure becomes the correct answer, and an answer that takes the actual state into account fails.

This article covered one of the 134 tasks in aws-bench, and the results here cannot be extended as-is to the whole. Also, aws-bench is a research preview, and the official procedure prepares a clean test account for each scenario. Even so, as long as the reference answer presupposes a specific command, the judgment leans toward matching that string. If you use the benchmark in your own environment, it is safer to include a step of reading the contents of answers judged correct as well.