I checked why Kiro CLI failed when it answered the steps that can be executed in the CloudFormation recovery task of aws-bench

Introduction

I have written four articles about aws-bench, a benchmark for AI agents published by AWS. The first article covered the benchmark's structure, the second covered building a mini test environment that runs in a single account. The third and fourth articles dealt with the results of Step Functions and VPC Flow Logs diagnostic tasks measured in that mini environment.

https://dev.classmethod.jp/articles/aws-bench-overview/

https://dev.classmethod.jp/articles/aws-bench-mini-tasks-local-runner/

https://dev.classmethod.jp/articles/aws-bench-diagnose-sfn-kiro-cli-17models/

https://dev.classmethod.jp/articles/aws-bench-check-vpc-kiro-cli-17models/

This time as well, these are results measured in the same mini test environment. The task covered is fix-cloudformation-update-failed-stack. It asks models to identify the cause of a stack stopped at UPDATE_FAILED and provide recovery procedures.

The correct answers for the two preceding tasks were 51 out of 51 for Step Functions and 22 out of 51 for VPC Flow Logs. This time it is 12. The reason for this difference lies not in the difficulty of the task, but in the limited materials available for evaluation.

Hereafter, one cell refers to one run of one model, and the denominator of 51 represents 17 models × 3 runs.

The task definition references dataset commit 2daf77d2d41c21bae00bf8227fc463be51f721d0. The benchmark itself is edfab47710a505b7e73de66584a84f8b9bb6e26c. Note that claude-sonnet-5 used for evaluation is also included among the 17 models being evaluated. The impact of this overlap on evaluation has not been verified in this article.

Identifiers in the article are obscured. Stack names, account IDs, and logical IDs have been replaced with dummies.

Comparisons are made only within this task; totals across other task results or overall model rankings are not presented.

Task Content

The task definition is in the public dataset. The following are key points; the original instruction text and reference answers can be confirmed at the linked location.

https://github.com/aws-bench/aws-bench-datasets/tree/2daf77d2d41c21bae00bf8227fc463be51f721d0/tasks/troubleshooting-multiservice/fix-cloudformation-update-failed-stack

Resources Deployed in Advance

The resources to be diagnosed are created by a single CDK stack included with the scenario.

Anomaly Created by the Setup Script

After deployment, the setup script runs and places the stack in UPDATE_FAILED.

1. Retrieve the original template
2. Replace the FunctionVersion of AWS::Lambda::Alias with a non-existent version number
3. Run the update specifying DisableRollback=True
4. Confirm that the stack has entered UPDATE_FAILED

The method of creating the anomaly is described with code in the second article. The key to understanding the results is step 3's DisableRollback=True. Because rollback is disabled, the stack stops at UPDATE_FAILED rather than UPDATE_ROLLBACK_FAILED.

Request to the Agent

The gist of the instruction is "the specified stack is in UPDATE_FAILED. How do you fix it and continue the deployment?" The state is explicitly stated here. After replacing placeholders with the actual stack name, it is passed to the agent in the isolated container. The agent examines AWS using read-only temporary credentials and writes the final answer to /logs/agent/agent-output.txt. Reference answers and rubrics are not passed to the container.

Answer Evaluation Method

Evaluation is binary: whether the answer is equivalent to the reference answer or not. There is no partial credit.

The reference answer identifies two causes and one recovery procedure. The causes are that non-existent function versions are hardcoded in two Lambda aliases, and that !GetAtt Function.Version should be used instead of hardcoding. The recovery procedure is to run continue-update-rollback passing the logical IDs of the two aliases to --resources-to-skip.

Of the failing conditions listed in the rubric, the following two apply to this task:

• Contradicts the reference answer regarding the actual infrastructure state or resource configuration
• The proposed fix would not actually work (missing parameters, wrong target)

Four others remain: reversed judgment of correctness, wrong cause, omission of a primary cause, and misleading a practitioner with a false assertion.

Since this task requires presenting a fix, the second condition — "would not actually work" — applies. The relationship between this condition and the recovery procedure in the reference answer determines the results.

Official Evaluation Results

Correct answers were 12/51. Timeouts were 0.

The dividing line between pass and fail was whether --resources-to-skip was explicitly stated. All 12 correct answers included continue-update-rollback, and 11 of those also specified --resources-to-skip. None of the 39 failing answers included --resources-to-skip. On the other hand, continue-update-rollback itself was mentioned in 26 of the 39 failing answers. The presence or absence of the option alone explains the pass/fail outcome for 50 out of 51 cells.

Reading the Answer Content

The actual state of the target stack is UPDATE_FAILED, with DisableRollback set to true. The continue-update-rollback required by the reference answer is an API for UPDATE_ROLLBACK_FAILED stacks and cannot be used on a stack in this state.

https://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_ContinueUpdateRollback.html

The options available for UPDATE_FAILED are retrying the update, update-stack with --disable-rollback, and rollback-stack.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stack-failure-options.html

The evaluation reasons for the 39 failing answers mostly follow the same pattern: the diagnosis of the cause matches the reference answer, but the recovery procedure does not reach continue-update-rollback --resources-to-skip. Answers that explicitly stated this API cannot be used also failed. In run 2, claude-opus-5 even provided the reason why it cannot be used.

Step 2 - Resume the deployment. The stack is in UPDATE_FAILED with DisableRollback=true,
which means it is directly updatable; you do NOT need continue-update-rollback (that API
only applies to UPDATE_ROLLBACK_FAILED, which is not the case here).

This answer instead proposes a direct update with a corrected template (Option A) and rollback-stack (Option B). Both are operations that can be executed against a stack in UPDATE_FAILED according to the documentation.

Option B - return to the last known good state first, then deploy the fix:

aws cloudformation rollback-stack \
  --stack-name troubleshooting-multiservice-cloudformation-example-us-east-1 \
  --role-arn arn:aws:iam::111122223333:role/cdk-hnb659fds-cfn-exec-role-...

The evaluation marked this as failing because "the operation differs from what the reference answer requires."

Here is an example from the passing side. Run 1 of claude-opus-4.5 presents the exact same command as the reference answer.

2. **Continue the stack update** with the corrected template:

   aws cloudformation continue-update-rollback \
     --stack-name troubleshooting-multiservice-cloudformation-example-us-east-1 \
     --resources-to-skip <logical IDs of the two aliases> \
     --region us-east-1

Only the answer and the reference answer are available for evaluation; the actual state of the stack is not included in either. As a result, an answer that writes continue-update-rollback exactly as the reference answer does will pass, even if it is a procedure the API would reject. The aforementioned condition "the proposed fix would not actually work" is also applied without any material to verify whether it would actually work.

There is also a discrepancy within the dataset itself. The CDK stack's comments state that the setup script places the stack in UPDATE_ROLLBACK_FAILED. However, what the setup script actually creates and confirms is UPDATE_FAILED. It is consistent to suppose the reference answer was written with UPDATE_ROLLBACK_FAILED in mind, but this is speculation.

Mini environment-specific circumstances did not affect the results of this task. Of the 51 answers, only runs 2 and 3 of claude-opus-5 referenced resources other than the target stack. Both referred to another stack included in the same scenario, which is also placed in the same account in the official procedure. The evaluation reasons for these two cases were also based solely on continue-update-rollback, and the references to the other stack were not cited as reasons for failure.

Results Recounted Under a Different Criterion

For this task, a simple correction of adding missed answers on the failing side is not sufficient. The passing side has the same problem — procedures that cannot be executed. Therefore, I set aside the official pass/fail distinction and applied the same criterion to all 51 cells.

The criterion is: "Does the answer include a recovery operation that can actually be executed against a stack in UPDATE_FAILED with DisableRollback set to true?" Specifically, I counted answers that presented rollback-stack or update-stack with --disable-rollback. continue-update-rollback was not counted as applicable even when accompanied by --resources-to-skip. The evaluation was performed by string matching against the answer text; I did not actually execute the APIs to verify.

Applicable answers were 10/51. All 10 included rollback-stack; none qualified solely through the update-stack approach. Only 2 of these 10 overlap with the 12 official correct answers.

The 8 models not shown in the table are 0/3 under both the official evaluation and the alternative criterion.

Of the 12 official correct answers, 10 contain no operations that can be executed against the actual state and list only continue-update-rollback as the recovery procedure. Conversely, claude-opus-5 and claude-opus-4.7 produced executable procedures in all 3 runs, but their official evaluation results are 1/3 each.

Credits and Time Per Answer

I calculated the credits and time per answer for only the 10 applicable cases under the alternative criterion, excluding answers that did not qualify. The counts in the table represent the number of times an executable procedure was included, separate from the official correct answer count.

claude-opus-4.5, which scored 3/3 in the official evaluation, does not appear in this table. This is because all 3 runs contained no executable operations and listed only continue-update-rollback. In its place, claude-opus-4.8 and deepseek-3.2, which scored 0/3 in the official evaluation, appear. Even within the same set of answers, changing the criterion causes different models to appear in the table.

Summary

The results of this task reflected the evaluation mechanism more than model capability. When evaluation materials consist only of the answer and the reference answer, procedures that cannot be executed become correct answers, while answers grounded in the actual state fail.

The task covered here is one of 134 tasks in aws-bench, and the results here cannot be directly extended to the whole. Additionally, aws-bench is in research preview, and the official procedure involves preparing a clean test account for each scenario. Nevertheless, as long as the reference answer presupposes a specific command, the evaluation gravitates toward matching that string. If you use this benchmark in your own environment, it is safer to include a step of reading the actual content of answers that were evaluated as correct.