Trying aws-bench's Step Functions Diagnostic Task with 17 Models in Kiro CLI
This page has been translated by machine translation. View original
Introduction
I have written two articles so far about aws-bench, the AI agent benchmark published by AWS. The first covered how the benchmark works, and the second covered building a mini test environment that runs in a single account.
This time, I will present the results I actually measured in that mini test environment. I focus on a single task, diagnose-step-functions-failing-executions. In this task, the agent investigates why executions of a Step Functions state machine are failing.
The target was 17 models, each solving the task three times under the same conditions. From here on, one cell refers to one run of one model, and the denominator of 51 represents 17 models × 3 runs.
| Item | Details |
|---|---|
| Target models | 17. All models selectable from kiro-cli at measurement time, except auto and qwen3-coder-next |
| Number of runs | 3 runs under identical conditions. 51 cells per task |
| Agent | kiro-cli 2.15.1 run headless. Isolated in a Docker container, with no benchmark materials from the host mounted |
| AWS permissions | read-only role only. This task is diagnosis only and does not modify any resources |
| Per-cell limit | 300 seconds for the agent. Shortened from the official 600 seconds, but all 51 cells finished within the limit |
| Judging | Fixed to claude-sonnet-5 on the host side |
| Credits | A comparison value obtained by multiplying the headless run's Credits by the model multiplier. Not converted to monetary amounts |
| Measurement period | 2026-07-29 14:16 to 2026-07-30 03:34 |
The task definition refers to dataset commit 2daf77d2d41c21bae00bf8227fc463be51f721d0. The benchmark itself is edfab47710a505b7e73de66584a84f8b9bb6e26c. Note that claude-sonnet-5, used for judging, is also among the 17 models under evaluation. This article does not examine the effect of this overlap on judging.
Task Content
The task definition is in the public dataset. The following is a summary; the original instruction text and reference answer can be found at the link.
| Item | Details |
|---|---|
| Type | introspection (read and diagnose only) |
| Target services | Step Functions, Lambda |
| Region | ap-southeast-2 |
| Official timeout | Agent 600 seconds / Judge 240 seconds |
Resources Deployed in Advance
The resources to be diagnosed are created by a single CDK stack bundled with the scenario.
| Resource | Role |
|---|---|
| Lambda function (Python 3.12, inline code) | Processing function called from the state machine |
| CloudWatch Logs log group | Log output destination for the Lambda above |
| IAM role | Permissions for the state machine to call the Lambda |
| Step Functions state machine | Calls the Lambda in the InvokeProcessingLambda state |
The Planted Anomaly
The main body of the Lambda inline code is just the following single line.
def handler(event, context):
raise RuntimeError("Configuration validation failed: missing required field 'targetBucket'")
The error message reads as "required field targetBucket is missing", but in reality the code does not inspect the input. Since the handler body is just one line that raises an exception, it fails on any input.
After deployment, a setup script runs, starts five executions of the state machine, and waits for them to finish. The anomaly itself is embedded on the CDK side, and this script is responsible for producing the failed execution history and Lambda error logs. It is idempotent: if an execution with the same name already exists, it is skipped.
Request to the Agent
The gist of the instruction is a single sentence: "Executions are failing on the specified state machine. Why?" After replacing the placeholder with the actual ARN, it is passed to the agent in the isolated container. The agent investigates AWS with read-only temporary credentials and writes its final answer to /logs/agent/agent-output.txt. The reference answer and rubric are not passed to the container.
How Answers Are Judged
Judging is binary: whether the answer is equivalent to the reference answer. There is no partial credit.
| Item | Details |
|---|---|
| Judging criterion | answers_equivalent. Whether a practitioner following the answer would reach the same conclusion and take the same actions as one following the reference answer |
| Rubric | judge_prompt.md. A file shared by all 21 troubleshooting-multiservice tasks, with no task-specific description |
| What is given to the judge model | Only the agent's answer and the reference answer. The actual AWS state is not given |
| Gist of the reference answer | All executions fail at InvokeProcessingLambda, and the cause is that the Lambda raises a RuntimeError about a missing targetBucket |
The rubric fails any answer that falls under one of the following.
- The judgment of whether something is broken or healthy is reversed
- The cause is wrong, so it neither explains the problem nor resolves it
- It contradicts the reference answer about the actual infrastructure state or resource configuration
- The proposed fix would not actually work (missing parameters, wrong target)
- The main cause given in the reference answer is missing
- It asserts incorrect information about the infrastructure and misleads the practitioner
Because this task does not ask for a fix to be presented, in practice only agreement on the cause is tested.
Official Judging Results
All 17 models passed on all three runs, for 51/51.
All 51 judgment reasons also mention targetBucket, passing on the basis of pointing to the same root cause. Since there were no fails, there is nothing in this task to examine regarding judging discrepancies.
The contents of the answers were not uniform. Answers that went as far as noting that the Lambda code does not reference the input and throws an exception unconditionally numbered 10 out of 51. The breakdown is 3 from claude-opus-5, 3 from claude-opus-4.7, 2 from claude-opus-4.8, and 2 from claude-sonnet-5. The following is an excerpt from the second run of claude-opus-5.
That Lambda's handler raises an unconditional RuntimeError on every invocation
...
is irrelevant to the failure because the handler never reads the event.
Primary fix - replace the unconditional raise in the Lambda with real validation and real work.
On the other hand, some answers suggested that adding targetBucket to the state machine input would fix it. The following is an excerpt from the second run of minimax-m2.1. Because the handler does not read the input, this fix does not resolve the problem.
To fix this issue, ensure that when invoking the state machine, the execution input
includes a `targetBucket` field with a valid S3 bucket name.
gpt-5.6-luna and claude-haiku-4.5 also proposed addressing it on the input side in the same way. In the third run of claude-haiku-4.5, as a proposal to fix the Lambda side, it wrote "remove the required field from the validation logic in index.py". The handler has no validation logic at all. However, this difference did not affect the pass/fail result.
Credits and Time per Correct Answer
Since every cell was correct, the credits and time for the three runs divided by 3 are directly the per-answer values. The table is in descending order of credits.
| Model | Multiplier | Correct | Credits per correct answer | Time per correct answer |
|---|---|---|---|---|
claude-opus-4.7 |
2.2 | 3/3 | 6.23 | 100 sec |
claude-opus-5 |
2.2 | 3/3 | 5.60 | 92 sec |
claude-opus-4.8 |
2.2 | 3/3 | 5.52 | 87 sec |
claude-opus-4.6 |
2.2 | 3/3 | 2.91 | 52 sec |
gpt-5.6-sol |
2.4 | 3/3 | 2.83 | 64 sec |
claude-opus-4.5 |
2.2 | 3/3 | 2.20 | 41 sec |
claude-sonnet-5 |
1.3 | 3/3 | 1.60 | 86 sec |
claude-sonnet-4 |
1.3 | 3/3 | 1.38 | 65 sec |
claude-sonnet-4.6 |
1.3 | 3/3 | 1.32 | 64 sec |
claude-sonnet-4.5 |
1.3 | 3/3 | 1.15 | 58 sec |
gpt-5.6-terra |
1.2 | 3/3 | 0.70 (0.56) | 34 sec |
glm-5 |
0.5 | 3/3 | 0.51 | 43 sec |
deepseek-3.2 |
0.25 | 3/3 | 0.19 | 77 sec |
gpt-5.6-luna |
0.6 | 3/3 | 0.19 (0.04) | 29 sec |
minimax-m2.5 |
0.25 | 3/3 | 0.15 | 49 sec |
claude-haiku-4.5 |
0.4 | 3/3 | 0.07 | 27 sec |
minimax-m2.1 |
0.15 | 3/3 | 0.05 | 34 sec |
The values in parentheses for gpt-5.6-luna and gpt-5.6-terra were calculated using the Amazon Bedrock price reduction rates announced on July 30, 2026. They assume the same rates are reflected in Kiro's model multipliers. Luna has an 80% reduction and Terra a 20% reduction; Sol is unchanged, so no parenthetical value is given. This measurement was taken before the price reduction announcement, and no revision of Kiro's multipliers has been announced. The table order is descending by the measured values outside the parentheses.
The top model, claude-opus-4.7, is at 6.23 and the bottom, minimax-m2.1, is at 0.05, so the payment for the same correct answer differs by about 120 times. The multiplier range is 16 times, from 0.15 to 2.4, so the multiplier alone cannot explain it. There are also differences in credit consumption before the multiplier is applied, and the product with the multiplier widens the gap.
The time range is 27 to 100 seconds, which did not spread as much as credits. Also, the order of credits and the order of time do not match. deepseek-3.2 and gpt-5.6-luna are tied at 0.19 per answer, but their times are 77 seconds and 29 seconds. A cheaper model is not necessarily faster.
Summary
Since all 17 models answered correctly on all three runs, there is no room to choose by performance on this task. Even minimax-m2.1 with a 0.15 multiplier passed, so the difficulty was one that even inexpensive open models can solve.
What remains as selection criteria is payment and time. If you run a simple investigation like the task presented here on Kiro, gpt-5.6-luna and claude-haiku-4.5 are candidates. The former is 0.19 and 29 seconds (0.04 under the assumed price reduction), and the latter is 0.07 and 27 seconds. Both have low credits and the shortest times, and they met the requirement of identifying the cause. The Opus family and gpt-5.6-sol were 2.20 to 6.23, which is 31 to 89 times claude-haiku-4.5. Their times were also long at 41 to 100 seconds, so for this requirement they were excessive in both cost and time.
However, what you get from inexpensive models is only "what is failing where". gpt-5.6-luna and claude-haiku-4.5 proposed, as fixes, passing targetBucket in the execution input, and removing the required field from the Lambda's validation logic. Both rest on the premise that the Lambda validates its input, but that validation logic does not exist. This was the result of inferring the implementation from the wording of the error message, and since the handler does not read the input, fixing it as written would not stop the failures.
On the other hand, answers that read the Lambda code and wrote that it always fails regardless of input numbered 10 out of 51. Furthermore, those that carried that observation consistently through to the fix proposal and explicitly stated that the place to fix is the Lambda code numbered 7. These were 3 from claude-opus-5, 2 from claude-opus-4.7, and 2 from claude-opus-4.8, and 2 of them stated that adding targetBucket to the input would not fix it. The remaining 3 (1 from claude-opus-4.7 and 2 from claude-sonnet-5) wrote that it fails unconditionally. However, in their fix proposals they listed adding targetBucket to an environment variable or the Payload as the leading option, so their observation and remedy are inconsistent.
If you prioritize time and cost and the task you give goes only as far as detecting the anomaly, an inexpensive model may be able to handle it. In contrast, in situations where you need it to read the code and arrive at a root-cause fix policy, it is preferable to assign an Opus-family model, spending the time and cost. The task here falls into the former. Assigning an Opus-family model would still yield results, but the payment would be dozens of times higher, so the cost-effectiveness for this task is poor.
Note that in other tasks in the same dataset, the correct answer rate itself diverged. I plan to introduce those in a separate article.
