I tried the Step Functions diagnostic task from aws-bench with 17 models in Kiro CLI
This page has been translated by machine translation. View original
Introduction
I have written two previous articles about aws-bench, an AI agent benchmark published by AWS. The first article covered the benchmark's mechanism, and the second covered building a mini test environment to run it on a single account.
This time, I will introduce the results actually measured in that mini test environment. The task covered is diagnose-step-functions-failing-executions. The content involves investigating the cause of failing executions in a Step Functions state machine.
The target was 17 models, each run 3 times under the same conditions. Hereafter, one cell refers to one run of one model, and the denominator of 51 represents 17 models × 3 runs.
| Item | Content |
|---|---|
| Target models | 17. All models selectable from kiro-cli at the time of measurement, excluding auto and qwen3-coder-next |
| Number of runs | 3 runs under identical conditions. 51 cells per task |
| Agent | kiro-cli 2.15.1 run headlessly. Isolated in a Docker container; benchmark materials on the host side are not mounted |
| AWS permissions | Read-only role only. This task is diagnosis only and does not modify resources |
| Per-cell limit | 300 seconds for agent. Shortened from the official 600 seconds, but all 51 cells completed within the limit |
| Judgment | Fixed to claude-sonnet-5 on the host side |
| Credits | A comparative value obtained by multiplying the Credits from headless execution by the model multiplier. Not converted to monetary amounts |
| Measurement period | 2026-07-29 14:16 to 2026-07-30 03:34 |
The task definition references dataset commit 2daf77d2d41c21bae00bf8227fc463be51f721d0. The benchmark itself is edfab47710a505b7e73de66584a84f8b9bb6e26c. Note that claude-sonnet-5 used for judgment is also included among the 17 evaluated models. The impact of this overlap on judgment is not examined in this article.
Task Content
The task definition is available in the public dataset. The following is a summary; the original instruction text and reference answer can be found at the linked location.
| Item | Content |
|---|---|
| Type | introspection (read and diagnose only) |
| Target services | Step Functions, Lambda |
| Region | ap-southeast-2 |
| Official timeout | Agent 600 seconds / Judgment 240 seconds |
Resources Deployed in Advance
The resources to be diagnosed are created by a single CDK stack included with the scenario.
| Resource | Role |
|---|---|
| Lambda function (Python 3.12, inline code) | Processing function called by the state machine |
| CloudWatch Logs log group | Log output destination for the above Lambda |
| IAM role | Permissions for the state machine to call Lambda |
| Step Functions state machine | Calls Lambda in the InvokeProcessingLambda state |
The Injected Anomaly
The Lambda inline code contains only the following single line as the processing body.
def handler(event, context):
raise RuntimeError("Configuration validation failed: missing required field 'targetBucket'")
The error message reads as though the required field targetBucket is missing, but in reality the code does not inspect the input. Since the handler body consists solely of a single exception line, it will fail regardless of any input.
After deployment, the setup script runs, starts 5 executions of the state machine, and waits for them to complete. The anomaly itself is embedded on the CDK side, and this script serves the role of creating failed execution history and Lambda error logs. It is built to be idempotent, skipping executions if one with the same name already exists.
The Request to the Agent
The gist of the instruction is a single sentence: "Executions are failing in the specified state machine. Why?" Placeholders are replaced with actual ARNs before being passed to the agent in the isolated container. The agent investigates AWS using read-only temporary credentials and writes the final answer to /logs/agent/agent-output.txt. The reference answer and rubric are not passed to the container.
How Answers Are Judged
Judgment is binary—whether the answer is equivalent to the reference answer or not. There is no partial credit.
| Item | Content |
|---|---|
| Judgment criterion | answers_equivalent. Would a practitioner acting on the answer reach the same conclusion and take the same action as one following the reference answer? |
| Rubric | judge_prompt.md. A file shared across all 21 tasks in troubleshooting-multiservice; contains no task-specific descriptions |
| What is passed to the judgment model | Only the agent's answer and the reference answer. The actual AWS state is not passed |
| Summary of reference answer | All executions fail at InvokeProcessingLambda, and the cause is that Lambda throws a RuntimeError for missing targetBucket |
The rubric marks any answer that falls into any of the following categories as failing.
- The judgment of whether something is broken or normal is reversed
- The cause is wrong, and the explanation neither describes the problem nor leads to resolution
- The answer contradicts the reference answer regarding the actual state or resource configuration of the infrastructure
- The proposed fix would not actually work (missing parameters, wrong target)
- A primary cause identified in the reference answer is missing
- False information about the infrastructure is stated with certainty, misleading a practitioner
Since this task does not ask for a proposed fix, in practice only the alignment of the identified cause matters.
Official Judgment Results
All 17 models passed all 3 runs, resulting in 51/51.
All 51 judgment reasons also mentioned targetBucket, pointing to the same root cause and passing. Since there were no failures, there are no cases in this task where judgment discrepancies can be examined.
The content of the answers was not uniform. Only 10 out of 51 answers went as far as noting that the Lambda code throws an exception unconditionally without referencing the input. The breakdown is claude-opus-5 with 3, claude-opus-4.7 with 3, claude-opus-4.8 with 2, and claude-sonnet-5 with 2. The following is an excerpt from the 2nd run of claude-opus-5.
That Lambda's handler raises an unconditional RuntimeError on every invocation
...
is irrelevant to the failure because the handler never reads the event.
Primary fix - replace the unconditional raise in the Lambda with real validation and real work.
On the other hand, some answers proposed adding targetBucket to the state machine input as a fix. The following is an excerpt from the 2nd run of minimax-m2.1. Since the handler does not read the input, this fix would not resolve the issue.
To fix this issue, ensure that when invoking the state machine, the execution input
includes a `targetBucket` field with a valid S3 bucket name.
Similarly, gpt-5.6-luna and claude-haiku-4.5 also proposed addressing the issue on the input side. In the 3rd run of claude-haiku-4.5, it wrote as a fix on the Lambda side to "remove the required field from the validation logic in index.py." The handler contains no validation logic whatsoever. However, this difference had no effect on pass/fail outcomes.
Credits and Time Required per Correct Answer
Since all cells are correct answers, dividing the total credits and time across 3 runs by 3 gives the per-item values directly. The table is sorted in descending order of credits.
| Model | Multiplier | Correct answers | Credits per correct answer | Time per correct answer |
|---|---|---|---|---|
claude-opus-4.7 |
2.2 | 3/3 | 6.23 | 100s |
claude-opus-5 |
2.2 | 3/3 | 5.60 | 92s |
claude-opus-4.8 |
2.2 | 3/3 | 5.52 | 87s |
claude-opus-4.6 |
2.2 | 3/3 | 2.91 | 52s |
gpt-5.6-sol |
2.4 | 3/3 | 2.83 | 64s |
claude-opus-4.5 |
2.2 | 3/3 | 2.20 | 41s |
claude-sonnet-5 |
1.3 | 3/3 | 1.60 | 86s |
claude-sonnet-4 |
1.3 | 3/3 | 1.38 | 65s |
claude-sonnet-4.6 |
1.3 | 3/3 | 1.32 | 64s |
claude-sonnet-4.5 |
1.3 | 3/3 | 1.15 | 58s |
gpt-5.6-terra |
1.2 | 3/3 | 0.70 (0.56) | 34s |
glm-5 |
0.5 | 3/3 | 0.51 | 43s |
deepseek-3.2 |
0.25 | 3/3 | 0.19 | 77s |
gpt-5.6-luna |
0.6 | 3/3 | 0.19 (0.04) | 29s |
minimax-m2.5 |
0.25 | 3/3 | 0.15 | 49s |
claude-haiku-4.5 |
0.4 | 3/3 | 0.07 | 27s |
minimax-m2.1 |
0.15 | 3/3 | 0.05 | 34s |
The values in parentheses for gpt-5.6-luna and gpt-5.6-terra were calculated using the price reduction rates announced for Amazon Bedrock on July 30, 2026. These are values assuming the same rates would be reflected in Kiro's model multipliers. Luna saw an 80% price reduction and Terra a 20% reduction; Sol remained unchanged and is therefore not shown with a parenthetical value. This measurement was conducted before the price reduction announcement, and no revision to Kiro's multipliers has been announced. The table is sorted in descending order of the actual measured values outside the parentheses.
The top-ranked claude-opus-4.7 scored 6.23 and the bottom-ranked minimax-m2.1 scored 0.05, meaning the cost for the same correct answer differed by approximately 120 times. The range of model multipliers spans 16x, from 0.15 to 2.4, so the multiplier alone cannot explain the gap. There are also differences in raw credit consumption before the multiplier is applied, and the gap widens when multiplied by the multiplier.
The range of time required was 27 to 100 seconds, which did not diverge as much as credits. Furthermore, the ordering by credits and the ordering by time do not match. deepseek-3.2 and gpt-5.6-luna are tied at 0.19 credits per item, but their times are 77 seconds and 29 seconds respectively. A cheaper model is not necessarily faster.
Summary
Since all 17 models gave correct answers across all 3 runs, there is no room to select based on performance for this task. Given that even minimax-m2.1 with a multiplier of 0.15 passed, the difficulty level was manageable even for inexpensive open models.
What remains as selection criteria are cost and time. For simple investigations like the task introduced here when running with Kiro, gpt-5.6-luna and claude-haiku-4.5 are candidates. The former is 0.19 credits and 29 seconds (0.04 under the assumed price reduction), and the latter is 0.07 credits and 27 seconds. Both have low credit consumption, the shortest time required, and fulfill the requirement of identifying the cause. The Opus family and gpt-5.6-sol ranged from 2.20 to 6.23, which is 31 to 89 times higher than claude-haiku-4.5. Their time requirements were also long at 41 to 100 seconds, making them excessive in both cost and time for this requirement.
However, what can be obtained from inexpensive models is limited to "where and what is failing." gpt-5.6-luna and claude-haiku-4.5 proposed as fixes either passing targetBucket in the execution input or removing the required field from the Lambda validation logic. Both are premised on Lambda validating the input, but that validation logic does not exist. This was the result of inferring the implementation from the error message text; since the handler does not read the input, following the proposed fixes as written would not stop the failures.
In contrast, only 10 out of 51 answers read the Lambda code and stated that it would always fail regardless of input. Furthermore, only 7 answers were consistent in carrying that observation through to a proposed fix, explicitly stating that the Lambda code itself is what needs to be fixed. claude-opus-5 accounted for 3, claude-opus-4.7 for 2, and claude-opus-4.8 for 2; among these, 2 explicitly stated that adding targetBucket to the input would not fix the issue. The remaining 3 (claude-opus-4.7 with 1 and claude-sonnet-5 with 2) wrote that the failure was unconditional, but in their proposed fixes still listed adding targetBucket to environment variables or Payload as the top recommendation, creating an inconsistency between observation and remediation.
When time and cost are the priority and the task given is limited to detecting anomalies, there is a possibility that inexpensive models can handle it. In contrast, for situations requiring reading through the code and determining a root cause fix, it is preferable to allocate Opus-class models at greater time and cost. The task introduced this time falls into the former category. Opus-class models will also deliver results, but the cost is tens of times higher, making the cost-effectiveness poor for this particular task.
Note that among other tasks in the same dataset, there were some where the correct answer rate itself diverged. Those will be introduced in a separate article.
