I Checked Why the Pass/Fail Results Differed Across 17 Kiro CLI Models on aws-bench's VPC Flow Logs Diagnosis Task
This page has been translated by machine translation. View original
Introduction
I have written three articles so far about aws-bench, the AI agent benchmark published by AWS. The first covered how the benchmark works, the second covered building a mini test environment that runs in a single account, and the third covered the results of the Step Functions diagnosis task measured in that mini environment.
These are again results measured in the same mini test environment. This time I cover one task, check-vpc-flow-log-destinations. It is a task that has the agent investigate why a VPC's Flow Log appears to be delivering to multiple destinations. In the previous Step Functions task, all 17 models answered correctly in all 3 runs, but this time the correct answers dropped to less than half.
Hereafter, one cell means one run of one model, and the denominator of 51 represents 17 models × 3 runs.
| Item | Details |
|---|---|
| Target models | 17. All models selectable in kiro-cli at the time of measurement, except auto and qwen3-coder-next |
| Number of runs | 3 runs under identical conditions. 51 cells per task |
| Agent | kiro-cli 2.15.1 run headless. Isolated in a Docker container, with no benchmark materials from the host mounted |
| AWS permissions | Read-only role only. This task is diagnosis only, with no resource changes |
| Per-cell limit | Agent 300 seconds. Shortened from the official 600 seconds, but all 51 cells finished within the limit |
| Judging | Fixed on the host side to claude-sonnet-5 |
| Credits | Values for comparison, calculated by multiplying the headless run's Credits by the model multiplier. Not converted to money |
| Measurement period | 2026-07-29 14:16 to 2026-07-30 03:34 |
The task definition refers to dataset commit 2daf77d2d41c21bae00bf8227fc463be51f721d0. The benchmark itself is edfab47710a505b7e73de66584a84f8b9bb6e26c. Note that claude-sonnet-5, which was used for judging, is also among the 17 models being evaluated. The effect of this overlap on the judging is not verified in this article.
Log group names, Flow Log IDs, and bucket names in the article are masked.
Comparisons are made only within this task. I do not add up results with other tasks or produce an overall model ranking. The differences from the previous task are touched on in the summary.
Task Contents
Below are the key points of the task definition. The original text of the instructions and the reference answer can be checked at the link.
The type is introspection (read-only, diagnosis only), and the target services are VPC Flow Logs, S3, and CloudWatch Logs. The region is us-east-1, and the official timeouts are 600 seconds for the agent / 240 seconds for the judge.
Resources deployed in advance
The resources to be diagnosed are created by a single CDK stack bundled with the scenario.
| Resource | Role |
|---|---|
| VPC (10.2.0.0/16, 2 AZs, public subnets only, no NAT) | Diagnosis target |
| S3 bucket | The actual destination of the Flow Log |
| One VPC Flow Log (destination S3, all traffic, aggregation interval 600 seconds) | The only active Flow Log |
| 2 CloudWatch Logs log groups | They merely have names that look like they are for a Flow Log; nothing is written to them by any Flow Log |
The planted anomaly
This task has no broken resources and no setup script. A confusing state is simply left in place.
The only configured destination is a single S3 one. However, there are two log groups whose names end in VpcFlowLogLogGroup1 and VpcFlowLogLogGroup2, so it looks as if there are multiple destinations. The comments on the CDK side assume they are leftovers from a previously failed stack deployment.
No data remains at the destinations. The Flow Log is ACTIVE and DeliverLogsStatus is also SUCCESS, but because the VPC has no instances and 0 ENIs, the S3 bucket contains no objects at all. The two log groups also have empty log streams. Which one is actually used must be judged from the configuration, not from the actual data.
This is not a state specific to the mini environment. The CDK stack creates only the VPC, the bucket, one Flow Log, and two log groups, and even if the whole scenario is deployed by the official procedure, no resources to be recorded are placed in this VPC. It is a task of diagnosing in a state where no valid log records exist.
Request to the agent
The gist of the instruction is: "The Flow Log of the specified VPC seems to be going to multiple destinations. Please check what is actually configured and which one is really being used." The placeholder is replaced with the actual VPC ID and passed to the agent in the isolated container. The agent investigates AWS with read-only temporary credentials and writes its final answer to /logs/agent/agent-output.txt. The reference answer and rubric are not passed to the container.
How answers are judged
Judging is a binary decision on whether the answer is equivalent to the reference answer. There is no partial credit.
| Item | Details |
|---|---|
| Criterion | answers_equivalent. Whether a practitioner who follows the answer would reach the same conclusion and take the same actions as one who follows the reference answer |
| Rubric | judge_prompt.md. A file common to the 21 troubleshooting-multiservice tasks, with no task-specific description |
| What is passed to the judge model | Only the agent's answer and the reference answer. The actual AWS state is not passed |
The reference answer lists two points.
- There is one active Flow Log, delivering all traffic to an S3 bucket at 600-second intervals
- There are two log groups with names that look like they are for a Flow Log, but no Flow Log data is written to either
The rubric fails an answer that falls under any of the following.
- The judgment of broken versus healthy is reversed
- The cause is wrong, so it neither explains the problem nor resolves it
- It contradicts the reference answer about the actual state of the infrastructure or the resource configuration
- It enumerates more resources than the reference answer shows
- The proposed fix would not actually work (missing parameters, wrong target)
- A major cause listed in the reference answer is missing
- It asserts incorrect information about the infrastructure and misleads the practitioner
Since this task does not ask for a fix to be presented, what is effectively tested is agreement on the cause and configuration. Most of the judgment reasons were one of two: "contradicts the reference answer about the actual state of the infrastructure or the resource configuration" and "a major cause listed in the reference answer is missing."
The judgment reasons that mention "enumerates more resources than the reference answer shows" number 4 out of 51. Of these, 2 considered this clause and still passed, and the remaining 2 (run 3 of claude-haiku-4.5 and run 2 of claude-opus-4.7) used it as grounds for fail.
Official judgment results
There were 22 correct answers out of 51, and 0 timeouts.
| Model | Multiplier | Run 1 | Run 2 | Run 3 | Correct |
|---|---|---|---|---|---|
claude-opus-4.5 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.6 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.8 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-5 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.7 |
2.2 | pass | fail | pass | 2/3 |
gpt-5.6-sol |
2.4 | pass | pass | fail | 2/3 |
deepseek-3.2 |
0.25 | pass | fail | pass | 2/3 |
claude-haiku-4.5 |
0.4 | pass | fail | fail | 1/3 |
claude-sonnet-5 |
1.3 | fail | pass | fail | 1/3 |
gpt-5.6-terra |
1.2 | fail | pass | fail | 1/3 |
glm-5 |
0.5 | fail | fail | pass | 1/3 |
claude-sonnet-4 |
1.3 | fail | fail | fail | 0/3 |
claude-sonnet-4.5 |
1.3 | fail | fail | fail | 0/3 |
claude-sonnet-4.6 |
1.3 | fail | fail | fail | 0/3 |
gpt-5.6-luna |
0.6 | fail | fail | fail | 0/3 |
minimax-m2.1 |
0.15 | fail | fail | fail | 0/3 |
minimax-m2.5 |
0.25 | fail | fail | fail | 0/3 |
In this task, results split by grade within the Claude lineup. The 5 Opus models got 14 of 15 correct, while the 4 Sonnet models got only 1 of 12 correct. The grouping into Opus / Sonnet is an observation limited to this task.
What separated pass from fail
Whether the answer named the two log groups that are not configured as destinations separated pass from fail. Whether they were named and the official pass / fail judgment match in all 51 cases.
Most of the failing answers also got the main finding right: the only active Flow Log is the one going to S3. The difference was whether they could explain the reason the user felt there were "multiple destinations."
The correct answers name these two. Here is an excerpt from run 1 of claude-opus-5.
NOT USED (2):
CWL group ExampleStack-Vpc-us-east-1-VpcFlowLogLogGroup1
CWL group ExampleStack-Vpc-us-east-1-VpcFlowLogLogGroup2
The failing answers end with the conclusion that there is no destination other than S3. Here is an excerpt from run 1 of gpt-5.6-luna.
No separate CloudWatch Logs, Kinesis Data Firehose, or second S3 flow-log
destination is configured.
How the two were found
These two log groups are not configured as the destination of any Flow Log. They cannot be traced from the result of ec2:DescribeFlowLogs, so you have to investigate the CloudWatch Logs side yourself. From the execution log of each run, I classified how the log groups were investigated.
| How log groups were investigated | pass | fail |
|---|---|---|
| Listed all without filtering by name | 20 | 5 |
| Filtered by specifying a prefix | 1 | 11 |
| Never queried | 1 | 13 |
The 12 that filtered by prefix specified /aws/vpc or flow by guess and came up empty. The actual names start with the stack name and end with VpcFlowLogLogGroup1. Guessing the prefix correctly is difficult.
The 2 exceptions on the pass side traced them through CloudFormation. Run 2 of gpt-5.6-sol used cloudformation list-stack-resources, and run 2 of gpt-5.6-terra used describe-stack-resources and get-template to get the names of the two, and then checked the log groups by name.
The 5 that failed even after listing everything either finished without touching the two even though they were in the list, or cited unrelated log groups.
Breakdown of the 29 failures
| Breakdown | Count |
|---|---|
| Stated that no CloudWatch destination exists, or did not mention the two | 23 |
| Cited log groups unrelated to this task | 6 |
No answer would be overturned on review of the judgments, so the official judgment results are the totals as they are.
The destinations being empty was not a reason for failure. 42 of 51 answers mentioned that there is no data at the destinations, split into 21 pass and 21 fail. The judgment reasons also did not use emptiness as grounds for failure.
For the former 23, the judge model wrote that they "contradict the actual state." The statement that they are not configured as destinations is itself true, but many answers wrote that "there are no log groups for the Flow Log." The log groups do exist; they are just not destinations of the Flow Log.
The latter 6 involve a circumstance specific to the mini environment. The existing AWS account running the mini environment had one log group with a Flow Log-like name unrelated to this task. The 6 found it and reported it as a confusing destination. The judge model failed them because both the count and the name differed from the reference answer. The affected runs are runs 1 and 3 of claude-sonnet-5, run 2 of claude-opus-4.7, runs 2 and 3 of claude-sonnet-4.6, and run 3 of gpt-5.6-sol.
This noise does not occur with the official procedure, because a clean test account is used for each scenario.
Credits and time per correct answer
Credits and elapsed time per answer were calculated only for the 22 answers judged correct. The table is in descending order of credits.
| Model | Multiplier | Correct count | Credits per correct answer | Time per correct answer |
|---|---|---|---|---|
claude-opus-4.7 |
2.2 | 2/3 | 10.44 | 131 sec |
claude-opus-5 |
2.2 | 3/3 | 10.00 | 150 sec |
claude-opus-4.8 |
2.2 | 3/3 | 8.96 | 143 sec |
claude-opus-4.6 |
2.2 | 3/3 | 6.37 | 105 sec |
claude-opus-4.5 |
2.2 | 3/3 | 4.39 | 77 sec |
gpt-5.6-sol |
2.4 | 2/3 | 3.97 | 120 sec |
claude-sonnet-5 |
1.3 | 1/3 | 2.73 | 136 sec |
glm-5 |
0.5 | 1/3 | 1.56 | 76 sec |
gpt-5.6-terra |
1.2 | 1/3 | 1.51 | 72 sec |
deepseek-3.2 |
0.25 | 2/3 | 0.76 | 200 sec |
claude-haiku-4.5 |
0.4 | 1/3 | 0.19 | 63 sec |
The value for gpt-5.6-terra does not reflect the Amazon Bedrock price reduction announced on July 30, 2026. This measurement was made before the announcement, and no revision of the multiplier on the Kiro side has been announced either. If we assume the 20% price reduction rate for Terra is reflected equally in the multiplier, the credits per answer would be equivalent to 1.21.
The 6 models with 0 correct answers were excluded from the table because a per-answer value cannot be computed. The credits spent on these 6 models did not produce a single correct answer.
Even among the 5 Opus models with the same multiplier of 2.2, the credits per answer differed by a factor of 2.4, from 4.39 to 10.44. claude-opus-4.5 answered correctly in all 3 runs while having the lowest credits and the shortest time. The cheaper ones were the older 4.5 and 4.6, while the three models from 4.7 onward cluster at 8.96 to 10.44.
Summary
Most models got the main finding right. What separated pass from fail was whether they looked in places that cannot be traced from the Flow Log configuration. ec2:DescribeFlowLogs only reveals the one going to S3, and answers that stopped there made up most of the failures. The two confusing log groups are not tied to any Flow Log, so no amount of careful reading of the destination configuration would surface them.
In practice too, when the user's premise that there "appear to be multiple destinations" is itself mistaken, confirming that the configuration is correct does not complete the explanation. If you leave an investigation to an AI agent, you need to instruct it to include resources that do not appear in the configuration.
In the previous Step Functions task, all 17 models answered correctly, and I named the inexpensive claude-haiku-4.5 and gpt-5.6-luna as candidates. This time those same two models scored 1/3 and 0/3 respectively. When choosing a model, the surest approach is to prepare tasks and test scenarios close to your actual workload and measure with them.
