I checked why Kiro CLI 17 models had mixed pass/fail results on the VPC Flow Logs diagnostic task in aws-bench
This page has been translated by machine translation. View original
Introduction
I have written three articles so far about aws-bench, an AI agent benchmark published by AWS. The first article covered the benchmark's mechanism, the second covered building a mini test environment running on a single account, and the third covered the results of the Step Functions diagnostic task measured in that mini environment.
This time as well, the results were measured in the same mini test environment. The task covered is check-vpc-flow-log-destinations. It is a task that investigates why a VPC's Flow Log appears to be delivered to multiple destinations. In the previous Step Functions task, all 17 models answered correctly across all 3 runs, but this time correct answers dropped to less than half.
Hereafter, 1 cell refers to 1 run of 1 model, and the denominator 51 represents 17 models × 3 runs.
| Item | Content |
|---|---|
| Target models | 17. All models selectable from kiro-cli at the time of measurement, excluding auto and qwen3-coder-next |
| Number of runs | 3 runs under identical conditions. 51 cells per task |
| Agent | kiro-cli 2.15.1 run headless. Isolated in a Docker container, with no mount of benchmark materials from the host side |
| AWS permissions | Read-only role only. This task involves diagnostics only, with no resource modifications |
| Per-cell limit | 300 seconds for the agent. Shortened from the official 600 seconds, but all 51 cells completed within the limit |
| Judgment | Fixed to claude-sonnet-5 on the host side |
| Credits | A comparative value obtained by multiplying the Credits of headless execution by the model multiplier. Not converted to a monetary amount |
| Measurement period | 2026-07-29 14:16 ~ 2026-07-30 03:34 |
The task definition references dataset commit 2daf77d2d41c21bae00bf8227fc463be51f721d0. The benchmark itself is edfab47710a505b7e73de66584a84f8b9bb6e26c. Note that claude-sonnet-5, used for judgment, is also included among the 17 models being evaluated. The impact of this overlap on judgment is not examined in this article.
Log group names, Flow Log IDs, and bucket names in the article have been redacted.
Comparisons are made only within this task. Results are not aggregated with other tasks, nor are overall model rankings produced. Differences from the previous task are mentioned in the summary.
Task Contents
Below are the key points of the task definition. The original text of the instructions and reference answers can be found at the link.
The type is introspection (read and diagnostic only), and the target services are VPC Flow Logs, S3, and CloudWatch Logs. The region is us-east-1, and the official timeouts are 600 seconds for the agent and 240 seconds for judgment.
Resources Deployed in Advance
The resources to be diagnosed are created by a single CDK stack included with the scenario.
| Resource | Role |
|---|---|
| VPC (10.2.0.0/16, 2 AZs, public subnets only, no NAT) | Diagnostic target |
| S3 bucket | Actual destination for Flow Log |
| 1 VPC Flow Log (destination: S3, all traffic, aggregation interval 600 seconds) | The only active Flow Log |
| 2 CloudWatch Logs log groups | Simply named to look like they are for Flow Logs, but no Flow Log writes to either of them |
The Planted Anomaly
This task has no broken resources and no setup script. A confusing state is simply left in place.
The configured destination is only 1, an S3 bucket. However, because there are 2 log groups whose names end in VpcFlowLogLogGroup1 and VpcFlowLogLogGroup2, it appears as though there are multiple destinations. According to a comment on the CDK side, the assumed scenario is that these are remnants of a previously failed stack deployment.
There is no data remaining at any destination. The Flow Log is ACTIVE and DeliverLogsStatus is also SUCCESS, but because there are no instances in the VPC and the ENI count is 0, there are no objects in the S3 bucket. The 2 log groups also have empty log streams. Which one is actually being used must be determined from configuration rather than actual data.
This is not a state unique to the mini environment. The CDK stack only creates a VPC, a bucket, 1 Flow Log, and 2 log groups, and even if the entire scenario is deployed following the official procedure, no resources that would be subject to logging are placed in this VPC. This is a task that requires diagnostics in a state where no valid log records exist.
The Request Given to the Agent
The gist of the instructions is: "The Flow Log for the specified VPC appears to be going to multiple destinations. Please check what is actually configured and which one is truly being used." After replacing the placeholder with the actual VPC ID, it is passed to the agent in the isolated container. The agent examines AWS using read-only temporary credentials and writes its final answer to /logs/agent/agent-output.txt. The reference answer and rubric are not passed to the container.
How Answers Are Judged
Judgment is binary: whether the answer is equivalent to the reference answer. There are no partial marks.
| Item | Content |
|---|---|
| Judgment criteria | answers_equivalent. Whether a practitioner acting on the answer would reach the same conclusion and take the same actions as one acting on the reference answer |
| Rubric | judge_prompt.md. A file shared across all 21 tasks in troubleshooting-multiservice, with no task-specific content |
| What is passed to the judgment model | Only the agent's answer and the reference answer. The actual state of AWS is not passed |
The reference answer lists 2 points.
- There is only 1 active Flow Log, delivering all traffic to an S3 bucket at 600-second intervals
- There are 2 log groups with names that look like they are for Flow Logs, but neither has Flow Log data written to it
The rubric marks the following types of answers as failing.
- The judgment of whether something is broken or normal is reversed
- The cause is wrong, does not explain the problem, and does not lead to a resolution
- Contradicts the reference answer regarding the actual state of the infrastructure or resource configuration
- Lists more resources than the reference answer indicates
- The proposed fix would not actually work (missing parameters, wrong target)
- A major cause identified in the reference answer is missing
- Asserts incorrect information about the infrastructure definitively, misleading practitioners
Since this task does not ask for a proposed fix, what is effectively being assessed is agreement on cause and configuration. Most judgment reasons cited were either "contradicts the reference answer regarding the actual state of the infrastructure or resource configuration" or "a major cause identified in the reference answer is missing."
Judgment reasons mentioning "lists more resources than the reference answer indicates" appeared in 4 out of 51 cases. Of these, 2 were considered and resulted in pass, while the remaining 2 (claude-haiku-4.5 run 3 and claude-opus-4.7 run 2) were the basis for a fail.
Official Judgment Results
Correct answers totaled 22 out of 51 cells, with 0 timeouts.
| Model | Multiplier | Run 1 | Run 2 | Run 3 | Correct |
|---|---|---|---|---|---|
claude-opus-4.5 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.6 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.8 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-5 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.7 |
2.2 | pass | fail | pass | 2/3 |
gpt-5.6-sol |
2.4 | pass | pass | fail | 2/3 |
deepseek-3.2 |
0.25 | pass | fail | pass | 2/3 |
claude-haiku-4.5 |
0.4 | pass | fail | fail | 1/3 |
claude-sonnet-5 |
1.3 | fail | pass | fail | 1/3 |
gpt-5.6-terra |
1.2 | fail | pass | fail | 1/3 |
glm-5 |
0.5 | fail | fail | pass | 1/3 |
claude-sonnet-4 |
1.3 | fail | fail | fail | 0/3 |
claude-sonnet-4.5 |
1.3 | fail | fail | fail | 0/3 |
claude-sonnet-4.6 |
1.3 | fail | fail | fail | 0/3 |
gpt-5.6-luna |
0.6 | fail | fail | fail | 0/3 |
minimax-m2.1 |
0.15 | fail | fail | fail | 0/3 |
minimax-m2.5 |
0.25 | fail | fail | fail | 0/3 |
In this task, results diverged within the Claude family by grade. The 5 Opus-series models answered correctly in 14 out of 15 cases, while the 4 Sonnet-series models answered correctly in only 1 out of 12 cases. The Opus-series / Sonnet-series distinction is limited to observations within this task.
What Determined Pass or Fail
Whether or not the 2 unconfigured log groups were specifically named determined the outcome. The presence or absence of naming and the official pass/fail judgment matched in all 51 cases.
The main finding — that there is only 1 active Flow Log destined for S3 — was identified correctly by nearly all failing answers as well. The difference came down to whether the answer could explain the cause of why the user felt there were "multiple destinations."
The passing answers specifically named these 2 log groups. The following is an excerpt from claude-opus-5 run 1.
NOT USED (2):
CWL group ExampleStack-Vpc-us-east-1-VpcFlowLogLogGroup1
CWL group ExampleStack-Vpc-us-east-1-VpcFlowLogLogGroup2
The failing answers concluded that no destination other than S3 existed. The following is an excerpt from gpt-5.6-luna run 1.
No separate CloudWatch Logs, Kinesis Data Firehose, or second S3 flow-log
destination is configured.
How They Reached the 2 Log Groups
These 2 log groups are not configured as destinations for any Flow Log. They cannot be found by following the results of ec2:DescribeFlowLogs, so it is necessary to investigate the CloudWatch Logs side independently. From the execution logs of each run, the methods used to investigate log groups were classified.
| Method of investigating log groups | pass | fail |
|---|---|---|
| Listed all without filtering by name | 20 | 5 |
| Narrowed down by specifying a prefix | 1 | 11 |
| Never queried at all | 1 | 13 |
The 12 cases that filtered by prefix specified /aws/vpc or flow by guesswork and came up empty. The actual name begins with the stack name and ends with VpcFlowLogLogGroup1. It is difficult to guess the correct prefix.
The 2 exceptions on the passing side found them via CloudFormation. gpt-5.6-sol run 2 used cloudformation list-stack-resources, and gpt-5.6-terra run 2 used describe-stack-resources and get-template to obtain the names of the 2 log groups before confirming them by name.
The 5 cases that listed all but still failed either finished without mentioning the 2 log groups despite them appearing in the list, or cited unrelated log groups.
Breakdown of the 29 Failing Cases
| Breakdown | Count |
|---|---|
| Stated that no CloudWatch destination existed, or did not mention the 2 log groups | 23 |
| Listed log groups unrelated to this task | 6 |
No answers were overturned upon re-examining the judgments, and the official judgment results serve as the final totals.
The fact that destinations had no data was not a reason for failing. Among the 51 cases, 42 mentioned that there was no data at the destinations, split between 21 passes and 21 fails. The judgment reasons also did not cite the emptiness as grounds for failure.
For the 23 cases in the former category, the judgment model wrote "contradicts actual state." The statement that the destinations were not configured is itself factual, but many answers wrote "there are no log groups for Flow Logs." The log groups themselves exist; they are simply not configured as Flow Log destinations.
The 6 cases in the latter category involved circumstances specific to the mini environment. The existing AWS account running the mini environment happened to have 1 log group with a Flow Log-like name that was unrelated to this task. These 6 cases found it and reported it as a confusing destination. The judgment model marked them as failing because both the count and the name differed from the reference answer. The relevant cases are claude-sonnet-5 runs 1 and 3, claude-opus-4.7 run 2, claude-sonnet-4.6 runs 2 and 3, and gpt-5.6-sol run 3.
This noise does not occur in the official procedure, as a clean test account is used for each scenario.
Credits and Time Required per Correct Answer
Looking only at the 22 cases judged as correct, the credits and time required per correct answer were calculated. The table is sorted in descending order of credits.
| Model | Multiplier | Correct answers | Credits per correct answer | Time per correct answer |
|---|---|---|---|---|
claude-opus-4.7 |
2.2 | 2/3 | 10.44 | 131 sec |
claude-opus-5 |
2.2 | 3/3 | 10.00 | 150 sec |
claude-opus-4.8 |
2.2 | 3/3 | 8.96 | 143 sec |
claude-opus-4.6 |
2.2 | 3/3 | 6.37 | 105 sec |
claude-opus-4.5 |
2.2 | 3/3 | 4.39 | 77 sec |
gpt-5.6-sol |
2.4 | 2/3 | 3.97 | 120 sec |
claude-sonnet-5 |
1.3 | 1/3 | 2.73 | 136 sec |
glm-5 |
0.5 | 1/3 | 1.56 | 76 sec |
gpt-5.6-terra |
1.2 | 1/3 | 1.51 | 72 sec |
deepseek-3.2 |
0.25 | 2/3 | 0.76 | 200 sec |
claude-haiku-4.5 |
0.4 | 1/3 | 0.19 | 63 sec |
The value for gpt-5.6-terra does not reflect the Amazon Bedrock price reduction announced on July 30, 2026. The measurement was conducted before the announcement, and no multiplier revision from the Kiro side has been announced either. If the 20% price reduction for Terra is assumed to be equally reflected in the multiplier, the credits per correct answer would be equivalent to approximately 1.21.
The 6 models with 0 correct answers were excluded from the table since a per-answer value cannot be calculated. The credits spent on these 6 models produced not a single correct answer.
Even among the 5 Opus-series models with the same 2.2 multiplier, credits per correct answer ranged from 4.39 to 10.44, a 2.4× difference. claude-opus-4.5 answered correctly all 3 times while having the smallest credits and time of all. The cheaper ones were the older 4.5 and 4.6, while the 3 models from 4.7 onward clustered between 8.96 and 10.44.
Summary
Most models identified the main finding correctly. What determined pass or fail was whether the agent looked in places that could not be reached from the Flow Log configuration. ec2:DescribeFlowLogs reveals only the single S3 entry, and the majority of failing answers stopped there. The 2 misleading log groups are not linked to any Flow Log, so no matter how carefully the destination configuration is read, they will not appear.
In real-world practice as well, when a user's premise of "multiple destinations appear to exist" is itself incorrect, simply confirming that the configuration is correct does not complete the explanation. If you are entrusting an investigation to an AI agent, you need to instruct it to include resources that do not appear in the configuration as part of the investigation scope.
In the previous Step Functions task, all 17 models answered correctly, and claude-haiku-4.5 and gpt-5.6-luna were suggested as cost-effective candidates. Those same 2 models scored 1/3 and 0/3 respectively this time. When selecting a model, the most reliable approach is to prepare tasks or test scenarios close to your actual workload and measure against them.
