I checked why Kiro CLI 17 models had mixed pass/fail results on the VPC Flow Logs diagnostic task in aws-bench
This page has been translated by machine translation. View original
Introduction
I have written three articles so far about aws-bench, an AI agent benchmark published by AWS. The first article covered the benchmark's mechanism, the second covered building a mini test environment running on a single account, and the third covered the results of Step Functions diagnostic tasks measured in that mini environment.
This time as well, the results were measured in the same mini test environment. The task covered is check-vpc-flow-log-destinations. It is a task that investigates why a VPC's Flow Log appears to be delivered to multiple destinations. In the previous Step Functions task, all 17 models answered correctly across all 3 runs, but this time correct answers fell to less than half.
Hereafter, 1 cell refers to 1 run of 1 model, and the denominator 51 represents 17 models × 3 runs.
| Item | Details |
|---|---|
| Target models | 17. All models selectable from kiro-cli at the time of measurement, excluding auto and qwen3-coder-next |
| Number of runs | 3 runs under identical conditions. 51 cells per task |
| Agent | kiro-cli 2.15.1 run headless. Isolated in a Docker container, with no mount of benchmark materials from the host side |
| AWS permissions | Read-only role only. This task involves diagnostics only, with no resource modifications |
| Per-cell limit | 300 seconds for the agent. Shortened from the official 600 seconds, but all 51 cells completed within the limit |
| Judgment | Fixed to claude-sonnet-5 on the host side |
| Credits | A comparative value obtained by multiplying the Credits from headless execution by the model multiplier. Not converted to a monetary amount |
| Measurement period | 2026-07-29 14:16 to 2026-07-30 03:34 |
The task definition references dataset commit 2daf77d2d41c21bae00bf8227fc463be51f721d0. The benchmark itself is edfab47710a505b7e73de66584a84f8b9bb6e26c. Note that claude-sonnet-5 used for judgment is also included among the 17 models being evaluated. The impact of this overlap on judgment has not been verified in this article.
Log group names, Flow Log IDs, and bucket names in the article have been redacted.
Comparisons are made only within this task. Results will not be combined with other tasks, nor will overall model rankings be produced. Differences from the previous task will be mentioned in the summary.
Task Overview
The following is a summary of the task definition. The original text of the instructions and reference answers can be found at the linked page.
The type is introspection (read and diagnostics only), and the target services are VPC Flow Logs, S3, and CloudWatch Logs. The region is us-east-1, and the official timeouts are 600 seconds for the agent and 240 seconds for judgment.
Pre-deployed Resources
The resources to be diagnosed are created by a single CDK stack included with the scenario.
| Resource | Role |
|---|---|
| VPC (10.2.0.0/16, 2 AZs, public subnets only, no NAT) | Diagnostic target |
| S3 bucket | Actual destination for Flow Log |
| 1 VPC Flow Log (destination: S3, all traffic, aggregation interval 600 seconds) | The only active Flow Log |
| 2 CloudWatch Logs log groups | Named to appear related to Flow Logs, but not written to by any Flow Log |
Planted Anomaly
This task has no broken resources and no setup script. A confusing state is simply left in place.
The configured destination is only one S3 bucket. However, because two log groups exist with names ending in VpcFlowLogLogGroup1 and VpcFlowLogLogGroup2, it appears as though there are multiple destinations. According to a comment on the CDK side, these are assumed to be remnants of a previously failed stack deployment.
There is no data remaining at the destinations. The Flow Log is ACTIVE and DeliverLogsStatus is also SUCCESS, but since there are no instances in the VPC and zero ENIs, there are no objects in the S3 bucket. Both log groups also have empty log streams. Determining which destination is actually being used must be based on configuration rather than actual data.
This is not a state unique to the mini environment. The CDK stack only creates a VPC, a bucket, one Flow Log, and two log groups; even when deploying the entire scenario following official procedures, no resources subject to recording are placed in this VPC. This is a task that requires diagnosis in a state where no valid log records exist.
Request to the Agent
The summary of the instructions is: "The Flow Log for a specified VPC appears to be going to multiple destinations. Please verify what is actually configured and which ones are truly in use." After replacing the placeholder with the actual VPC ID, it is passed to the agent in the isolated container. The agent investigates AWS using read-only temporary credentials and writes the final answer to /logs/agent/agent-output.txt. Reference answers and rubrics are not passed to the container.
How Answers Are Judged
Judgment is binary: whether the answer is equivalent to the reference answer or not. There is no partial credit.
| Item | Details |
|---|---|
| Judgment criterion | answers_equivalent. Whether a practitioner following the answer would reach the same conclusion and take the same actions as one following the reference answer |
| Rubric | judge_prompt.md. A file common to all 21 tasks under troubleshooting-multiservice, with no task-specific content |
| What is passed to the judgment model | Only the agent's answer and the reference answer. The actual AWS state is not passed |
The reference answer raises two points.
- There is only one active Flow Log, delivering all traffic to an S3 bucket at 600-second intervals
- Two log groups with names that appear to be for Flow Logs exist, but neither has Flow Log data written to it
The rubric marks the following types of answers as failing:
- The judgment of whether something is broken or normal is reversed
- The cause is wrong, and the explanation neither describes the problem nor leads to resolution
- The answer contradicts the reference answer regarding the actual state of the infrastructure or resource configuration
- The answer lists more resources than the reference answer indicates
- The proposed fix would not actually work (missing parameters, wrong target)
- A primary cause identified in the reference answer is missing
- False information about the infrastructure is asserted with certainty, leading practitioners astray
Since this task does not require proposing a fix, what is essentially being tested is alignment on cause and configuration. The majority of judgment reasons cited were "contradicts the reference answer regarding the actual state of the infrastructure or resource configuration" and "a primary cause identified in the reference answer is missing."
Judgment reasons that referenced "lists more resources than the reference answer indicates" appeared in 4 out of 51 cases. Of these, 2 were considered and still resulted in a pass, while the remaining 2 (claude-haiku-4.5 run 3 and claude-opus-4.7 run 2) used this clause as grounds for a fail.
Official Judgment Results
Correct answers totaled 22 out of 51 cases, with 0 timeouts.
| Model | Multiplier | Run 1 | Run 2 | Run 3 | Correct |
|---|---|---|---|---|---|
claude-opus-4.5 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.6 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.8 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-5 |
2.2 | pass | pass | pass | 3/3 |
claude-opus-4.7 |
2.2 | pass | fail | pass | 2/3 |
gpt-5.6-sol |
2.4 | pass | pass | fail | 2/3 |
deepseek-3.2 |
0.25 | pass | fail | pass | 2/3 |
claude-haiku-4.5 |
0.4 | pass | fail | fail | 1/3 |
claude-sonnet-5 |
1.3 | fail | pass | fail | 1/3 |
gpt-5.6-terra |
1.2 | fail | pass | fail | 1/3 |
glm-5 |
0.5 | fail | fail | pass | 1/3 |
claude-sonnet-4 |
1.3 | fail | fail | fail | 0/3 |
claude-sonnet-4.5 |
1.3 | fail | fail | fail | 0/3 |
claude-sonnet-4.6 |
1.3 | fail | fail | fail | 0/3 |
gpt-5.6-luna |
0.6 | fail | fail | fail | 0/3 |
minimax-m2.1 |
0.15 | fail | fail | fail | 0/3 |
minimax-m2.5 |
0.25 | fail | fail | fail | 0/3 |
In this task, results diverged along grade lines within the Claude family. The 5 Opus models achieved 14 correct out of 15 cases, while the 4 Sonnet models achieved only 1 correct out of 12 cases. The Opus/Sonnet distinction is an observation limited to this task.
What Determined Pass or Fail
Whether or not the two log groups that were not configured were specifically named was what determined pass or fail. The presence or absence of naming and the official judgment of pass/fail matched across all 51 cases.
The primary finding — that there is only one active Flow Log, directed to S3 — was identified by nearly all answers, including those that failed. The difference came down to whether the answer could explain the cause of why the user perceived "multiple destinations."
The passing answers specifically named these two log groups. The following is an excerpt from claude-opus-5 run 1:
NOT USED (2):
CWL group ExampleStack-Vpc-us-east-1-VpcFlowLogLogGroup1
CWL group ExampleStack-Vpc-us-east-1-VpcFlowLogLogGroup2
The failing answers concluded by stating that no destination other than S3 exists. The following is an excerpt from gpt-5.6-luna run 1:
No separate CloudWatch Logs, Kinesis Data Firehose, or second S3 flow-log
destination is configured.
How the Two Were Found
These two log groups are not configured as the destination of any Flow Log. Since they cannot be traced from ec2:DescribeFlowLogs results, it is necessary to investigate the CloudWatch Logs side independently. From the execution logs of each run, the methods used to investigate log groups were classified.
| Method of investigating log groups | pass | fail |
|---|---|---|
| Listed all without filtering by name | 20 | 5 |
| Filtered by specifying a prefix | 1 | 11 |
| Never queried at all | 1 | 13 |
The 12 cases that filtered by prefix guessed values like /aws/vpc or flow, and got no results. The actual names begin with the stack name, ending in VpcFlowLogLogGroup1. Guessing the correct prefix is difficult.
The 2 exceptions on the passing side found them via CloudFormation. gpt-5.6-sol run 2 used cloudformation list-stack-resources, and gpt-5.6-terra run 2 used describe-stack-resources and get-template to obtain the names of both log groups, then confirmed them by name.
The 5 cases that listed all but still failed either ended without mentioning the two groups despite their presence in the list, or cited unrelated log groups.
Breakdown of the 29 Failed Cases
| Category | Count |
|---|---|
| Stated that no CloudWatch destination exists, or did not mention the two log groups | 23 |
| Named log groups unrelated to this task | 6 |
No answers were overturned upon reviewing the judgments, and the official judgment results are reflected directly in the tallies.
The fact that destinations were empty was not a reason for failure. 42 out of 51 answers mentioned that the destinations had no data, split between 21 pass and 21 fail. The judgment reasons also did not cite emptiness as grounds for failure.
For the 23 cases in the former category, the judgment model wrote "contradicts the actual state." The statement that they were not configured as a destination is itself factual, but many answers stated "there are no log groups for Flow Logs." The log groups do exist — they are simply not configured as Flow Log destinations.
The 6 cases in the latter category involve circumstances specific to the mini environment. The existing AWS account running the mini environment had one log group with a Flow Log-like name that was unrelated to this task. These 6 cases found it and reported it as a confusing destination. The judgment model ruled them as failures because both the count and the names differed from the reference answer. The affected cases are claude-sonnet-5 runs 1 and 3, claude-opus-4.7 run 2, claude-sonnet-4.6 runs 2 and 3, and gpt-5.6-sol run 3.
This noise does not occur with the official procedure, since clean test accounts are used for each scenario.
Credits and Time per Correct Answer
For only the 22 cases judged as correct, credits and time per correct answer were calculated. The table is sorted in descending order of credits.
| Model | Multiplier | Correct count | Credits per correct answer | Time per correct answer |
|---|---|---|---|---|
claude-opus-4.7 |
2.2 | 2/3 | 10.44 | 131 seconds |
claude-opus-5 |
2.2 | 3/3 | 10.00 | 150 seconds |
claude-opus-4.8 |
2.2 | 3/3 | 8.96 | 143 seconds |
claude-opus-4.6 |
2.2 | 3/3 | 6.37 | 105 seconds |
claude-opus-4.5 |
2.2 | 3/3 | 4.39 | 77 seconds |
gpt-5.6-sol |
2.4 | 2/3 | 3.97 | 120 seconds |
claude-sonnet-5 |
1.3 | 1/3 | 2.73 | 136 seconds |
glm-5 |
0.5 | 1/3 | 1.56 | 76 seconds |
gpt-5.6-terra |
1.2 | 1/3 | 1.51 | 72 seconds |
deepseek-3.2 |
0.25 | 2/3 | 0.76 | 200 seconds |
claude-haiku-4.5 |
0.4 | 1/3 | 0.19 | 63 seconds |
The value for gpt-5.6-terra does not reflect the Amazon Bedrock price reduction announced on July 30, 2026. The measurement was conducted before the announcement, and no multiplier revision from the Kiro side has been announced either. If we assume that Terra's 20% price reduction is reflected equally in the multiplier, the credits per correct answer would be approximately 1.21.
The 6 models with zero correct answers were excluded from the table, as a per-answer value cannot be calculated. The credits spent on these 6 models produced not a single correct answer.
Even among the 5 Opus models with the same 2.2 multiplier, credits per correct answer ranged from 4.39 to 10.44 — a 2.4x spread. claude-opus-4.5 achieved correct answers in all 3 runs while having the lowest credits and time of all. The older 4.5 and 4.6 versions were cheaper, while the 3 models from 4.7 onward clustered between 8.96 and 10.44.
Summary
Most models identified the primary findings. What determined pass or fail was whether the model went to look in places that cannot be traced from Flow Log configuration. ec2:DescribeFlowLogs only reveals the single S3-destined entry, and answers that stopped there made up the majority of failures. The two confusing log groups are not linked to any Flow Log, so no matter how carefully one reads the destination settings, they will not appear.
In practice as well, when a user's premise of "multiple destinations appear to exist" is itself mistaken, simply confirming that the configuration is correct does not complete the explanation. If entrusting investigation to an AI agent, it is necessary to instruct it to include resources that do not appear in the configuration within the scope of investigation.
In the previous Step Functions task, all 17 models answered correctly, and I cited the inexpensive claude-haiku-4.5 and gpt-5.6-luna as candidates. Those same two models scored 1/3 and 0/3 respectively this time. When selecting a model, the reliable approach is to prepare tasks and test scenarios that closely resemble your actual workload and measure accordingly.
