I Tried the aws-bench Diagnostic Tasks with Almost Zero AWS Costs, Using Kiro Credits

Introduction

I recently introduced aws-bench, a benchmark for AI agents released by AWS.

https://dev.classmethod.jp/articles/aws-bench-overview/

To run the benchmark by following the official Getting started procedure, you need an AWS Organizations environment that can issue new AWS accounts. In addition, some scenarios use expensive AWS resources, so it was not something you could try casually.

So, instead of aiming to reproduce the official environment, I selected four diagnostic tasks and prepared a mini test environment that runs in a single existing AWS account.

Mini Test Approach

We create intentional anomalies in the AWS environment to be diagnosed, have an agent using a read-only role inside an isolated container investigate it, and produce an answer file.

The answer file is evaluated by a separate Kiro in charge of judging.

Account issuance via Organizations, the official orchestration, the Bedrock judge, drift verification, and resource state verification for tasks that make changes were omitted.

Building the Four Diagnostic Task Environments with CDK

In the official scenarios, CDK stacks and the necessary setup create AWS resources containing intentional anomalies, deploying 30 stacks across 7 regions. This time, I narrowed it down to the resources for four tasks plus a shared read-only role, and deployed 5 stacks directly across 3 regions.

To build the four task environments, I used materials from the official dataset.

https://github.com/aws-bench/aws-bench-datasets

The code that defines the five stacks this time is environment.ts.

// Common to all tasks (read-only role)
new QARolesStack(app, `${envId}-QARoles-us-east-1`, { env: { account, region: 'us-east-1' } });

// Task: check-vpc-flow-log-destinations
new Cloudformation_vesgrw3ay(app, `${envId}-cloudformation-vesgrw3ay-us-east-1`, { env: { account, region: 'us-east-1' } });

// Task: diagnose-step-functions-failing-executions
new Lambda_mw9wjm2q7(app, `${envId}-lambda-mw9wjm2q7-ap-southeast-2`, { env: { account, region: 'ap-southeast-2' } });

// Task: fix-cloudformation-update-failed-stack
new Cloudformation_t9dx4pgqw(app, `${envId}-cloudformation-t9dx4pgqw-us-east-1`, { env: { account, region: 'us-east-1' } });

// Task: lambda-appconfig-stale-value-troubleshoot
new Lambda_1sscp39dx(app, `${envId}-lambda-1sscp39dx-us-west-2`, { env: { account, region: 'us-west-2' } });

The table below summarizes the main resources, the planted anomalies, and the diagnostic content for the four tasks.

The name fix-cloudformation-update-failed-stack contains fix, but this mini test does not have the agent perform recovery operations. Instead, it has the agent write the cause of the failure and the necessary recovery operations in the diagnostic report.

The CDK stack definitions and the logic that creates the anomalous states are adapted from the official materials. To make it work in a single account, I modified the Dockerfile, the deployment procedure, and how the authentication profile is resolved in setup.

The Anomalous State Created by fix-cloudformation-update-failed-stack

Among the four tasks, I'll use fix-cloudformation-update-failed-stack, which creates its anomalous state after deployment, as an example. In CDK, two Lambda functions and aliases pointing to the current versions are created.

const simpleEmailServiceAlias = new lambda.Alias(this, 'SimpleEmailServiceLambdaAlias', {
    aliasName: 'LATEST',
    version: simpleEmailServiceLambda.currentVersion,
});

const getDetectorOutcomeAlias = new lambda.Alias(this, 'GetDetectorOutcomeLambdaAlias', {
    aliasName: 'LATEST',
    version: getDetectorOutcomeLambda.currentVersion,
});

After deployment, the setup script rewrites the original CloudFormation template. It replaces the FunctionVersion of AWS::Lambda::Alias with a nonexistent version number and updates the stack. Because DisableRollback=True is specified, the stack in the UPDATE_FAILED state can still be investigated after the update fails.

for resource in template.get("Resources", {}).values():
    if resource.get("Type") != "AWS::Lambda::Alias":
        continue

    ref = resource["Properties"].get("FunctionName", {}).get("Ref", "")
    resource["Properties"]["FunctionVersion"] = (
        "593" if "SimpleEmailService" in ref else "268"
    )

update_and_wait(cfn, template, disable_rollback=True)

final_status = cfn.describe_stacks(StackName=STACK_NAME)["Stacks"][0]["StackStatus"]
if final_status != "UPDATE_FAILED":
    raise RuntimeError(f"Expected stack to reach UPDATE_FAILED but got {final_status}")

In this task, the agent writes the cause of UPDATE_FAILED and the operations required for recovery in the diagnostic report.

Having Kiro Isolated in a Container Perform the Diagnosis

Initially, I ran Kiro in headless mode with all tools trusted (--trust-all-tools) in the project directory where the official benchmark materials were placed. However, from the execution logs, I found traces of the agent referencing materials used for scoring, such as the correct answer files. In this state, access to locally placed evaluation materials could affect the answers, rather than the ability to investigate the AWS environment.

So, as in the official aws-bench, this mini test also follows a configuration in which the agent is isolated and run in a Docker container. Only Kiro's credentials, the task instructions, read-only AWS credentials, and the output destination for the answer are passed to the agent container. In the examples that follow, the actual ARNs and the original text of the reference answers are withheld. The agent is not given access to scoring materials such as the correct answers and rubrics.

The shared QARoles stack creates a read-only role for the agent's AWS investigation. Kiro receives the instructions for each task, uses this role to look at AWS configurations, logs, and execution history, and writes a diagnostic report.

For the agent container, I use a Python slim image containing AWS CLI v2 and Kiro CLI, and run Kiro headless inside it. The benchmark materials on the host are not mounted. Only the execution environment and the necessary inputs are placed on the container side, and judging is done on the host side.

For example, in the Step Functions task, the agent follows the execution history and Lambda error logs using read-only AWS operations, and identifies the absence of targetBucket in the input passed to Lambda as the cause.

Handing the Step Functions Task to Kiro in the Isolated Container

The runner replaces the placeholders in the task instructions with identifiers within the single-account environment, and then runs Kiro in headless mode inside the container. The following instruction is given to the agent.

My Step Functions state machine <resolved StateMachine ARN>
in ap-southeast-2 has failing executions. Why?

When the container starts, only the authentication volume and the temporary per-task output destination are mounted. The temporary AWS credentials are passed via environment variables.

docker run --rm \
  -v <kiro-auth-volume>:/root/.local/share/kiro-cli \
  -v <per-task-output-directory>:/logs/agent \
  -e TASK_INSTRUCTION="<resolved instruction>" \
  <isolation-image> \
  'kiro-cli chat \
    --model <model-id> \
    --trust-all-tools \
    --no-interactive \
    "$TASK_INSTRUCTION

Benchmark safety rule: investigate with read-only AWS operations only.
Do NOT create, update, delete, deploy, invoke, or otherwise modify any AWS resource.

Write only your final answer to:
/logs/agent/agent-output.txt"'

The only things given to the agent as task instructions are this problem statement and constraints. The host's benchmark materials, the reference answer ground_truth.json, and the official rubric judge_prompt.md are not passed to the container. The reference answer and the rubric are read only by the downstream Kiro that judges the agent's answer.

Using the Official Scoring Materials with a Local Kiro

The official dataset includes the task instructions, the reference answer, and the rubric (judge_prompt.md). The placeholders contained in the task instructions and the reference answer are replaced with identifiers within the single-account environment before use.

Scoring the Step Functions Task

In this Step Functions task, the scoring-side Kiro judges the agent's diagnostic result agent-output.txt. During judging, it refers to the resolved reference answer and the official rubric. The reference answer includes the task instructions and the expected diagnostic content.

The original text of the official rubric can be found in the following file on GitHub.

https://github.com/aws-bench/aws-bench-datasets/blob/2daf77d2d41c21bae00bf8227fc463be51f721d0/tasks/troubleshooting-multiservice/diagnose-step-functions-failing-executions/tests/judge_prompt.md

In this example, we judge the answer to the Step Functions task instruction shown in the previous section. The reference answer expects that the input passed to InvokeProcessingLambda lacks targetBucket, causing the Lambda to fail.

The scoring-side Kiro is made to read agent-output.txt, ground_truth-resolved.json, and judge_prompt.md. These are, respectively, the agent's diagnostic result, the resolved reference answer, and the official rubric.

It checks whether the answer matches the cause, failure state, and target resource shown in the reference answer. Differences in wording and additional correct information are tolerated.

On the other hand, if the cause differs or the main cause is missing, the answer fails. For tasks that ask for a recovery measure, an answer that presents an infeasible recovery measure also fails.

Each judgment is saved in the following JSON format.

{
  "criterion": "answers_equivalent",
  "result": "<pass|fail>",
  "reason": "<brief explanation>",
  "task": "diagnose-step-functions-failing-executions"
}

Cost Factors

The test environment uses VPC Flow Logs, CloudWatch Logs, Amazon S3, and so on, but the AWS usage charges are nearly zero. Running the AI agent uses Kiro credits, which, within the range of bundled credits, can be used at $0.02 per credit.

Summary

Even when it is hard to prepare AWS Organizations or expensive resources, a small single-account environment can keep AWS usage charges nearly at zero. You can now try a flow in which an agent performs diagnosis using Kiro credits and the answer is judged by a local Kiro. I plan to introduce the execution results in a separate article.

Reference Links

• aws-bench mini tasks local runner