I tried the aws-bench diagnostic task with almost zero AWS charges, using Kiro credits

Introduction

The other day, I introduced aws-bench, an AI agent benchmark published by AWS.

https://dev.classmethod.jp/articles/aws-bench-overview/

Running the benchmark using the official Getting started procedure requires an AWS Organizations environment capable of provisioning new AWS accounts. Additionally, some scenarios use expensive AWS resources, making it difficult to try casually.

Therefore, rather than aiming to replicate the official environment, I selected four diagnostic tasks and prepared a mini test environment that runs on a single existing AWS account.

Mini Test Policy

I intentionally create anomalies in the target AWS environment, have an agent using a read-only role inside an isolated container investigate them, and generate an answer file.

The answer file is evaluated by a separate Kiro acting as the judge.

I omitted account provisioning via Organizations, official orchestration, Bedrock judge, drift validation, and resource state validation for mutating tasks.

Building Four Diagnostic Task Environments with CDK

In the official scenarios, CDK stacks and the required setup create AWS resources containing intentional anomalies, deploying 30 stacks across 7 regions. This time, I narrowed it down to resources for four tasks plus a shared read-only role, deploying 5 stacks directly across 3 regions.

I used materials from the official dataset to build the four task environments.

https://github.com/aws-bench/aws-bench-datasets

The code defining the 5 stacks for this time is environment.ts.

// Common to all tasks (read-only role)
new QARolesStack(app, `${envId}-QARoles-us-east-1`, { env: { account, region: 'us-east-1' } });

// Task: check-vpc-flow-log-destinations
new Cloudformation_vesgrw3ay(app, `${envId}-cloudformation-vesgrw3ay-us-east-1`, { env: { account, region: 'us-east-1' } });

// Task: diagnose-step-functions-failing-executions
new Lambda_mw9wjm2q7(app, `${envId}-lambda-mw9wjm2q7-ap-southeast-2`, { env: { account, region: 'ap-southeast-2' } });

// Task: fix-cloudformation-update-failed-stack
new Cloudformation_t9dx4pgqw(app, `${envId}-cloudformation-t9dx4pgqw-us-east-1`, { env: { account, region: 'us-east-1' } });

// Task: lambda-appconfig-stale-value-troubleshoot
new Lambda_1sscp39dx(app, `${envId}-lambda-1sscp39dx-us-west-2`, { env: { account, region: 'us-west-2' } });

The following table summarizes the main resources, injected anomalies, and diagnostic content for the four tasks.

Although the name fix-cloudformation-update-failed-stack contains fix, this mini test does not execute recovery operations. Instead, the failure cause and necessary recovery operations are recorded in the diagnostic report.

The CDK stack definitions and the logic for creating anomalous states are repurposed from the official materials. I modified the Dockerfile, deployment procedures, and the method for resolving authentication profiles in setup to work with a single account.

Anomalous State Created by fix-cloudformation-update-failed-stack

Among the four tasks, I'll use fix-cloudformation-update-failed-stack, which creates an anomalous state after deployment, as an example. In CDK, two Lambda functions and aliases pointing to the current versions are created.

const simpleEmailServiceAlias = new lambda.Alias(this, 'SimpleEmailServiceLambdaAlias', {
    aliasName: 'LATEST',
    version: simpleEmailServiceLambda.currentVersion,
});

const getDetectorOutcomeAlias = new lambda.Alias(this, 'GetDetectorOutcomeLambdaAlias', {
    aliasName: 'LATEST',
    version: getDetectorOutcomeLambda.currentVersion,
});

After deployment, the setup script rewrites the original CloudFormation template. It replaces the FunctionVersion in AWS::Lambda::Alias with a non-existent version number and updates the stack. Since DisableRollback=True is specified, the stack in UPDATE_FAILED state can be investigated even after the update fails.

for resource in template.get("Resources", {}).values():
    if resource.get("Type") != "AWS::Lambda::Alias":
        continue

    ref = resource["Properties"].get("FunctionName", {}).get("Ref", "")
    resource["Properties"]["FunctionVersion"] = (
        "593" if "SimpleEmailService" in ref else "268"
    )

update_and_wait(cfn, template, disable_rollback=True)

final_status = cfn.describe_stacks(StackName=STACK_NAME)["Stacks"][0]["StackStatus"]
if final_status != "UPDATE_FAILED":
    raise RuntimeError(f"Expected stack to reach UPDATE_FAILED but got {final_status}")

In this task, the agent records the cause of UPDATE_FAILED and the operations required for recovery in the diagnostic report.

Having Kiro Isolated in a Container Perform the Diagnosis

Initially, I was running Kiro in headless mode with all tools trusted (--trust-all-tools) in the project directory where the official benchmark materials were stored. However, from the execution logs, I found evidence that the agent had referenced materials used for scoring, such as the correct answer files. In this state, access to locally stored evaluation materials—rather than the ability to investigate the AWS environment—could influence the answers.

Therefore, following the same approach as the official aws-bench, I adopted a configuration where the agent is isolated in a Docker container for execution in this mini test as well. Only Kiro's credentials, task instructions, read-only AWS credentials, and the answer output destination are passed to the agent container. In the examples below, actual ARNs and the original text of reference answers are omitted. Access to scoring materials such as correct answers and rubrics is not granted.

The shared QARoles stack creates a read-only role for the agent's AWS investigation. Kiro receives instructions for each task, uses this role to reference AWS configurations, logs, and execution histories, and writes a diagnostic report.

The container for the agent uses a Python slim image containing AWS CLI v2 and Kiro CLI, within which Kiro runs in headless mode. Benchmark materials on the host are not mounted. Only the execution environment and necessary inputs are placed on the container side, and judgment is performed on the host side.

For example, in the Step Functions task, the agent traces the execution history and Lambda error logs using read-only AWS operations and identifies the missing targetBucket in the input passed to Lambda as the cause.

Passing the Step Functions Task to Kiro in an Isolated Container

The runner replaces placeholders in the task instructions with identifiers within the single-account environment, then runs Kiro in headless mode inside the container. The following instructions are passed to the agent.

My Step Functions state machine <resolved StateMachine ARN>
in ap-southeast-2 has failing executions. Why?

When starting the container, only the authentication volume and the per-task temporary output destination are mounted. Temporary AWS credentials are passed via environment variables.

docker run --rm \
  -v <kiro-auth-volume>:/root/.local/share/kiro-cli \
  -v <per-task-output-directory>:/logs/agent \
  -e TASK_INSTRUCTION="<resolved instruction>" \
  <isolation-image> \
  'kiro-cli chat \
    --model <model-id> \
    --trust-all-tools \
    --no-interactive \
    "$TASK_INSTRUCTION

Benchmark safety rule: investigate with read-only AWS operations only.
Do NOT create, update, delete, deploy, invoke, or otherwise modify any AWS resource.

Write only your final answer to:
/logs/agent/agent-output.txt"'

Only this problem statement and constraints are passed to the agent as task instructions. The host's benchmark materials, reference answer ground_truth.json, and official rubric judge_prompt.md are not passed to the container. The reference answer and rubric are read only by the subsequent Kiro that judges the agent's answer.

Using Official Scoring Materials with Local Kiro

The official dataset includes task instructions, reference answers, and rubrics (judge_prompt.md). The placeholders contained in the task instructions and reference answers are replaced with identifiers within the single-account environment before use.

Scoring the Step Functions Task

In this Step Functions task, the scoring Kiro judges the agent's diagnostic result agent-output.txt. During judgment, it references the resolved reference answer and the official rubric. The reference answer includes the task instructions and expected diagnostic content.

The original text of the official rubric can be found in the following file on GitHub.

https://github.com/aws-bench/aws-bench-datasets/blob/2daf77d2d41c21bae00bf8227fc463be51f721d0/tasks/troubleshooting-multiservice/diagnose-step-functions-failing-executions/tests/judge_prompt.md

In this example, the answer to the Step Functions task instruction shown in the previous section is judged. The reference answer expects that the input passed to InvokeProcessingLambda is missing targetBucket, causing Lambda to fail.

The scoring Kiro reads agent-output.txt, ground_truth-resolved.json, and judge_prompt.md—the agent's diagnostic result, the resolved reference answer, and the official rubric, respectively.

It verifies whether the cause, failure state, and target resources match those indicated by the reference answer. Differences in expression and additional correct information are acceptable.

On the other hand, if the cause differs or the main cause is missing, the result is a failure. For tasks that require recovery steps, providing infeasible recovery steps also results in a failure.

Each judgment is saved in the following JSON format.

{
  "criterion": "answers_equivalent",
  "result": "<pass|fail>",
  "reason": "<brief explanation>",
  "task": "diagnose-step-functions-failing-executions"
}

Cost Factors

The test environment uses VPC Flow Logs, CloudWatch Logs, Amazon S3, and other services, but AWS charges are nearly zero. Running the AI agent uses Kiro credits, and within the included credits, it can be used at $0.02 per credit.

Summary

Even when it is difficult to prepare AWS Organizations or expensive resources, a small-scale single-account environment can keep AWS charges nearly zero. It is now possible to try a workflow where Kiro is used to diagnose with agent credits and the answers are judged by a local Kiro. The execution results will be introduced in a separate article.

Reference Links

• aws-bench mini tasks local runner