I Checked Out the Fidelity Scores and Caveats of AWS Emulators, and Skills for Agents
This page has been translated by machine translation. View original
Hello. I'm Takeda from the Service Development Department.
On November 11, 2026, "AWS Emulator," an official AWS local emulator, was released. Just by changing the destination of the AWS CLI or AWS SDK to a local endpoint, you can try creating and operating resources without an AWS account.
The startup steps and basic operations for S3, DynamoDB, and so on are summarized in the following article.
This time, I ran it on my local macOS and checked the advantages unique to an official AWS tool and the current caveats. I also tried the agent skill that was released alongside it.
Test environment
- macOS (Apple silicon), Rancher Desktop (Docker Engine 29.5.3)
- AWS CLI 2.36.30
- Claude Code (
aws-emulatorskill v1) - AWS Emulator 1.0.322
- Tested on November 12, 2026
With Rancher Desktop, start it with --user
When I ran the docker run from the user guide as is on Rancher Desktop, it exited with the following error.
emulator serve failed: server I/O error: Operation not permitted (os error 1)
This is because the user inside the container (UID 1000) does not have write permission to the mounted directory. Adding --user, which the user guide describes as being for Linux, lets it start.
mkdir -p ~/.aws/emulator
PORT=9297
docker run --rm --memory=2g --memory-swap=2g -p 127.0.0.1:$PORT:$PORT \
--user "$(id -u):$(id -g)" \
-e HOME="$HOME" -v "$HOME/.aws/emulator:$HOME/.aws/emulator" \
aws-emulator -p $PORT
I have not tried it with Docker Desktop.
Advantages of being provided officially
Compared with third-party emulators, the fact that AWS itself builds it is directly an advantage.
Fidelity tests and scores are public
How closely it behaves like AWS is measured by fidelity tests. AWS publishes tests verified against real AWS in aws-emulator-evaluation-suite. Tests that do not pass yet are also included. The results for each release are attached to the GitHub release as scores.json.gz. I aggregated the results of 1.0.322 by service.
| Service | Passed / Number of tests |
|---|---|
| Amazon S3 | 763 / 924 |
| Amazon DynamoDB | 661 / 808 |
| Amazon SQS | 286 / 304 |
| AWS IAM | 359 / 463 |
| AWS CloudFormation | 354 / 451 |
| AWS Lambda | 293 / 616 |
I aggregated tests whose kind is behavioral, counting by the service name at the beginning of the test ID. Skipped tests and tests that span multiple services are excluded. You can check how far the APIs you want to use are reproduced before running anything.
The fidelity tests even check the error codes and messages for invalid calls. When I tried the too-short table name that the user guide gives as an example, the following ValidationException was returned. Real AWS returns the same message.
An error occurred (ValidationException) when calling the CreateTable operation: 1 validation error detected: Value 'ab' at 'tableName' failed to satisfy constraint: Member must have length greater than or equal to 3
Unimplemented APIs fail with 501
Unimplemented operations return NotImplemented with status code 501. A test failure immediately tells you that you are calling an unsupported API.
$ aws athena list-work-groups
An error occurred (NotImplemented) when calling the ListWorkGroups operation: operation ListWorkGroups is scaffolded but not implemented in this emulator
Authentication that is hard to confuse with real AWS
The emulator generates access keys at every startup and validates SigV4 signatures. Access key IDs start with EMULATOR, and requests signed with any other key are rejected.
An error occurred (AccessDenied) when calling the ListBuckets operation: Sign the request with the emulator credentials
When I started two emulators on different ports, their keys and state were separate. Calling 9300 with the key for 9297 is rejected with the same error. You can assign one to each parallel CI job or each agent.
Keeping up with new APIs
Each service is generated from the same API model as the AWS SDK, so when services or APIs are added to AWS, they also appear in the emulator. Behavior is implemented over successive releases.
According to the AWS News Blog, agents are responsible for the implementation and maintenance. They are split into agents that find differences from real AWS and the API models and file them as issues, and agents that pick up the issues to implement and review them. The repository does not accept PRs, and users report through issues.
Skill for coding agents
Alongside AWS Emulator, an aws-emulator skill for coding agents has been published. Running the following command with AWS CLI 2.35.0 or later places SKILL.md and 8 reference documents in ~/.claude/skills/aws-emulator/. If you omit --agent, all detected agents are targeted.
aws agent-toolkit add-skill --skill-name aws-emulator --region us-east-1 --agent claude-code
I prepared a small sample app that calls DynamoDB, S3, SQS, and Athena with boto3, and gave Claude Code two requests identical to the example prompts in the user guide.
Delegating the setup
The first request was "Install AWS Emulator and point my project at it." When I asked, it finished everything from building the image to verifying the connection in about four and a half minutes. The environment variables for connecting are output to ~/.aws/emulator/9297.env. You load and use them in a subshell, like ( . ~/.aws/emulator/9297.env && aws s3 ls ).
On Rancher Desktop, the docker run that the skill instructs also failed with the same error as before. Because it uses -d --rm, neither the container nor its logs remain. This error is not listed in the skill's troubleshooting. The agent removed --rm and restarted it, investigated the cause from the logs, and added --user.
Delegating the compatibility check
The second request was "Will my app in this repo work on AWS Emulator?" The agent first identifies the AWS calls from the application code. Next it reads the source of the fidelity tests and selects the tests that directly verify each call. Finally it cross-checks them against the release's score report. It does not call the emulator or AWS. It took about 12 minutes.
The verdict was WILL NOT WORK, with the following breakdown.
| Item | Result |
|---|---|
| DynamoDB, S3, and SQS calls (12 items) | VERIFIED |
athena:StartQueryExecution |
VERIFIED |
athena:ListWorkGroups |
FAILING |
| Assumption that region and credentials are received from the environment (2 items) | UNVERIFIED |
Afterward I ran the sample app against the emulator. The tests going through DynamoDB, S3, and SQS succeeded, and the Athena report failed with a 501 on ListWorkGroups. For the range that could be run, this matches the per-item results. StartQueryExecution stopped before that point, so I have not confirmed it.
The verdict is binary, and a single item without backing makes it WILL NOT WORK. To see which APIs are problematic, it seems best to look at the per-item results. On macOS, installing GNU grep (ggrep) is a prerequisite.
Current caveats
IAM policy evaluation does not match AWS
The user guide explicitly says not to use it for security testing. I created an IAM user with no policies attached inside the emulator and tried operations with its access key.
S3 GetObject and PutObject, DynamoDB Scan and PutItem, SQS SendMessage, and IAM ListUsers were denied. On the other hand, S3 CreateBucket and DeleteBucket, DynamoDB ListTables, SNS CreateTopic, and SSM PutParameter succeeded. These are all operations that would be denied on AWS.
The user guide says S3 bucket policies are not checked. In 1.0.322, when I attached a bucket policy that denies s3:ListBucket, even the emulator's own key got AccessDenied. Whether policies are evaluated differs by operation. Check permissions on real AWS.
Use one emulator for one account and one region
There is only one account, 000000000000, and cross-account behavior is not reproduced. Any value is accepted for the region. A DynamoDB table created in us-east-1 was not visible when I specified ap-northeast-1. However, cross-region behavior is unsupported, and whether things are separated per region also depends on the service. When trying multiple regions, start an emulator for each region.
State and credentials disappear when stopped
There is no persistence option. On restart, buckets and tables disappear, and the access keys and certificates are regenerated. Create the resources used in tests every time. According to the user guide, long-running operation and heavy-load usage have not been tested.
Fidelity scores vary by service
According to the user guide, what is currently close to AWS is the control plane APIs that create, retrieve, and delete resources. Support for the data plane and IaC tools varies by service. With the same aggregation as before, EC2 was 576 of 1,299 and Athena was 50 of 306. Tests spanning multiple services were 2,667 of 6,380.
Bedrock's converse can be called, but the model does not run. A fixed message like the following is returned.
No language model runs in this emulator. anthropic.claude-3-haiku-20240307-v1:0 generated no text for this request. The request carried 1 message(s) and 1 input token(s).
Telemetry is on by default
Usage and error data are sent to AWS. Request and response contents and credentials are said not to be included. To disable it, pass -t off to aws-emulator or set the environment variable EMULATOR_TELEMETRY=off. With Docker, add -e EMULATOR_TELEMETRY=off to docker run.
Summary
With AWS Emulator, you can check how far each API is reproduced through tests and scores. It looks usable for finding API call mistakes before deployment and as an environment where coding agents can iterate through trial and error. Permission checks and tests that depend on the data plane should still be done on real AWS.
There were 12 releases between November 7 and 11 alone. Security fixes go only into the latest version, so don't pin the version, and review the scores for each release.
I hope this is helpful to someone.
