I tried out the rate limiting feature added to AgentCore Gateway
This page has been translated by machine translation. View original
Introduction
Hello, I'm Kamino from the Consulting Division, a huge fan of cheese naan.
On August 6, 2026, new features including Rate Limiting were added to Amazon Bedrock AgentCore Gateway!
It feels like a feature truly worthy of the Gateway name has been added! Let's give it a try right away!
Prerequisites
The environment used for verification is as follows.
| Item | Details |
|---|---|
| Region | ap-northeast-1 (Tokyo) |
| boto3 / botocore | 1.43.67 |
| Python / Package Manager | 3.12 / uv |
| Gateway | AgentCore Gateway (MCP, AWS_IAM authentication) |
| Target | Inference Target (Bedrock Mantle connector) |
First, let me briefly introduce what the rate limiting feature does.
Rate Limiting
This is a feature that groups traffic passing through the Gateway by units called "dimensions" and allows you to set an allowed rate for each group.
Here is a rough overview of its characteristics.
- The grouping method is determined by dimension keys.
- In addition to target name (targetName) and model ID (qualifiedModelId), you can also specify JWT claims like $.context.jwt.sub to split by calling user.
- Rates for each group are determined by entries.
- In addition to the number of requests, you can also limit token count and concurrent connections for inference targets.
- Setting the rate to 0 allows you to block specific callers.
- When multiple rate limits are configured, they are evaluated with AND logic, and only requests that pass all of them are executed.
The intended use cases seem to be backend protection, per-user quotas, and inference cost caps (TPM control). Indeed, it seems quite useful for inference targets connected to LLMs!
The official documentation is here.
Now that we have an overview of what it is, let's try it out!
Preparing the Verification Gateway
For verification, I'll create a Gateway with one Inference Target using the Bedrock Mantle connector. This is a feature that allows the Gateway to serve as an entry point for LLM APIs, and it's introduced in detail in the article below.
Creating the IAM Role
First, create the service role for the Gateway. Allow bedrock-agentcore.amazonaws.com in the trust policy, and attach Bedrock invocation and Mantle permissions to the permission policy.
# Trust policy
aws iam create-role \
--role-name rate-limit-demo-gateway-role \
--assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "bedrock-agentcore.amazonaws.com"},
"Action": "sts:AssumeRole",
"Condition": {"StringEquals": {"aws:SourceAccount": "<ACCOUNT_ID>"}}
}]
}'
# Permission policy
aws iam put-role-policy \
--role-name rate-limit-demo-gateway-role \
--policy-name inference-permissions \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "bedrock-mantle:*",
"Resource": "arn:aws:bedrock-mantle:ap-northeast-1:<ACCOUNT_ID>:project/*"
}
]
}'
Creating the Gateway and Inference Target
Next, create the Gateway and the inference target.
import boto3
client = boto3.client("bedrock-agentcore-control", region_name="ap-northeast-1")
# Create Gateway
gateway = client.create_gateway(
name="rate-limit-demo-gateway",
roleArn="arn:aws:iam::<ACCOUNT_ID>:role/rate-limit-demo-gateway-role",
protocolType="MCP",
authorizerType="AWS_IAM",
)
print(gateway["gatewayId"], gateway["gatewayUrl"])
# Register Bedrock Mantle connector inference target
client.create_gateway_target(
gatewayIdentifier=gateway["gatewayId"],
name="bedrock-mantle",
credentialProviderConfigurations=[
{"credentialProviderType": "GATEWAY_IAM_ROLE"}
],
targetConfiguration={
"inference": {"connector": {"source": {"connectorId": "bedrock-mantle"}}}
},
)
Run boto3 with the latest version specified using uv.
uv run --with 'boto3>=1.43.67' setup.py
Setting a TPM Limit
Let's try limiting token consumption to 300 tokens per minute using the per-model dimension (qualifiedModelId).
import boto3
client = boto3.client("bedrock-agentcore-control", region_name="ap-northeast-1")
response = client.create_gateway_rate_limit(
gatewayIdentifier="rate-limit-demo-gateway-xxxx",
dimensionKeys=["qualifiedModelId"],
description="Per-model TPM limit for demo",
entries=[
{
"dimensions": {"qualifiedModelId": "*"},
"tokens": [{"rate": 300, "period": "minute"}],
}
],
)
print(response["rateLimitId"])
uv run --with 'boto3>=1.43.67' create_rate_limit.py
This time, we're specifying the token count.
Since we're using a wildcard (*), this becomes a limit common to all models, but by specifying a model ID explicitly, you can differentiate settings like "tighten Opus but loosen Haiku."
It takes a little time to take effect, so wait a few minutes before testing.
Triggering HTTP Status Code 429
With the limit in effect, let's call Claude Haiku 4.5 via the Gateway's /inference/v1/messages endpoint. Firing off requests that consume approximately 210 tokens each (21 input + 191 output) in rapid succession...
--- call 1: HTTP 200 (Success)
--- call 2: HTTP 200 (Success)
--- call 3: HTTP 200 (Success)
--- call 4: HTTP 429
{"type":"error","error":{"type":"rate_limit_error","message":"Token rate limit exceeded",
"limitKey":"0bqftbvw7w","metric":"tokens","retryAfter":0.2}}
It stopped with HTTP 429!
The response tells us which metric was exceeded (tokens), which rate limit was hit (limitKey), and how many seconds to wait (retryAfter). Since the error format matches the Anthropic Messages API standard rate_limit_error, the built-in retry mechanisms of the Anthropic SDK and OpenAI SDK can respond to it as-is.
It appears that the 300 tokens/minute setting operates as a token bucket that replenishes 5 tokens per second, and even right after a 429, the next request succeeded after a short wait.
Since it doesn't behave as a strict window of exactly 300 per minute, it may be better to think of it as throttling rather than strict counting.
requests and tokens Can Be Used Together
You can specify both token count (tokens) and request count (requests) in a single entry simultaneously.
In this case, they are evaluated with AND logic, and a 429 is returned as soon as either limit is reached. If you want to cap both the number of calls with RPM and the cost with TPM, you can express it like this.
entries=[
{
"dimensions": {"qualifiedModelId": "*"},
"requests": [{"rate": 2, "period": "minute"}],
"tokens": [{"rate": 300, "period": "minute"}],
}
]
Let's test this too. This time, firing small requests that consume almost no tokens (approximately 32 tokens per call) in rapid succession to hit the RPM before reaching the TPM...
--- call 6: HTTP 200 (Success)
--- call 7: HTTP 429
{"type":"error","error":{"type":"rate_limit_error","message":"Rate limit exceeded",
"limitKey":"0bqftbvw7w","metric":"requests","retryAfter":30.0}}
This time the metric is requests!
The total token consumption is around 200, which hasn't reached the TPM (300), so the request count limit triggered first. Even with the same rate limit, you can tell from the error which limit was hit. The retryAfter: 30.0 also seems consistent with the behavior of 2 calls/minute = replenishment every 30 seconds.
Setting Different Limits per Model
Since multiple entries can be written, you can set different limits for each model.
As a test, I set only Haiku to rate 0 (= complete block) while leaving all other models at 100 calls/minute via wildcard.
entries=[
{
"dimensions": {"qualifiedModelId": "anthropic.claude-haiku-4-5"},
"requests": [{"rate": 0, "period": "minute"}],
},
{
"dimensions": {"qualifiedModelId": "*"},
"requests": [{"rate": 100, "period": "minute"}],
},
]
haiku : HTTP 429 {"metric":"requests","retryAfter":60.0}
deepseek : HTTP 200
deepseek : HTTP 200
deepseek : HTTP 200
While Haiku is blocked from the very first call, DeepSeek successfully processed requests at the same time without any issues!
This seems useful for differentiating settings like strictly limiting expensive models only, or temporarily stopping a specific model!
MCP Targets Can Be Limited the Same Way
This time I tested with an inference target, but you can apply the same mechanism to MCP targets that expose tools like Lambda. Use targetName as the dimension and limit the number of requests with requests.
response = client.create_gateway_rate_limit(
gatewayIdentifier="my-gateway-xxxx",
dimensionKeys=["targetName"],
description="Per-target RPM limit",
entries=[
{
"dimensions": {"targetName": "SupportTarget"},
"requests": [{"rate": 3, "period": "minute"}],
},
{
"dimensions": {"targetName": "*"},
"requests": [{"rate": 100, "period": "minute"}],
},
],
)
The differences from inference targets are that tokens cannot be used (since there's no concept of tokens) and that targetName is used as the dimension. Incidentally, specifying toolName allows you to narrow it down to the tool level.
When I actually limited a Lambda mock tool to 3 calls/minute and fired requests in rapid succession, I confirmed that HTTP 429 was returned in the same way!
I get the impression it's convenient when you want to control the rate limit of tool executions from the outside!
Points to Note
Here are some points I personally found worth noting, summarized as bullet points.
- Rate limiting is fail-open by default.
- If there is a problem on the limiting service side or if the dimension cannot be resolved, the request will succeed. It seems the correct positioning is to use it as traffic flow control rather than as a security boundary.
- Requests may momentarily exceed the configured rate.
- For token limits, the input token count is estimated before execution, and the actual consumption is recorded after the response is returned. During the brief period until the response is fully returned, the consumption is not yet finalized, so rapid-fire requests can temporarily exceed the limit. This is likely why up to the 4th call (approximately 840 tokens worth) went through in our verification with a 300 tokens/minute setting.
- For use cases where you cannot afford even 1 token of overage (such as confirmed billing control), please use it in combination with a backend-side check.
- The same thing happens with request counts. When firing requests in rapid succession from a state where no calls had been made for a while against a 2 calls/minute limit, 14 consecutive calls went through. Even with a low rate setting, the accumulated tokens in the bucket can all be consumed at once, so it is recommended to observe for at least a few minutes before making a judgment.
- For token limits, the input token count is estimated before execution, and the actual consumption is recorded after the response is returned. During the brief period until the response is fully returned, the consumption is not yet finalized, so rapid-fire requests can temporarily exceed the limit. This is likely why up to the 4th call (approximately 840 tokens worth) went through in our verification with a 300 tokens/minute setting.
- Gateways already have limits managed by AWS (service quotas) by default.
- Rate limiting is a feature to further restrict within those limits, so setting a value larger than the quota has no effect. What is actually applied is the smaller of the value you set and the service quota.
- Rate limiting is executed before the policy evaluation of AgentCore Policy.
- Requests that hit the limit are rejected with 429 before proceeding to policy evaluation, so please use this as a reference when troubleshooting when using both Policy and rate limiting together.
The fail-open behavior and the mechanism for recording token limit consumption are described in the best practices documentation, and the service quota values are in the quota documentation.
Additionally, the official blog introduces a more advanced design example of segmenting users by JWT claims (Basic / Advanced / Beta) and setting different limits for each tier. This is a useful reference when considering full-fledged design for multi-tenant scenarios.
Conclusion
Rate limiting has been implemented and can now be achieved with just the gateway configuration! It's really starting to feel more and more like a true gateway...!!!
The per-model TPM limiting I tested this time I believe is a useful feature for setting an upper limit on LLM costs, and by using JWT claim-based dimensions, it can also be applied to per-user quotas in multi-tenant SaaS.
I'm also currently verifying the Temporal Policies announced at the same time, so I'll write a separate article on that soon. The Gateway is increasingly becoming a place where agent behavior is governed with greater depth!
I hope this article was helpful in some way.
Thank you for reading all the way to the end!
