[Update] I tried out AgentCore Gateway's newly supported Bedrock InvokeModel API and model fallback

Introduction

Hello, I'm Kamino from the Consulting Department, and I have terrible back pain.
I feel like it's gradually getting better when I stretch every day.

By the way, Amazon Bedrock AgentCore Gateway's inference targets now support Bedrock's InvokeModel and InvokeModelWithResponseStream APIs! In addition, Gateway rules now include a condition for routing by model name (matchModels) and fallback with a circuit breaker.

https://aws.amazon.com/jp/about-aws/whats-new/2026/10/agentcore-gateway-invokemodel-fallback/

Since Gateway's Bedrock connector only supported Mantle before, this is great news!
Let's try both features including fallback this time!

Update Details

Bedrock Runtime Connector Support

Inference targets are a feature that places LLM providers such as Bedrock and OpenAI behind the Gateway, allowing you to call various LLMs from a single entry point. When I tried it in a previous article, only Bedrock Mantle was supported for calling Bedrock models.

https://dev.classmethod.jp/articles/agentcore-gateway-inference-target/

Starting now, Bedrock Runtime's invoke_model and others are also supported!

Improved Failure Handling

The other update is about failure handling. Previously, when the primary provider returned a 503 or became unresponsive, the error was passed directly back to the calling application, so if you wanted to retry with a different model, you had to write retry and switching logic in each application.

With the fallback feature added in this update, you only need to write the order and conditions for switching in the Gateway rules! When the primary model fails, the Gateway automatically reroutes to the secondary target, so the application doesn't need to be aware of this at all. Note that you need to choose a model that can be called with the same request body format.

Incidentally, the Gateway also has a circuit breaker that temporarily skips primaries with repeated failures, which is also helpful as it eliminates the need to manage state yourself.

Let's take a closer look at the update details.

Added Settings

InvokeModel API

Paths in the same format as Bedrock have been added under Gateway's /inference.

The calling methods and conversion table are summarized in the documentation below.

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-target-inference-invokemodel.html

A bedrock-runtime connector has also been added. Using this as a target allows you to call Bedrock Runtime models. Since it retrieves the model list from Bedrock, the Gateway execution role needs bedrock:List* permissions in addition to bedrock:InvokeModel and others.

As of October 25, 2026, Bedrock Runtime was not yet displayed in the connector options when adding an inference target in the console...

Therefore, you create it using the SDK or CLI. With boto3, you can simply specify bedrock-runtime as the connector ID.

import boto3

client = boto3.client("bedrock-agentcore-control", region_name="us-east-1")
client.create_gateway_target(
    gatewayIdentifier="<GATEWAY_ID>",
    name="bedrock-runtime",
    credentialProviderConfigurations=[{"credentialProviderType": "GATEWAY_IAM_ROLE"}],
    targetConfiguration={"inference": {"connector": {"source": {"connectorId": "bedrock-runtime"}}}},
)

Rule Conditions and Actions

Gateway rules are a mechanism that applies actions to requests matching conditions.

Explanations of model-based routing and fallback are in the documentation below, and detailed specifications for each item are in the API reference.

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-rules-fallback.html

https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/API_CreateGatewayRule.html

Condition for Routing by Model Name

matchModels has been added to conditions, allowing rules to be applied based on the model name in the request. The comparison method can be chosen between StringEquals for exact match or StringLike (the default) which supports * and ?. Up to 100 entries can be listed in a single condition, and the action is executed if any one of them matches.

conditions=[{
    "matchModels": {
        "anyOf": [
            {"value": "primary-chat", "operator": "StringEquals"},  # exact match
            {"value": "team-*"},  # omitting operator defaults to StringLike
        ]
    }
}]

Replacing the Destination Model

model has been added to staticRoute (and each destination in weightedRoute). When specified, it replaces the model name in the request with this model ID before sending to the target. If omitted, the model name from the request is used as-is.

actions=[{
    "routeToTarget": {
        "staticRoute": {
            "targetName": "bedrock-mantle",
            "model": "openai.gpt-oss-120b",  # Send with this model ID replacing the original
        }
    }
}]

Fallback

Similarly, fallback has been added to each destination in staticRoute and weightedRoute.
The specific configuration items for fallback are as follows.

In triggerOn, you specify which HTTP status codes trigger switching, either as individual values like 429 or ranges like 5XX. triggerOn and timeout can also be written per fallback target.

"staticRoute": {
    "targetName": "primary",
    "fallback": {
        "triggerOn": {
            "statusCodes": ["429", "5XX"],
            "errors": ["CONNECTION_ERROR", "FIRST_BYTE_TIMEOUT", "UNROUTABLE"],
        },
        "firstByteTimeoutMilliseconds": 3000,
        "circuitBreaker": {"mode": "ENABLED"},
        "targetFallback": [
            {"targetName": "bedrock-mantle", "model": "openai.gpt-oss-120b"},
        ],
    },
}

The documentation also noted the following points.

• No switching after streaming has started returning responses
• Gateway invocation charges apply per attempt. A request that fails on the primary and succeeds on the secondary counts as 2 invocations
• The caller does not receive information about which target responded. This is recorded in the routing logs output by the Gateway

Now that we have a rough understanding of the updates, let's try them out!

Prerequisites

This time I tested in the following environment.

Since we can't actually make the primary provider error out, I created a dummy provider as a Lambda function that returns OpenAI-compatible Chat Completions. It switches between the following 3 states via environment variables.

The primary returns a fixed message, and Bedrock returns the model's actual response. You can tell which one responded from the returned body and model name.

uv init --no-readme agentcore-gw-fallback
cd agentcore-gw-fallback
uv python pin 3.14
uv add 'boto3>=1.43.118' requests

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Gateway service role
aws iam create-role \
  --role-name fallback-demo-gateway-role \
  --assume-role-policy-document "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [{
      \"Effect\": \"Allow\",
      \"Principal\": {\"Service\": \"bedrock-agentcore.amazonaws.com\"},
      \"Action\": \"sts:AssumeRole\",
      \"Condition\": {\"StringEquals\": {\"aws:SourceAccount\": \"${ACCOUNT_ID}\"}}
    }]
  }"

aws iam put-role-policy \
  --role-name fallback-demo-gateway-role \
  --policy-name inference-permissions \
  --policy-document "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [
      {
        \"Effect\": \"Allow\",
        \"Action\": [\"bedrock:InvokeModel\", \"bedrock:InvokeModelWithResponseStream\", \"bedrock:List*\"],
        \"Resource\": \"*\"
      },
      {
        \"Effect\": \"Allow\",
        \"Action\": \"bedrock-mantle:*\",
        \"Resource\": \"arn:aws:bedrock-mantle:*:${ACCOUNT_ID}:project/*\"
      },
      {
        \"Effect\": \"Allow\",
        \"Action\": [\"bedrock-agentcore:GetWorkloadAccessToken\", \"bedrock-agentcore:GetResourceApiKey\"],
        \"Resource\": \"*\"
      },
      {
        \"Effect\": \"Allow\",
        \"Action\": \"secretsmanager:GetSecretValue\",
        \"Resource\": \"arn:aws:secretsmanager:us-east-1:${ACCOUNT_ID}:secret:bedrock-agentcore-identity!*\"
      }
    ]
  }"

# Lambda execution role simulating the primary
aws iam create-role \
  --role-name fallback-demo-primary-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

aws iam attach-role-policy \
  --role-name fallback-demo-primary-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

import json
import os
import time

def handler(event, context):
    mode = os.environ.get("MODE", "ok")
    body = json.loads(event.get("body") or "{}")
    print(json.dumps({"mode": mode, "model": body.get("model"), "path": event.get("rawPath")}))

    if mode == "error":
        return {"statusCode": 503, "body": json.dumps({"error": {"message": "primary is unavailable"}})}
    if mode == "slow":
        time.sleep(10)

    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps({
            "id": "chatcmpl-primary",
            "object": "chat.completion",
            "model": body.get("model"),
            "choices": [{
                "index": 0,
                "message": {"role": "assistant", "content": "response from primary"},
                "finish_reason": "stop",
            }],
        }),
    }

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

(cd primary && zip -q ../primary.zip app.py)

aws lambda create-function \
  --function-name fallback-demo-primary \
  --runtime python3.13 \
  --handler app.handler \
  --timeout 15 \
  --role arn:aws:iam::${ACCOUNT_ID}:role/fallback-demo-primary-role \
  --zip-file fileb://primary.zip \
  --environment 'Variables={MODE=ok}' \
  --region us-east-1

aws lambda wait function-active-v2 --function-name fallback-demo-primary --region us-east-1

aws lambda create-function-url-config \
  --function-name fallback-demo-primary \
  --auth-type NONE \
  --region us-east-1

aws lambda add-permission \
  --function-name fallback-demo-primary \
  --statement-id public-url \
  --action lambda:InvokeFunctionUrl \
  --principal '*' \
  --function-url-auth-type NONE \
  --region us-east-1

aws lambda add-permission \
  --function-name fallback-demo-primary \
  --statement-id public-invoke \
  --action lambda:InvokeFunction \
  --principal '*' \
  --invoked-via-function-url \
  --region us-east-1

import sys
import time

import boto3

PRIMARY_URL = sys.argv[1].rstrip("/")  # Lambda function URL
account_id = boto3.client("sts").get_caller_identity()["Account"]
client = boto3.client("bedrock-agentcore-control", region_name="us-east-1")

gateway = client.create_gateway(
    name="fallback-demo-gateway",
    roleArn=f"arn:aws:iam::{account_id}:role/fallback-demo-gateway-role",
    protocolType="MCP",
    authorizerType="AWS_IAM",
)
gateway_id = gateway["gatewayId"]
print(gateway_id, gateway["gatewayUrl"])

while client.get_gateway(gatewayIdentifier=gateway_id)["status"] != "READY":
    time.sleep(5)

# API key attached to the primary (dummy value for verification)
api_key = client.create_api_key_credential_provider(
    name="fallback-demo-primary-key",
    apiKey="dummy-key",
)

# Primary: Lambda for simulating failures (returns OpenAI-compatible Chat Completions)
client.create_gateway_target(
    gatewayIdentifier=gateway_id,
    name="primary",
    credentialProviderConfigurations=[
        {
            "credentialProviderType": "API_KEY",
            "credentialProvider": {
                "apiKeyCredentialProvider": {
                    "providerArn": api_key["credentialProviderArn"],
                    "credentialLocation": "HEADER",
                    "credentialParameterName": "Authorization",
                    "credentialPrefix": "Bearer ",
                }
            },
        }
    ],
    targetConfiguration={
        "inference": {
            "provider": {
                "endpoint": PRIMARY_URL,
                "operations": [
                    {"path": "/v1/chat/completions", "models": [{"model": "primary-chat"}]}
                ],
            }
        }
    },
)

# Secondary: Bedrock Mantle connector
client.create_gateway_target(
    gatewayIdentifier=gateway_id,
    name="bedrock-mantle",
    credentialProviderConfigurations=[{"credentialProviderType": "GATEWAY_IAM_ROLE"}],
    targetConfiguration={"inference": {"connector": {"source": {"connectorId": "bedrock-mantle"}}}},
)

# For InvokeModel: Bedrock Runtime connector
client.create_gateway_target(
    gatewayIdentifier=gateway_id,
    name="bedrock-runtime",
    credentialProviderConfigurations=[{"credentialProviderType": "GATEWAY_IAM_ROLE"}],
    targetConfiguration={"inference": {"connector": {"source": {"connectorId": "bedrock-runtime"}}}},
)

# Wait until all targets are READY
while True:
    statuses = {t["name"]: t["status"] for t in client.list_gateway_targets(gatewayIdentifier=gateway_id)["items"]}
    if all(status == "READY" for status in statuses.values()):
        break
    time.sleep(5)
print(statuses)

bash setup_iam.sh
bash deploy_primary.sh
uv run setup.py <Lambda function URL>

import json
import sys
import time

import boto3
import requests
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest

GATEWAY_URL = "https://<GATEWAY_ID>.gateway.bedrock-agentcore.us-east-1.amazonaws.com"
credentials = boto3.Session().get_credentials()

def call(model):
    body = {
        "model": model,
        "max_tokens": 512,
        "messages": [{"role": "user", "content": "Hello"}],
    }
    req = AWSRequest(
        method="POST",
        url=f"{GATEWAY_URL}/inference/v1/chat/completions",
        data=json.dumps(body),
        headers={"Content-Type": "application/json"},
    )
    SigV4Auth(credentials, "bedrock-agentcore", "us-east-1").add_auth(req)
    start = time.time()
    res = requests.post(req.url, data=req.body, headers=dict(req.headers), timeout=60)
    elapsed = time.time() - start
    try:
        data = res.json()
        summary = data.get("model"), (data["choices"][0]["message"]["content"] or "")[:30]
    except Exception:
        summary = res.text[:120]
    print(f"{res.status_code} {elapsed:5.2f}s {summary}")

if __name__ == "__main__":
    model = sys.argv[1]
    for _ in range(int(sys.argv[2]) if len(sys.argv) > 2 else 1):
        call(model)

# Usage: bash set_mode.sh ok|error|slow
aws lambda update-function-configuration \
  --function-name fallback-demo-primary \
  --environment "Variables={MODE=$1}" \
  --region us-east-1 > /dev/null
aws lambda wait function-updated-v2 --function-name fallback-demo-primary --region us-east-1

# Usage: bash count_primary.sh <start UNIX timestamp> <end UNIX timestamp>
aws logs filter-log-events \
  --log-group-name /aws/lambda/fallback-demo-primary \
  --start-time $(( $1 * 1000 )) \
  --end-time $(( $2 * 1000 )) \
  --filter-pattern '"mode"' \
  --region us-east-1 \
  --query 'length(events)'

Now that we're ready, let's try it out!

Calling the Gateway with Bedrock SDK's InvokeModel

First, let's try InvokeModel!

All you need to do is point the endpoint_url to the Gateway's /inference when creating the boto3 bedrock-runtime client.

import json
import sys

import boto3

from call import GATEWAY_URL

# Just point the connection to Gateway's /inference. Authentication is the usual SigV4
client = boto3.client(
    "bedrock-runtime",
    region_name="us-east-1",
    endpoint_url=f"{GATEWAY_URL}/inference",
)

CLAUDE = "us.anthropic.claude-haiku-4-5-20251001-v1:0"
claude_body = json.dumps({
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 128,
    "messages": [{"role": "user", "content": "Hello. Reply in one sentence"}],
})

def invoke():
    response = client.invoke_model(modelId=CLAUDE, body=claude_body, contentType="application/json")
    data = json.loads(response["body"].read())
    print(data["model"], data["content"][0]["text"])

def stream():
    body = json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 256,
        "messages": [{"role": "user", "content": "Explain AgentCore Gateway in 3 sentences"}],
    })
    response = client.invoke_model_with_response_stream(
        modelId=CLAUDE, body=body, contentType="application/json"
    )
    for event in response["body"]:
        chunk = json.loads(event["chunk"]["bytes"])
        if chunk["type"] == "content_block_delta":
            print(repr(chunk["delta"]["text"]))
        else:
            print(chunk["type"])

if __name__ == "__main__":
    {"invoke": invoke, "stream": stream}[sys.argv[1]]()

Call Claude Haiku 4.5 with both InvokeModel and InvokeModelWithResponseStream.

uv run invoke_model.py invoke
uv run invoke_model.py stream

anthropic.claude-haiku-4-5-20251001-v1:0 Hello!

'ata'
'flow'
'con'
'trol effi'
'cient'
'ly con'
'trols. Enter'
'prises'
(omitted)
'operability'
'reali'
'zes.'
content_block_stop
message_delta
message_stop

Both worked right away! They can now be aggregated through the Gateway.
Next, let's try the fallback feature.

What Happens When the Primary Fails Without Rules

First, for comparison, let's set the primary to 503 state without creating any rules and make a call.

bash set_mode.sh error
uv run call.py primary-chat 3

503  1.21s {"error": {"message": "primary is unavailable"}}
503  1.08s {"error": {"message": "primary is unavailable"}}
503  0.87s {"error": {"message": "primary is unavailable"}}

With default routing, the request is sent directly to the target handling primary-chat, so 503 is returned as-is.

Let's see how this changes with fallback!

Creating a Rule with matchModels and Fallback

Create a rule that sends requests for primary-chat to the primary and switches to Bedrock Mantle's openai.gpt-oss-120b on failure.

import time

import boto3

client = boto3.client("bedrock-agentcore-control", region_name="us-east-1")

rule = client.create_gateway_rule(
    gatewayIdentifier="<GATEWAY_ID>",
    priority=100,
    description="Switch to Bedrock's gpt-oss-120b when primary-chat fails",
    conditions=[
        {"matchModels": {"anyOf": [{"value": "primary-chat", "operator": "StringEquals"}]}}
    ],
    actions=[
        {
            "routeToTarget": {
                "staticRoute": {
                    "targetName": "primary",
                    "fallback": {
                        "targetFallback": [
                            {"targetName": "bedrock-mantle", "model": "openai.gpt-oss-120b"}
                        ],
                    },
                }
            }
        }
    ],
)
while client.get_gateway_rule(gatewayIdentifier="<GATEWAY_ID>", ruleId=rule["ruleId"])["status"] != "ACTIVE":
    time.sleep(5)
print(rule["ruleId"], "ACTIVE")

Since no model is specified for the primary side, the primary-chat from the request is sent as-is. For the fallback target, a model is specified to replace it with a model ID that Bedrock can handle. Since triggerOn is omitted, switching occurs on 503 (5XX) which is included in the default conditions.

uv run create_rule.py

950fdf64-bfef-4309-9b76-9bbf9b04c263 ACTIVE

Once it becomes ACTIVE, leave the primary in 503 state and send the same request.

uv run call.py primary-chat 3

200  1.74s ('openai.gpt-oss-120b', 'Hello! Please feel free to ask if you have any requests or questions. 😊')
200  1.57s ('openai.gpt-oss-120b', 'Hello! Please let me know if you have any requests or questions')
200  1.93s ('openai.gpt-oss-120b', 'Hello! How can I help you today? 🙂')

It switched! Without changing anything in the calling code, we got a 200 response from the fallback gpt-oss-120b.

When the primary was restored to ok, it immediately returned to the original response.

bash set_mode.sh ok
uv run call.py primary-chat 2

200  1.39s ('primary-chat', 'response from primary')
200  0.90s ('primary-chat', 'response from primary')

It works!

Switch on Timeout When Primary Is Unresponsive

Provider failures don't always return 5XX — there are also cases where you're left waiting with no response coming back. So we'll test a configuration where the primary is made to wait for 10 seconds, and if the first byte doesn't arrive within 3 seconds, it switches over.

For the rule update, we call it with firstByteTimeoutMilliseconds and circuitBreaker added to the settings from when it was created. The circuit breaker is made switchable via arguments so we can compare it later.

import sys
import time

import boto3

client = boto3.client("bedrock-agentcore-control", region_name="us-east-1")
GATEWAY_ID = "<GATEWAY_ID>"
RULE_ID = sys.argv[1]
breaker = sys.argv[2] if len(sys.argv) > 2 else "ENABLED"

client.update_gateway_rule(
    gatewayIdentifier=GATEWAY_ID,
    ruleId=RULE_ID,
    priority=100,
    description="primary-chatが失敗・3秒無応答ならBedrockのgpt-oss-120bへ切り替える",
    conditions=[
        {"matchModels": {"anyOf": [{"value": "primary-chat", "operator": "StringEquals"}]}}
    ],
    actions=[
        {
            "routeToTarget": {
                "staticRoute": {
                    "targetName": "primary",
                    "fallback": {
                        "firstByteTimeoutMilliseconds": 3000,  # added
                        "circuitBreaker": {"mode": breaker},  # added
                        "targetFallback": [
                            {"targetName": "bedrock-mantle", "model": "openai.gpt-oss-120b"}
                        ],
                    },
                }
            }
        }
    ],
)
time.sleep(5)
while client.get_gateway_rule(gatewayIdentifier=GATEWAY_ID, ruleId=RULE_ID)["status"] != "ACTIVE":
    time.sleep(5)
print("ACTIVE")

First, let's try it with the circuit breaker disabled.

uv run update_rule.py <RULE_ID> DISABLED
bash set_mode.sh slow
uv run call.py primary-chat 3

200  4.57s ('openai.gpt-oss-120b', 'こんにちは！今日はどんなことをお手伝いできますか？😊')
200  4.73s ('openai.gpt-oss-120b', 'こんにちは！どのようなお手伝いが必要ですか？お気軽に教えてく')
200  4.73s ('openai.gpt-oss-120b', 'こんにちは！本日はいかがでしょうか？何かお手伝いできることが')

The primary was cut off at 3 seconds, and the secondary's response came back in approximately 4.7 seconds!

Comparing With and Without Circuit Breaker

You wouldn't want to wait 3 seconds every time for a primary that stays unresponsive. By enabling the circuit breaker, targets that keep failing will be skipped for a while and requests will be sent directly to the secondary from the start.

With the same slow-failure state, we called it 20 times in a row and compared the elapsed time and the number of times the primary Lambda was called.

import sys
import time

import call

results = []
start = time.time()
for i in range(int(sys.argv[1])):
    t = time.time()
    call.call("primary-chat")
    results.append(time.time() - t)
print(f"Total {time.time() - start:.1f}s / per call " + " ".join(f"{r:.1f}" for r in results))

# Call 20 times with breaker disabled, wait for logs to be ingested, then count the number of calls in that period
START=$(date +%s); uv run breaker_test.py 20; END=$(date +%s)
sleep 60
bash count_primary.sh $START $END

# Enable it and do the same
uv run update_rule.py <RULE_ID> ENABLED
START=$(date +%s); uv run breaker_test.py 20; END=$(date +%s)
sleep 60
bash count_primary.sh $START $END

Total 93.5s / per call 4.2 4.5 4.7 4.5 4.7 4.9 4.7 4.6 4.3 4.6 5.2 4.6 5.7 4.5 4.8 4.5 4.7 5.1 4.4 4.4
20

Total 70.1s / per call 4.8 4.9 5.1 4.4 4.5 4.7 4.5 4.4 4.5 4.6 5.0 1.5 1.3 3.1 1.4 1.2 2.7 1.3 1.3 4.7
12

The circuit breaker is working!

By the way, how many failures it takes to open is not documented and cannot be configured...
It seems to open after a certain number of consecutive failures — in this case it was around 10 to 11 times.

Creating Aliases with Wildcards and Model Replacement

Separate from fallback, combining StringLike in matchModels with model replacement lets you map names that don't exist on any target to Bedrock models. Let's create a rule that processes all model names starting with team- using gpt-oss-120b.

Before that, let's change the primary — which was left on slow from the circuit breaker verification — back to ok, and confirm it's responding normally.

bash set_mode.sh ok
uv run call.py primary-chat

200  1.25s ('primary-chat', 'primaryからの応答です')

import time

import boto3

client = boto3.client("bedrock-agentcore-control", region_name="us-east-1")

rule = client.create_gateway_rule(
    gatewayIdentifier="<GATEWAY_ID>",
    priority=50,
    description="team-で始まるモデル名をBedrockのgpt-oss-120bで処理する",
    conditions=[
        {"matchModels": {"anyOf": [{"value": "team-*", "operator": "StringLike"}]}}
    ],
    actions=[
        {
            "routeToTarget": {
                "staticRoute": {
                    "targetName": "bedrock-mantle",
                    "model": "openai.gpt-oss-120b",
                }
            }
        }
    ],
)
while client.get_gateway_rule(gatewayIdentifier="<GATEWAY_ID>", ruleId=rule["ruleId"])["status"] != "ACTIVE":
    time.sleep(5)
print(rule["ruleId"], "ACTIVE")

Let's compare the results of sending requests before and after creating the rule.

404  0.79s {"error":{"message":"Model 'team-standard' not found on any target.","type":"not_found_error","code":"404"}}

team-standard: 200  2.64s ('openai.gpt-oss-120b', 'こんにちは！今日はどのようなお手伝いが必要ですか？気になるこ')
team-fast:     200  1.31s ('openai.gpt-oss-120b', 'こんにちは！ご機嫌いかがですか？何かお手伝いできることがあれ')
primary-chat:  200  0.98s ('primary-chat', 'primaryからの応答です')

Before the rule was created, names not found on any target returned 404, but after creation, names starting with team- were processed by gpt-oss-120b. primary-chat, which doesn't match the condition, continues to be answered by the primary as before.

You could specify an alias like team-standard in your application and let the Gateway rule decide the actual model ID. It's nice not having to change application settings when swapping out models — this looks useful!

Notes

I also separately tested streaming, rate limiting, and combinations with Policy. The following are behaviors confirmed in this verification environment. I plan to summarize the configuration and detailed verification results in a separate article.

• With streaming, if the primary returns 503, it switches to the secondary and continues to deliver via streaming. However, if the primary goes down after it has started returning data, there's no switchover and it ends midway (in this case, no error event was delivered either)
• Fallback does not occur for 429 from the Gateway's own rate limiting or 403 from Policy rejection. The same applies even when added to triggerOn
• Rate limiting and Policy are evaluated at the fallback destination as well. If the secondary target is rate-limited, it will stop at the secondary after a 503 from the primary
• When Policy Engine is attached with ENFORCE, streaming responses arrived all at once at the end
• If you attach guardrailIdentifier to InvokeModel, routes that involve conversion (providers other than Bedrock) are rejected with 400
• The Converse API returned 400 via Gateway. Only InvokeModel and InvokeModelWithResponseStream are supported

Cleanup

Delete the rules, targets, Gateway, and API keys, then delete the Lambda functions and IAM roles. Since the Lambda function URL is publicly accessible without authentication, be sure not to forget to delete it.

import time

import boto3

client = boto3.client("bedrock-agentcore-control", region_name="us-east-1")
GATEWAY_ID = "<GATEWAY_ID>"

for rule in client.list_gateway_rules(gatewayIdentifier=GATEWAY_ID)["gatewayRules"]:
    client.delete_gateway_rule(gatewayIdentifier=GATEWAY_ID, ruleId=rule["ruleId"])
for target in client.list_gateway_targets(gatewayIdentifier=GATEWAY_ID)["items"]:
    client.delete_gateway_target(gatewayIdentifier=GATEWAY_ID, targetId=target["targetId"])
while client.list_gateway_targets(gatewayIdentifier=GATEWAY_ID)["items"]:
    time.sleep(5)
client.delete_gateway(gatewayIdentifier=GATEWAY_ID)
client.delete_api_key_credential_provider(name="fallback-demo-primary-key")

aws lambda delete-function --function-name fallback-demo-primary --region us-east-1
aws logs delete-log-group --log-group-name /aws/lambda/fallback-demo-primary --region us-east-1

aws iam delete-role-policy --role-name fallback-demo-gateway-role --policy-name inference-permissions
aws iam delete-role --role-name fallback-demo-gateway-role
aws iam detach-role-policy --role-name fallback-demo-primary-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam delete-role --role-name fallback-demo-primary-role

uv run cleanup.py
bash cleanup.sh

Closing

Once again, this was an update that expanded the scope of Gateway!
Fallback and routing seem like great features that are very fitting for a Gateway!

I hope this article was helpful in some way. Thank you for reading to the end!