![[Update] Amazon Bedrock AgentCore Identity Now Supports Private Certificate Authorities (CAs)!](https://devio2024-media.developers.io/image/upload/f_auto,q_auto/c_limit,w_1200/v1790875048/eyecatch/edq4dfptctqlcwjhffhs.png)
[Update] Amazon Bedrock AgentCore Identity Now Supports Private Certificate Authorities (CAs)!
This page has been translated by machine translation. View original
Introduction
Hello, I'm Jinno from the Consulting Department, and I love cheese naan.
Amazon Bedrock AgentCore Identity now supports certificates issued by a private certificate authority (CA)!
It's a somewhat niche update, but I think it's welcome news for those who manage their own IdP in-house and issue certificates from an internal CA!
This is an Identity-side update following last month's update on Gateway targets, so I'll summarize the main changes and try connecting to Keycloak inside a VPC.
Wait, what's different from last time...? Didn't it already support private CAs before? You might be thinking that...!!
Last time was about Gateway connecting to targets such as MCP servers, while this time is about AgentCore Identity and Gateway's JWT authorization connecting to the IdP. The difference is whether the connection destination is a target or an IdP.

Main changes
Until now, even when AgentCore Identity connected to an IdP inside a VPC via VPC Lattice, the IdP needed a certificate from a public CA.
For that reason, when I previously tried it with Keycloak, I obtained a public domain and placed an internal ALB with an ACM public certificate in front of Keycloak.
Starting now, you just place the CA certificate PEM (a text-format certificate file) in S3 or Secrets Manager and reference it, and you can connect directly to the IdP while keeping its private CA certificate! It's nice not to have to prepare an ALB purely for terminating TLS or a public domain.

The target is the custom OAuth 2.0 credential provider (outbound authentication), which gets tokens from the IdP so that an agent can call external services.
The CA can be any of AWS Private CA, an internal CA, or a self-signed root CA.
The official blog introduces that the same setting can also be used for authorization that validates calls to Runtime and Gateway with JWTs (inbound authentication), so this time I'll try that with Gateway as well.
By the way, if you combine the old and new CA certificates into a single PEM, you can swap them with minimal downtime.
You can also remove trust in a private CA by removing the reference and updating.
Prerequisites
This time I'll test with the configuration in the diagram below!

I run Keycloak on an EC2 instance in a private subnet and accept HTTPS with a server certificate issued from a self-signed root CA created with OpenSSL.
The hostname idp.internal.example.com is registered in a Route 53 private hosted zone so it can be resolved from inside the VPC, and AgentCore Identity connects to Keycloak via a VPC Lattice resource gateway (the entry point that routes traffic from AgentCore into the VPC).
With this configuration, I'll create a credential provider and try to get a token from the AWS CLI.
| Item | Details |
|---|---|
| Region | ap-northeast-1 |
| IdP | Keycloak 26.8.2 (EC2 t4g.medium, Amazon Linux 2023, Docker) |
| Certificate creation | OpenSSL 4.1.0 (macOS) |
| Environment setup | AWS CDK (aws-cdk-lib 2.277.0, TypeScript, Node.js 26.13.0 + pnpm 10.28.1) |
| Provider creation and token retrieval | AWS CLI 2.40.2 |
| Connection mode | managed Lattice (managedVpcResource) |
The following commands are run in Bash. The code used for the verification is in the following repository.
The main files in the repository are as follows.
agentcore-identity-private-ca/
├── certs.sh
├── env.sh
├── keycloak/realm-demo.json
└── cdk/
├── bin/app.ts
└── lib/idp-stack.ts
Let's get started!
Trying it out
First, let's prepare the certificates.
Creating the certificates
I create a private CA with CA:TRUE, which indicates it is a CA, and a server certificate with Keycloak's hostname in the SAN.
# Private CA
openssl req -x509 -new -key ca.key -sha256 -days 3650 \
-subj "/CN=$CA_NAME" \
-addext "basicConstraints=critical,CA:TRUE" \
-addext "keyUsage=critical,keyCertSign,cRLSign" \
-out ca.pem
# Server certificate (put Keycloak's hostname in the SAN)
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
-days 365 -sha256 \
-extfile <(printf "subjectAltName=DNS:%s\nbasicConstraints=CA:FALSE\nextendedKeyUsage=serverAuth\n" "$HOST") \
-out server.pem
I clone the repository, run certs.sh, and check the contents of the certificates.
git clone https://github.com/yuu551/agentcore-identity-private-ca.git
cd agentcore-identity-private-ca
./certs.sh
openssl x509 -in certs/ca.pem -noout -subject -ext basicConstraints
openssl verify -CAfile certs/ca.pem certs/server.pem
subject=CN=Demo Private Root CA
X509v3 Basic Constraints: critical
CA:TRUE
certs/server.pem: OK
The CA certificate has CA:TRUE, and the server certificate can be verified with this CA.
Setting up the environment
I use CDK to create the VPC, Keycloak (EC2), the private hosted zone, and the S3 bucket for the CA certificate all at once.
Keycloak runs on Docker and accepts HTTPS with the server certificate, and I restricted the connection source to the security group attached to the resource gateway's ENI.
Of the certificates created by certs.sh, the server certificate is used by Keycloak for HTTPS, and the CA certificate is used by AgentCore Identity to verify Keycloak's certificate.
The server certificate's private key is not included in the CDK assets; it is registered in Secrets Manager before deployment and retrieved when the EC2 instance starts.

// Keycloak accepts only port 443 from the resource gateway's ENI
idpSg.addIngressRule(latticeSg, ec2.Port.tcp(443));
// UserData (don't add -x, because secret values would remain in the cloud-init log)
'set -euo pipefail',
// Get the private key from Secrets Manager with 600 permissions, and accept HTTPS with the server certificate signed by the private CA
`(umask 077; aws secretsmanager get-secret-value --region ${this.region} --secret-id ${SERVER_KEY_SECRET} --query SecretString --output text > /opt/kc/tls/server.key)`,
'--https-certificate-file=/opt/keycloak/conf/tls/server.pem',
'--https-certificate-key-file=/opt/keycloak/conf/tls/server.key',
keycloak/realm-demo.json is the Keycloak realm definition; a realm is a unit for managing clients and users together.
Here, it creates the client agentcore-client used by AgentCore Identity in the demo realm, and also includes a setting that puts agentcore-gateway in the audience (the aud claim, which indicates the token's intended recipient) that Gateway will check later.
The values used in subsequent commands are loaded with env.sh after the stack is deployed.
env.sh
# Load the values used in subsequent commands from the stack outputs
export AWS_REGION=ap-northeast-1
out() { aws cloudformation describe-stacks --stack-name PrivateCaIdp \
--query "Stacks[0].Outputs[?OutputKey=='$1'].OutputValue" --output text; }
PE=$(out PrivateEndpoint) # JSON passed to privateEndpoint (VPC, subnets, SG)
B=$(out BucketName) # Bucket where the CA certificate PEM is placed
GW_ROLE=$(out GatewayRoleArn) # Role of the Gateway used to try inbound authentication
KC_ID=$(out KeycloakInstanceId)
CLIENT_SECRET=$(aws secretsmanager get-secret-value --secret-id "$(out ClientSecretArn)" \
--query SecretString --output text)
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
DISCOVERY_URL=https://idp.internal.example.com/realms/demo/.well-known/openid-configuration
export AWS_REGION=ap-northeast-1
# Register the server certificate's private key in Secrets Manager beforehand
aws secretsmanager create-secret --name private-ca-demo/keycloak-server-key \
--secret-string file://certs/server.key
(cd cdk && pnpm install && pnpm cdk deploy PrivateCaIdp)
source env.sh
Creating the credential provider
I create a credential provider in AgentCore Identity with the CA certificate specified. In the JSON, I add certificateConfigurations at the same level as privateEndpoint (used to connect to the IdP inside the VPC) and pass the URI of the CA certificate placed in S3.
The meaning of each item is shown in the following diagram.

AgentCore Identity reads the CA certificate using the credentials of the principal that performs the provider creation or update, so that principal needs s3:GetObject on the S3 object.
CONFIG=$(jq -n \
--arg url "$DISCOVERY_URL" --arg secret "$CLIENT_SECRET" --argjson pe "$PE" \
--arg ca "s3://$B/ca/private-ca.pem" --arg owner "$ACCOUNT" '{
customOauth2ProviderConfig: {
oauthDiscovery: {discoveryUrl: $url},
clientId: "agentcore-client",
clientSecret: $secret,
privateEndpoint: $pe,
certificateConfigurations: [{s3: {uri: $ca, bucketOwnerAccountId: $owner}}]
}}')
aws bedrock-agentcore-control create-oauth2-credential-provider \
--name keycloak-private-ca \
--credential-provider-vendor CustomOauth2 \
--oauth2-provider-config-input "$CONFIG"
About 3 minutes after the initial CREATING status, it became READY, and when I retrieved the configuration, the reference to the CA certificate had been saved.
{
"status": "READY",
"oauth2ProviderConfigOutput": {
"customOauth2ProviderConfig": {
"certificateConfigurations": [
{"s3": {"uri": "s3://privatecaidp-bucket83908e77-xxxx/ca/private-ca.pem", "bucketOwnerAccountId": "<アカウントID>"}}
]
}
}
}
Getting a token
I create a workload identity representing the agent and get a token with the M2M (client credentials) flow. M2M is a method where the token is issued using only the client ID and secret, without a user login, and from the AWS CLI you call two APIs in order as shown in the following diagram.

I'll reuse the two calls and the token decoding later, so I put them together into the functions token and claims.
aws bedrock-agentcore-control create-workload-identity --name private-ca-demo-agent
token() {
local wat=$(aws bedrock-agentcore get-workload-access-token \
--workload-name private-ca-demo-agent --query workloadAccessToken --output text)
aws bedrock-agentcore get-resource-oauth2-token \
--workload-identity-token "$wat" \
--resource-credential-provider-name "${1:-keycloak-private-ca}" \
--scopes openid --oauth2-flow M2M \
--query accessToken --output text
}
# Decode the payload of the retrieved JWT and check its contents
claims() {
token "$@" | cut -d. -f2 | uv run --no-project python -c \
"import base64, json, sys; p = sys.stdin.read().strip(); print(json.dumps(json.loads(base64.urlsafe_b64decode(p + '=' * (-len(p) % 4))), indent=2))"
}
claims
{
"iss": "https://idp.internal.example.com/realms/demo",
"aud": ["agentcore-gateway", "account"],
"azp": "agentcore-client",
"clientAddress": "10.0.2.166"
}
We got a token from a Keycloak that can't be reached from the internet! When I checked the clientAddress (the connection source as seen from Keycloak) against the ENIs in the VPC, it was the IP address of the resource gateway.
What happens if you don't specify the CA certificate
For comparison, I also created a provider without certificateConfigurations (keycloak-no-ca). It also reaches READY, but fails when getting a token.
aws bedrock-agentcore-control create-oauth2-credential-provider \
--name keycloak-no-ca \
--credential-provider-vendor CustomOauth2 \
--oauth2-provider-config-input "$(jq 'del(.customOauth2ProviderConfig.certificateConfigurations)' <<< "$CONFIG")"
# Run after it becomes READY
token keycloak-no-ca
aws: [ERROR]: An error occurred (ValidationException) when calling the GetResourceOauth2Token operation: HTTP request failed against private endpoint
By default only public CAs are trusted, so Keycloak's certificate can't be verified and the request fails, but the error doesn't mention certificates, so it could be a little confusing.
I also tried two other patterns. A provider given the server certificate PEM (keycloak-leaf-cert) ended up CREATE_FAILED with the reason "Leaf certificates are not allowed, must provide CA certificates only". When specifying a CA certificate without privateEndpoint, the creation request itself is rejected with the error "The private certificate authority feature can only be used with private IdP".

Since a provider becomes READY even when certificateConfigurations is omitted, it's reassuring to get a token once after creation and confirm that you can connect to Keycloak.
Trying it with Gateway's JWT authorization (inbound authentication) too
You can also add certificateConfigurations to Gateway's authorization settings (customJWTAuthorizer) in the same form as the provider!
For the allowed audience, I specify agentcore-gateway, which was put in the token via the realm definition, and I use the token obtained with the earlier provider as-is to call Gateway. Gateway retrieves and validates the discovery URL at creation time, and checks the JWT signature and aud at invocation time.

AUTHZ=$(jq -n \
--arg url "$DISCOVERY_URL" --argjson pe "$PE" \
--arg ca "s3://$B/ca/private-ca.pem" --arg owner "$ACCOUNT" '{
customJWTAuthorizer: {
discoveryUrl: $url,
allowedAudience: ["agentcore-gateway"],
privateEndpoint: $pe,
certificateConfigurations: [{s3: {uri: $ca, bucketOwnerAccountId: $owner}}]
}}')
# Note the ID of the created Gateway for use in invocation and cleanup
GW_ID=$(aws bedrock-agentcore-control create-gateway \
--name private-ca-inbound \
--protocol-type MCP \
--role-arn "$GW_ROLE" \
--authorizer-type CUSTOM_JWT \
--authorizer-configuration "$AUTHZ" \
--query gatewayId --output text)
After waiting about 2 minutes for it to become READY, I call tools/list with the token attached. No targets are registered, so if JWT validation passes, an empty tool list is returned.
GW_URL=$(aws bedrock-agentcore-control get-gateway --gateway-identifier "$GW_ID" \
--query gatewayUrl --output text)
curl -s -X POST "$GW_URL" \
-H "Authorization: Bearer $(token)" \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
{"jsonrpc":"2.0","id":1,"result":{"tools":[]}}
JWT validation passed and we got a response!!
Now let's think about what happens at certificate renewal time.
Swapping the CA certificate
You're probably curious about swapping during certificate renewal!
Using a PEM that combines the old and new CAs, I'll try the following 3-step swap introduced in the official blog.

AgentCore Identity reads and stores the PEM contents at creation and update time, so after rewriting the file in S3 you also need to call the update API. First I create a new CA and update with a PEM that combines it with the old CA.
OUT=certs-new CA_NAME="Demo Private Root CA 2" ./certs.sh
cat certs/ca.pem certs-new/ca.pem > certs/bundle.pem
aws s3 cp certs/bundle.pem s3://$B/ca/private-ca.pem
# Update with the same settings as at creation (including certificateConfigurations)
aws bedrock-agentcore-control update-oauth2-credential-provider \
--name keycloak-private-ca \
--credential-provider-vendor CustomOauth2 \
--oauth2-provider-config-input "$CONFIG"
M2M tokens are returned as the same one during their validity period (5 minutes this time), so I proceeded while checking whether a token was newly issued using the decoded iat (issued-at time).
The update returned to READY in a dozen or so seconds, and even after replacing Keycloak's certificate with one signed by the new CA, I could get newly issued tokens.
Finally, I updated so that only the new CA is trusted, and I could still get new tokens afterwards, so switching to the new CA works without problems!
Replacing Keycloak's certificate and updating with only the new CA
# Overwrite server/server.pem, the same as at first startup, so that the new certificate is used even if the EC2 instance is recreated
aws s3 cp certs-new/server.pem s3://$B/server/server.pem
aws secretsmanager put-secret-value --secret-id private-ca-demo/keycloak-server-key \
--secret-string file://certs-new/server.key
CID=$(aws ssm send-command --instance-ids "$KC_ID" --document-name AWS-RunShellScript \
--parameters "commands=[\"aws s3 cp s3://$B/server/server.pem /opt/kc/tls/server.pem\",\"aws secretsmanager get-secret-value --region $AWS_REGION --secret-id private-ca-demo/keycloak-server-key --query SecretString --output text > /opt/kc/tls/server.key\",\"chown -R 1000:1000 /opt/kc\",\"docker restart keycloak\"]" \
--query Command.CommandId --output text)
aws ssm wait command-executed --command-id "$CID" --instance-id "$KC_ID"
# If the iat differs from before the replacement, it's a newly issued token (the same iat is cached, so wait about 5 minutes and rerun)
claims | grep '"iat"'
# Remove the old CA after confirming a token with a new iat
aws s3 cp certs-new/ca.pem s3://$B/ca/private-ca.pem
aws bedrock-agentcore-control update-oauth2-credential-provider \
--name keycloak-private-ca \
--credential-provider-vendor CustomOauth2 \
--oauth2-provider-config-input "$CONFIG"
# After it returns to READY, confirm a token with a new iat in the same way
claims | grep '"iat"'
Cleanup
I delete the Gateway and credential providers, then delete the CDK stack, and also remove the private key registered in Secrets Manager and the Lambda log groups.
aws bedrock-agentcore-control delete-gateway --gateway-identifier "$GW_ID"
for p in keycloak-private-ca keycloak-no-ca; do
aws bedrock-agentcore-control delete-oauth2-credential-provider --name "$p"
done
# Run this only if you also tried the pattern that passes the server certificate
# aws bedrock-agentcore-control delete-oauth2-credential-provider --name keycloak-leaf-cert
aws bedrock-agentcore-control delete-workload-identity --name private-ca-demo-agent
# Before deleting the stack, note the Lambda function names to identify the log groups that remain
FNS=$(aws cloudformation describe-stack-resources --stack-name PrivateCaIdp \
--query "StackResources[?ResourceType=='AWS::Lambda::Function'].PhysicalResourceId" --output text)
(cd cdk && pnpm cdk destroy PrivateCaIdp)
aws secretsmanager delete-secret --secret-id private-ca-demo/keycloak-server-key \
--force-delete-without-recovery
for fn in $FNS; do
aws logs delete-log-group --log-group-name "/aws/lambda/$fn"
done
The resource gateway's ENIs were automatically removed within a minute after deleting the provider and Gateway.
Conclusion
Compared with before, when I had to prepare an internal ALB and a public domain, I think it's great that you now just place a PEM and reference it! It is a somewhat niche update, but I think it can be put to good use with IdPs used on internal company networks!
I hope this article was of some help. Thank you for reading to the end!
