I checked out the GenAI Observability dashboard from the Observability feature in Amazon Bedrock AgentCore
Introduction
Hello, I am Kanno from the Consulting Department, who supports the supermarket La Mu.
Today I want to introduce Amazon Bedrock AgentCore Observability! It's a metrics collection function.
Hearing the name, did you intuitively think "observability...? Sounds difficult..."? I hope to unravel this in this article!
AgentCore Observability
AgentCore Observability is a feature that helps trace, debug, and monitor agent performance.
It's a useful feature when you want to observe behavior while developing and operating agents!
Looking at the diagram in the official documentation, it was a feature like the following.
It collects information from AgentCore Runtime, Memory, and Gateway, converts it to OTEL (Open Telemetry) logs, and makes it visible.
The main telemetry is divided into three categories: metrics, structured logs, and spans and traces. Each has its characteristics, so let's look at them in order!
First, metrics represent basic indicators such as call counts, latency, duration, token usage, session counts, throttling, user errors, and system error aggregations.
Since they are provided as CloudWatch metrics, they are relatively easy to understand. They are useful for understanding agent performance and usage.
Next, structured logs are JSON-formatted logs that capture event ingestion, long-term memory extraction, integration procedures, and various operations. These logs record detailed operational logs of the agent in JSON format compliant with the OpenTelemetry standard. They include detailed information about what decisions the agent made, what tools it used, what prompts it processed, etc., which is very useful for debugging.
Finally, spans and traces - traces record the complete execution path from agent invocation to response, while spans represent individual operation units within that path. They have a hierarchical structure, so you can visually confirm how much time each process took.
In AgentCore Observability, sessions exist as units that bundle traces, imagining the entire conversation. It looks like the following:
Dashboard Integration
This is a built-in dashboard. It's a convenient feature that can be viewed from CloudWatch.
It's provided as a feature called GenAI Observability.
Enabling Transaction Search
To allow span ingestion, you need to enable Transaction Search once per account.
Press the Enable Transaction Search button below to enable it.
### Required Dependencies
When collecting OTEL logs in AgentCore Runtime, you need to include aws-opentelemetry-distro as a dependency. However, when deploying with the AgentCore starter toolkit, opentelemetry-instrument is automatically enabled and ready for visualization.
The image below illustrates this concept.
Log Groups
When using Runtime, two types of log streams are output.
- runtime-logs
- OpenTelemetry standard logs
- YYYY/MM/DD/[runtime-logs]xxxx
- Application logs
There are quite a lot of features...!!
In this article, I'd like to focus mainly on the GenAI Observability dashboard!
Preparation
Agent to be used this time
For this test, we'll check what we can see with the agent we've created so far.
We'll verify using an agent that utilizes the Memory feature.
https://github.com/yuu551/tech-learning-assistant/tree/main## Try it out
I'll keep the runtime-session-id in chat.py the same value.
runtime_session_id = "sample-id-xxxx"
Let's immediately ask the agent a question.
python chat.py "Tell me about S3 bucket security" --user "test_user" --session "session_test" [11:05:31]
"Practical advice:\n- Check AWS official documentation regularly\n- Carefully configure security groups and bucket policies\n- Always keep the principle of least privilege in mind\n\nDo you have any questions or want to know more details about specific implementations?"
python chat.py "I'm not good with security permissions... What specific settings should I configure?" --user "test_user" --session "session_test"
"Here's a detailed explanation of S3 bucket security configuration points:\n\n1. Bucket access settings\n- Block public access\n - Enable the \"Block all public access\" option\n - Prohibit all public access by default\n\n2. Bucket policy configuration\n- Apply the principle of least privilege\n ```json\n {\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"RestrictBucketAccess\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": \"s3:*\",\n \"Resource\": [\n \"arn:aws:s3:::your-bucket-name\",\n \"arn:aws:s3:::your-bucket-name/*\"\n ],\n \"Condition\": {\n \"NotIpAddress\": {\n \"aws:SourceIp\": \"allowed IP address/range\"\n }\n }\n }\n ]\n }\n ```\n\n3. IAM user/role permission management\n- Allow only specifically needed operations\n- Separate read/write permissions\n- Regularly review permissions\n\n4. Enable encryption\n- Enable server-side encryption\n - SSE-S3 (AWS managed keys)\n - SSE-KMS (Customer managed keys)\n\n5. Versioning and tracking\n- Enable versioning\n- Record access logs with CloudTrail\n\nAs practical advice, I recommend the following:\n- Reference AWS official documentation\n- Configure security groups incrementally\n- Be mindful of the least privilege principle\n\nDo you have concerns about specific implementations or configurations? Let's carefully check them together."
I'll also ask a question that uses a tool. It's the weakness analysis tool.
python chat.py "Analyze my weaknesses" --session "session_test" --user "test_user" [11:21:58]
"From the analysis, I can see that you have particular difficulty with security permission settings in cloud computing. Focusing on S3 bucket security, here's specific advice:\n\n1. IAM (Identity and Access Management) basics\n- Set bucket access rights using the \"principle of least privilege\"\n- Grant only the minimum necessary permissions to users and roles\n\n2. Key points for bucket policy configuration\n- Finely control read, write, and delete permissions\n- Restrict access from specific IP addresses\n- Enable encryption\n\n3. Specific configuration examples\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": \"arn:aws:iam::xxxxxxxxxxxx:user/username\"},\n \"Action\": [\n \"s3:GetObject\",\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::your-bucket-name/*\"\n }\n ]\n}\n```\n\nI recommend learning step by step. Start with one setting and gradually deepen your understanding.\n\nPlease let me know if there are additional details or specific questions you'd like to know. We can learn together step by step."
```Now let's look at the CloudWatch dashboard.
Click on GenAI observability on the left tab. Click the tab to select `Bedrock AgentCore`.
A pretty cool dashboard appears.
There are three Views (Agents view, Sessions view, and Traces view), and you can check metrics and behaviors from each perspective.
### Agents view
You can check the behavior of the deployed Agent.
It's designed so you can see at a glance the number of Sessions and Traces, as well as whether there are any errors or throttling.
You can also click on the endpoints that are deployed from the agent list, so click the link when you want to see details on a per-agent basis.
This allows you to see more details on a per-agent basis.
Also, by pressing the Sessions tab and Traces tab, you can check the list of Sessions and Traces within this agent.
I think this is the flow when looking at details on a per-agent basis.
I'll explain Sessions and Traces later.
### Sessions view
You can check the list of Sessions for the agent.
The Session ID with the arbitrary name I created earlier also exists. Let's click on it.
When clicked, you can now see a summary of this Session. You can also see the linked Traces.
Click to see more details.### Traces View
Here you can view all traces in a list. Click on the latest trace.
When you click on it, a graphical dashboard appears. It visualizes the Spans within the Traces.
You can check the elapsed time for each span, which makes it easier to identify bottlenecks!
**Trajectory**
Viewing it as a flow makes it easy to understand. This way you can clearly see the flow of the agent starting up, launching tools, and so on.
The `identity_weak_areas` tool that was executed for weakness analysis is also clearly recorded.
Also, looking at the Spans below, you can easily see which AI took action.
They are arranged in chronological order. You can see logs by opening one of the Events.
Let's look at Event 1.
```json
{
"resource": {
"attributes": {
"deployment.environment.name": "bedrock-agentcore:default",
"aws.local.service": "my-agent.DEFAULT",
"service.name": "my-agent.DEFAULT",
"cloud.region": "us-west-2",
"aws.log.stream.names": "runtime-logs",
"telemetry.sdk.name": "opentelemetry",
"aws.service.type": "gen_ai_agent",
"telemetry.sdk.language": "python",
"cloud.provider": "aws",
"cloud.resource_id": "arn:aws:bedrock-agentcore:us-west-2:xxxxxxxxxxxx:runtime/my-agent-xxxx/runtime-endpoint/DEFAULT:DEFAULT",
"aws.log.group.names": "/aws/bedrock-agentcore/runtimes/my-agent-xxxx-DEFAULT",
"telemetry.sdk.version": "1.33.1",
"cloud.platform": "aws_bedrock_agentcore",
"telemetry.auto.version": "0.11.0-aws"
}
},
"scope": {
"name": "opentelemetry.instrumentation.botocore.bedrock-runtime",
"schemaUrl": "https://opentelemetry.io/schemas/1.30.0"
},
"timeUnixNano": 1756785750184271639,
"observedTimeUnixNano": 1756785750184290707,
"severityNumber": 9,
"severityText": "",
"body": {
"content": [
{
"text": "You are an excellent technical learning assistant.\n You support engineers in technical learning, record their understanding, and suggest effective learning methods.\n \n The following tools are available:\n - analyze_learning_progress: Analyze learning progress (can also specify a particular technical field)\n - identify_weak_areas: Identify weak areas\n - suggest_review_topics: Suggest topics to review\n - get_session_summary: Get a summary of the learning session\n \n Please keep the following points in mind:\n - Include specific examples when answering technical questions\n - Check understanding as you proceed\n - When weak areas are identified, suggest learning methods accordingly\n - Provide encouragement and constructive feedback\n - Use tools as needed to understand the learning situation\n "
}
]
},
"attributes": {
"event.name": "gen_ai.system.message",
"gen_ai.system": "aws.bedrock"
},
"flags": 1,
"traceId": "68b66c556c14723d4f7168399c6e91af",
"spanId": "ad1a0ebc825b849e"
}
```Event1 was a log that conveyed the system prompt to the AI.
Let's look at Event2 as well.
```json
{
"resource": {
"attributes": {
"deployment.environment.name": "bedrock-agentcore:default",
"aws.local.service": "my-agent.DEFAULT",
"service.name": "my-agent.DEFAULT",
"cloud.region": "us-west-2",
"aws.log.stream.names": "runtime-logs",
"telemetry.sdk.name": "opentelemetry",
"aws.service.type": "gen_ai_agent",
"telemetry.sdk.language": "python",
"cloud.provider": "aws",
"cloud.resource_id": "arn:aws:bedrock-agentcore:us-west-2:xxxxxxxxxxxx:runtime/my-agent-xxxx/runtime-endpoint/DEFAULT:DEFAULT",
"aws.log.group.names": "/aws/bedrock-agentcore/runtimes/my-agent-xxxx-DEFAULT",
"telemetry.sdk.version": "1.33.1",
"cloud.platform": "aws_bedrock_agentcore",
"telemetry.auto.version": "0.11.0-aws"
}
},
"scope": {
"name": "opentelemetry.instrumentation.botocore.bedrock-runtime",
"schemaUrl": "https://opentelemetry.io/schemas/1.30.0"
},
"timeUnixNano": 1756779796818481971,
"observedTimeUnixNano": 1756779796818488400,
"severityNumber": 9,
"severityText": "",
"body": {
"content": [
{
"text": "Tell me about S3 bucket security"
}
]
},
"attributes": {
"event.name": "gen_ai.user.message",
"gen_ai.system": "aws.bedrock"
},
"flags": 1,
"traceId": "68b6551285d1b1a532aa55b39fa596eb",
"spanId": "a02614dcd6a51732"
}
It continues with a specific question. It's easy to understand. The history includes previous question history as well.
Event6 contained a question about analyzing weaknesses.
{
"resource": {
"attributes": {
"deployment.environment.name": "bedrock-agentcore:default",
"aws.local.service": "my-agent.DEFAULT",
"service.name": "my-agent.DEFAULT",
"cloud.region": "us-west-2",
"aws.log.stream.names": "runtime-logs",
"telemetry.sdk.name": "opentelemetry",
"aws.service.type": "gen_ai_agent",
"telemetry.sdk.language": "python",
"cloud.provider": "aws",
"cloud.resource_id": "arn:aws:bedrock-agentcore:us-west-2:xxxxxxxxxxxx:runtime/my-agent-xxxx/runtime-endpoint/DEFAULT:DEFAULT",
"aws.log.group.names": "/aws/bedrock-agentcore/runtimes/my-agent-xxxx-DEFAULT",
"telemetry.sdk.version": "1.33.1",
"cloud.platform": "aws_bedrock_agentcore",
"telemetry.auto.version": "0.11.0-aws"
}
},
"scope": {
"name": "opentelemetry.instrumentation.botocore.bedrock-runtime",
"schemaUrl": "https://opentelemetry.io/schemas/1.30.0"
},
"timeUnixNano": 1756779796818529300,
"observedTimeUnixNano": 1756779796818534000,
"severityNumber": 9,
"severityText": "",
"body": {
"content": [
{
"text": "Analyze my weaknesses"
}
]
},
"attributes": {
"event.name": "gen_ai.user.message",
"gen_ai.system": "aws.bedrock"
},
"flags": 1,
"traceId": "68b6551285d1b1a532aa55b39fa596eb",
"spanId": "a02614dcd6a51732"
}
```By tracking this, it's easier to visually investigate what behaviors were performed in the flow.
## Conclusion
We've mainly looked at the dashboard, a feature of Amazon Bedrock AgentCore Observability!
It's convenient that by enabling it with a button and deploying with built-in functionality, the AI agent can be traced using the OpenTelemetry standard, allowing you to visually confirm the AI agent's behavior.
There are many aspects that can be observed, and I'd like to introduce how to use them in actual operations in the future!
I hope this article was helpful to you. Thank you very much for reading until the end!!
