I tried analyzing Amazon Q's chat logs and feedback logs with CloudWatch Logs Analytics for usage insights and agent improvement hints

I am Ishikawa from the Cloud Business Division. In Amazon Quick, you can deliver AI assistant chat conversations and user feedback to CloudWatch Logs. This time, we will walk through the entire process from log delivery configuration to analysis using CloudWatch Logs Analytics, based on the improvement scenario of "understanding who is using the chat agent and how, and using that to improve custom agents and consider new agent candidates".

https://docs.aws.amazon.com/quick/latest/userguide/monitoring-quicksuite-chat-feedback-cloudwatch.html

When you deploy Quick Chat from Amazon Quick to your organization, as an administrator you start to wonder about the following:

• Who is using which agent, and how much
• Whether user questions are being answered properly (what questions are going unanswered)
• Whether users are satisfied with the answers (what are the reasons for "Not Useful" ratings)
• Based on user question trends, what custom agent should be built next

Amazon Quick has vended logs (a mechanism where the AWS service itself generates logs and automatically delivers them to the destination, without the user needing to implement a log output mechanism), and can deliver logs for chat conversations, user feedback, agent/research time, index storage usage, and knowledge base file sync status to CloudWatch Logs, Amazon S3, or Amazon Data Firehose (standard rate charges apply for delivery).

In this article, we actually try delivering chat conversations (CHAT_LOGS) and feedback (FEEDBACK_LOGS) among these to CloudWatch Logs, and then analyzing the four perspectives mentioned at the beginning using Logs Insights.

For index storage usage (INDEX_USAGE_LOGS), please refer to the following article.

https://dev.classmethod.jp/articles/quick-vendedlogs-index-usage-logs/

What is Amazon Quick Chat and Feedback Log Delivery

Amazon Quick log delivery is configured using the CloudWatch Logs delivery API (PutDeliverySource / PutDeliveryDestination / CreateDelivery). The following 5 types can be specified for logType.

The prerequisites are as follows.

• Amazon Quick instance with Enterprise or Professional subscription
• IAM permissions to configure log delivery (such as quicksight:AllowVendedLogDeliveryForResource)
• Delivery destination (one of CloudWatch Logs log group, S3 bucket, or Firehose stream)

As a note, chat conversation logs record the question text entered by the user and the AI's response text as-is. Since there is a possibility that confidential or personal information may be included, the official documentation recommends using CloudWatch Logs masking policies (data protection policies). We recommend designing this together with control of IAM principals who can view the logs.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html

Trying It Out

Prerequisites

• Verification region: ap-northeast-1 (Tokyo)
• Amazon Quick account: Enterprise edition (region is Tokyo)

The following 4 objectives were set as scenarios.

1. Understanding who is using which agent and how
2. Quantitative understanding of usage (usage volume, success rate, latency)
3. Improving custom agents (analyzing reasons for low ratings)
4. Considering new agent candidates (extracting questions that could not be answered)

1. Checking the Delivery Configuration Template

First, use the DescribeConfigurationTemplates API to check the log delivery template for the quicksight service. For each logType, you can check the "fields included by default" and the "fields that can be specified".

% aws logs describe-configuration-templates \
    --service quicksight \
    --region ap-northeast-1
{
    "configurationTemplates": [
        ...
        {
            "service": "quicksight",
            "logType": "CHAT_LOGS",
            "resourceType": "account",
            "deliveryDestinationType": "CWL",
            "defaultDeliveryConfigValues": {
                "recordFields": [
                    "resource_arn",
                    "event_timestamp",
                    "logType",
                    "accountId",
                    "user_arn",
                    "user_type",
                    "status_code",
                    "conversation_id",
                    "system_message_id",
                    "message_scope",
                    "user_message_id",
                    "user_message",
                    "agent_id",
                    "flow_id",
                    "system_text_message",
                    "user_selected_resources",
                    "action_connectors",
                    "cited_resource",
                    "file_attachment"
                ],
                "fieldDelimiter": ""
            },
            "allowedFields": [
                {
                    "name": "timestamp",
                    "mandatory": false
                },
                {
                    "name": "resource_arn",
                    "mandatory": true
                },
                {
                    "name": "resource_id",
                    "mandatory": false
                },
                {
                    "name": "event_timestamp",
                    "mandatory": true
                },
                {
                    "name": "logType",
                    "mandatory": true
                },
                {
                    "name": "accountId",
                    "mandatory": true
                },
                {
                    "name": "user_arn",
                    "mandatory": false
                },
                {
                    "name": "user_type",
                    "mandatory": false
                },
                {
                    "name": "namespace",
                    "mandatory": false
                },
                {
                    "name": "status_code",
                    "mandatory": false
                },
                {
                    "name": "conversation_id",
                    "mandatory": false
                },
                {
                    "name": "system_message_id",
                    "mandatory": false
                },
                {
                    "name": "latency",
                    "mandatory": false
                },
                {
                    "name": "time_to_first_token",
                    "mandatory": false
                },
                {
                    "name": "message_scope",
                    "mandatory": false
                },
                {
                    "name": "user_message_id",
                    "mandatory": false
                },
                {
                    "name": "user_message",
                    "mandatory": false
                },
                {
                    "name": "agent_id",
                    "mandatory": false
                },
                {
                    "name": "flow_id",
                    "mandatory": false
                },
                {
                    "name": "system_text_message",
                    "mandatory": false
                },
                {
                    "name": "surface_type",
                    "mandatory": false
                },
                {
                    "name": "web_search",
                    "mandatory": false
                },
                {
                    "name": "user_selected_resources",
                    "mandatory": false
                },
                {
                    "name": "action_connectors",
                    "mandatory": false
                },
                {
                    "name": "cited_resource",
                    "mandatory": false
                },
                {
                    "name": "file_attachment",
                    "mandatory": false
                }
            ],
            "allowedOutputFormats": [
                "plain",
                "json"
            ],
            "allowedActionForAllowVendedLogsDeliveryForResource": "quicksight:AllowVendedLogDeliveryForResource",
            "allowedFieldDelimiters": [
                "\t",
                " ",
                ","
            ]
        },
        ...
    ]
}

Note that the 5 fields latency, time_to_first_token, namespace, surface_type, and web_search above are included in allowedFields, but are not included in the default recordFields. If you want to perform latency analysis, you need to explicitly specify them in the CreateDelivery described later.

2. Creating the Log Group

Create the log group that will serve as the delivery destination. Since accumulating conversation logs indefinitely poses risks from both cost and information management perspectives, we also set a retention period. If you start the log group name with the /aws/vendedlogs/ prefix, you can configure delivery settings without worrying about resource policy size limits.

% aws logs create-log-group \
    --log-group-name /aws/vendedlogs/quick/chat \
    --region ap-northeast-1

% aws logs put-retention-policy \
    --log-group-name /aws/vendedlogs/quick/chat \
    --retention-in-days 30 \
    --region ap-northeast-1

Similarly, create /aws/vendedlogs/quick/feedback for feedback logs.

% aws logs create-log-group \
    --log-group-name /aws/vendedlogs/quick/feedback \
    --region ap-northeast-1

% aws logs put-retention-policy \
    --log-group-name /aws/vendedlogs/quick/feedback \
    --retention-in-days 30 \
    --region ap-northeast-1

3. Creating the Delivery Source

Create delivery sources for CHAT_LOGS and FEEDBACK_LOGS respectively.

% aws logs put-delivery-source \
    --name quick-chat-logs-source \
    --resource-arn "arn:aws:quicksight:ap-northeast-1:123456789012:account/123456789012" \
    --log-type CHAT_LOGS \
    --region ap-northeast-1
{
    "deliverySource": {
        "name": "quick-chat-logs-source",
        "arn": "arn:aws:logs:ap-northeast-1:123456789012:delivery-source:quick-chat-logs-source",
        "resourceArns": [
            "arn:aws:quicksight:ap-northeast-1:123456789012:account/123456789012"
        ],
        "service": "quicksight",
        "logType": "CHAT_LOGS"
    }
}

% aws logs put-delivery-source \
    --name quick-feedback-logs-source \
    --resource-arn "arn:aws:quicksight:ap-northeast-1:123456789012:account/123456789012" \
    --log-type FEEDBACK_LOGS \
    --region ap-northeast-1
{
    "deliverySource": {
        "name": "quick-feedback-logs-source",
        "arn": "arn:aws:logs:ap-northeast-1:123456789012:delivery-source:quick-feedback-logs-source",
        "resourceArns": [
            "arn:aws:quicksight:ap-northeast-1:123456789012:account/123456789012"
        ],
        "service": "quicksight",
        "logType": "FEEDBACK_LOGS"
    }
}

The official documentation states that "logs become available in the region specified as the resource ARN in PutDeliverySource". In the author's account, a CHAT_LOGS delivery configuration was initially created in the Northern Virginia (us-east-1) region, but no logs arrived even after chatting in Quick in the Tokyo region. After recreating with the resource ARN region matching the identity region of the Quick account (Tokyo in this case), logs started arriving. Please create the delivery source in the identity region of your Quick account.

4. Creating the Delivery Destination

Register the delivery destination (log group) as a delivery destination.

% aws logs put-delivery-destination \
    --name quick-chat-logs-dest \
    --delivery-destination-configuration \
    "destinationResourceArn=arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/vendedlogs/quick/chat:*" \
    --region ap-northeast-1

% aws logs put-delivery-destination \
    --name quick-feedback-logs-dest \
    --delivery-destination-configuration \
    "destinationResourceArn=arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/vendedlogs/quick/chat:*" \
    --region ap-northeast-1

5. Creating the Delivery (Explicitly Specifying Optional Fields)

Create a delivery that links the delivery source and delivery destination. As confirmed in step 1, latency-related fields are not included by default, so we explicitly specify all fields with --record-fields.

aws logs create-delivery \
    --delivery-source-name quick-chat-logs-source \
    --delivery-destination-arn "arn:aws:logs:ap-northeast-1:123456789012:delivery-destination:quick-chat-logs-dest" \
    --record-fields resource_arn event_timestamp logType accountId user_arn \
      user_type namespace status_code conversation_id system_message_id \
      latency time_to_first_token message_scope user_message_id user_message \
      agent_id flow_id system_text_message surface_type web_search \
      user_selected_resources action_connectors cited_resource file_attachment \
    --region ap-northeast-1

{
    "delivery": {
        "id": "Y03kIsmF1wiUcd5D",
        "arn": "arn:aws:logs:ap-northeast-1:123456789012:delivery:Y03kIsmF1wiUcd5D",
        "deliverySourceName": "quick-chat-logs-source",
        "deliveryDestinationArn": "arn:aws:logs:ap-northeast-1:123456789012:delivery-destination:quick-chat-logs-dest",
        "deliveryDestinationType": "CWL",
        "recordFields": [
            "resource_arn", "event_timestamp", "logType", "accountId",
            "user_arn", "user_type", "namespace", "status_code",
            "conversation_id", "system_message_id", "latency",
            "time_to_first_token", "message_scope", "user_message_id",
            "user_message", "agent_id", "flow_id", "system_text_message",
            "surface_type", "web_search", "user_selected_resources",
            "action_connectors", "cited_resource", "file_attachment"
        ]
    }
}

Create the FEEDBACK_LOGS side in the same way (with namespace additionally specified).

aws logs create-delivery \
    --delivery-source-name quick-feedback-logs-source \
    --delivery-destination-arn "arn:aws:logs:ap-northeast-1:123456789012:delivery-destination:quick-feedback-logs-dest" \
    --record-fields resource_arn event_timestamp logType accountId user_arn \
      user_type namespace status_code conversation_id research_id \
      system_message_id user_message_id feedback_type feedback_reason \
      feedback_details rating \
    --region ap-northeast-1

Finally, confirm the delivery configuration with DescribeDeliveries.

aws logs describe-deliveries --region ap-northeast-1

6. Trying a Chat in Quick

Now that the configuration is complete, let's actually have a conversation in Quick chat. To make the results "look good" in the analysis section, we intentionally mixed the following patterns for a total of 11 messages and 2 conversations.

Note that the content entered in the chat is recorded directly in the logs, so during verification as well, we are careful not to enter confidential information or actual personal information, just as in production operations.

7. Confirming Log Delivery

After the chat, in the author's environment, feedback logs arrived in the log group in about 2 minutes, and chat logs arrived in about 3 minutes. Let's look at one of the delivered chat logs.

aws logs filter-log-events \
    --log-group-name /aws/vendedlogs/quick/chat \
    --region ap-northeast-1 \
    --max-items 1

% aws logs filter-log-events \
    --log-group-name /aws/vendedlogs/quick/chat \
    --region ap-northeast-1 \
    --max-items 1
{
    "events": [
        {
            "logStreamName": "QUICK_SUITE_CHAT_LOGS",
            "timestamp": 1783787914186,
            "message": "{\"user_arn\":\"arn:aws:quicksight:ap-northeast-1:123456789012:user/default/cm-ishikawa.satoru/cm-ishikawa.satoru\",\"user_type\":\"ADMIN_PRO\",\"namespace\":\"default\",\"status_code\":\"success\",\"conversation_id\":\"979613ec-65f0-40c7-a838-d45d371c62d8\",\"system_message_id\":\"6e06e65d-d6f6-4bbf-917d-fbc433260cf0\",\"latency\":61124,\"time_to_first_token\":61124,\"message_scope\":\"all_resources\",\"user_message_id\":\"ff05e8da-1dea-492c-856e-24b2b19da523\",\"user_message\":\"Superstore Performance Analytics Suite ダッシュボードから、売上のハイライトを教えて\",\"agent_id\":\"SYSTEM\",\"flow_id\":\"-\",\"system_text_message\":\"「Superstore Performance Analytics Suite」ダッシュボードの **Executive Overview** シートから、売上のハイライトをまとめます。\\n\\n## 主要KPI\\n\\n| 指標 | 値 |\\n|------|-----|\\n| 総売上 | 約 2億2,718万円 |\\n| 総利益 | 約 2,739万円 |\\n| 総オーダー数 | 2,818件 |\\n| 平均割引率 | 約 10.8% |\\n\\n## 地域別売上\\n\\n| 地域 | 売上 |\\n|------|------|\\n| 関西地方 | 約 5,690万円 |\\n| 関東地方 | 約 4,852万円 |\\n| 中部地方 | 約 4,290万円 |\\n| 九州 | 約 2,798万円 |\\n| 中国地方 | 約 1,947
万円 |\\n| 東北地方 | 約 1,698万円 |\\n| 四国 | 約 977万円 |\\n| 北海道 | 約 465万円 |\\n\\n**関西地方**が最大の売上を誇り、全体の約25%を占めています。\\n\\n## 顧客区分別売上\\n\\n| 顧客区分 | 売上 |\\n|----------|------|\\n| 消費
者 | 約 1億1,419万円（約50%） |\\n| 大企業 | 約 7,059万円（約31%） |\\n| 小規模事業所 | 約 4,239万円（約19%） |\\n\\n**消費者**セグメントが売上の約半分を
占める最大の顧客区分です。\\n\\n## カテゴリ別売上（顧客区分別）\\n\\n全顧客区分を通じて、**家具・家電・事務用品**の3カテゴリが均衡しており、消費者向けの家具（約4,170万円）が最も高い売上を記録しています。\\n\\n## サブカテゴリ別利益 (上位・下位)\\n\\n- **利益トップ3**: 椅子（約702万円）、コピー機（約492万円）、本棚（約488万円）\\n- **唯一の赤字**: テーブル（約 -318万円）がマイナス利益となっており、注意が必要です。\\n\\n---\\n\\n詳細な分析が必要な場合は、以下のオプションをご検討ください。\\n\\n\",\"surface_type\":\"WEB_EXPERIENCE\",\"web_search\":true,\"user_selected_resources\":[{\"resourceId\":\"ALL\",\"resourceType\":\"space\"}],\"action_connectors\":[{\"actionConnectorId\":\"quicksuite-documentation\"},{\"actionConnectorId\":\"quicksuite-websearch\"}],\"cited_resource\":[],\"file_attachment\":[],\"resource_arn\":\"arn:aws:quicksight:ap-northeast-1:123456789012:account/123456789012\",\"event_timestamp\":1783787914186,\"logType\":\"CHAT_LOGS\",\"accountId\":\"123456789012\"}",
            "ingestionTime": 1783788030550,
            "eventId": "39779799762480394727843288965236226612872707756014698496"
        }
    ],
    "searchedLogStreams": [],
    "NextToken": "eyJuZXh0VG9rZW4iOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAxfQ=="
}

In addition to the question text (user_message) and the full response text (system_text_message), user_arn, conversation_id, latency, message_scope, and whether web_search was used are all recorded. This is data that allows you to trace everything about who asked what, when, and what was returned.

Worth noting is that even for questions that were answered with "I couldn't find that", the status_code is success. It seems that when the agent returns a polite response as a conversation, it is treated as a success and does not become no_answer_found. This point is addressed with text matching workarounds in the analysis described later.

Let's also look at one feedback log.

{
    "logStreamName": "QUICK_SUITE_FEEDBACK_LOGS",
    "timestamp": 1783788302000,
    "message": "{\"user_arn\":\"arn:aws:quicksight:ap-northeast-1:123456789012:user/default/cm-ishikawa.satoru/cm-ishikawa.satoru\",\"user_type\":\"ADMIN_PRO\",\"namespace\":\"default\",\"status_code\":\"success\",\"conversation_id\":\"979613ec-65f0-40c7-a838-d45d371c62d8\",\"research_id\":\"-\",\"system_message_id\":\"de34b8df-227f-4b53-a75d-e8e15ab273f5\",\"user_message_id\":\"\",\"feedback_type\":\"Not Useful\",\"feedback_reason\":\"Incomplete answer\",\"feedback_details\":\"返品率を聞いたのに返品件数で回答された。率での分析と簡潔なまとめが欲しい。\",\"rating\":\"-\",\"resource_arn\":\"arn:aws:quicksight:ap-northeast-1:123456789012:account/123456789012\",\"event_timestamp\":1783788302000,\"logType\":\"FEEDBACK_LOGS\",\"accountId\":\"123456789012\"}",
    "ingestionTime": 1783788424595,
    "eventId": "39779808411021593150798372453140582184638924035319463936"
}

The reason selected in the UI (incomplete answer) was recorded as the English code Incomplete answer, and the free-text comment was recorded as-is in Japanese in feedback_details. Since it can be joined to chat logs via conversation_id, you can trace back to "what actually happened in the conversation that received a low rating".

8. Analyzing with CloudWatch Logs Analytics

Now for the main topic: analysis. We will run Logs Insights queries in line with the 4 objectives mentioned at the beginning.

Objective 1: Understanding who is using which agent and how

Target log group: /aws/vendedlogs/quick/chat

SOURCE "arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/vendedlogs/quick/chat" START=-604800s END=0s |
fields @timestamp
| stats count(*) as messages, count_distinct(conversation_id) as conversations,
        avg(latency) as avg_latency_ms
  by user_arn, agent_id
| sort messages desc

Usage volume by user and by agent is visible at a glance. From these results, you can judge "whether usage is concentrated among certain users" and "whether there are departments that were onboarded but are not using it", and link this to deployment measures for non-using departments. Note that chats using the default agent (My Assistant) were recorded as SYSTEM for agent_id. If you want to analyze by custom agent, the prerequisite is an operation where different agents are created and used according to purpose.

Objective 2: Quantitative Understanding of Usage

Target log group: /aws/vendedlogs/quick/chat

Aggregating counts by status × scope

SOURCE "arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/vendedlogs/quick/chat" START=-604800s END=0s |
stats count(*) as cnt by status_code, message_scope
| sort cnt desc

Aggregating latency distribution

SOURCE "arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/vendedlogs/quick/chat" START=-604800s END=0s |
filter ispresent(latency)
| stats count(*) as cnt, avg(latency) as avg_ms,
        pct(latency, 50) as p50_ms, pct(latency, 90) as p90_ms, max(latency) as max_ms

In this verification, the p50 was approximately 51 seconds and p90 was approximately 80 seconds. For questions involving data retrieval and aggregation from dashboards, there were cases exceeding 1.5 minutes, so continuous monitoring of latency is effective for measuring the user experience. Time-series trends can be aggregated by adding by bin(1h) as hour.

Note that the actual values of message_scope were all_resources / specific_resource (singular). Since the documentation examples show specific_resources, we recommend checking the actual log values when using this as a query condition.

Objective 3: Improving Custom Agents

Target log group: /aws/vendedlogs/quick/feedback

Total counts by feedback_type and feedback_reason

Check the total counts by feedback_type and feedback_reason.

SOURCE "arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/vendedlogs/quick/feedback" START=-604800s END=0s |
stats count(*) as cnt by feedback_type, feedback_reason
| sort cnt desc

Detailed comments for Not Useful

Additionally, list the detailed comments where feedback_type is Not Useful.

SOURCE "arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/vendedlogs/quick/feedback" START=-604800s END=0s |
filter feedback_type = 'Not Useful'
| fields @timestamp, feedback_reason, feedback_details, conversation_id
| sort @timestamp desc

The output of this query can be used directly as an issue list for agent improvements. You can tailor your response approach for each reason code — for example, "Many Incomplete answers → knowledge gaps or misinterpretation of the question's intent" or "Many Too wordy → revise the agent's response policy (prompt) to be more concise."

Furthermore, you can expand the logs to view more details.

Objective 4: Considering New Agent Candidates

As mentioned earlier, since questions that could not be answered still result in a status_code of success, we extract them using pattern matching on the response text.

SOURCE "arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/vendedlogs/quick/chat" START=-604800s END=0s |
filter system_text_message like /見つかりませんでした|お伝えすることはできません|確認することができません/
| fields user_message, substr(system_text_message, 0, 60) as answer_head
| sort @timestamp asc

5 out of 52 cases were extracted as "unanswered questions." Looking at the breakdown, questions about expense reimbursement were repeated 3 times with different phrasings. If a similar trend is observed across multiple users, it becomes possible to make decisions like "create an expense reimbursement agent by incorporating expense policy documents into a Space" — backed by quantitative evidence from the logs. This was truly an experience of "looking at the logs reveals what agent to build next."

Observations

Here is a summary of the insights gained from actually trying this out.

• Detecting unanswered questions requires more than just status_code: Even when an agent politely responded with "not found" to a question outside its knowledge base, the status_code was still success. To extract unanswered questions, you need to combine pattern matching on the response text (in this case, regular expressions). Since this depends on the agent's response style, it is practical to refine the patterns as you operate.
• Latency analysis requires explicitly specifying fields: latency / time_to_first_token are not included in the default recordFields. If you forget to specify them at CreateDelivery time, you won't be able to retrieve past data when you need it later, so it is recommended to specify all fields from the start.
• Create the delivery source in the identity region: If you use the wrong region, it won't result in an error — instead, it becomes a silent failure where "logs don't arrive," so caution is required.
• Be mindful of personal information contained in conversation logs: Individuals can be identified via user_arn, and question and response text are recorded as-is. Design access control for the log group, retention periods, and masking policies together. Notifying users (that their conversations are being recorded) is also important from an organizational governance perspective. Note that the chat UI also had a private mode toggle (a mode that does not update memory or history).
• Feedback requires deliberate collection practices: Analysis of FEEDBACK_LOGS only works when users actually press the thumbs up/down buttons. If you inform users at rollout to "please give a low rating with a reason if it wasn't helpful," the improvement cycle becomes easier to sustain.
• For production use, consider S3 + Athena / Quick dashboard: This time we completed everything with Logs Insights alone, but if long-term storage, sharing with departments, or joining with other business data becomes necessary, you can evolve the setup by switching (or adding) the delivery destination to S3 and visualizing with Athena or Quick's own dashboards.

Closing

We delivered Amazon Q's chat and feedback logs to CloudWatch Logs and analyzed them with Logs Insights. Without building any additional analytics infrastructure, you can get an end-to-end picture of "who is using it and how," "what questions are going unanswered," "what are the reasons for low ratings," and "what agent should be built next."

Deploying an AI assistant is not "done once it's rolled out" — it only takes hold when you run an improvement cycle based on usage logs. Once you enable Amazon Q's AI features, we recommend setting up log delivery at an early stage. We hope this article proves useful to someone.