Here is a summary of Amazon CloudWatch metrics monitoring candidates for Amazon Connect two-way SMS operations

Introduction

In Amazon Connect, you can use SMS-enabled numbers obtained through AWS End User Messaging SMS to allow customers and agents to communicate via SMS.

The official documentation describes the process for setting up SMS messaging in Amazon Connect: obtaining an SMS-enabled number with AWS End User Messaging SMS, enabling two-way SMS on that number, and specifying Amazon Connect as the message destination.

https://docs.aws.amazon.com/connect/latest/adminguide/setup-sms-messaging.html

In this article, I have organized candidate metrics for CloudWatch alarm monitoring, assuming a configuration where customers and operators communicate via SMS using Amazon Connect two-way SMS.

Please note that this article reflects content I personally organized while considering monitoring design for Amazon Connect two-way SMS. The actual monitoring items and thresholds you adopt should be adjusted according to your business requirements, SLA, inquiry volume, and operator structure.

This article assumes the following:

• SMS usage is intended for domestic use within Japan
• The SMS sender is assumed to be a short code
• Customers and operators communicate via SMS
• Voice calls are not used
• Monitoring is centered on CloudWatch standard metrics
• CloudWatch alarms are created as needed

Conclusion

When monitoring Amazon Connect two-way SMS with CloudWatch alarms, it is easier to organize the targets by dividing them into the following two main areas.

Amazon Connect side
  ├ Queue backlog
  ├ Customer wait time
  ├ Concurrent chat count
  ├ Chat quota pressure
  ├ Flow errors
  └ Queue capacity exceeded

AWS End User Messaging SMS side
  ├ SMS delivery rate
  ├ Blocks by Protect
  └ SMS costs

The main CloudWatch Namespaces used are as follows.

The monitoring item candidates organized this time are as follows.

In addition, the following service quotas are also kept in mind as a perspective for monitoring with CloudWatch metrics.

The key point is that on the Amazon Connect side, you focus primarily on chat-related metrics, while SMS sending, delivery, costs, and Protect blocks are checked in the AWS/SMSVoice namespace on the AWS End User Messaging SMS side.

Also, rather than looking at NumberOfTextMessagePartsDelivered alone for SMS delivery status, combining it with NumberOfTextMessagePartsSent to view it as a delivery rate makes it easier to determine declines in reachability.

How SMS Is Handled in Amazon Connect

SMS in Amazon Connect is treated as a subtype of chat.

In the Amazon Connect SMS setup procedure as well, to identify SMS contacts, an example is described where you check the Subtype of Segment attributes within a flow and branch on the value connect:SMS.

In Attribute to check section, set Namespace to Segment attributes and key to Subtype.

In the Attribute to check section, set Namespace to Segment attributes and key to Subtype.

https://docs.aws.amazon.com/connect/latest/adminguide/setup-sms-messaging.html

In the Conditions to check section, set condition to Equals and value to connect:SMS.

In the Conditions to check section, set the condition to Equals and the value to connect:SMS.

https://docs.aws.amazon.com/connect/latest/adminguide/setup-sms-messaging.html

Therefore, when monitoring Amazon Connect via CloudWatch, rather than looking at SMS-specific metrics, you will primarily be looking at chat-related metrics.

CloudWatch Standard Metrics Cannot Extract SMS Only

SMS in Amazon Connect is treated as a subtype of chat.

Therefore, when monitoring Amazon Connect via CloudWatch, you will primarily look at metrics in the MetricGroup=Chats category, such as ConcurrentActiveChats and SuccessfulChatsPerInterval.

On the other hand, CloudWatch standard metrics and CloudWatch alarms cannot extract and monitor SMS only.

Amazon Connect CloudWatch metrics provide dimensions such as InstanceId and MetricGroup for chat-related metrics, but there are no dimensions for filtering by chat subtypes such as SMS or Web Chat.

https://docs.aws.amazon.com/ja_jp/connect/latest/adminguide/monitoring-cloudwatch.html

This point is easy to confuse with the Amazon Connect GetMetricDataV2 API.

With the GetMetricDataV2 API, you can filter SMS metrics by specifying connect:SMS for contact/segmentAttributes/connect:Subtype.

https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html

However, the GetMetricDataV2 API is an Amazon Connect metric retrieval API and is a separate mechanism from CloudWatch standard metrics. Therefore, just because you can filter SMS with the GetMetricDataV2 API does not mean you can directly specify the same conditions in a CloudWatch alarm.

To summarize:

If you are currently using a configuration that only uses SMS, the MetricGroup=Chats metrics are easy to treat as SMS monitoring.

On the other hand, if you plan to use chat channels other than SMS in the future, such as Web Chat, WhatsApp, or Apple Messages for Business, values may be mixed in the CloudWatch standard metrics.

If you want to isolate only SMS for CloudWatch alarming, you would consider a configuration where Amazon Connect contact events are received via EventBridge, and Lambda determines only SMS contacts to publish CloudWatch custom metrics.

Amazon Connect contact events include SegmentAttributes. The official documentation states that SegmentAttributes can be used to indicate the channel subtype, with connect:SMS given as an example.

https://docs.aws.amazon.com/ja_jp/connect/latest/adminguide/contact-events.html

Metrics to Monitor on the Amazon Connect Side

On the Amazon Connect side, you primarily look at metrics in the AWS/Connect namespace.

The Amazon Connect CloudWatch metrics documentation describes metrics such as ConcurrentActiveChats, ConcurrentActiveChatsPercentage, ChatsBreachingActiveChatQuota, and SuccessfulChatsPerInterval.

https://docs.aws.amazon.com/connect/latest/adminguide/monitoring-cloudwatch.html

Monitoring Queue Backlog

To check whether SMS inquiries are accumulating in the queue without being assigned to operators, the following are considered as monitoring candidates.

QueueSize is a metric for viewing the number of contacts in the queue. When SMS inquiries surge or when there are insufficient operators, contacts may continue to accumulate in the queue.

LongestQueueWaitTime is a metric for viewing the wait time of the contact that has been waiting the longest in the queue. Although SMS is a more asynchronous channel than phone calls, slow initial responses can affect the customer experience, so wait time is also included as a monitoring target.

QueueCapacityExceededError is a metric for checking the possibility that contacts could not be accepted due to queue capacity being exceeded. When this occurs, check the queue design, routing settings, number of operators, and concurrent handling capacity.

Example alarm conditions:

QueueSize >= 10 continues for 5 minutes

LongestQueueWaitTime >= 300 seconds

QueueCapacityExceededError > 0

Thresholds should be adjusted according to the number of inquiries, number of operators, business hours, and SLA. The above are examples only.

Monitoring Concurrent Chat Count and Quota

Since SMS is treated as a subtype of chat, check the quota for the number of concurrent active chats.

Since SMS is also included in Concurrent active chats per instance, check this when estimating the number of concurrent SMS conversations.

This quota includes chats that are waiting and chats where the customer has been unresponsive for a long time.

This includes SMS, WhatsApp, and Apple Messages for Business. It also includes chats that are waiting.

This includes SMS, WhatsApp, and Apple Messages for Business. It also includes chats that are waiting.

https://docs.aws.amazon.com/connect/latest/adminguide/amazon-connect-service-limits.html

Therefore, in operations where SMS conversations remain for a long time, the number of concurrent active chats may increase beyond expectations.

The service quota related to this monitoring item is Concurrent active chats per instance. By using ConcurrentActiveChatsPercentage and ChatsBreachingActiveChatQuota, you can monitor the quota usage rate and exceedances of concurrent active chats in CloudWatch.

Example alarm conditions:

ConcurrentActiveChatsPercentage >= 80%

ConcurrentActiveChatsPercentage >= 90%

ChatsBreachingActiveChatQuota > 0

ConcurrentActiveChatsPercentage is used for the purpose of detecting quota usage rate early. For example, setting 80% as Warning and 90% as Critical makes it easier to detect before approaching the limit.

ChatsBreachingActiveChatQuota is a metric that shows the number of chat start requests that exceeded the concurrent active chat limit. If even one occurs, check it because there is a possibility that new SMS contacts could not be started.

SuccessfulChatsPerInterval is a metric for viewing trends in the number of successfully started chats. It is convenient to display on a dashboard for the purpose of understanding normal inflow trends, but there may be large variations depending on business hours and day of the week. Therefore, rather than setting a fixed-threshold alarm from the start, it seems better to understand the normal values first and then consider alarm conditions.

Monitoring Flow Errors

If, after receiving an SMS, the flow performs queue routing, automated responses, attribute settings, external integrations, and so on, flow errors should also be included as monitoring targets.

The Amazon Connect documentation describes ContactFlowErrors as the number of times an error branch in a flow was executed, and ContactFlowFatalErrors as the number of times flow execution failed due to a system error.

https://docs.aws.amazon.com/connect/latest/adminguide/monitoring-cloudwatch.html

Example alarm conditions:

ContactFlowErrors > 0

ContactFlowFatalErrors > 0

If these occur in a flow that all incoming SMS must pass through, there is a possibility that customer SMS messages will not be routed correctly. When this happens, check the change history of the relevant flow, the status of external integration destinations, and queue settings.

Metrics to Monitor on the AWS End User Messaging SMS Side

SMS send counts, delivery counts, costs, and Protect blocks are checked not on the Amazon Connect side, but in the AWS/SMSVoice namespace on the AWS End User Messaging SMS side.

The AWS End User Messaging SMS CloudWatch metrics documentation describes metrics such as TextMessageMonthlySpend, NumberOfTextMessagePartsSent, NumberOfTextMessagePartsDelivered, and TextMessagesBlockedByProtect.

https://docs.aws.amazon.com/sms-voice/latest/userguide/monitoring-cloudwatch.html

Monitoring SMS Delivery Rate

To check whether SMS messages are actually being delivered, combine NumberOfTextMessagePartsSent and NumberOfTextMessagePartsDelivered.

NumberOfTextMessagePartsSent is a metric for checking the number of SMS message parts sent.

NumberOfTextMessagePartsDelivered is a metric for checking the number of SMS message parts delivered to recipients.

SMS messages may be split into multiple parts depending on the character count and character type. Therefore, they are treated as message part counts rather than message counts.

With NumberOfTextMessagePartsDelivered alone, it is difficult to determine whether the delivery count is low because there was no sending, or because messages were sent but not delivered. Also, if the volume of sending itself varies by day, it becomes difficult to determine a deterioration in reachability from the absolute number of deliveries alone.

Therefore, for CloudWatch alarms, it seems better to use Metric Math with NumberOfTextMessagePartsSent (the number of SMS message parts sent) and NumberOfTextMessagePartsDelivered (the number of SMS message parts delivered) to view the delivery rate.

Example Metric Math expression:

100 * NumberOfTextMessagePartsDelivered / NumberOfTextMessagePartsSent

The purpose is to monitor the percentage of sent SMS messages that are successfully delivered, and to detect declines in SMS reachability.

Example alarm conditions:

Delivery rate < 95%

Delivery rate < 90%

There are some caveats regarding the SMS delivery rate.

The AWS End User Messaging SMS documentation explains that for multipart messages, the carrier infrastructure may return more than one delivery confirmation for the same message, and NumberOfTextMessagePartsDelivered may exceed NumberOfTextMessagePartsSent.

When a message is split into multiple parts, the carrier infrastructure may return more than one delivery confirmation for the same message.

When a message is split into multiple parts, the carrier infrastructure may return more than one delivery confirmation for the same message.

https://docs.aws.amazon.com/sms-voice/latest/userguide/monitoring-cloudwatch.html

Therefore, it seems better not to treat a delivery rate exceeding 100% as an anomaly in itself, and instead monitor for declines.

Also, there may be a difference in the recording timing of NumberOfTextMessagePartsSent and NumberOfTextMessagePartsDelivered. With a short evaluation period, both metrics may not be available within the same period, potentially resulting in unintended alarms. First, verify the reflection timing with test sends, and then adjust starting with a reasonably long evaluation period such as 15 or 30 minutes.

Monitoring SMS Sending Rate

The service quota related to the sending rate of AWS End User Messaging SMS is Number of SMS messages that can be sent each second, sending rate.

The sending rate is considered not in terms of the "number of messages" of SMS, but in terms of the number of SMS message parts that can be sent per second. This is because SMS messages may be split into multiple parts depending on the length and character type of the body text.

This "number of SMS message parts that can be sent per second" is called MPS, or Message Parts per Second.

When using a short code for domestic use within Japan, the MPS limit for short codes is not published as a fixed value in the AWS official documentation. Since the sending rate for dedicated short codes varies by country or region, the actual limit should be confirmed at the time of short code application or as a configured value after provisioning.

https://docs.aws.amazon.com/sms-voice/latest/userguide/sms-limitations-mps.html

With CloudWatch standard metrics, you can check the actual number of sent message parts using NumberOfTextMessagePartsSent. However, this is not a metric that directly represents the usage rate of the sending rate quota.

Therefore, for the sending rate, the approach will be to approximately confirm it as the number of message parts per evaluation period, after the actual MPS limit becomes known.

For example, if the short code limit is X MPS, the estimate per minute is as follows:

X MPS × 60 seconds = Maximum number of message parts per minute

To detect at 80%:

X MPS × 60 seconds × 0.8

To detect at 90%:

X MPS × 60 seconds × 0.9

However, this is an approximation based on a 1-minute average. It cannot accurately detect instantaneous per-second spikes.

Monitoring Protect Blocks

AWS End User Messaging SMS Protect is a protection feature designed to suppress fraudulent use and unintended sending of SMS.

In Protect configuration, you can set rules by country. For example, you can block sending to specific countries, or filter to reduce the impact of fraudulent traffic such as AIT (Artificially Inflated Traffic).

https://docs.aws.amazon.com/sms-voice/latest/userguide/protect-configuration.html

https://docs.aws.amazon.com/sms-voice/latest/userguide/filter-and-monitor-messages.html

To check whether SMS messages are being blocked by this Protect configuration, the following are considered as monitoring candidates.

Example alarm conditions:

TextMessagesBlockedByProtect > 0

If even one occurrence is detected, check the Protect settings, destination country, and sender settings.

Also, to view the Protect block rate, use the following formula described in the documentation:

100 * AVG(TextMessagesBlockedByProtect)

The AWS End User Messaging SMS documentation explains that TextMessagesBlockedByProtect has a value of 1 when blocked by Protect and 0 when not blocked, and that the percentage of SMS messages blocked by Protect can be determined by 100 * AVG(TextMessagesBlockedByProtect).

The percentage of SMS messages blocked by Protect can be determined by 100 * AVG(TextMessagesBlockedByProtect).

The percentage of SMS messages blocked by Protect can be determined by 100 * AVG(TextMessagesBlockedByProtect).

https://docs.aws.amazon.com/sms-voice/latest/userguide/monitoring-cloudwatch.html

One important point to note here is not to use the following formula:

100 * TextMessagesBlockedByProtect / NumberOfTextMessagePartsSent

NumberOfTextMessagePartsSent is described as not including messages blocked by Protect or service limits. Therefore, it is not suitable as the denominator for calculating the percentage blocked by Protect.

Monitoring SMS Costs

To monitor SMS costs, the following is considered as a monitoring candidate.

TextMessageMonthlySpend is a metric that expresses the amount spent on SMS sending during the current month in US dollars.

https://docs.aws.amazon.com/sms-voice/latest/userguide/monitoring-cloudwatch.html

The service quota related to this monitoring item is Spending threshold.

Spending threshold is the monthly cost limit available for SMS sending. If this limit is reached, SMS sending may be stopped. Therefore, TextMessageMonthlySpend is important not only for monitoring cost increases, but also for detecting when costs are approaching the limit before it impacts SMS sending.

Example alarm conditions:

80% of monthly budget

100% of monthly budget

SMS costs increase with the number of sends, destination countries, and number of message parts. It is a good idea to monitor TextMessageMonthlySpend so that abnormal sending or sudden surges in inquiries can also be detected from a cost perspective.

Since prices and limits may change depending on the account, region, usage, and time period, please check the actual values in Service Quotas, the AWS End User Messaging SMS settings screen, and the official pricing page.

What Is Difficult to See with CloudWatch Standard Metrics Alone

Items such as operator first response time, average response time, per-agent handling status, first-contact resolution rate, and failure reasons at the SMS body level are difficult to directly alarm on with CloudWatch standard metrics alone.

To monitor these items, consider using Amazon Connect real-time metrics, historical metrics, real-time metric rules, contact events, Contact Trace Records (CTR), EventBridge, Lambda, CloudWatch custom metrics, and so on.

Also, if you want to check delivery events or failure reasons at the SMS level, configuring AWS End User Messaging SMS event destinations to output events to CloudWatch Logs, Amazon Data Firehose, Amazon SNS, and similar services is also an option.

Summary of Monitoring Item Candidates

The CloudWatch monitoring item candidates organized this time are summarized as follows.

Items marked with ◯ in Priority alarm are items for which CloudWatch alarming should be considered first.

ConcurrentActiveChats, SuccessfulChatsPerInterval, NumberOfTextMessagePartsSent, NumberOfTextMessagePartsDelivered, and Protect block rate are positioned as items where you first grasp normal trends on a dashboard and then consider alarming as needed.

Conclusion

When monitoring Amazon Connect two-way SMS with CloudWatch, it is easier to organize by separating the Amazon Connect side (queues, concurrent chat counts, flow errors) from the AWS End User Messaging SMS side (delivery rate, Protect blocks, costs).

For SMS delivery status, rather than looking at NumberOfTextMessagePartsDelivered alone, it seems better to view it as a delivery rate using Metric Math. Also, since the sending rate for domestic short codes within Japan is not published as a fixed value, you should confirm the actual limit before considering monitoring conditions.

The actual thresholds and service quotas should be adjusted according to business requirements, SMS body length, short code sending rate, and SMS sending spending threshold.