Check if you are using third-party monitoring tools when CloudWatch GetMetricData API pricing is high
This article was published more than one year ago. Please be aware that the information may be outdated.
This page has been translated by machine translation. View original
Good evening, Chiba (Yuki) here.
I received a request from a customer about CloudWatch fees being consistently high.
After reviewing the costs again, we found that CloudWatch monthly fees remained high, specifically with $0.01 per 1,000 metrics requested using GetMetricData API - Asia Pacific (Tokyo) costing around 200 USD per month.
Display on the pricing page
This item can be confirmed in Cost Explorer as usage type APN1-CW:GMD-Metrics.
Initially, I wanted to investigate who was executing this GetMetricData API through CloudTrail, but calls to this API are not recorded in CloudTrail. (2026 update: As of 2024, this is now possible. More details below.)
In the end, we discovered that these API calls were from a third-party monitoring tool, but I'd like to share the details of what we learned during our investigation.
2026 Update: GetMetricData API can be recorded as CloudTrail data events
In the original article, I emphasized that "GetMetricData API is not recorded as a CloudTrail event." As of June 2024, there was an update allowing these to be recorded as CloudTrail data events.
(For reference, there's also an article introducing analysis methods based on this update.)
Unlike "management events" that are recorded for free by default, you need to explicitly configure recording as "data events," which comes with a cost. Nevertheless, it's good to remember that there are ways to check if necessary.
For the rest of this article, I've only added strikethrough lines to parts stating "not possible." The rest of the content remains unchanged, so please interpret accordingly if there are any inconsistencies in the text.
Summary First
GetMetricDataAPI callsare notrecorded in CloudTrail events- When
GetMetricDatafees are high, they are often caused by third-party monitoring tools - Consider switching to CloudWatch Metric Stream if your tool supports it
GetMetricData API fees are high!
To reiterate, in this case, the CloudWatch GetMetricData API calls were costing about 200 USD/month.
Checking the price breakdown, for the Tokyo region it is:
- $0.01 per 1,000 metrics requested
It's important to note that the basis is not "number of API calls" but rather "number of metrics requested."
With GetMetricData, you can retrieve up to 500 metrics in a single API call. If you execute GetMetricData twice specifying 500 metrics each time, that alone will cost $0.01.
In the example at the beginning, more than 20 million metrics were requested in one month.
GetMetricData API is not recorded as a CloudTrail event
When a lot of costs are incurred from GetMetricData API calls, figuring out who is making these calls isn't straightforward.
In such cases, the first thing that comes to mind is checking CloudTrail event history, but GetMetricData is outside the scope of CloudTrail recording.
You can check which CloudWatch actions are recorded in CloudTrail from the following documentation:
Considering all CloudWatch actions, recorded ones are actually fewer. I wrote about this in the following entry, so please refer to it as well.
High costs from GetMetricData API calls are likely from third-party tools
What can we do if we can't check from CloudTrail? There was a helpful reference for this situation.
In the following AWS forum post, someone had the same issue of "high costs from GetMetricData API calls and wanting to identify the source."
There was an answer from someone at AWS in the forum, which I quote here (machine-translated). (Please refer to the link above for the original text.)
Thank you for posting to the AWS forums. I understand that your GetMetricData charges are very high, and you'd like to know why.
Unfortunately, there are no tools to check the source of GetMetricData because these API calls are not recorded in CloudTrail.
Therefore, to check this, you would need to contact the AWS Technical Support team. They will contact the internal team, who will do a log dive to find the source of these API calls.
However, before doing that, I'd like to inform you that these high charges for API calls are usually caused by third-party tools such as DataDog, Splunk, etc. This is because these third-party tools have a high number of API calls per minute, resulting in high charges. We've seen many cases where DataDog brings very high charges to customers.
So, before raising your concern to the AWS technical team, please make sure you're not using third-party monitoring tools like DataDog.
If you're actually using DataDog, you'll need to contact DataDog support to ask if they can reduce the number of API calls being made.
It seems that third-party tools are often the cause. While it may be possible to identify the caller of GetMetricData API by contacting the technical support team, it's necessary to first check if third-party monitoring tools are being used.
In this case, when we asked the customer "Are you using any third-party monitoring tools?", it turned out they were indeed using one, which pretty much identified the cause at that stage.
Confirming through other approaches
Checking other API calls from CloudTrail
Although we've largely identified the cause, let's continue investigating through other approaches to solidify our understanding.
While GetMetricData is not recorded in CloudTrail, the same executor might be calling other APIs.
By filtering with the event source as CloudWatch service endpoint monitoring.amazonaws.com, we were able to confirm an executor making regular calls to DescribeAlarmHistory.
Since the executor was an IAM role for third-party tool integration, this helped confirm our cause identification.
As an aside, by filtering events with this IAM role (session name) as the username, we could see it performing Get and List operations on various services.
Checking the CallCount metric in AWS/Usage
While we can't identify the executor of API calls, we can check the number of executions from CloudWatch metrics. The relevant metric is CallCount in the AWS/Usage namespace.
In the Japanese management console, it's expressed as "使用" (Usage) group. By filtering with the Service dimension set to CloudWatch, you can check only the API call count for the CloudWatch service.
Looking at the daily call count for the GetMetricData API, it was about 3,700. In our example, over 20 million metrics were requested in a month, so calculating 20 million / (3,700 * 30 days), we can determine that each API call requested about 180 metrics.
Slightly off-topic, but you can also check the count for metrics that charge based on API call counts from here.
Consider switching to CloudWatch Metric Streams
The GetMetricData API we've been looking at was called by third-party tools (polling). From CloudWatch's perspective, it's being pulled. In contrast, there's also a mechanism called CloudWatch Metric Streams that pushes from CloudWatch.
Instead of polling at fixed intervals, it streams when values are updated, keeping latency low. Additionally, the pricing structure is different from GetMetricData API, potentially reducing costs.
- GetMetricData format
- $0.01 per 1,000 metrics requested
- CloudWatch Metric Streams format
- $0.003 per 1,000 metrics updated
- Kinesis Data Firehose usage fee (from $0.036 per GB of ingested data)
- Possibly additional costs (data transfer fees for cross-region or fees for data transformation in Firehose)
Very roughly speaking, if the interval at which the GetMetricData API is called and the interval at which values are updated by CloudWatch Metric Stream are the same, the latter seems cheaper. If the ratio is up to 1:3, there still might be benefits to the latter.
These intervals likely depend on the specifications of the third-party tool, so I recommend checking before adopting.
Conclusion
When CloudWatch GetMetricData API fees are high, it's likely caused by third-party monitoring tools, so it's worth checking.
Not being able to identify the caller in CloudTrail is a stumbling point, but it's good to investigate assuming monitoring tools are the cause, alongside other approaches. Of course, it could also be caused by custom scripts rather than third-party tools.
Although not covered here, approaches such as checking the last activity of IAM users or IAM roles to identify those with regular activity, or narrowing down those accessing the Amazon CloudWatch service from Access Advisor, could also be effective.
I hope this helps if you find your CloudWatch fees are high.
That's all from Chiba Yuki (@batchicchi).
