New Relicでクラスメソッドメンバーズ用のAWSコストと使用状況レポート(MCUR)を可視化してみた

こんにちは、ゲームソリューション部のsoraです。
今回は、New Relicでクラスメソッドメンバーズ用のAWSコストと使用状況レポート(MCUR)を可視化してみたことについて書いていきます。

前回以下記事に記載した通り、CURについてはNew Relicでグラフ化したものの、MCURでは取得できる情報が異なり設定が異なる部分があるため、重複する箇所は多いものの書いていきます。

構成

クラスメソッドメンバーズのユーザガイド記載の手順に従い、CloudFormationでS3バケットを作成してMCURをS3に発行します。
S3バケットへのレポート発行をトリガーとしてLambda関数が起動し、New Relicへレポートデータを送ります。
New Relic側では、Parsing Ruleによりレポートデータのmessageを変換して可視化します。

MCUR配置用のS3の作成

クラスメソッドメンバーズのユーザガイド記載の設定手順に従い、AWSマネジメントコンソールにログイン後にCloudFormationテンプレートからスタックを作成します。
スタックの作成が完了すると、名前がcm-cur-[アカウントID]のS3バケットが作成されています。

New RelicのAPI Keyの作成

New Relic転送用のLambda関数で使用するNew RelicのAPIキーを作成します。
New Relicの自分の名前のところからAPI Keys＞Create a keyで、Key TypeをIngest - Licenseで作成します。
作成後はCopy keyでkey valueをコピーしておきます。

New Relic転送用のLambda関数の作成

New Relic転送用のLambda関数の作成します。
Lambda＞アプリケーションでアプリケーションを作成します。
サーバレスアプリケーションで[カスタムIAMロールまたはリソースポリシーを作成するアプリを表示する]にチェックを入れて検索し、NewRelic-log-ingestion-s3を選択します。
設定のNRLicenseKeyに先ほど作成したNew RelicのAPI Key valueを入力します。
次にS3へのオブジェクト配置をトリガーとするため、トリガー追加を行います。
バケットにはCURの出力先であるバケットを指定します。
これでAWS側での準備は完了です。

New RelicでのParsing Ruleの作成

ここからNew Relic側での設定を行います。
Logs＞ParsingでAWSから受け取ったレポートを変換するルールを作成します。
Filter logs based on NRQLにバケット名を入力します。（`aws.s3_bucket_name` = 'cm-cur-[アカウントID]'）
※New Relic公式手順のバケット名を囲む記号が誤っているのか変換が上手くいかずに私はここで詰まりました。‘Amazon S3バケット名’ではなく'Amazon S3バケット名'で入力しました。 MCURをNew Relicに配信する前であり、New Relic側にMCURのログがなかったため、Choose your matching logのPaste logに以下を入力しました。

"identity/LineItemId","identity/TimeInterval","bill/BillingEntity","bill/BillType","bill/PayerAccountId","bill/BillingPeriodStartDate","bill/BillingPeriodEndDate","bill/InvoicingEntity","lineItem/UsageAccountId","lineItem/LineItemType","lineItem/UsageStartDate","lineItem/UsageEndDate","lineItem/ProductCode","lineItem/UsageType","lineItem/Operation","lineItem/AvailabilityZone","lineItem/ResourceId","lineItem/UsageAmount","lineItem/NormalizationFactor","lineItem/NormalizedUsageAmount","lineItem/CurrencyCode","lineItem/UnblendedRate","lineItem/UnblendedCost","lineItem/LineItemDescription","lineItem/TaxType","lineItem/LegalEntity","product/ProductName","product/acceleratorSize","product/acceleratorType","product/accessType","product/activityType","product/addonFeature","product/alarmType","product/apiType","product/attachmentType","product/availability","product/availabilityZone","product/bitRate","product/brokerEngine","product/bundle","product/cacheEngine","product/cacheMemorySizeGb","product/callingType","product/capacitystatus","product/category","product/clientLocation","product/clockSpeed","product/cloudSearchVersion","product/codec","product/computeFamily","product/computeType","product/concurrencyscalingfreeusage","product/contentType","product/country","product/countsAgainstQuota","product/cputype","product/currentGeneration","product/data","product/dataTransfer","product/dataTransferQuota","product/databaseEdition","product/databaseEngine","product/datatransferout","product/dedicatedEbsThroughput","product/deploymentLocation","product/deploymentOption","product/describes","product/description","product/device","product/deviceType","product/directConnectLocation","product/directorySize","product/directoryType","product/directoryTypeDescription","product/dominantnondominant","product/durability","product/ebsOptimized","product/ecu","product/edition","product/elasticGraphicsType","product/endpoint","product/endpointType","product/engine","product/engineCode","product/enhancedNetworkingSupport","product/enhancedNetworkingSupported","product/entityType","product/eventType","product/executionFrequency","product/executionLocation","product/feeCode","product/feeDescription","product/fileSystemType","product/frameRate","product/freeOverage","product/freeQueryTypes","product/freeTier","product/freeTrial","product/freeUsageIncluded","product/frequencyMode","product/fromLocation","product/fromLocationType","product/georegioncode","product/gets","product/gpu","product/gpuMemory","product/graphqloperation","product/group","product/groupDescription","product/highAvailability","product/indexingSource","product/ingestType","product/input","product/inputMode","product/instance","product/instanceCapacity10xlarge","product/instanceCapacity12xlarge","product/instanceCapacity24xlarge","product/instanceCapacity2xlarge","product/instanceCapacity4xlarge","product/instanceCapacity8xlarge","product/instanceCapacityLarge","product/instanceCapacityXlarge","product/instanceFamily","product/instanceFunction","product/instanceType","product/instanceTypeFamily","product/instances","product/instancesku","product/intelAvx2Available","product/intelAvxAvailable","product/intelTurboAvailable","product/io","product/license","product/licenseModel","product/licenseType","product/lineType","product/location","product/locationType","product/logsSource","product/logsType","product/machineLearningProcess","reservation/ModificationStatus","reservation/NormalizedUnitsPerReservation","reservation/NumberOfReservations","reservation/RecurringFeeForUsage","reservation/ReservationARN","reservation/StartTime","reservation/TotalReservedNormalizedUnits","reservation/TotalReservedUnits","reservation/UnitsPerReservation","reservation/UnusedAmortizedUpfrontFeeForBillingPeriod","reservation/UnusedNormalizedUnitQuantity","reservation/UnusedQuantity","reservation/UnusedRecurringFee","reservation/UpfrontValue","resourceTags/user:CmBillingGroup","savingsPlan/TotalCommitmentToDate","savingsPlan/SavingsPlanARN","savingsPlan/SavingsPlanRate","savingsPlan/UsedCommitment","savingsPlan/SavingsPlanEffectiveCost","savingsPlan/AmortizedUpfrontCommitmentForBillingPeriod","savingsPlan/RecurringCommitmentForBillingPeriod","savingsPlan/Region","savingsPlan/PaymentOption","savingsPlan/EndTime","savingsPlan/InstanceTypeFamily","savingsPlan/PurchaseTerm","savingsPlan/OfferingType","savingsPlan/StartTime"

私はParsing rule and outputには以下を入力します。
※MCURではCURより取得できる項目が少ないため、ここがCURのときとは異なっており、注意するポイントとなります。

%{GREEDYDATA:log:csv({"columns": ["identity/LineItemId","identity/TimeInterval","bill/BillingEntity","bill/BillType","bill/PayerAccountId","bill/BillingPeriodStartDate","bill/BillingPeriodEndDate","bill/InvoicingEntity","lineItem/UsageAccountId","lineItem/LineItemType","lineItem/UsageStartDate","lineItem/UsageEndDate","lineItem/ProductCode","lineItem/UsageType","lineItem/Operation","lineItem/AvailabilityZone","lineItem/ResourceId","lineItem/UsageAmount","lineItem/NormalizationFactor","lineItem/NormalizedUsageAmount","lineItem/CurrencyCode","lineItem/UnblendedRate","lineItem/UnblendedCost","lineItem/LineItemDescription","lineItem/TaxType","lineItem/LegalEntity","product/ProductName","product/acceleratorSize","product/acceleratorType","product/accessType","product/activityType","product/addonFeature","product/alarmType","product/apiType","product/attachmentType","product/availability","product/availabilityZone","product/bitRate","product/brokerEngine","product/bundle","product/cacheEngine","product/cacheMemorySizeGb","product/callingType","product/capacitystatus","product/category","product/clientLocation","product/clockSpeed","product/cloudSearchVersion","product/codec","product/computeFamily","product/computeType","product/concurrencyscalingfreeusage","product/contentType","product/country","product/countsAgainstQuota","product/cputype","product/currentGeneration","product/data","product/dataTransfer","product/dataTransferQuota","product/databaseEdition","product/databaseEngine","product/datatransferout","product/dedicatedEbsThroughput","product/deploymentLocation","product/deploymentOption","product/describes","product/description","product/device","product/deviceType","product/directConnectLocation","product/directorySize","product/directoryType","product/directoryTypeDescription","product/dominantnondominant","product/durability","product/ebsOptimized","product/ecu","product/edition","product/elasticGraphicsType","product/endpoint","product/endpointType","product/engine","product/engineCode","product/enhancedNetworkingSupport","product/enhancedNetworkingSupported","product/entityType","product/eventType","product/executionFrequency","product/executionLocation","product/feeCode","product/feeDescription","product/fileSystemType","product/frameRate","product/freeOverage","product/freeQueryTypes","product/freeTier","product/freeTrial","product/freeUsageIncluded","product/frequencyMode","product/fromLocation","product/fromLocationType","product/georegioncode","product/gets","product/gpu","product/gpuMemory","product/graphqloperation","product/group","product/groupDescription","product/highAvailability","product/indexingSource","product/ingestType","product/input","product/inputMode","product/instance","product/instanceCapacity10xlarge","product/instanceCapacity12xlarge","product/instanceCapacity24xlarge","product/instanceCapacity2xlarge","product/instanceCapacity4xlarge","product/instanceCapacity8xlarge","product/instanceCapacityLarge","product/instanceCapacityXlarge","product/instanceFamily","product/instanceFunction","product/instanceType","product/instanceTypeFamily","product/instances","product/instancesku","product/intelAvx2Available","product/intelAvxAvailable","product/intelTurboAvailable","product/io","product/license","product/licenseModel","product/licenseType","product/lineType","product/location","product/locationType","product/logsSource","product/logsType","product/machineLearningProcess","reservation/ModificationStatus","reservation/NormalizedUnitsPerReservation","reservation/NumberOfReservations","reservation/RecurringFeeForUsage","reservation/ReservationARN","reservation/StartTime","reservation/TotalReservedNormalizedUnits","reservation/TotalReservedUnits","reservation/UnitsPerReservation","reservation/UnusedAmortizedUpfrontFeeForBillingPeriod","reservation/UnusedNormalizedUnitQuantity","reservation/UnusedQuantity","reservation/UnusedRecurringFee","reservation/UpfrontValue","resourceTags/user:CmBillingGroup","savingsPlan/TotalCommitmentToDate","savingsPlan/SavingsPlanARN","savingsPlan/SavingsPlanRate","savingsPlan/UsedCommitment","savingsPlan/SavingsPlanEffectiveCost","savingsPlan/AmortizedUpfrontCommitmentForBillingPeriod","savingsPlan/RecurringCommitmentForBillingPeriod","savingsPlan/Region","savingsPlan/PaymentOption","savingsPlan/EndTime","savingsPlan/InstanceTypeFamily","savingsPlan/PurchaseTerm","savingsPlan/OfferingType","savingsPlan/StartTime"], "noPrefix": false})}

New Relicでの取り込み+変換確認

New Relic側での設定も完了したため、レポートデータを確認していきます。
まずはSELECTでデータが取り込めているかの確認をします。

SELECT
    `message`
FROM
    Log
WHERE
    allColumnSearch('cm-cur-[アカウントID]', insensitive : true) SINCE 1 day ago
limit 10

ログを選択して中身を見てみると、Parsing Ruleにて変換されていることがわかります。

New Relicでのグラフ化

変換されていることも確認できたため、グラフ化してみます。
New Relic公式ページ記載のものと若干選択する項目などを変えています。
明細項目の詳細は以下をご確認ください。（MCURには一部含まれない項目があります）
明細項目の詳細（AWS公式）

SELECT
    max(cost)
FROM
    (
        SELECT
            sum(numeric(`log.lineItem/UnblendedCost`)) as cost
        FROM
            Log
        WHERE
            `aws.s3_bucket_name` = 'cm-cur-[アカウントID]' TIMESERIES 6 hour facet `log.lineItem/ProductCode`
    ) SINCE 7 days ago TIMESERIES 24 hours facet `log.lineItem/ProductCode`

stacked bar chart(積み上げ棒グラフ)にすると以下のようになります。

参考情報

メンバーズポータルユーザーガイド コストと使用状況レポート（CUR）
AWS コストと使用状況レポート(CUR)を分析しやすく可視化

最後に

今回は、New Relicでクラスメソッドメンバーズ用のAWSコストと使用状況レポート(MCUR)を可視化してみたことを記事にしました。
どなたかの参考になると幸いです。