ログ出力をサポートしたAmazon EventBridge Pipes を試してみた

EventBridge Pipes がデバッグや障害時の調査に役立つログ出力をサポートしました。

#Amazon EventBridge Pipes

#Amazon EventBridge

#Amazon SQS

#AWS

suzuki.ryo

2023.11.17

この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。

2023年11月14日付けで Amazon EventBridge Pipes がログ出力をサポートするアップデートがありました。

今回、 CloudWatch ログへのログ記録を試す機会がありましたので、紹介させて頂きます。

Introducing logging support for Amazon EventBridge Pipes

構成図

SQS-event-bridge-pipes

EventBridge 設定

EventBridge Pipes

2つのSQSを結ぶパイプを作成しました。

パイプ設定1

パイプ設定

ログ設定として、CloudWatch Logsへのログ出力を有効化しました。

パイプ設定2

検証

awscliを利用して、ソースに指定したSQSに対しメッセージを送信。

パイプ設定で指定可能なログレベル別のログを確認しました。

aws sqs send-message --queue-url  "https://sqs.ap-northeast-1.amazonaws.com/00000000/sqs-1" --message-body "test"

TRACE

最も詳細なログが確認できました。

2023-11-16T23:35:57.955+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145357955,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"ExecutionStarted","logLevel":"INFO"}
2023-11-16T23:35:57.955+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145357955,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"PollingStageSucceeded","logLevel":"TRACE"}
2023-11-16T23:35:57.965+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145357965,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"TargetStageEntered","logLevel":"INFO"}
2023-11-16T23:35:57.965+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145357965,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"TargetTransformationStarted","logLevel":"TRACE"}
2023-11-16T23:35:57.965+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145357965,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"TargetTransformationSucceeded","logLevel":"TRACE"}
2023-11-16T23:35:57.965+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145357965,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"TargetInvocationStarted","logLevel":"TRACE"}
2023-11-16T23:35:58.035+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145358035,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"TargetInvocationSucceeded","logLevel":"TRACE"}
2023-11-16T23:35:58.035+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145358035,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"TargetStageSucceeded","logLevel":"INFO"}
2023-11-16T23:35:58.036+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145358036,"executionId":"00000000-c2ec-4dc7-b5ef-ddbb91f13cea","messageType":"ExecutionSucceeded","logLevel":"INFO"}

INFO

2023-11-16T23:38:16.048+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145496048,"executionId":"00000000-c1ef-412b-98c1-09a35d3f79a0","messageType":"TargetStageEntered","logLevel":"INFO"}
2023-11-16T23:38:16.111+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145496111,"executionId":"00000000-c1ef-412b-98c1-09a35d3f79a0","messageType":"TargetStageSucceeded","logLevel":"INFO"}
2023-11-16T23:38:16.111+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145496111,"executionId":"00000000-c1ef-412b-98c1-09a35d3f79a0","messageType":"ExecutionSucceeded","logLevel":"INFO"}

ERROR

ターゲットのSQS、SendMessage を拒否するアクセスポリシーを反映し、メッセージ転送が失敗する状態でのログ記録を確認しました。

ターゲット(SQS2)のアクセスポリシー

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "deny-sqs-actions",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "SQS:SendMessage",
      "Resource": "*"
    }
  ]
}

エラーログ内容

2023-11-17T00:18:32.023+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:00000000:pipe/sqs-pipe","timestamp":1700147912023,"executionId":"3e1c9794-efbc-4755-89e3-9382d47dde62","messageType":"TargetInvocationFailed","logLevel":"ERROR","error":{"message":"Target invocation failed with error from SQS. Not allowed to invoke arn:aws:sqs:ap-northeast-1:00000000:sqs-2.","httpStatusCode":403,"awsService":"sqs","requestId":"c2466a96-61d3-5958-8dee-374654db4818","exceptionType":"AccessDenied","resourceArn":"arn:aws:sqs:ap-northeast-1:00000000:sqs-2"}}
2023-11-17T00:18:32.024+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:00000000:pipe/sqs-pipe","timestamp":1700147912024,"executionId":"3e1c9794-efbc-4755-89e3-9382d47dde62","messageType":"TargetStageFailed","logLevel":"ERROR","error":{"message":"Target invocation failed with error from SQS. Not allowed to invoke arn:aws:sqs:ap-northeast-1:00000000:sqs-2.","httpStatusCode":403,"awsService":"sqs","requestId":"c2466a96-61d3-5958-8dee-374654db4818","exceptionType":"AccessDenied","resourceArn":"arn:aws:sqs:ap-northeast-1:00000000:sqs-2"}}
2023-11-17T00:18:32.024+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:00000000:pipe/sqs-pipe","timestamp":1700147912024,"executionId":"3e1c9794-efbc-4755-89e3-9382d47dde62","messageType":"ExecutionFailed","logLevel":"ERROR","error":{"message":"Target invocation failed with error from SQS. Not allowed to invoke arn:aws:sqs:ap-northeast-1:00000000:sqs-2.","httpStatusCode":403,"awsService":"sqs","requestId":"c2466a96-61d3-5958-8dee-374654db4818","exceptionType":"AccessDenied","resourceArn":"arn:aws:sqs:ap-northeast-1:00000000:sqs-2"}}

SQSの権限不足で失敗、3回リトライを試みたログが記録されていました。

実行データを含める

実行データを含める指定を有効にしてみました。

2023-11-16T23:40:54.311+09:00   {"resourceArn":"arn:aws:pipes:ap-northeast-1:000000000000:pipe/sqs-pipe","timestamp":1700145654311,"executionId":"00000000-730d-4d93-b5ca-0d2c8c8d7aba","messageType":"ExecutionStarted","logLevel":"INFO","payload":"{\"Records\":[{\"messageId\":\"00000000-c558-4a6a-8259-2bda3d838c46\",\"receiptHandle\":\"00000000/00000000/00000000/00000000/00000000/h+LcId0MNS+QMfw==\",\"body\":\"test\",\"attributes\":{\"ApproximateReceiveCount\":\"1\",\"SentTimestamp\":\"1700145654276\",\"SenderId\":\"00000000:00000000\",\"ApproximateFirstReceiveTimestamp\":\"1700145654278\"},\"messageAttributes\":{},\"md5OfBody\":\"098f6bcd4621d373cade4e832627b4f6\",\"eventSource\":\"aws:sqs\",\"eventSourceARN\":\"arn:aws:sqs:ap-northeast-1:000000000000:sqs-1\",\"awsRegion\":\"ap-northeast-1\"}]}"}