I checked the cost of S3 delivery with the new ALB log feature "CloudWatch Logs integration"

I checked the cost of S3 delivery with the new ALB log feature "CloudWatch Logs integration"

# ALB ログの CloudWatch Logs 統合:コスト・形式・運用設計の整理 --- ## 1. `APN1-VendedLog-Bytes` と「S3配信無料」の整合確認 ### 公式の料金体系 | 経路 | 課金項目 | 備考 | |------|----------|------| | ALB → CloudWatch Logs (Vended Logs) | `VendedLog-Bytes` | **5 GB/月まで無料、超過分は割引単価** | | CloudWatch Logs → S3 配信 | **無料** | サブスクリプションフィルター経由のS3エクスポートは課金なし | | S3 ストレージ | S3標準料金 | CloudWatch側の話ではない | | Logs Insights クエリ | スキャンバイト課金 | 必要時のみ使用で制御可能 | ### 「設定前の日次範囲内」だった理由の解釈 ``` 設定前に観測していた VendedLog-Bytes = ALBがもともとVended Logsとして送信していたバイト数 設定後も同じ値の範囲内 = S3への配信設定はVended Logsの「送信量」を増やさない (ログの生成源はALBであり、S3配信は下流の転送に過ぎない) ``` **結論:整合している。** S3配信を追加しても `VendedLog-Bytes` の増加要因にはならない。ヘルスチェックのトラフィックが一定であれば、設定前後で同じ範囲に収まるのは正常な挙動。 --- ## 2. 形式別ファイルサイズの比較 ### 理論的な大小関係 ``` Parquet < JSON (gzip圧縮時) ≈ plain (gzip圧縮時) < JSON (非圧縮) < plain (非圧縮) ``` ### 具体的な特性比較 | 形式 | 圧縮 | サイズ感 | 特徴 | |------|------|----------|------| | **plain** (テキスト) | なし | 最大 | スペース区切り、人間が直読可能 | | **plain** | gzip | 中 | ALBログは繰り返しパターンが多く圧縮効率良好 | | **JSON** | なし | 大 | キー名の繰り返しでplainより大きくなる場合あり | | **JSON** | gzip | 中〜小 | キー繰り返しが圧縮で吸収される | | **Parquet** | Snappy/ZSTD内包 | 最小 | 列指向+辞書符号化で数値・IPが大幅圧縮 | ### ヘルスチェックログの場合の実態 ``` ヘルスチェックは - 送信元IPが固定(ELBのヘルスチェックIP数種類) - パスが固定(/health 等) - ステータスが固定(200のみ) - 繰り返しパターンが極めて高い → Parquetの列指向圧縮が最も効果を発揮するケース → JSON/plainでもgzip有効なら十分小さくなる ``` ### 実測値の目安(ALBログ 1万リクエスト分) ``` plain非圧縮 : ~3–5 MB plain gzip : ~300–500 KB JSON非圧縮 : ~5–8 MB JSON gzip : ~300–600 KB Parquet : ~100–250 KB(内部圧縮込み) ``` --- ## 3. 「S3保存を軸に、必要時だけ Logs Insights」構成 ### アーキテクチャ図 ``` ALB │ │ Vended Logs (無料枠内) ▼ CloudWatch Logs │ ├─[サブスクリプションフィルター]──→ S3バケット ←── 常時保存・低コスト │ (S3配信は無料) │ │ ├── Parquet形式 │ │ → Athena でアドホック分析 │ │ │ └── plain/JSON形式 │ → grep・jq でローカル確認 │ └─[Logs Insights] ←── 必要な時だけ使用(スキャン課金に注意) ▲ │ 障害調査・リアルタイム分析時のみ │ 保持期間を短く設定(例:3日〜1週間) ``` ### CloudWatch Logs の保持期間設定 ```bash # ロググループの保持期間を短縮(Logs Insightsスキャン対象を最小化) aws logs put-retention-policy \ --log-group-name "aws-waf-logs-alb" \ --retention-in-days 3 ``` **設計思想:** - 長期保存・検索 → S3 + Athena(安い) - 直近の障害調査 → Logs Insights(高いが使用頻度を限定) - 定常監視 → CloudWatch Metrics / アラーム(ログを読まない) ### Logs Insights 使用コストの試算 ``` Logs Insightsの料金:$0.005 per GB scanned(東京リージョン) ヘルスチェックログ 1日分 = 約50MB(圧縮前)として 1週間保持 = 350MB 1クエリのスキャンコスト = 350MB × $0.005/GB ≈ $0.002 → 1日10回クエリしても月 $0.6 未満 → ただしアクセスログ全体を長期保持すると急増するため S3に逃がしてからCWLの保持期間を削ることが重要 ``` --- ## 4. レガシー設定からの移行方法 ### レガシー設定とは ``` 旧来の方法: ALB → S3 (ALBアクセスログ直接配信) └── ALBコンソールの「アクセスログ」設定で S3バケットを直接指定するだけ 新しい方法: ALB → CloudWatch Logs → S3 └── CloudWatch Logs統合を有効化し、 サブスクリプションフィルターでS3へ ``` ### 移行手順 #### Step 1: 現状確認 ```bash # レガシー設定(ALB直接S3配信)の確認 aws elbv2 describe-load-balancer-attributes \ --load-balancer-arn <ALB_ARN> \ --query 'Attributes[?Key==`access_logs.s3.enabled`]' # CloudWatch Logs統合の確認 aws elbv2 describe-load-balancer-attributes \ --load-balancer-arn <ALB_ARN> \ --query 'Attributes[?starts_with(Key, `access_logs.s3`) || starts_with(Key, `routing.http`)]' ``` #### Step 2: CloudWatch Logs 統合の有効化 ```bash # ロググループ作成 aws logs create-log-group \ --log-group-name "/aws/elasticloadbalancing/alb-access-logs" # ALB に CloudWatch Logs 統合を設定 aws elbv2 modify-load-balancer-attributes \ --load-balancer-arn <ALB_ARN> \ --attributes \ Key=access_logs.s3.enabled,Value=false \ Key=connection_logs.s3.enabled,Value=false # ※ CloudWatch Logs への配信はコンソールから # 「モニタリング」→「ログ配信を編集」で設定 ``` #### Step 3: S3配信の設定(Parquet推奨) ```bash # S3バケットへのログ配信設定(コンソール操作を経由する場合) # AWS CLIでの直接設定 aws logs put-delivery-destination \ --name "alb-logs-to-s3" \ --delivery-destination-configuration \ destinationResourceArn=arn:aws:s3:::your-bucket/alb-logs/ aws logs put-delivery-source \ --name "alb-access-logs" \ --resource-arn <ALB_ARN> \ --log-type ACCESS_LOGS aws logs put-delivery \ --delivery-source-name "alb-access-logs" \ --delivery-destination-arn <DESTINATION_ARN> \ --s3-delivery-configuration \ suffixPath="{yyyy}/{MM}/{dd}/{HH}/" \ enableHiveCompatiblePath=true ``` #### Step 4: 移行期間の並走と切り替え ``` 移行スケジュール(推奨): Week 1: 新設定(CWL → S3)を有効化、旧設定も並走 └── 両方のS3パスでデータが来ていることを確認 Week 2: データ欠損がないことを確認 └── Athenaでクエリして件数・タイムスタンプを照合 Week 3: 旧設定(ALB直接S3配信)を無効化 └── access_logs.s3.enabled = false ``` #### Step 5: Athena テーブル定義(Parquet形式の場合) ```sql CREATE EXTERNAL TABLE alb_logs_parquet ( type string, time string, elb string, client_ip string, client_port int, target_ip string, target_port int, request_processing_time double, target_processing_time double, response_processing_time double, elb_status_code int, target_status_code int, received_bytes bigint, sent_bytes bigint, request_verb string, request_url string, request_proto string, user_agent string, ssl_cipher string, ssl_protocol string, target_group_arn string, trace_id string, domain_name string, chosen_cert_arn string, matched_rule_priority int, request_creation_time string, actions_executed string, redirect_url string, error_reason string, target_port_list string, target_status_code_list string, classification string, classification_reason string ) STORED AS PARQUET LOCATION 's3://your-bucket/alb-logs/' TBLPROPERTIES ('parquet.compress'='SNAPPY'); ``` --- ## 5. まとめ:設計判断のチェックリスト ``` ✅ VendedLog-Bytesが設定前後で同範囲 → S3配信追加はVended Logsの送信量に影響しない。正常。 ✅ S3配信コストは無料 → CWL → S3 のサブスクリプションフィルター経由配信は課金なし。 ✅ ファイル形式選択 → 長期保存・Athena分析 → Parquet → 人間が直読・grep → plain gzip → 他システム連携・汎用性 → JSON gzip ✅ Logs Insights はCWL保持期間を短く保ちスキャン量を制限 → 保持3〜7日に設定し、長期分はS3+Athenaへ委譲 ✅ レガシー移行は並走期間を設けて確認後に旧設定を無効化 → データ欠損リスクを排除してから切り替え ```
2026.07.27

This page has been translated by machine translation. View original

Introduction

On July 23, 2026, there was an update integrating ALB logging into the CloudWatch Logs delivery feature (Vended Logs).

https://aws.amazon.com/jp/about-aws/whats-new/2026/07/amazon-cloudwatch-logs/

A previous article introduced log output to CloudWatch Logs.

https://dev.classmethod.jp/articles/alb-access-logs-cloudwatch-logs-vended/

This article examines the cost impact when delivering ALB logs to S3 using the CloudWatch Logs integration.

In this article, the conventional S3 output configured via ALB attributes (such as access_logs.s3.*) is referred to as the "legacy configuration."

Verification Details

This verification targeted ALB health check logs. Since they are less affected by actual access traffic and are output at regular intervals, they were considered suitable for comparing file sizes and output counts across formats.

The same health check logs were delivered in parallel to S3 in three formats — JSON, plain, and Parquet — to examine file sizes and the cost impact of S3 delivery via CloudWatch Logs integration.

Verification Environment

Item Value
Region ap-northeast-1
Load Balancer 1 Application Load Balancer (<ALB_NAME>)
Target Log Type ALB_HEALTH_CHECK_LOGS
Number of Log Fields 10
Health Check Interval 30 seconds
Log File Output Interval 5 minutes (per ALB node)
Output Formats JSON / plain (space-delimited) / Parquet
Measurement Period 2026-07-22 to 2026-07-27 (S3 delivery configuration implemented on 07-24)

In this comparison, to avoid the impact of destination differences, all three formats were delivered to the same S3 bucket <LOG_BUCKET>, with only the prefix differing per format.

All size and cost figures below are based solely on health check logs.

Comparison by Format

File Size

One set of files containing 10 records, output from the same time and the same ALB node, was extracted and compared.

Format File Size Per Record (estimated) Ratio to plain (%)
plain (gz) 283 B 28.3 B 100%
JSON (gz) 393 B 39.3 B 139%
Parquet (internal GZIP) 4,042 B 404.2 B 1,429%

"Per Record (estimated)" is a reference value obtained by dividing the compressed file size by 10 records. Compression efficiency varies with the number of records, so it does not scale proportionally when the record count increases.

plain (Space-Delimited)

The plain format arranges values separated by spaces, the same as the legacy configuration.

http 2026-07-27T00:25:02.444270Z 0.022732728 <TARGET_IP>:3000 <TARGET_GROUP> PASS 200 - app/<ALB_NAME>/<ALB_ID> <ALB_NODE_IP>

JSON

JSON outputs each record as a single-line object.

{"type":"http","time":"2026-07-27T00:25:02.444270Z","latency":"0.022732728","target_addr":"<TARGET_IP>:3000","target_group_id":"<TARGET_GROUP>","status":"PASS","status_code":"200","reason_code":"-","elb":"app/<ALB_NAME>/<ALB_ID>","ip_address":"<ALB_NODE_IP>"}

The ratio changes with the number of fields (in the access log measurements in the previous article, JSON was 2.53 times the size of plain).

Since JSON pairs keys with values, it can be handled without relying on field position as with plain. JSON is easier to work with when anticipating future field additions.

Parquet

Reading Parquet files with pyarrow allowed us to check the proportion of metadata.

  • rows=10, row_groups=1, columns=10
  • Compression: GZIP
  • Row group data section: 1,070 B
  • Footer and schema metadata: approximately 2,972 B (about 73% of the entire file)

For output at the 5-minute interval and 10-record scale in this verification, Parquet is not suitable. Metadata accounts for approximately 73% of the total, and the large file size was more prominent than the benefits of columnar format compression and scan range narrowing. In addition, the official announcement states that Parquet conversion incurs an additional charge, with the unit price shown in the official blog for the US East (N. Virginia) region being $0.035/GB. There is no reason to choose this format for storage in S3 at this scale even in the Tokyo region.

Although outside the scope of this verification, if you want to store data in Parquet, another option is to batch-convert logs delivered in JSON or plain format using Athena CTAS or Glue ETL. Since the number of records per file can be increased, the proportion of metadata can be kept lower compared to saving small files directly as Parquet.

Actual Cost Measurement

Daily usage by usage type for ap-northeast-1 was checked in Cost Explorer. We compare 2026-07-22 and 07-23 (before configuration) with 2026-07-25 (the day after configuration).

Date S3 Recording APN1-VendedLog-Bytes Usage (MB equivalent)
2026-07-22 None 10.57
2026-07-23 None 10.75
2026-07-25 Yes 10.73

APN1-VendedLog-Bytes in Cost Explorer was converted to MB units for comparison. Even after configuring the CloudWatch Logs integration, this usage type remained within the range of 10.57–10.75 MB observed before configuration.

The approximately 10 MB recorded even before configuration is attributable to VPC flow logs running in the same account. Cost allocation tags were set on the flow log group, confirming that this usage type was generated only by the flow logs. No Vended Logs deliveries other than the ALB logs in this verification were configured.

The official announcement states that delivering ALB logs to Amazon S3 is free of charge for delivery. Delivery to CloudWatch Logs and Amazon Data Firehose is charged as Vended Logs. The comparison results for APN1-VendedLog-Bytes in this verification are also consistent with this pricing structure.

Configuration Procedure (Reference)

This section summarizes the configuration for delivering the same logs in parallel in multiple formats to S3, as done in this verification.

Parallel delivery in 3 formats via CLI

Create one Delivery Source, and create a Delivery Destination and Delivery for each of JSON, plain, and Parquet.

When switching formats in production, separate S3 buckets per format if possible to avoid mixing with existing logs. If using the same bucket, at minimum separate the prefixes.

First, create the Delivery Source from which the logs originate.

aws logs put-delivery-source \
  --name alb-health-check-logs \
  --resource-arn arn:aws:elasticloadbalancing:ap-northeast-1:<ACCOUNT_ID>:loadbalancer/app/<ALB_NAME>/<ALB_ID> \
  --log-type ALB_HEALTH_CHECK_LOGS

In the verification, a Delivery Destination targeting the same S3 bucket was created for each format. Only --output-format is changed.

aws logs put-delivery-destination \
  --name alb-vended-logs-s3-json \
  --output-format json \
  --delivery-destination-configuration destinationResourceArn=arn:aws:s3:::<LOG_BUCKET>

aws logs put-delivery-destination \
  --name alb-vended-logs-s3-plain \
  --output-format plain \
  --delivery-destination-configuration destinationResourceArn=arn:aws:s3:::<LOG_BUCKET>

aws logs put-delivery-destination \
  --name alb-vended-logs-s3-parquet \
  --output-format parquet \
  --delivery-destination-configuration destinationResourceArn=arn:aws:s3:::<LOG_BUCKET>

Finally, link them with Delivery. Change the destination ARN and suffixPath per format to prevent output from mixing within the same bucket.

aws logs create-delivery \
  --delivery-source-name alb-health-check-logs \
  --delivery-destination-arn arn:aws:logs:ap-northeast-1:<ACCOUNT_ID>:delivery-destination:alb-vended-logs-s3-json \
  --s3-delivery-configuration 'suffixPath=health-check-logs/json/{region}/{yyyy}/{MM}/{dd}/'

Run create-delivery for plain and Parquet as well, changing the destination ARN and suffixPath accordingly.

S3 Bucket Policy

When using S3 as the destination, grant the following permissions to delivery.logs.amazonaws.com.

  • Object write (s3:PutObject)
  • Bucket ACL retrieval (s3:GetBucketAcl)
{
  "Effect": "Allow",
  "Principal": {
    "Service": "delivery.logs.amazonaws.com"
  },
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::<LOG_BUCKET>/*",
  "Condition": {
    "StringEquals": {
      "aws:SourceAccount": "<ACCOUNT_ID>"
    }
  }
}
{
  "Effect": "Allow",
  "Principal": {
    "Service": "delivery.logs.amazonaws.com"
  },
  "Action": "s3:GetBucketAcl",
  "Resource": "arn:aws:s3:::<LOG_BUCKET>",
  "Condition": {
    "StringEquals": {
      "aws:SourceAccount": "<ACCOUNT_ID>"
    }
  }
}

In this verification, to simplify the setup, the condition in both statements was limited to aws:SourceAccount only, and aws:SourceArn was not specified. In production, it is recommended to also specify aws:SourceArn to restrict the delivery source.

Reasons to Migrate from Legacy Configuration to CloudWatch Logs Integration

As mentioned above, when delivering ALB logs to S3 via CloudWatch Logs integration, no Vended Logs delivery charges apply. S3 storage charges and request charges continue to occur as before. Note that when delivering in parallel in multiple formats as in this verification, the number of PutObject requests increases proportionally with the number of formats.

Log delivery definitions can be separated from ALB attributes. In this verification as well, all ALB log-related attributes remained disabled, yet logs could be delivered to S3 through the Delivery defined on the CloudWatch Logs side.

Even in a configuration that prioritizes cost by limiting log storage to S3only, the new method allows delivery to CloudWatch Logs log groups to be added as needed. This enables searching via Logs Insights and near-real-time review, which is useful during events or for incident investigation.

Additional delivery to a log group incurs CloudWatch Logs charges. It is also possible to keep storage to S3 only at normal times and add delivery to a log group only during the period before an event or when reproducing an incident. Please consider this as needed.

Summary

For new deployments storing ALB logs to S3, the CloudWatch Logs integration should be considered the first choice. Delivering to S3 incurs no Vended Logs delivery charges, output formats can be selected, and delivery to a log group can be added as needed for investigation via Logs Insights.

For format selection, plain is preferable if storage size is the priority, while JSON is easier to handle if you anticipate field additions or programmatic processing.

For environments using existing legacy configurations, it is recommended to review them when a maintenance window for the log analysis environment becomes available. Configure new log delivery to a different destination while keeping the legacy configuration enabled. Once the operation of log analysis and log processing pipelines has been verified, disabling the legacy configuration allows for a safe cutover.

Share this article

AWSのお困り事はクラスメソッドへ

Related articles