developersIO
I checked the cost of S3 delivery with the new ALB log feature "CloudWatch Logs integration"

I checked the cost of S3 delivery with the new ALB log feature "CloudWatch Logs integration"

# ALB Access Logs → CloudWatch Logs Integration: Cost, Format & Architecture Review --- ## 1. `APN1-VendedLog-Bytes` Cost Observation ### What You Observed | Timing | Daily `APN1-VendedLog-Bytes` Usage | |---|---| | Before configuration | Baseline range observed | | After configuration | **Within the same baseline range** | ### Why This Aligns With Official Guidance **Official AWS Statement:** > "Logs delivered by AWS services to CloudWatch Logs — including ALB access logs — are delivered free of charge (Vended Logs). No charge applies for ingestion." **Mechanism behind zero additional cost:** ``` ALB → CloudWatch Logs (Vended Log delivery) ↓ [No ingestion charge] APN1-VendedLog-Bytes = $0.00/GB ↓ CloudWatch Logs → S3 Export [S3 delivery via subscription filter = also $0 transfer] ↓ S3 Storage = charged at standard S3 rates ``` **Why the Cost Explorer value stayed in range:** - `APN1-VendedLog-Bytes` tracks **volume** but at **$0 unit price** for Vended Logs - Health check logs are high-frequency but small per-record - The byte volume itself was already within your pre-existing daily fluctuation band - **No billing surprise = expected behavior, consistent with official guidance** ✅ --- ## 2. File Size Differences: JSON vs Plain Text vs Parquet ### Comparative Analysis (ALB Health Check Log Typical Values) ``` Single ALB health check log record (raw): ~200-350 bytes uncompressed Format comparison for 1M health check records: ``` | Format | Approx Size | Compression | Notes | |---|---|---|---| | **Plain Text** | ~280 MB | None by default | One record per line, human-readable | | **JSON** | ~380 MB | None by default | Field names repeated per record, ~35% larger than plain | | **Parquet** | ~18-45 MB | Snappy (built-in) | Columnar, ~85-90% smaller than plain | ### Detailed Format Characteristics #### Plain Text ``` 2024-01-15T10:23:45Z GET /health 200 0.001 - - - 2024-01-15T10:23:46Z GET /health 200 0.001 - - - ``` - ✅ Human-readable, simple grep/awk - ✅ Smallest metadata overhead - ❌ No schema enforcement - ❌ Athena queries require regex parsing #### JSON ```json {"timestamp":"2024-01-15T10:23:45Z","type":"http","request":"GET /health HTTP/1.1","response":200,"response_time":0.001} {"timestamp":"2024-01-15T10:23:46Z","type":"http","request":"GET /health HTTP/1.1","response":200,"response_time":0.001} ``` - ✅ Self-describing schema - ✅ Direct Athena JSON SerDe support - ✅ Easy programmatic parsing - ❌ Field names stored per record = significant size overhead - ❌ ~35-40% larger than equivalent plain text #### Parquet ``` [Binary columnar format] timestamp column: [val1, val2, val3, ...] ← all timestamps together response column: [200, 200, 200, ...] ← high compression ratio ``` - ✅ **Optimal for Athena** — columnar scan, pays only for queried columns - ✅ 85-90% size reduction vs plain text - ✅ Built-in Snappy compression - ✅ Lowest S3 storage cost at scale - ❌ Not human-readable - ❌ Requires Spark/Athena/Glue to read - ❌ Slightly higher write CPU overhead ### Health Check Log Specific Context Health check logs are particularly well-suited for Parquet because: ``` Health check record characteristics: - High cardinality in: timestamp - Low cardinality in: request_url (/health), response_code (200), elb_status_code - Repetitive values → columnar compression extremely effective Example: response_code column = [200,200,200,...200] for 99.9% of records → Run-length encoding compresses this to near-zero bytes ``` --- ## 3. Recommended Architecture: S3 as Primary Store, Logs Insights On-Demand ### Design Principle ``` "Store everything in S3, query S3 with Athena by default, use Logs Insights only when real-time/operational need exists" ``` ### Architecture Diagram ``` ┌─────────────────────────────────────────────────────────────┐ │ ALB │ └─────────────────────┬───────────────────────────────────────┘ │ Vended Log (free ingestion) ▼ ┌─────────────────────────────────────────────────────────────┐ │ CloudWatch Logs Log Group │ │ /aws/alb/access-logs (retention: 1-3 days) │ │ │ │ [Short retention = minimal CWL storage cost] │ └──────────────┬──────────────────────────────────────────────┘ │ Subscription Filter (real-time) │ or │ Export Task (batch, free) ▼ ┌─────────────────────────────────────────────────────────────┐ │ Amazon S3 │ │ │ │ s3://your-bucket/alb-logs/ │ │ ├── format=parquet/ ← primary archive │ │ │ └── year=2024/month=01/day=15/ │ │ │ └── *.parquet │ │ └── format=json/ ← optional, recent only │ │ └── year=2024/month=01/day=15/ │ │ └── *.json.gz │ └──────────────┬──────────────────────────────────────────────┘ │ ┌───────┴────────┐ ▼ ▼ ┌────────────┐ ┌──────────────────────────────────────────┐ │ Athena │ │ CloudWatch Logs Insights │ │ │ │ │ │ Default │ │ Use ONLY when: │ │ query path │ │ • Incident in progress (real-time need) │ │ │ │ • Last 1-3 days data (within retention) │ │ Cost: │ │ • Quick ad-hoc during investigation │ │ $5/TB │ │ │ │ scanned │ │ Cost: $0.0058/GB scanned │ │ │ │ (vs Athena: more expensive at scale) │ └────────────┘ └──────────────────────────────────────────┘ ``` ### When to Use Each Query Tool | Scenario | Tool | Reason | |---|---|---| | Daily batch analysis | **Athena** | Low cost, Parquet columnar scan | | Historical trend (>3 days) | **Athena** | Data only in S3 | | Active incident investigation | **Logs Insights** | Real-time, no S3 export lag | | Last 30 min anomaly | **Logs Insights** | Immediate availability | | Cost report generation | **Athena** | Scheduled, batch-oriented | | SLA compliance audit | **Athena** | Historical, large dataset | ### CWL Retention Strategy ```yaml # Optimal retention for cost control CloudWatch Log Group Retention: 1 day # minimum viable # covers: S3 export lag + real-time incident window # If export has SLA concerns, extend to: CloudWatch Log Group Retention: 3 days # safe buffer # Avoid: CloudWatch Log Group Retention: 30 days # unnecessary; data is in S3 # CWL storage = $0.033/GB/month # S3 storage = $0.023/GB/month # ~43% more expensive in CWL ``` ### S3 Lifecycle Policy ```json { "Rules": [ { "ID": "alb-logs-lifecycle", "Filter": {"Prefix": "alb-logs/"}, "Status": "Enabled", "Transitions": [ { "Days": 30, "StorageClass": "S3_INTELLIGENT_TIERING" }, { "Days": 90, "StorageClass": "GLACIER_INSTANT_RETRIEVAL" } ], "Expiration": { "Days": 365 } } ] } ``` --- ## 4. Migration From Legacy Configuration ### Legacy vs Current Configuration Overview ``` Legacy (Classic): ALB → S3 (direct, via built-in ALB access log feature) ↓ Fixed format (space-delimited text) Delivered every 5-60 minutes No real-time access No CloudWatch integration Current (Integrated): ALB → CloudWatch Logs → S3 (via export or subscription) ↓ ↓ Real-time access Structured format (JSON/Parquet) Logs Insights Athena queryable Metric filters Lifecycle management ``` ### Migration Steps #### Step 1: Inventory Existing Configuration ```bash # Check current ALB access log settings aws elbv2 describe-load-balancer-attributes \ --load-balancer-arn <your-alb-arn> \ --query 'Attributes[?Key==`access_logs.s3.enabled` || Key==`access_logs.s3.bucket` || Key==`access_logs.s3.prefix`]' # Check if CWL integration already enabled aws elbv2 describe-load-balancer-attributes \ --load-balancer-arn <your-alb-arn> \ --query 'Attributes[?contains(Key, `connection_logs`) || contains(Key, `cloudwatch`)]' ``` #### Step 2: Enable CloudWatch Logs Integration (Non-destructive) ```bash # Enable ALB → CWL (does NOT disable legacy S3 direct delivery) aws elbv2 modify-load-balancer-attributes \ --load-balancer-arn <your-alb-arn> \ --attributes \ Key=access_logs.s3.enabled,Value=true \ Key=connection_logs.s3.enabled,Value=true # Separately enable CWL delivery # (via Console: EC2 > Load Balancers > Select ALB > Monitoring tab > Edit) ``` #### Step 3: Parallel Running Period ``` Week 1-2: Run both legacy S3 direct + new CWL→S3 simultaneously Compare: - Record counts match? - Timestamps align? - No gaps in CWL delivery? Validation query (Athena): ``` ```sql -- Compare daily record counts between legacy and new pipeline SELECT date_trunc('hour', from_iso8601_timestamp(time)) as hour, count(*) as record_count, 'new_pipeline' as source FROM alb_logs_new_parquet WHERE day = '2024-01-15' GROUP BY 1 UNION ALL SELECT date_trunc('hour', parse_datetime(time, 'yyyy-MM-dd''T''HH:mm:ss.SSSSSS''Z''')) as hour, count(*) as record_count, 'legacy_direct_s3' as source FROM alb_logs_legacy WHERE day = '2024-01-15' GROUP BY 1 ORDER BY 1, 3; ``` #### Step 4: Update Glue Catalog ```python # Create new Glue table for Parquet format import boto3 glue = boto3.client('glue', region_name='ap-northeast-1') glue.create_table( DatabaseName='alb_logs_db', TableInput={ 'Name': 'alb_access_logs_parquet', 'StorageDescriptor': { 'Columns': [ {'Name': 'time', 'Type': 'string'}, {'Name': 'type', 'Type': 'string'}, {'Name': 'elb', 'Type': 'string'}, {'Name': 'client_ip', 'Type': 'string'}, {'Name': 'target_ip', 'Type': 'string'}, {'Name': 'request_processing_time', 'Type': 'double'}, {'Name': 'target_processing_time', 'Type': 'double'}, {'Name': 'response_processing_time', 'Type': 'double'}, {'Name': 'elb_status_code', 'Type': 'int'}, {'Name': 'target_status_code', 'Type': 'int'}, {'Name': 'request', 'Type': 'string'}, {'Name': 'user_agent', 'Type': 'string'}, ], 'Location': 's3://your-bucket/alb-logs/format=parquet/', 'InputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat', 'OutputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat', 'SerdeInfo': { 'SerializationLibrary': 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe', }, }, 'PartitionKeys': [ {'Name': 'year', 'Type': 'string'}, {'Name': 'month', 'Type': 'string'}, {'Name': 'day', 'Type': 'string'}, ], 'TableType': 'EXTERNAL_TABLE', } ) ``` #### Step 5: Disable Legacy Direct S3 Delivery (After Validation) ```bash # Only after confirming new pipeline is stable (2+ weeks) # and all dashboards/alerts updated to new table # Option A: Disable legacy S3 direct delivery aws elbv2 modify-load-balancer-attributes \ --load-balancer-arn <your-alb-arn> \ --attributes Key=access_logs.s3.enabled,Value=false # Note: CWL Vended Log delivery continues independently ``` #### Step 6: Migration Checklist ``` Pre-migration: □ Document existing S3 bucket/prefix for legacy logs □ Export existing Athena table DDL □ List all dashboards/alerts querying legacy table □ Establish record count baseline (7-day average) During parallel run: □ Daily record count comparison (legacy vs new) □ Verify Parquet files are readable via Athena □ Confirm S3 lifecycle policies applied to new prefix □ Test Logs Insights queries on CWL log group Post-migration: □ Update all Athena queries to new table □ Update CloudWatch dashboards □ Update alerting (metric filters if applicable) □ Set legacy S3 prefix Glacier transition (keep for history) □ Disable legacy S3 direct delivery □ Document new architecture ``` --- ## 5. Summary | Topic | Key Takeaway | |---|---| | **Cost (`APN1-VendedLog-Bytes`)** | Staying within pre-config range confirms $0 Vended Log ingestion — consistent with official guidance | | **Format: JSON vs Plain** | JSON ~35% larger due to repeated field names; use when schema self-description is needed | | **Format: Parquet** | 85-90% smaller than plain text; optimal for S3+Athena workloads | | **Health check logs** | Highly repetitive values make Parquet compression especially effective | | **Architecture** | S3(Parquet) + Athena as default; CWL Logs Insights only for real-time/incident use | | **CWL Retention** | 1-3 days maximum; longer retention in CWL wastes ~43% vs equivalent S3 cost | | **Migration** | Run parallel pipelines for 2 weeks, validate record counts, then cut over |
Application Load Balancer（ALB）CloudWatch LogsAmazon S3AWS Cost ExplorerAWS
2026.07.27

This page has been translated by machine translation. View original

Introduction

On July 23, 2026, there was an update integrating ALB logging into the CloudWatch Logs delivery feature (Vended Logs).

https://aws.amazon.com/jp/about-aws/whats-new/2026/07/amazon-cloudwatch-logs/

A previous article introduces log output to CloudWatch Logs.

https://dev.classmethod.jp/articles/alb-access-logs-cloudwatch-logs-vended/

This article examines the cost impact when delivering ALB logs to S3 using the CloudWatch Logs integration.

Note that in this article, the conventional S3 output configured via ALB attributes (such as access_logs.s3.*) is referred to as the "legacy configuration."

Verification Details

This time, we targeted ALB health check logs. We chose these because they are less affected by actual access traffic and output logs at regular intervals, making it easier to compare file sizes and output counts across formats.

We delivered the same health check logs in parallel to S3 in three formats — JSON, plain, and Parquet — and verified the file sizes and cost impact of S3 delivery via the CloudWatch Logs integration.

Verification Environment

Item Value
Region ap-northeast-1
Load Balancer 1 Application Load Balancer (<ALB_NAME>)
Target Log Type ALB_HEALTH_CHECK_LOGS
Number of Log Fields 10
Health Check Interval 30 seconds
Log File Output Interval 5 minutes (per ALB node)
Output Formats JSON / plain (space-delimited) / Parquet
Measurement Period 2026-07-22 to 2026-07-27 (S3 delivery configuration implemented on 07-24)

In this comparison, to avoid the impact of differences in destination, all three formats were delivered to the same S3 bucket <LOG_BUCKET>, with only the prefix differing per format.

All size and cost figures that follow are based solely on health check logs.

Comparison by Format

File Size

We extracted one set of files containing 10 records output from the same timestamp and the same ALB node for comparison.

Format File Size Per-Record Equivalent Ratio to plain (%)
plain (gz) 283 B 28.3 B 100%
JSON (gz) 393 B 39.3 B 139%
Parquet (internal GZIP) 4,042 B 404.2 B 1,429%

The "per-record equivalent" is a reference value obtained by dividing the compressed file size by 10 records. Compression efficiency varies with the number of records, so it does not scale proportionally when the record count increases.

plain (space-delimited)

The plain format, like the legacy configuration, lists values separated by spaces.

http 2026-07-27T00:25:02.444270Z 0.022732728 <TARGET_IP>:3000 <TARGET_GROUP> PASS 200 - app/<ALB_NAME>/<ALB_ID> <ALB_NODE_IP>

JSON

JSON outputs each record as a single-line object.

{"type":"http","time":"2026-07-27T00:25:02.444270Z","latency":"0.022732728","target_addr":"<TARGET_IP>:3000","target_group_id":"<TARGET_GROUP>","status":"PASS","status_code":"200","reason_code":"-","elb":"app/<ALB_NAME>/<ALB_ID>","ip_address":"<ALB_NODE_IP>"}

The ratio changes with differing field counts (in the previous article's access log measurements, JSON was 2.53 times the size of plain).

Since JSON pairs keys with values, it can be handled without relying on positional values as in plain format. If you anticipate field additions, JSON is easier to work with.

Parquet

Reading a Parquet file with pyarrow allowed us to check the proportion of metadata.

  • rows=10, row_groups=1, columns=10
  • Compression format is GZIP
  • Row group data section is 1,070 B
  • Footer and schema metadata is approximately 2,972 B (approximately 73% of the entire file)

For output at 5-minute intervals with 10 records as in this case, Parquet is not suitable. Metadata accounts for approximately 73% of the total, and the large file size was more prominent than the advantages of columnar format compression and scan range narrowing. In addition, the official announcement states that Parquet conversion incurs a separate charge, with the official blog showing a unit price for Northern Virginia of $0.035/GB. There is no reason to choose this format at this scale for S3 storage, even in the Tokyo region.

While outside the scope of this verification, if you want to accumulate data in Parquet, another approach is to batch-convert logs delivered in JSON or plain format using Athena CTAS or Glue ETL. Since you can increase the number of records per file, this keeps the metadata ratio lower than saving small files directly as Parquet.

Actual Cost Measurement

We checked daily usage by usage type for ap-northeast-1 in Cost Explorer. We compare 2026-07-22 and 07-23 (before configuration) with 2026-07-25 (the day after configuration).

Date S3 Recording APN1-VendedLog-Bytes Usage (MB equivalent)
2026-07-22 None 10.57
2026-07-23 None 10.75
2026-07-25 Yes 10.73

We converted APN1-VendedLog-Bytes from Cost Explorer to MB units for comparison. Even after configuring the CloudWatch Logs integration, this usage type remained within the 10.57–10.75 MB range observed before configuration.

Note that the approximately 10 MB being recorded even before configuration is attributable to VPC flow logs running in the same account. We set cost allocation tags on the flow log log group and confirmed that this usage type was occurring only from flow logs. No Vended Logs delivery other than the ALB logs in this verification was configured.

The official announcement states that delivering ALB logs to Amazon S3 is free of charge. Delivery to CloudWatch Logs and Amazon Data Firehose is billed as Vended Logs. The comparison results for APN1-VendedLog-Bytes this time are also consistent with this pricing model.

Configuration Procedure (Reference)

Here we summarize the configuration for delivering the same logs in parallel in multiple formats to S3 as done in this verification.

Parallel delivery in 3 formats via CLI

Create one Delivery Source, then create a Delivery Destination and Delivery for each of JSON, plain, and Parquet.

When switching formats in production, use separate S3 buckets per format if possible to avoid mixing with existing logs. If using the same bucket, at minimum separate the prefixes.

First, create the Delivery Source from which logs originate.

aws logs put-delivery-source \
  --name alb-health-check-logs \
  --resource-arn arn:aws:elasticloadbalancing:ap-northeast-1:<ACCOUNT_ID>:loadbalancer/app/<ALB_NAME>/<ALB_ID> \
  --log-type ALB_HEALTH_CHECK_LOGS

In the verification, we created a Delivery Destination targeting the same S3 bucket for each format, changing only --output-format.

aws logs put-delivery-destination \
  --name alb-vended-logs-s3-json \
  --output-format json \
  --delivery-destination-configuration destinationResourceArn=arn:aws:s3:::<LOG_BUCKET>

aws logs put-delivery-destination \
  --name alb-vended-logs-s3-plain \
  --output-format plain \
  --delivery-destination-configuration destinationResourceArn=arn:aws:s3:::<LOG_BUCKET>

aws logs put-delivery-destination \
  --name alb-vended-logs-s3-parquet \
  --output-format parquet \
  --delivery-destination-configuration destinationResourceArn=arn:aws:s3:::<LOG_BUCKET>

Finally, link them with Delivery. Change the destination ARN and suffixPath per format to prevent output from mixing within the same bucket.

aws logs create-delivery \
  --delivery-source-name alb-health-check-logs \
  --delivery-destination-arn arn:aws:logs:ap-northeast-1:<ACCOUNT_ID>:delivery-destination:alb-vended-logs-s3-json \
  --s3-delivery-configuration 'suffixPath=health-check-logs/json/{region}/{yyyy}/{MM}/{dd}/'

Run create-delivery for plain and Parquet as well, changing the destination ARN and suffixPath accordingly.

S3 Bucket Policy

When using S3 as a destination, grant delivery.logs.amazonaws.com the following permissions.

  • Object write (s3:PutObject)
  • Bucket ACL retrieval (s3:GetBucketAcl)
{
  "Effect": "Allow",
  "Principal": {
    "Service": "delivery.logs.amazonaws.com"
  },
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::<LOG_BUCKET>/*",
  "Condition": {
    "StringEquals": {
      "aws:SourceAccount": "<ACCOUNT_ID>"
    }
  }
}

{
  "Effect": "Allow",
  "Principal": {
    "Service": "delivery.logs.amazonaws.com"
  },
  "Action": "s3:GetBucketAcl",
  "Resource": "arn:aws:s3:::<LOG_BUCKET>",
  "Condition": {
    "StringEquals": {
      "aws:SourceAccount": "<ACCOUNT_ID>"
    }
  }
}

In this verification, to simplify testing, both statements use only aws:SourceAccount as the condition and do not specify aws:SourceArn. In production, we recommend also specifying aws:SourceArn to restrict the delivery source.

Reasons to Migrate from the Legacy Configuration to CloudWatch Logs Integration

As mentioned above, when delivering ALB logs to S3 via the CloudWatch Logs integration, no Vended Logs delivery charges apply. S3 storage fees and request fees continue to apply as before. Note that when delivering in parallel in multiple formats as in this verification, the number of PutObject requests increases proportionally with the number of formats.

Log delivery definitions can be separated from ALB attributes. In this verification as well, all ALB log-related attributes remained disabled, yet logs could be delivered to S3 through the Delivery defined on the CloudWatch Logs side.

Even in a configuration that prioritizes cost and restricts log storage to S3, the new method allows you to add delivery to a CloudWatch Logs log group as needed. Since you can use Logs Insights for searching and near-real-time review, this is useful during events or incident investigations.

Adding delivery to a log group incurs CloudWatch Logs charges. You can also keep logs stored only in S3 at all times and add log group delivery only before an event or during an incident reproduction period. Please consider this as needed.

Summary

For new builds storing ALB logs to S3, we consider CloudWatch Logs integration to be the first choice. Delivering to S3 incurs no Vended Logs delivery charges, you can choose the output format, and when needed, you can add delivery to a log group to investigate with Logs Insights.

For format, plain is preferable if storage size is the priority, while JSON is easier to handle if you anticipate field additions or programmatic processing.

For environments already using the legacy configuration, we recommend reviewing it when you have a maintenance window for your log analysis environment. Configure new log delivery to a separate output destination while keeping the legacy configuration enabled. Once you have verified the operation of your log analysis and log processing pipelines, you can safely switch by disabling the legacy configuration.

Share this article

AWSのお困り事はクラスメソッドへ