I tried optimizing CloudFront VPC origin failover wait time from production logs

Introduction

In the CloudFront VPC origin incident that occurred on July 16, 2026, a timeout wait of approximately 30 seconds occurred before origin failover was triggered, during which no response was returned to users.

https://dev.classmethod.jp/articles/cloudfront-vpc-origin-incident-20260716-log-analysis/

https://dev.classmethod.jp/articles/cloudfront-vpc-origin-failure-inter-region-vpc-peering-bypass/

This article analyzes response time distributions from CloudFront Standard Logging v2 in a production environment to derive timeout settings that reduce failover wait times. Additionally, we verify failover behavior during connection failures and response delays in a test environment, and examine how failover is recorded in Standard Logging v2 and Real-time Logs.

As a result, we derived settings that failover in approximately 10 seconds during response delays and approximately 2 seconds during connection failures, and confirmed expected behavior in the test environment.

Verification Details

Test Environment

We analyzed logs from the CloudFront distribution of dev.classmethod.jp. The origin is configured with a VPC Origin in ap-northeast-1 (Internal ALB → Lambda). Logs use Standard Logging v2, and the fields used for analysis are time-to-first-byte, origin-fbl, x-edge-result-type, sc-status, and time-taken.

A separate environment with the same configuration as production was built for timeout behavior verification.

Deriving Appropriate Values from Production Logs

We analyzed 33,670 records spanning 3 hours from 2026-07-21 00:00 to 03:00 UTC from Standard Logging v2. The target is origin requests with x-edge-result-type = Miss and HTML pages (SSR responses).

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/standard-logging-v2.html

We limited the scope to Miss and HTML pages because timeout settings only affect requests to the origin. Static resources are served from cache and rarely reach the origin, so they were excluded. The majority of Misses are SSR pages, which represent the group with the largest response times.

We aggregated time_to_first_byte - origin-fbl as a measure of overhead including connection establishment from edge locations to the VPC origin.

The p99 is at most 0.217 seconds, providing sufficient margin even with ConnectionTimeout set to 1 second. Since VPC origins connect via the AWS backbone, even geographically distant edges such as GRU (São Paulo) and LHR (London) showed stable p99 values below 0.024 seconds.

We aggregated origin-fbl, the time until the origin returns the first byte. This serves as the basis for OriginReadTimeout.

The maximum value of fetchWithTimeout on the application side is 5 seconds. Since origin-fbl was 1.538 seconds even at p99.9 and 3.241 seconds at the observed maximum, we set OriginReadTimeout to 5 seconds.

The derived configuration values are as follows.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html

Failover wait time during response delays: OriginReadTimeout 5s × ConnectionAttempts 2 = 10 seconds. During connection failures: ConnectionTimeout 1s × ConnectionAttempts 2 = 2 seconds.

Demonstrating Timeout Behavior

In the test environment, we intentionally configured the failover destination S3 with AccessDenied, so when failover is triggered, a 403 is returned.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html

We removed the inbound rule from the ALB Security Group to make TCP connections impossible and verified ConnectionTimeout=5 behavior (set to 5 seconds in testing to make behavior easier to observe).

We delayed Lambda responses by 10 seconds and verified OriginReadTimeout=5 behavior.

In both verifications, the time to failover matched the Timeout × ConnectionAttempts formula in the official documentation. The actual measured values include approximately 0.5 seconds of overhead due to the round trip between the measurement client and the edge.

We verified the boundary between normal responses and failover using the derived configuration values. We expected that delays under 5 seconds would result in normal responses, and delays over 5 seconds would trigger failover after two attempts in approximately 10 seconds.

Responses under 5 seconds were returned normally from the VPC origin, and responses exceeding 5 seconds triggered failover after two attempts in approximately 10 seconds.

Logging Strategy: Detecting Failover Activation

Detection with Standard Logging v2 (Adopted)

In the previous article, origin-fbl, origin-lbl, and timestamp(ms) were already added to the configuration. Extraction of failover candidates and classification by type are as follows.

• Primary extraction: time-taken ≧ 2 seconds and sc-status is 4xx/5xx
• Connection failure candidates: time-taken around 2 seconds
• Response delay candidates: time-taken around 10 seconds

Since origin-fbl for normal requests is approximately 1.5 seconds even at p99.9, and time-taken for HTML pages tends to be close to this, requests with time-taken ≧ 2 seconds and sc-status of 4xx/5xx can be extracted as failover candidates. Note that this condition alone cannot detect failovers in configurations where the failover destination returns 200 normally.

Detection with Real-time Logs (Reference)

We used Real-time Logs to confirm r-host (the origin that actually responded) and sr-reason (the failover reason), which are not available in Standard Logging v2.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/real-time-logs.html

During failover, Failover:504 was recorded in sr-reason, and r-host had switched from the primary origin (ALB) to the failover destination (S3).

While Standard Logging v2 is limited to extracting failover candidates, Real-time Logs allow confirmation of the responding origin and failover reason on the CloudFront side via r-host and sr-reason.

Summary

By determining timeout values based on actual production log data, we were able to derive settings that reduce failover wait time during response delays from the previous approximately 30 seconds to approximately 10 seconds, and trigger failover in approximately 2 seconds during connection failures. The verifications in this article also confirmed that timeout-triggered failovers due to connection failures and response delays activate as configured.

On the other hand, failover during response delays still involves a wait of approximately 10 seconds. While this is a significant reduction from the previous approximately 30 seconds, it is still long compared to a response within a few seconds as a guideline for LCP, and the impact on user experience remains. In cases like the July 16, 2026 incident, where routing around the issue for hours until full recovery is necessary, we would also like to consider approaches such as temporarily treating the failover destination as the primary system or swapping primary and secondary to reach a healthy origin with shorter wait times.