I Tried Optimizing the Failover Wait Time for CloudFront VPC Origins Using Production Logs

Introduction

During the CloudFront VPC origin incident that occurred on July 16, 2026, it took about 30 seconds of timeout waiting before origin failover kicked in, and during that time users received no response.

https://dev.classmethod.jp/articles/cloudfront-vpc-origin-incident-20260716-log-analysis/

https://dev.classmethod.jp/articles/cloudfront-vpc-origin-failure-inter-region-vpc-peering-bypass/

In this article, I analyze the response time distribution from the production CloudFront Standard Logging v2 and derive timeout settings that shorten the failover wait time. I also verify the failover behavior in a test environment for both connection failures and response delays, and look at how failover is recorded in Standard Logging v2 and Real-time Logs.

As a result, I derived settings that fail over in about 10 seconds on response delay and about 2 seconds on connection failure, and confirmed the expected behavior in the test environment.

Verification Details

Test Environment

The analysis target was the logs of the CloudFront distribution for dev.classmethod.jp. The origin is configured as a VPC Origin in ap-northeast-1 (Internal ALB → Lambda). The logs are in Standard Logging v2, and the fields used for the analysis were time-to-first-byte, origin-fbl, x-edge-result-type, sc-status, and time-taken.

For verifying the timeout behavior, I built a separate environment with the same configuration as production.

Deriving Appropriate Values from Production Logs

I analyzed 33,670 records from three hours of Standard Logging v2, 00:00–03:00 UTC on 2026-07-21. The targets were origin requests with x-edge-result-type = Miss that were HTML pages (SSR responses).

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/standard-logging-v2.html

I limited the targets to Miss and HTML pages because timeout settings only affect requests to the origin. Static resources are served from the cache and rarely reach the origin, so they were excluded. Most Misses are SSR pages, which are the group of requests with the longest response times.

I aggregated time_to_first_byte - origin-fbl as a rough measure of overhead, including establishing the connection from the edge location to the VPC origin.

The p99 was at most 0.217 seconds, so setting ConnectionTimeout to 1 second leaves ample margin. Because VPC origins connect over the AWS backbone, p99 was stable at 0.024 seconds or less even at geographically distant edges such as GRU (São Paulo) and LHR (London).

I then aggregated origin-fbl, the time until the origin returns the first byte. This serves as the basis for OriginReadTimeout.

The maximum value of fetchWithTimeout on the application side is 5 seconds. Since origin-fbl was 1.538 seconds even at p99.9 and 3.241 seconds at the observed maximum, I set OriginReadTimeout to 5 seconds.

The derived settings are as follows.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html

Failover wait time on response delay: OriginReadTimeout 5s × ConnectionAttempts 2 = 10 seconds. On connection failure: ConnectionTimeout 1s × ConnectionAttempts 2 = 2 seconds.

Demonstrating the Timeout Behavior

In the test environment, an S3 bucket configured to return AccessDenied is intentionally used as the failover target, so a 403 is returned when failover is triggered.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html

I removed the inbound rule from the ALB's Security Group to make TCP connections impossible, and checked the behavior with ConnectionTimeout=5 (set to 5 seconds in the test to make the behavior easier to observe).

I delayed the Lambda response by 10 seconds and checked the behavior with OriginReadTimeout=5.

In both tests, the time until failover matched Timeout × ConnectionAttempts as described in the official documentation. The measured values include about 0.5 seconds of overhead because they include the round trip between the measuring client and the edge.

Using the derived settings, I checked the boundary between a normal response and failover. I expect responses under 5 seconds to be normal, and responses over 5 seconds to fail over after two attempts in about 10 seconds.

Responses under 5 seconds were returned normally from the VPC origin, and for responses over 5 seconds, failover occurred in about 10 seconds after two attempts.

Logging Strategy: Detecting Failover Activation

Detection with Standard Logging v2 (Adopted)

In the previous article, I already configured the additional fields origin-fbl, origin-lbl, and timestamp(ms). Extracting failover candidates and classifying their types is as follows.

• Primary extraction: time-taken ≧ 2 seconds and sc-status is 4xx/5xx
• Connection failure candidates: time-taken around 2 seconds
• Response delay candidates: time-taken around 10 seconds

For normal requests, origin-fbl is around 1.5 seconds even at p99.9, and for HTML pages time-taken tends to be close to it, so requests with time-taken ≧ 2 seconds and sc-status of 4xx/5xx can be extracted as failover candidates. Note that in a configuration where the failover target returns 200 normally, this condition alone cannot detect failover.

Detection with Real-time Logs (Reference)

I checked r-host (the origin that actually responded) and sr-reason (the failover reason), which cannot be obtained in Standard Logging v2, using Real-time Logs.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/real-time-logs.html

During failover, Failover:504 was recorded in sr-reason, and r-host switched from the primary origin (ALB) to the failover target (S3).

With Standard Logging v2 you can only extract failover candidates, but with Real-time Logs, r-host and sr-reason let you confirm on the CloudFront side which origin responded and why failover occurred.

Summary

By determining timeout values based on the actual state of production logs, I was able to derive settings that shorten the failover wait time on response delay from the previous roughly 30 seconds to roughly 10 seconds, and fail over in about 2 seconds on connection failure. The verification in this article also confirmed that timeout-triggered failover caused by connection failures or response delays is activated according to the configured values.

On the other hand, failover on response delay still involves a wait of about 10 seconds. Although this is a significant reduction from the previous roughly 30 seconds, it is still long compared with the responses within a few seconds that serve as a guideline for LCP, so the impact on user experience remains. In cases like the incident on July 16, 2026, where a detour is needed for several hours until full recovery, I would like to consider ways to reach a healthy origin with an even shorter wait time, such as temporarily treating the failover target as the primary or swapping the primary and secondary.