CloudFront VPC Origin Outage (2026-07-16): Confirming from logs that approximately 30-second timeout waits were occurring

Introduction

An outage occurred with CloudFront VPC Origins on 2026-07-16 (JST 16:45–20:18). The AWS Service Health Dashboard's final report described the root cause as follows:

We identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections.

The cause was that the fleet managing private connections to VPC Origins reached an internal constraint, causing the routing configuration distribution to network processors to fail.

The interim countermeasure taken on the day of the outage (building a bypass configuration using Inter-Region VPC Peering) is introduced in the following article.

https://dev.classmethod.jp/articles/cloudfront-vpc-origin-failure-inter-region-vpc-peering-bypass/

In this article, we analyzed the impact of the outage using CloudFront access logs and ALB access logs on either side of the VPC Origin. The target system is configured with two routes using a CloudFront Origin Failover Group:

• primary: CloudFront → VPC Origin → Internal ALB (3AZ) → ECS
• secondary: CloudFront → Cloudflare Workers (for degraded operation)

Verification Details

Verification Environment

Outage Occurrence as Seen from ALB Logs

Before the outage (07:30–07:53), the ALB was processing normally across all AZs, with HTTP responses being mostly 200/308/404 (only 2 instances of 500) and an average response time of 0.097–0.102 seconds.

The following shows request counts per second from 07:53:30 to 07:53:55 (UTC).

The last requests per AZ were AZ-c: 07:53:47, AZ-d: 07:53:49, and AZ-a: 07:53:50, with requests ceasing in all AZs in succession within approximately 3 seconds. This shows that requests stopped arriving simultaneously across all AZs, not as a failure in a specific AZ.

There were no signs of failure on the ALB/ECS side, indicating that the failure occurred at a stage before reaching the ALB. This observation is consistent with the impact on VPC Origin connections reported by AWS.

Reality of Timeouts as Seen from CloudFront Logs

The following is time-series data aggregated per minute from CloudFront access logs (29,251 origin_request entries) around the time of the outage. The ok column shows the number of requests with sc_status of 2xx/3xx, and the err column shows the number of requests with sc_status of 0 or 5xx. Since 4xx is not included in either, some rows may not have ok+err equal to total.

The first error (sc_status=0) occurred at 07:53:45, and from 07:54 onward, the p95 response time settled at approximately 30.3 seconds. This value is consistent with the approximately 30 seconds corresponding to the configured timeouts (OriginReadTimeout 30 seconds, or ConnectionTimeout 10 seconds × 3 attempts), indicating a state where communication with the VPC Origin was stalling at some stage and reaching a timeout (which was dominant will be discussed later).

The breakdown of errors is as follows.

ClientHungUpRequest (1,872 cases) are cases where the viewer disconnected before the response was complete. The majority of the remainder are ClientCommError (classification recorded as communication interruption between CloudFront and the viewer), and based on the correlation with the outage period, these are estimated to have occurred against a backdrop of origin timeouts.

Errors occurred simultaneously across a wide range of CloudFront edge locations in multiple regions, with no bias toward any specific PoP.

By path, /articles/* accounted for 72.4% of errors, reflecting the traffic composition of the site.

Effectiveness and Limitations of Origin Failover

Even during the outage, there were 60–116 successful responses per minute, and the ok rate ranged between 10–27% throughout the duration of the outage (07:54–08:29). Since 2xx responses were being returned while the VPC Origin was unable to respond, it is estimated that Cloudflare Workers configured as the secondary origin was returning responses.

In Cloudflare Workers (devio-front-failback), Invocations, which were normally nearly 0, increased to a peak of 2.1k/minute.

Cloudflare Workers (devio-front-failback) Invocations

We also confirmed pages on which the degraded operation banner was actually displayed.

Banner displayed during degraded operation

However, there was a problem in the process leading up to the failover triggering. When CloudFront reaches a timeout while establishing a connection to or obtaining a response from the primary origin, it attempts the secondary according to the configured failover conditions. In the logs for this incident, a wait of approximately 30 seconds was observed, meaning users received the degraded page only after that wait.

When the origin failover group was configured, we anticipated cases where the primary origin would return an HTTP error (5XX, etc.) that was set as a failover trigger. However, what actually occurred was an event where "no response was returned from the origin at all," resulting in behavior where the secondary was attempted only after the timeout (approximately 30 seconds) elapsed.

Interim Measure Policy

The timeout settings subject to review are as follows.

The root cause of this outage is said to be a failure in routing configuration distribution for VPC Origins. Since p95 matches the OriginReadTimeout of 30 seconds, it is possible that a TCP connection was established but no response was returned, however, from the logs in this article, whether a connection was established cannot be directly determined. Shortening OriginReadTimeout is the primary improvement measure, but since balance with SSR response times during normal operation must be considered, the appropriate value will be determined after verification. Shortening ConnectionTimeout/ConnectionAttempts is also planned as a precaution against cases where connection establishment itself fails.

Improvements Identified Through This Analysis

To resolve the aforementioned constraints, we added fields to the CloudFront access logs: origin-fbl (origin first-byte latency), origin-lbl (origin last-byte latency), and timestamp(ms) (millisecond-precision timestamp).

Summary

Through this log analysis, we confirmed that during the VPC Origin outage, a timeout wait of approximately 30 seconds was occurring on the CloudFront side. While origin failover was functioning, the wait time until a degraded response was served became a challenge, as it required waiting for the primary origin's timeout judgment.

Additionally, for timeout types and per-request behavior that could not be identified this time, by utilizing the added log fields, we expect to be able to capture origin-side latency and time correlations in greater detail going forward.

Going forward, we plan to verify behavior in a reproduction environment, then optimize CloudFront timeout settings and origin failover settings, and incorporate them as preventive measures against recurrence.