Confirming from the Logs That a CloudFront VPC Origin Outage (2026-07-16) Caused Timeout Waits of About 30 Seconds

Introduction

On 2026-07-16 (JST 16:45–20:18), a failure occurred in CloudFront VPC origins. The final report on the AWS Service Health Dashboard explains the root cause as follows.

We identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections.

The cause was that the fleet managing private connections to VPC origins reached an internal constraint, and distribution of routing configuration to the network processors failed.

The temporary workaround on the day of the failure (building a bypass configuration using Inter-Region VPC peering) is covered in the following article.

https://dev.classmethod.jp/articles/cloudfront-vpc-origin-failure-inter-region-vpc-peering-bypass/

In this article, I analyzed the impact of the failure using the CloudFront access logs and ALB access logs located before and after the VPC origin. The target system is configured with a CloudFront Origin Failover Group with the following two paths.

• primary: CloudFront → VPC Origin → Internal ALB (3 AZs) → ECS
• secondary: CloudFront → Cloudflare Workers (for degraded operation)

Verification

Verification environment

Onset of the failure as seen in the ALB logs

Before the failure (07:30–07:53), the ALB was processing normally in all AZs, HTTP responses were almost all 200/308/404 (only 2 were 500), and the average response time was 0.097–0.102 seconds.

The following is the number of requests per second from 07:53:30 to 07:53:55 (UTC).

The last request in each AZ was AZ-c: 07:53:47, AZ-d: 07:53:49, and AZ-a: 07:53:50, so they stopped one after another within about 3 seconds. This shows that it was not a failure of a specific AZ, but that requests stopped reaching all AZs at once.

There were no signs of failure on the ALB/ECS side, so we can conclude that the failure occurred at a stage before reaching the ALB. This observation is consistent with the impact on VPC origin connections announced by AWS.

The reality of the timeouts as seen in the CloudFront logs

The following is a time series of the CloudFront access logs before and after the failure (29,251 origin_request entries), aggregated per minute. The ok column shows the number of requests with sc_status 2xx/3xx, and the err column shows the number of requests with sc_status 0 or 5xx. 4xx is not included in either, so some rows have ok+err not equal to total.

The first error (sc_status=0) occurred at 07:53:45, and from 07:54 onward the p95 response time is stuck at about 30.3 seconds. This value is consistent with about 30 seconds, which corresponds to the configured timeouts (OriginReadTimeout 30 seconds, or ConnectionTimeout 10 seconds × 3 attempts), meaning communication with the VPC origin stalled at some stage and reached the timeout (which one was dominant is discussed later).

The breakdown of errors is as follows.

ClientHungUpRequest (1,872 cases) are cases where the viewer closed the connection before the response completed. Most of the rest are ClientCommError (a category recorded as a communication interruption between CloudFront and the viewer), and given the correlation with the failure period, they are presumed to have occurred against the background of origin timeouts.

Errors occurred simultaneously across a wide range of CloudFront edge locations in multiple regions, and no bias toward a specific PoP was observed.

By path, /articles/* accounted for 72.4% of the errors, a distribution that reflects the site's traffic composition.

Effect and limits of origin failover

Even during the failure, there were 60–116 successful responses per minute, and the ok rate stayed at 10–27% throughout the failure (07:54–08:29). Since 2xx responses were being returned while the VPC origin could not respond, it is presumed that the Cloudflare Workers configured as the secondary origin were returning the responses.

On Cloudflare Workers (devio-front-failback), Invocations, which are normally almost 0, increased to a maximum of 2.1k/minute.

Invocations of Cloudflare Workers (devio-front-failback)

I also confirmed pages on which the banner for degraded operation was actually displayed.

The banner displayed during degraded operation

However, there was a problem in the process until failover was triggered. When CloudFront reaches a timeout while establishing a connection to the primary origin or retrieving a response, it tries the secondary according to the configured failover criteria. In these logs, a wait of about 30 seconds was observed, and the structure was such that users were served the degraded page only after that wait.

When the origin failover group was configured, I had assumed the case where the primary origin returns an HTTP error (such as 5XX) set as a failover trigger. However, what actually occurred was the phenomenon of "no response at all from the origin," and the secondary was tried only after waiting for the timeout (about 30 seconds) to expire.

Policy for the interim measures

The timeout settings to be reviewed are as follows.

The cause of this failure is said to be the failure to distribute the routing configuration of the VPC origin. Since the p95 matches the OriginReadTimeout of 30 seconds, it is possible that the TCP connection was established but no response was returned, but the logs in this article do not let us directly determine whether the connection was established. Shortening OriginReadTimeout is the primary improvement measure, but because it is necessary to balance it against the SSR response time during normal operation, the appropriate value will be decided after verification. Shortening ConnectionTimeout/ConnectionAttempts is planned to be carried out as well, as preparation for cases where establishing the connection itself fails.

Improvements identified in this analysis

To remove the constraint mentioned above, I added fields to the CloudFront access logs. They are three: origin-fbl (origin first byte latency), origin-lbl (origin last byte latency), and timestamp(ms) (timestamp with millisecond precision).

Summary

This log analysis confirmed that during the VPC origin failure, CloudFront was in a state where it waited about 30 seconds for a timeout. Although origin failover was functioning, it had to wait for the primary origin's timeout determination, so the wait time until the degraded response became an issue.

In addition, for the timeout type and per-request behavior that could not be identified this time, by using the added log fields, we expect to be able to capture origin-side latency and time correlations in more detail going forward.

Going forward, after verifying the behavior in a reproduction environment, I plan to optimize CloudFront's timeout settings and origin failover settings and apply them as measures to prevent recurrence.