I tried bypassing CloudFront VPC origin failures using Inter-Region VPC peering

Introduction

From 16:45 JST (12:45 AM PDT) on July 16, 2026, to 20:18 JST (4:18 AM PDT), an outage occurred where 5xx errors increased for connections via CloudFront VPC origins. The following is the timeline according to the final report from the AWS Service Health Dashboard (SHD).

The root cause was that the fleet managing private connections to VPC origins reached an internal constraint. When this constraint was exceeded, routing configuration was no longer delivered correctly to network processors, affecting connections via VPC origins.

Between 12:45 AM and 4:18 AM PDT, we experienced increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. [...] We identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections.

As a workaround, users were advised to "change their origin type" to an origin type other than VPC origins. This article documents a practical example of following that recommended workaround—how we designed and built a route that bypasses VPC origins, and how long recovery took. The prerequisite configuration of the target service is explained in the following article.

https://dev.classmethod.jp/articles/developersio-public-subnet-less-ecs/

Observing the Outage

CloudFront access logs v2 were queried in Amazon Athena and aggregated per minute.

The following shows the count of status=0 (ClientCommError) and latency trends that increased in temporal correlation with the VPC origin outage.

The following shows the breakdown by major HTTP status. The normal rate is calculated as "(total − status=0 − 404) / total".

The progression of the outage:

• Normal through 16:52 (normal rate above 97%, P95 under 1 second)
• 16:53: Warning signs begin (39 ClientCommErrors appear)
• 16:54: Outage becomes apparent (status=0 reaches 200, P95 spikes to 30 seconds, per-minute request count in logs decreases)
• 16:55 onward: Normal rate hovers around 50% with no recovery

Since the processing time in the logs had stretched to approximately 30 seconds, most of the connection failures that occurred at 16:53 are believed to have been recorded in the 16:54 log entries.

Degraded Behavior of Existing Failover

This service had CloudFront origin failover configured, with Cloudflare Workers registered as the secondary origin. Even after connections via VPC origins began failing, the top page and article pages continued to be served normally from this fallback. However, dynamic features such as pre-publication previews were not covered by the fallback and were impacted, so we proceeded with recovery using a bypass configuration.

https://x.com/suzryo/status/2077679183439040762?s=20

Design Decisions for the Bypass Configuration

The approach was to add a route to CloudFront that does not go through VPC origins. Since the failure rate was high for requests going through VPC origins, we determined that bypassing that segment and reaching the existing backend (Tokyo region ALB → ECS) would restore functionality.

At the time we began work, there was no confirmation that the issue was specifically a VPC origin failure. Considering the possibility that the direct path from CloudFront into the Tokyo region VPC origin was the problem, we selected a public ALB in us-east-1 (N. Virginia) as the entry point for the bypass route.

The final route:

CloudFront → Virginia ALB (HTTPS) → Inter-Region VPC Peering → Tokyo NLB (fixed IP) → Tokyo ALB → ECS

Since the private IPs of an ALB are dynamic, a NLB (target type ALB) was placed on the Tokyo side to fix the target IP across the VPC peering connection.

Steps to Build the Bypass Configuration

The following describes the steps taken to reach the final configuration in order. Resource identifiers have been replaced with dummy values.

Create Inter-Region VPC Peering

A VPC peering connection was created with the N. Virginia default VPC (172.31.0.0/16) as the requester and the Tokyo production VPC (192.168.0.0/18) as the accepter.

aws ec2 create-vpc-peering-connection \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --peer-vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --peer-region ap-northeast-1 \
  --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Name,Value=temp-bypass-interregion}]' \
  --region us-east-1

VpcPeeringConnectionId: pcx-xxxxxxxxxxxxxxxxx
Status: initiating-request

Accept the request from the Tokyo region side.

aws ec2 accept-vpc-peering-connection \
  --vpc-peering-connection-id pcx-xxxxxxxxxxxxxxxxx \
  --region ap-northeast-1

Status: active

Route Table Configuration (Bidirectional)

Add routes to the peer's CIDR in the route tables of both VPCs.

# Virginia side: add route to Tokyo VPC CIDR
aws ec2 create-route \
  --route-table-id rtb-xxxxxxxxx \
  --destination-cidr-block 192.168.0.0/18 \
  --vpc-peering-connection-id pcx-xxxxxxxxxxxxxxxxx \
  --region us-east-1

# Tokyo side: add route to Virginia VPC CIDR
aws ec2 create-route \
  --route-table-id rtb-xxxxxxxxxxxxxxxxx \
  --destination-cidr-block 172.31.0.0/16 \
  --vpc-peering-connection-id pcx-xxxxxxxxxxxxxxxxx \
  --region ap-northeast-1

Security Group Configuration

Created a security group for the ALB on the Virginia side and allowed HTTP/HTTPS from the internet.

aws ec2 create-security-group \
  --group-name temp-bypass-alb-sg \
  --description "Temporary bypass ALB SG" \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --region us-east-1

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxxxxxxxxxx \
  --protocol tcp --port 443 --cidr 0.0.0.0/0 \
  --region us-east-1

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxxxxxxxxxx \
  --protocol tcp --port 80 --cidr 0.0.0.0/0 \
  --region us-east-1

On the Tokyo side, HTTP from the Virginia VPC was allowed on the existing ALB's security group.

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxxxxxxxxxx \
  --protocol tcp --port 80 --cidr 172.31.0.0/16 \
  --region ap-northeast-1

Tokyo: Create Internal NLB (Fixed Private IP, Target Type ALB)

To fix the IP for targets across the peering connection, an internal NLB was created in Tokyo. Private IPs are explicitly assigned to each subnet.

aws elbv2 create-load-balancer \
  --name temp-bypass-nlb \
  --type network \
  --scheme internal \
  --subnet-mappings \
    SubnetId=subnet-xxxxxxxxxxxxxxxxx,PrivateIPv4Address=192.168.0.10 \
    SubnetId=subnet-xxxxxxxxxxxxxxxxx,PrivateIPv4Address=192.168.16.10 \
    SubnetId=subnet-xxxxxxxxxxxxxxxxx,PrivateIPv4Address=192.168.32.10 \
  --region ap-northeast-1

Fixed IPs: 192.168.0.10, 192.168.16.10, 192.168.32.10

Create a target group with target type alb, register the existing ALB, and create a TCP:80 listener.

aws elbv2 create-target-group \
  --name temp-bypass-nlb-tg \
  --protocol TCP \
  --port 80 \
  --target-type alb \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --region ap-northeast-1

aws elbv2 register-targets \
  --target-group-arn arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:targetgroup/temp-bypass-nlb-tg/xxxxxxxxxxxxxxxx \
  --targets Id=arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:loadbalancer/app/existing-alb/xxxxxxxxxxxxxxxx \
  --region ap-northeast-1

aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:loadbalancer/net/temp-bypass-nlb/xxxxxxxxxxxxxxxx \
  --protocol TCP \
  --port 80 \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:targetgroup/temp-bypass-nlb-tg/xxxxxxxxxxxxxxxx \
  --region ap-northeast-1

Virginia: Create Internet-Facing ALB (HTTPS, ACM Wildcard Certificate)

An internet-facing ALB was created in Virginia to serve as the CloudFront origin.

aws elbv2 create-load-balancer \
  --name temp-bypass-alb \
  --type application \
  --scheme internet-facing \
  --subnets subnet-xxxxxxxxxxxxxxxxx subnet-xxxxxxxxxxxxxxxxx \
  --security-groups sg-xxxxxxxxxxxxxxxxx \
  --region us-east-1

DNSName: temp-bypass-alb-xxxxxxxxxx.us-east-1.elb.amazonaws.com

Created a target group with target type ip, associated an ACM wildcard certificate with the HTTPS listener, and configured HTTP to redirect to HTTPS.

aws elbv2 create-target-group \
  --name temp-bypass-tg \
  --protocol HTTP \
  --port 80 \
  --target-type ip \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --region us-east-1

aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/temp-bypass-alb/xxxxxxxxxxxxxxxx \
  --protocol HTTPS \
  --port 443 \
  --certificates CertificateArn=arn:aws:acm:us-east-1:123456789012:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/temp-bypass-tg/xxxxxxxxxxxxxxxx \
  --region us-east-1

aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/temp-bypass-alb/xxxxxxxxxxxxxxxx \
  --protocol HTTP \
  --port 80 \
  --default-actions 'Type=redirect,RedirectConfig={Protocol=HTTPS,Port=443,StatusCode=HTTP_301}' \
  --region us-east-1

Register Virginia ALB → Tokyo NLB Fixed IPs as Targets

Register the fixed private IPs of the Tokyo NLB as targets in the Virginia ALB's target group. Since these are IP targets across VPC peering, AvailabilityZone=all is specified.

aws elbv2 register-targets \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/temp-bypass-tg/xxxxxxxxxxxxxxxx \
  --targets Id=192.168.0.10,Port=80,AvailabilityZone=all \
            Id=192.168.16.10,Port=80,AvailabilityZone=all \
            Id=192.168.32.10,Port=80,AvailabilityZone=all \
  --region us-east-1

Register DNS CNAME Record

bypass.example.com was configured to reach the ALB, and this hostname was set to match the domain covered by the ACM wildcard certificate attached to the ALB's HTTPS listener. This allows it to be used as an HTTPS origin from CloudFront.

Add Origin to CloudFront + Switch Failover Group Primary

A new origin bypass-alb-origin was added to the CloudFront distribution, specifying bypass.example.com as the domain with HTTPS only (TLSv1.2) as the protocol. The primary in the existing origin failover group was switched to bypass-alb-origin, while the secondary remained the Cloudflare Workers origin.

Recovery Confirmation

Normal operation via the bypass route was confirmed at 18:42 JST. After the outage began (16:53), we waited approximately 1 hour for AWS to recover, but when it did not, we started building the bypass configuration at 18:03. From start to recovery took approximately 39 minutes. Normal operation of features such as pre-publication previews was confirmed.

Summary

In responding to this CloudFront VPC origin outage, we followed the AWS Health guidance to "change their origin type" to something other than VPC origins, and switched to a bypass route that does not go through VPC origins. This allowed us to restore the affected dynamic features.

Since the bypass configuration was built for temporary recovery purposes, the additional NLB and ALB are planned to be removed to avoid unnecessary costs. Going forward, a point for consideration is whether to pre-provision public subnets in the Tokyo VPC to prepare for similar outages. Since this workload does not require blocking public access, adding public subnets alone would have no cost impact. On the other hand, the mere existence of public subnets introduces the risk of IPv4 address charges and unintended exposure. We will continue to evaluate these tradeoffs.