I tried bypassing CloudFront VPC origin failures using Inter-Region VPC peering

Introduction

From 16:45 JST (12:45 AM PDT) on July 16, 2026 to 20:18 JST (4:18 AM PDT), an outage occurred in which 5xx errors increased for connections via CloudFront VPC Origins. The following is the timeline according to the final report from the AWS Service Health Dashboard (SHD).

The root cause was that the fleet managing private connections to VPC origins reached an internal constraint. When this constraint was exceeded, the routing configuration was no longer correctly delivered to network processors, affecting connections via VPC origins.

Between 12:45 AM and 4:18 AM PDT, we experienced increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. [...] We identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections.

As a workaround, users were advised to "change their origin type" to an origin type other than VPC origins. This article documents a practical example of following that recommended workaround — how we designed and built a route that does not go through VPC origins, and how long it took to recover. The prerequisite configuration of the target service is explained in the following article.

https://dev.classmethod.jp/articles/developersio-public-subnet-less-ecs/

Observing the Outage

CloudFront access logs v2 were extracted using Amazon Athena and aggregated per minute.

The following shows the count of status=0 (ClientCommError) and latency trends that increased in temporal alignment with the VPC origins outage.

The breakdown by major HTTP status codes is shown below. The "normal rate" is calculated as "(total − status=0 − 404) / total."

The progression of the outage is as follows.

• Up to 16:52: Normal (normal rate above 97%, P95 under 1 second)
• 16:53: First signs (39 ClientCommErrors appear)
• 16:54: Outage becomes apparent (status=0 reaches 200, P95 spikes to 30 seconds, per-minute log counts decrease)
• 16:55 onward: Normal rate hovers around 50% with no recovery

Since processing times in the logs had extended to approximately 30 seconds, most of the connection failures that occurred at 16:53 are thought to have been recorded in the 16:54 logs.

Degraded Behavior of Existing Failover

This service had CloudFront origin failover configured, with Cloudflare Workers registered as the secondary origin. Even after connections via VPC origins began failing, the top page and article pages were being served normally via this fallback destination. However, dynamic features such as pre-publication previews were not covered by the fallback destination and were experiencing issues, so we implemented a recovery response using a bypass configuration.

https://x.com/suzryo/status/2077679183439040762?s=20

Design Decisions for the Bypass Configuration

The approach was to add a route to CloudFront that does not go through VPC origins. Since the failure rate was rising for requests going through VPC origins, we determined that routing around that segment and reaching the existing backend (ALB → ECS in the Tokyo region) would restore functionality.

At the time we began work, there was no certainty that it was a VPC origins outage. Considering the possibility that there was a problem with the route directly entering the Tokyo region VPC via VPC origins from CloudFront, we chose a public ALB in US East (N. Virginia, us-east-1) as the entry point for the route.

The final route is as follows.

CloudFront → Virginia ALB (HTTPS) → Inter-Region VPC Peering → Tokyo NLB (fixed IP) → Tokyo ALB → ECS

Since ALB private IPs can change, a NLB (target type ALB) is placed on the Tokyo side to fix the target across the VPC peering.

Steps to Build the Bypass Configuration

The steps to reach the final configuration are listed in order. Resource identifiers have been replaced with dummy values.

Creating the Inter-Region VPC Peering

A VPC peering connection was created with the default VPC in US East (N. Virginia) (172.31.0.0/16) as the requester and the production VPC in Tokyo (192.168.0.0/18) as the accepter.

aws ec2 create-vpc-peering-connection \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --peer-vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --peer-region ap-northeast-1 \
  --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Name,Value=temp-bypass-interregion}]' \
  --region us-east-1

VpcPeeringConnectionId: pcx-xxxxxxxxxxxxxxxxx
Status: initiating-request

Accept on the Tokyo region side.

aws ec2 accept-vpc-peering-connection \
  --vpc-peering-connection-id pcx-xxxxxxxxxxxxxxxxx \
  --region ap-northeast-1

Status: active

Route Table Configuration (Bidirectional)

Add routes to the other side's CIDR to the route tables of both VPCs.

# Virginia side: add route to Tokyo VPC CIDR
aws ec2 create-route \
  --route-table-id rtb-xxxxxxxxx \
  --destination-cidr-block 192.168.0.0/18 \
  --vpc-peering-connection-id pcx-xxxxxxxxxxxxxxxxx \
  --region us-east-1

# Tokyo side: add route to Virginia VPC CIDR
aws ec2 create-route \
  --route-table-id rtb-xxxxxxxxxxxxxxxxx \
  --destination-cidr-block 172.31.0.0/16 \
  --vpc-peering-connection-id pcx-xxxxxxxxxxxxxxxxx \
  --region ap-northeast-1

Security Group Configuration

A security group for the ALB was created on the Virginia side, allowing HTTP/HTTPS from the internet.

aws ec2 create-security-group \
  --group-name temp-bypass-alb-sg \
  --description "Temporary bypass ALB SG" \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --region us-east-1

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxxxxxxxxxx \
  --protocol tcp --port 443 --cidr 0.0.0.0/0 \
  --region us-east-1

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxxxxxxxxxx \
  --protocol tcp --port 80 --cidr 0.0.0.0/0 \
  --region us-east-1

On the Tokyo side, HTTP from the Virginia VPC was allowed in the existing ALB's security group.

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxxxxxxxxxx \
  --protocol tcp --port 80 --cidr 172.31.0.0/16 \
  --region ap-northeast-1

Tokyo: Creating an Internal NLB (Fixed Private IP, Target Type ALB)

An internal NLB was created in Tokyo to fix the IP as the target across the peering. Private IPs are explicitly assigned to each subnet.

aws elbv2 create-load-balancer \
  --name temp-bypass-nlb \
  --type network \
  --scheme internal \
  --subnet-mappings \
    SubnetId=subnet-xxxxxxxxxxxxxxxxx,PrivateIPv4Address=192.168.0.10 \
    SubnetId=subnet-xxxxxxxxxxxxxxxxx,PrivateIPv4Address=192.168.16.10 \
    SubnetId=subnet-xxxxxxxxxxxxxxxxx,PrivateIPv4Address=192.168.32.10 \
  --region ap-northeast-1

Fixed IPs: 192.168.0.10, 192.168.16.10, 192.168.32.10

Create a target group with target type alb, register the existing ALB, and create a TCP:80 listener.

aws elbv2 create-target-group \
  --name temp-bypass-nlb-tg \
  --protocol TCP \
  --port 80 \
  --target-type alb \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --region ap-northeast-1

aws elbv2 register-targets \
  --target-group-arn arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:targetgroup/temp-bypass-nlb-tg/xxxxxxxxxxxxxxxx \
  --targets Id=arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:loadbalancer/app/existing-alb/xxxxxxxxxxxxxxxx \
  --region ap-northeast-1

aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:loadbalancer/net/temp-bypass-nlb/xxxxxxxxxxxxxxxx \
  --protocol TCP \
  --port 80 \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:targetgroup/temp-bypass-nlb-tg/xxxxxxxxxxxxxxxx \
  --region ap-northeast-1

Virginia: Creating an Internet-Facing ALB (HTTPS, ACM Wildcard Certificate)

An internet-facing ALB was created in Virginia to serve as the CloudFront origin.

aws elbv2 create-load-balancer \
  --name temp-bypass-alb \
  --type application \
  --scheme internet-facing \
  --subnets subnet-xxxxxxxxxxxxxxxxx subnet-xxxxxxxxxxxxxxxxx \
  --security-groups sg-xxxxxxxxxxxxxxxxx \
  --region us-east-1

DNSName: temp-bypass-alb-xxxxxxxxxx.us-east-1.elb.amazonaws.com

A target group with target type ip was created, and an ACM wildcard certificate was associated with the HTTPS listener. HTTP is redirected to HTTPS.

aws elbv2 create-target-group \
  --name temp-bypass-tg \
  --protocol HTTP \
  --port 80 \
  --target-type ip \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --region us-east-1

aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/temp-bypass-alb/xxxxxxxxxxxxxxxx \
  --protocol HTTPS \
  --port 443 \
  --certificates CertificateArn=arn:aws:acm:us-east-1:123456789012:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/temp-bypass-tg/xxxxxxxxxxxxxxxx \
  --region us-east-1

aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/temp-bypass-alb/xxxxxxxxxxxxxxxx \
  --protocol HTTP \
  --port 80 \
  --default-actions 'Type=redirect,RedirectConfig={Protocol=HTTPS,Port=443,StatusCode=HTTP_301}' \
  --region us-east-1

Registering Virginia ALB → Tokyo NLB Fixed IPs as Targets

Register the fixed private IPs of the Tokyo NLB in the Virginia ALB's target group. Since these are IP targets across VPC peering, AvailabilityZone=all is specified.

aws elbv2 register-targets \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/temp-bypass-tg/xxxxxxxxxxxxxxxx \
  --targets Id=192.168.0.10,Port=80,AvailabilityZone=all \
            Id=192.168.16.10,Port=80,AvailabilityZone=all \
            Id=192.168.32.10,Port=80,AvailabilityZone=all \
  --region us-east-1

Registering a DNS CNAME Record

bypass.example.com was set up to reach the ALB, and this hostname was made to match the target domain of the ACM wildcard certificate configured on the ALB's HTTPS listener. This allows it to be used as an HTTPS origin from CloudFront.

Adding an Origin to CloudFront + Switching the Primary in the Failover Group

A new origin bypass-alb-origin was added to the CloudFront distribution. The domain bypass.example.com was specified with HTTPS only (TLSv1.2) as the protocol. The primary in the existing origin failover group was switched to bypass-alb-origin, while the secondary remains the Cloudflare Workers origin.

Recovery Confirmation

Normal operation via the bypass route was confirmed at 18:42 JST. After waiting approximately 1 hour for AWS to recover from the outage (which began at 16:53) without success, work on building the bypass configuration began at 18:03. It took approximately 39 minutes from the start of work to recovery. Normal operation of features such as pre-publication previews was confirmed.

Summary

In response to this CloudFront VPC Origins outage, we followed the guidance from AWS Health to "change their origin type" to one other than VPC origins and switched to a bypass route that does not go through VPC origins. This allowed us to restore the affected dynamic features.

Since the bypass configuration was built for temporary recovery purposes, the added NLB and ALB are scheduled to be removed to avoid unnecessary costs. Going forward, whether to prepare a public subnet in the Tokyo VPC in advance in case of a similar outage is a topic for consideration. Since this workload does not require blocking public access, simply adding a public subnet would have no cost impact. On the other hand, the very existence of a public subnet introduces the risk of IPv4 address charges and unintended exposure. We will consider these tradeoffs as we move forward.