I tried to verify when CloudFront's stale-if-error returns stale content during origin timeout

Introduction

In 2023, DevelopersIO configured stale-if-error on CloudFront origins so that edge cache responses could continue even during origin failures.

https://dev.classmethod.jp/articles/developersio-cdn-cloudfront/

However, during the incident on July 16, 2026, there was a situation where the configured stale-if-error did not appear to be working as expected.

https://dev.classmethod.jp/articles/cloudfront-vpc-origin-incident-20260716-log-analysis/

The configuration was in place, so why weren't users protected? We built a reproduction environment and measured the actual behavior.

The verification results showed that during the stale-while-revalidate period, stale cache is returned immediately with zero client impact, but with origin timeout-type failures, once entering the stale-if-error period, a synchronous wait of OriginReadTimeout seconds occurs on every request.

Verification Details

Verification Environment

We prepared a Lambda Function URL that intentionally sleeps for 30 seconds as the origin, and reproduced timeouts by switching CloudFront's OriginReadTimeout. stale-while-revalidate and stale-if-error are directives added as CloudFront native features in May 2023.

https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-cloudfront-stale-while-revalidate-stale-if-error-cache-control-directives

The Cache-Control: no-store (Override=true) in the response headers policy only overrides the viewer-facing response and does not affect edge cache control.

This is the Lambda handler code.

def handler(event, context):
    time.sleep(30)
    now = datetime.now(timezone.utc).isoformat()
    body = json.dumps({"timestamp": now, "message": "OK from slow origin"})
    return {
        "statusCode": 200,
        "headers": {
            "Content-Type": "application/json",
            "Cache-Control": "max-age=90, stale-while-revalidate=90, stale-if-error=180"
        },
        "body": body
    }

We fixed the DNS resolution destination IP with curl's --resolve and recorded the Age, X-Cache, x-amz-cf-pop headers and the body timestamp. We confirmed that x-amz-cf-pop was NRT12-P9 for all requests.

# Fix the DNS resolution destination IP for CloudFront domain
RESOLVED_IP=$(dig +short "$DOMAIN" | grep -E '^[0-9]' | head -1)

curl -s \
  --resolve "${DOMAIN}:443:${RESOLVED_IP}" \
  --max-time 65 \
  -w '%{http_code} %{time_total}' \
  -D "$TMPHEADERS" \
  -o "$TMPBODY" \
  "https://${DOMAIN}/"

AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFront stale-if-error verification with Lambda Function URL origin

Parameters:
  OriginReadTimeout:
    Type: Number
    Default: 60
    Description: CloudFront origin read timeout in seconds (Phase1=60, Phase2=20)
    MinValue: 1
    MaxValue: 60

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${AWS::StackName}-lambda-role'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  SlowOriginFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub '${AWS::StackName}-slow-origin'
      Runtime: python3.12
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Timeout: 60
      Code:
        ZipFile: |
          import time
          import json
          from datetime import datetime, timezone

          def handler(event, context):
              time.sleep(30)
              now = datetime.now(timezone.utc).isoformat()
              body = json.dumps({"timestamp": now, "message": "OK from slow origin"})
              return {
                  "statusCode": 200,
                  "headers": {
                      "Content-Type": "application/json",
                      "Cache-Control": "max-age=90, stale-while-revalidate=90, stale-if-error=180"
                  },
                  "body": body
              }

  SlowOriginFunctionUrl:
    Type: AWS::Lambda::Url
    Properties:
      AuthType: NONE
      TargetFunctionArn: !GetAtt SlowOriginFunction.Arn

  SlowOriginFunctionUrlPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt SlowOriginFunction.Arn
      Action: lambda:InvokeFunctionUrl
      Principal: '*'
      FunctionUrlAuthType: NONE

  CachePolicy:
    Type: AWS::CloudFront::CachePolicy
    Properties:
      CachePolicyConfig:
        Name: !Sub '${AWS::StackName}-cache-policy'
        MinTTL: 0
        DefaultTTL: 0
        MaxTTL: 3600
        ParametersInCacheKeyAndForwardedToOrigin:
          CookiesConfig:
            CookieBehavior: none
          HeadersConfig:
            HeaderBehavior: none
          QueryStringsConfig:
            QueryStringBehavior: none
          EnableAcceptEncodingGzip: true
          EnableAcceptEncodingBrotli: true

  ResponseHeadersPolicy:
    Type: AWS::CloudFront::ResponseHeadersPolicy
    Properties:
      ResponseHeadersPolicyConfig:
        Name: !Sub '${AWS::StackName}-response-headers'
        CustomHeadersConfig:
          Items:
            - Header: Cache-Control
              Value: no-store
              Override: true

  Distribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Enabled: true
        Comment: !Sub '${AWS::StackName} - stale-if-error verification'
        DefaultCacheBehavior:
          TargetOriginId: lambda-origin
          ViewerProtocolPolicy: https-only
          CachePolicyId: !Ref CachePolicy
          ResponseHeadersPolicyId: !Ref ResponseHeadersPolicy
          Compress: true
        Origins:
          - Id: lambda-origin
            DomainName: !Select
              - 2
              - !Split ['/', !GetAtt SlowOriginFunctionUrl.FunctionUrl]
            CustomOriginConfig:
              HTTPSPort: 443
              OriginProtocolPolicy: https-only
              OriginReadTimeout: !Ref OriginReadTimeout
              OriginSSLProtocols:
                - TLSv1.2
            ConnectionAttempts: 1
            ConnectionTimeout: 10
        HttpVersion: http2and3
        PriceClass: PriceClass_200

Outputs:
  DistributionDomainName:
    Value: !GetAtt Distribution.DomainName
  DistributionId:
    Value: !Ref Distribution
  FunctionUrl:
    Value: !GetAtt SlowOriginFunctionUrl.FunctionUrl

Phase 1: Verifying stale-while-revalidate under normal conditions

We set OriginReadTimeout=60s and verified the behavior of stale-while-revalidate under conditions where Lambda's 30-second response returns normally.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html#stale-while-revalidate

Here are the measurement results over 300 seconds. The initial Miss is excluded as a warm-up.

Even when Age exceeds max-age=90 and becomes Age=92, X-Cache remains Hit from cloudfront and response time stays at 30ms unchanged. This is because stale cache is returned immediately while revalidation happens in the background; the distinction between fresh/stale was confirmed by the body timestamp.

Phase 2: Verifying stale-if-error during timeout

We changed OriginReadTimeout to 20s. Since Lambda responds in 30 seconds, all origin requests time out after 20 seconds and return 504. Starting from a warmed-up cache state, we sent requests for 400 seconds and observed the behavioral transition from the stale-while-revalidate period through the stale-if-error period and beyond stale expiration. Phase classification is based on the Age value returned by CloudFront in responses.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html#stale-if-error-only

During the stale-while-revalidate period (seq 9-17), responses are returned immediately in 25-44ms, whereas during the stale-if-error period (seq 18-22), response times jump to 11-20 seconds.

The reason seq 18's response time was approximately 11 seconds is presumed to be that the preceding background revalidation had already joined an in-progress origin connection, so the response returned after only the remaining timeout wait time. From seq 19 onward, a full 20-second wait occurs every request.

The X-Cache header during the stale-if-error period is RefreshHit from cloudfront. Unlike the Hit from cloudfront during the stale-while-revalidate period, this header indicates that stale cache was returned after an attempted origin connection.

After stale-if-error expiration (Age exceeds max-age 90 + stale-if-error 180 = 270), CloudFront returns a 504 error to the client. Since the 504 response is cached for the ErrorCachingMinTTL period (default 10 seconds), subsequent requests return the cached 504 immediately without accessing the origin (response time 30ms).

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html#use-both-stale-directives

Log Correlation (Standard Logging v2)

We correlated the results observed with the curl script against CloudFront's Standard Logging v2 (output to CloudWatch Logs).

Summary

From this verification, when an origin timeout failure persists, the time during which clients can be protected without any delay (zero-impact protection time) is determined by max-age + stale-while-revalidate. This is because once the stale-if-error period is entered, although stale cache is returned, a synchronous wait of OriginReadTimeout seconds occurs on every request.

DevelopersIO's current primary settings and the zero-impact protection time for each path are as follows.

The reason the zero-impact protection time for the top page is kept relatively short at 3 minutes is to minimize the reflection lag for new posts, but we are considering extending stale-while-revalidate to better mitigate incident impact.