I Tested When CloudFront's stale-if-error Returns Stale Content During an Origin Timeout

Introduction

In 2023, DevelopersIO set stale-if-error on its CloudFront origins so that it could keep responding from the edge cache even during an origin failure.

https://dev.classmethod.jp/articles/developersio-cdn-cloudfront/

However, during the incident on July 16, 2026, stale-if-error, which should have been configured, appeared not to work as expected.

https://dev.classmethod.jp/articles/cloudfront-vpc-origin-incident-20260716-log-analysis/

Why was the user experience not protected even though the setting was in place? I built a reproduction environment and measured it.

The results show that during the stale-while-revalidate period, the stale cache is returned immediately with zero client impact, but with an origin timeout-type failure, once the stale-if-error period begins, a synchronous wait of OriginReadTimeout seconds occurs on every request.

Verification Details

Verification Environment

I prepared a Lambda Function URL that intentionally sleeps for 30 seconds as the origin, and reproduced the timeout by switching CloudFront's OriginReadTimeout. stale-while-revalidate and stale-if-error are directives that were added as native CloudFront features in May 2023.

https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-cloudfront-stale-while-revalidate-stale-if-error-cache-control-directives

The response headers policy's Cache-Control: no-store (Override=true) only overrides the response to viewers and does not affect edge cache control.

Here is the Lambda handler code.

def handler(event, context):
    time.sleep(30)
    now = datetime.now(timezone.utc).isoformat()
    body = json.dumps({"timestamp": now, "message": "OK from slow origin"})
    return {
        "statusCode": 200,
        "headers": {
            "Content-Type": "application/json",
            "Cache-Control": "max-age=90, stale-while-revalidate=90, stale-if-error=180"
        },
        "body": body
    }

I fixed the resolved IP with curl's --resolve and recorded the Age, X-Cache, and x-amz-cf-pop headers along with the timestamp in the body. I confirmed that x-amz-cf-pop was NRT12-P9 for all requests.

# CloudFront domainの名前解決先IPを固定
RESOLVED_IP=$(dig +short "$DOMAIN" | grep -E '^[0-9]' | head -1)

curl -s \
  --resolve "${DOMAIN}:443:${RESOLVED_IP}" \
  --max-time 65 \
  -w '%{http_code} %{time_total}' \
  -D "$TMPHEADERS" \
  -o "$TMPBODY" \
  "https://${DOMAIN}/"

AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFront stale-if-error verification with Lambda Function URL origin

Parameters:
  OriginReadTimeout:
    Type: Number
    Default: 60
    Description: CloudFront origin read timeout in seconds (Phase1=60, Phase2=20)
    MinValue: 1
    MaxValue: 60

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${AWS::StackName}-lambda-role'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  SlowOriginFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub '${AWS::StackName}-slow-origin'
      Runtime: python3.12
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Timeout: 60
      Code:
        ZipFile: |
          import time
          import json
          from datetime import datetime, timezone

          def handler(event, context):
              time.sleep(30)
              now = datetime.now(timezone.utc).isoformat()
              body = json.dumps({"timestamp": now, "message": "OK from slow origin"})
              return {
                  "statusCode": 200,
                  "headers": {
                      "Content-Type": "application/json",
                      "Cache-Control": "max-age=90, stale-while-revalidate=90, stale-if-error=180"
                  },
                  "body": body
              }

  SlowOriginFunctionUrl:
    Type: AWS::Lambda::Url
    Properties:
      AuthType: NONE
      TargetFunctionArn: !GetAtt SlowOriginFunction.Arn

  SlowOriginFunctionUrlPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt SlowOriginFunction.Arn
      Action: lambda:InvokeFunctionUrl
      Principal: '*'
      FunctionUrlAuthType: NONE

  CachePolicy:
    Type: AWS::CloudFront::CachePolicy
    Properties:
      CachePolicyConfig:
        Name: !Sub '${AWS::StackName}-cache-policy'
        MinTTL: 0
        DefaultTTL: 0
        MaxTTL: 3600
        ParametersInCacheKeyAndForwardedToOrigin:
          CookiesConfig:
            CookieBehavior: none
          HeadersConfig:
            HeaderBehavior: none
          QueryStringsConfig:
            QueryStringBehavior: none
          EnableAcceptEncodingGzip: true
          EnableAcceptEncodingBrotli: true

  ResponseHeadersPolicy:
    Type: AWS::CloudFront::ResponseHeadersPolicy
    Properties:
      ResponseHeadersPolicyConfig:
        Name: !Sub '${AWS::StackName}-response-headers'
        CustomHeadersConfig:
          Items:
            - Header: Cache-Control
              Value: no-store
              Override: true

  Distribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Enabled: true
        Comment: !Sub '${AWS::StackName} - stale-if-error verification'
        DefaultCacheBehavior:
          TargetOriginId: lambda-origin
          ViewerProtocolPolicy: https-only
          CachePolicyId: !Ref CachePolicy
          ResponseHeadersPolicyId: !Ref ResponseHeadersPolicy
          Compress: true
        Origins:
          - Id: lambda-origin
            DomainName: !Select
              - 2
              - !Split ['/', !GetAtt SlowOriginFunctionUrl.FunctionUrl]
            CustomOriginConfig:
              HTTPSPort: 443
              OriginProtocolPolicy: https-only
              OriginReadTimeout: !Ref OriginReadTimeout
              OriginSSLProtocols:
                - TLSv1.2
            ConnectionAttempts: 1
            ConnectionTimeout: 10
        HttpVersion: http2and3
        PriceClass: PriceClass_200

Outputs:
  DistributionDomainName:
    Value: !GetAtt Distribution.DomainName
  DistributionId:
    Value: !Ref Distribution
  FunctionUrl:
    Value: !GetAtt SlowOriginFunctionUrl.FunctionUrl

Phase 1: Confirming stale-while-revalidate under normal conditions

I set OriginReadTimeout=60s and confirmed the behavior of stale-while-revalidate under conditions where Lambda's 30-second response returns normally.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html#stale-while-revalidate

These are the results of 300 seconds of measurement. The initial Miss is excluded as warm-up.

Even when Age reaches 92, exceeding max-age=90, X-Cache remains Hit from cloudfront and the response time stays unchanged at 30ms. This is because CloudFront returns the stale cache immediately while revalidating in the background; I distinguished fresh from stale by the timestamp in the body.

Phase 2: Confirming stale-if-error during timeouts

I changed OriginReadTimeout to 20s. Since Lambda takes 30 seconds to respond, every origin request times out after 20 seconds and results in a 504. Starting from a warmed cache, I sent requests for 400 seconds and observed the behavior transitions from the stale-while-revalidate period through the stale-if-error period and on to after the stale limit was exceeded. The phase classification is based on the Age value that CloudFront returns in the response.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html#stale-if-error-only

During the stale-while-revalidate period (seq 9-17), responses are returned immediately in 25-44ms, whereas during the stale-if-error period (seq 18-22), the response time jumps to 11-20 seconds.

I infer that the response time of about 11 seconds for seq 18 was because the request joined an origin connection already in progress from the preceding background revalidation, so the response returned after only the remaining timeout wait. From seq 19 onward, a full 20-second wait occurs every time.

The X-Cache header during the stale-if-error period is RefreshHit from cloudfront. Unlike Hit from cloudfront in the stale-while-revalidate period, the header shows that the stale cache was returned after an attempted connection to the origin.

After stale-if-error is exceeded (Age exceeds max-age 90 + stale-if-error 180 = 270), CloudFront returns a 504 error to the client. Because 504 responses are cached for the ErrorCachingMinTTL (default 10 seconds), subsequent requests immediately returned the cached 504 without accessing the origin (response time 30ms).

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html#use-both-stale-directives

Log reconciliation (Standard Logging v2)

I reconciled the results observed with the curl script against CloudFront's Standard Logging v2 (output to CloudWatch Logs).

Summary

From this verification, in a situation where an origin timeout failure continues, the time during which clients can be protected without any delay (zero-impact protection time) is determined by max-age + stale-while-revalidate. This is because once the stale-if-error period begins, stale content is returned, but a synchronous wait of OriginReadTimeout seconds occurs on every request.

The current primary settings for DevelopersIO and the zero-impact protection time for each path are as follows.

The zero-impact protection time for the top page is kept short at 3 minutes in order to minimize the lag before new posts are reflected, but to mitigate the impact of failures, we would like to consider extending stale-while-revalidate.