I Tried Testing Whether CloudFront Origin Shield Improves Cross-Region Access Performance to S3

Introduction

The DevelopersIO front end is a Next.js app that now runs in multiple regions using the global endpoint of Lambda Web Functions (LWF).

https://dev.classmethod.jp/articles/developersio-lambda-web-functions-multiregion/

To draw the charts on author pages, the app fetches aggregated JSON stored in Virginia via CloudFront.

After switching to LWF, we received reports of missing charts, and the app logs also showed records of JSON fetches timing out at 500ms.

So, we measured in a test environment whether Origin Shield could eliminate the JSON fetch latency, and this article presents the results of introducing it to the LWF environment.

Measuring fetch time with no cache at the edge

I created a CloudFront distribution without Origin Shield.

The S3 bucket is placed in us-east-1, and I copied 200 files (gzip-compressed, 128 to 714 bytes) selected from the 1,183 production per-author aggregated JSON files. The distribution references this bucket via OAC (Origin Access Control). For the cache policy, I used a custom one that includes the query string k in the cache key. Changing the value of k causes the object to be cached separately, so I can measure from a no-cache state every time without invalidating the cache.

{
  "Name": "oss-verify-cachingoptimized-qs-k",
  "Comment": "Managed-CachingOptimized と同じ TTL・圧縮。クエリ文字列 k だけをキャッシュキーに含める（計測用）",
  "MinTTL": 1,
  "DefaultTTL": 86400,
  "MaxTTL": 31536000,
  "ParametersInCacheKeyAndForwardedToOrigin": {
    "EnableAcceptEncodingGzip": true,
    "EnableAcceptEncodingBrotli": true,
    "HeadersConfig": {"HeaderBehavior": "none"},
    "CookiesConfig": {"CookieBehavior": "none"},
    "QueryStringsConfig": {"QueryStringBehavior": "whitelist", "QueryStrings": {"Quantity": 1, "Items": ["k"]}}
  }
}

For measurement, I placed one AWS Lambda Event Function (conventional Lambda; Node.js 22, 512MB memory) in each of four regions. Each Event Function fetches the 200 files one by one in order and records the time for each fetch. The steps for one round are as follows.

1. Set the query string k to a new value.
2. The Event Function in us-east-1 fetches the 200 files. Since neither the edge nor Origin Shield has a cache, the request goes all the way to S3.
3. The Event Functions in the remaining three regions concurrently fetch the same 200 files with the same k.

In the step 3 responses, the x-cache header was Miss from cloudfront in all three regions before Origin Shield was enabled. After enabling it, all were Hit from cloudfront. Since the edge has no cache, this hit comes from Origin Shield. In other words, the values before enabling are the fetch times when the request went all the way to S3, and the values after enabling are the fetch times when it hit the Origin Shield cache.

// Event Function for measurement (conventional Lambda).
// event: { base, prefix, k, files: [..] }
// Fetches https://<ディストリビューション>/<prefix><file>?k=<k> in order and records the elapsed time
const now = () => performance.now();

export const handler = async (event) => {
  const rows = [];
  const url = (f, k) => `${event.base}/${event.prefix}${f}?k=${k}`;
  // Establish the connection (fetch with a different key; not included in the measurement)
  try { const w = await fetch(url(event.files[0], "warm-" + event.k)); await w.arrayBuffer(); } catch (e) {}
  for (const f of event.files) {
    const t0 = now();
    let status = 0, ttfb = 0, xc = "", pop = "", bytes = 0;
    try {
      const r = await fetch(url(f, event.k), { signal: AbortSignal.timeout(10000) });
      ttfb = now() - t0;
      status = r.status; xc = r.headers.get("x-cache") ?? ""; pop = r.headers.get("x-amz-cf-pop") ?? "";
      bytes = (await r.arrayBuffer()).byteLength;
    } catch (e) { status = -1; xc = String(e.name); }
    const t = now() - t0;
    rows.push([f, status, +ttfb.toFixed(1), +t.toFixed(1), xc, pop, bytes]);
  }
  return { region: process.env.AWS_REGION, k: event.k, columns: ["file", "status", "ttfb_ms", "total_ms", "x_cache", "pop", "bytes"], rows };
};

Measurement results

Without Origin Shield

These are the fetch times (ms) for 8 rounds (1,600 requests per region).

Before enabling it, ap-northeast-1 had a p50 of 493.8ms.

LWF can be expected to deliver higher CPU performance than an Event Function (512MB memory), but we confirmed that there is little margin against the 500ms timeout set in production.

https://dev.classmethod.jp/articles/lambda-web-functions-runtime-environment/

With Origin Shield

I enabled Origin Shield, specifying us-east-1, the same region as the S3 bucket.

These are the fetch times (ms) for 2 rounds after enabling it (400 requests per region).

Enabling Origin Shield reduced fetch times in the three regions other than us-east-1.

Estimating the additional cost

The price for requests to Origin Shield in us-east-1 is $0.0075 per 10,000 requests.

JSON downloads are at most 10,000 per day based on measurements. For the calculation, I assumed that 3/4 of them go through Origin Shield from regions other than us-east-1.

7,500 requests/day × 30 days = 225,000 requests → 22.5 × $0.0075 ≒ $0.17/month

Requests that hit the edge cache do not reach Origin Shield and are not charged, so the actual amount is expected to be even lower.

Summary

Enabling Origin Shield shortened the time to fetch small files from S3 by 170 to 330ms at p50. We also introduced it to the production environment, and the timeouts (500ms) that had been a problem were largely resolved.

If you run in multiple regions with Lambda Web Functions, reference S3 as the data source via CloudFront, and S3 access latency is a problem, please try Origin Shield first as a stopgap measure.

As a fundamental fix, one option is Amazon S3 Multi-Region Access Points (global endpoints), which replicate S3 across regions and allow access to the nearest bucket. I evaluated this as well. The results are covered in the next article.

https://dev.classmethod.jp/articles/s3-multi-region-access-points-cross-region-latency/

https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/MultiRegionAccessPoints.html