I tried to see if cross-region reference performance improves with S3 Multi-Region Access Points

Introduction

In the previous article, we covered a configuration where Lambda Web Functions were deployed across multiple regions. We addressed the latency when referencing S3 in Virginia via CloudFront using Origin Shield.

https://dev.classmethod.jp/articles/cloudfront-origin-shield-s3-cross-region-latency/

Origin Shield is a measure to suppress latency by improving cache hit rates and network performance. This time, we evaluated Amazon S3 Multi-Region Access Points (hereafter, MRAP) as a candidate for a fundamental solution.

We replicated S3 in Virginia to Tokyo (ap-northeast-1), Oregon (us-west-2), and Frankfurt (eu-central-1) to create an MRAP. We compared retrieval times from Lambda Event Functions (traditional Lambda) in 4 regions. The comparison covered both the S3 API and CloudFront access, with retrieval targeting either a single S3 bucket or MRAP. We also introduce the time it takes for updates to reach replicas, along with an estimate of additional costs.

How MRAP Works

MRAP is a single global endpoint that consolidates buckets across multiple regions. Requests enter the AWS network through AWS Global Accelerator and are routed to the nearest bucket based on network latency. Data between buckets is separately synchronized using S3 cross-region replication.

There are three key points to keep in mind when using it:

• Requests require SigV4A signing.
• Access is granted only when permitted by both the MRAP policy and each bucket's bucket policy.
• Buckets included in an MRAP cannot be added, changed, or removed after creation.

https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/MultiRegionAccessPoints.html

Configuration Built

We created versioning-enabled buckets in 4 regions — us-east-1, ap-northeast-1, us-west-2, and eu-central-1 — and configured replication rules from us-east-1 to the other three. Next, we copied the same 200 files from the previous article (gzip-compressed, 128–714 bytes) to the us-east-1 bucket, and when we checked approximately 23 seconds after completion, all three buckets were fully synchronized.

MRAP is created using the s3control API in us-west-2. Creation is asynchronous and took a few minutes to become READY.

aws s3control create-multi-region-access-point \
  --region us-west-2 \
  --account-id <ACCOUNT_ID> \
  --client-token <token> \
  --details file://mrap-create.json

{
  "Name": "oss-mrap-verify",
  "PublicAccessBlock": {"BlockPublicAcls": true, "IgnorePublicAcls": true, "BlockPublicPolicy": true, "RestrictPublicBuckets": true},
  "Regions": [
    {"Bucket": "oss-mrap-us-east-1-<SUFFIX>"},
    {"Bucket": "oss-mrap-ap-northeast-1-<SUFFIX>"},
    {"Bucket": "oss-mrap-us-west-2-<SUFFIX>"},
    {"Bucket": "oss-mrap-eu-central-1-<SUFFIX>"}
  ]
}

Reading from MRAP via Lambda

With the AWS SDK (v3) bundled in the Node.js 22 runtime, the SigV4A implementation could not be loaded, making it impossible to retrieve from MRAP as-is. We bundled @aws-sdk/client-s3 and @aws-sdk/signature-v4a with the application. Importing @aws-sdk/signature-v4a at the top of the code enabled retrieval.

https://docs.aws.amazon.com/sdkref/latest/guide/feature-s3-mrap.html

The call uses the standard GetObjectCommand, passing the MRAP ARN instead of a bucket name for Bucket. For <MRAP_ALIAS>, enter the Alias value returned by aws s3control get-multi-region-access-point (including the trailing .mrap).

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import "@aws-sdk/signature-v4a";

const s3 = new S3Client({ region: "us-east-1" });
const response = await s3.send(new GetObjectCommand({
  Bucket: "arn:aws:s3::<ACCOUNT_ID>:accesspoint/<MRAP_ALIAS>",
  Key: "path/to/object"
}));

The Lambda execution role was granted s3:GetObject on both the MRAP's arn:aws:s3::<ACCOUNT_ID>:accesspoint/<MRAP_ALIAS>/object/* and the /* of each of the 4 buckets.

Reading from MRAP via CloudFront

When specifying MRAP as a CloudFront origin, configure it as a Custom Origin rather than an S3 origin. The domain name is <MRAP_ALIAS>.accesspoint.s3-global.amazonaws.com.

OAC (Origin Access Control) is created with the signing protocol set to sigv4a. For the OAC type, specify s3 even when the origin is a Custom Origin.

{
  "Name": "oss-mrap-sigv4a-oac",
  "SigningProtocol": "sigv4a",
  "SigningBehavior": "always",
  "OriginAccessControlOriginType": "s3"
}

{
  "Id": "MRAP-oss-mrap",
  "DomainName": "<MRAP_ALIAS>.accesspoint.s3-global.amazonaws.com",
  "OriginAccessControlId": "<OAC_ID>",
  "CustomOriginConfig": {
    "HTTPPort": 80,
    "HTTPSPort": 443,
    "OriginProtocolPolicy": "https-only",
    "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]}
  }
}

Both the bucket policies of the 4 buckets and the MRAP policy granted s3:GetObject to the CloudFront service principal. Both were restricted to this distribution using aws:SourceArn.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3-mrap.html

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudFrontViaMRAP",
      "Effect": "Allow",
      "Principal": {"Service": "cloudfront.amazonaws.com"},
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::<BUCKET>/*",
      "Condition": {
        "StringEquals": {
          "aws:SourceArn": "arn:aws:cloudfront::<ACCOUNT_ID>:distribution/<DIST_ID>"
        }
      }
    }
  ]
}

Other settings such as cache policy are the same as the S3 origin distribution from the previous article.

Measurement Method

The measurement method is the same as the previous article. Event Functions (Node.js 22, 512MB) placed in 4 regions retrieve 200 files one by one in sequence and record the time for each retrieval. We ran 8 rounds (1,600 requests per region per path).

For CloudFront access, the query string k was changed each round to retrieve from the edge with no cache. Origin Shield was not used. The S3 API was used with AWS SDK for JavaScript v3. For the single S3 bucket case, the us-east-1 bucket name was specified in Bucket; for MRAP, the MRAP ARN was used.

// event: { mode: "cf" | "s3" | "mrap", base, bucket, prefix, k, files: [..] }
//   cf:   Retrieve https://<distribution>/<prefix><file>?k=<k> in sequence
//   s3:   Retrieve bucket directly via S3 API
//   mrap: Pass MRAP ARN to bucket and retrieve via S3 API (SigV4A)
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import "@aws-sdk/signature-v4a";

const now = () => performance.now();

export const handler = async (event) => {
  const rows = [];
  if (event.mode === "s3" || event.mode === "mrap") {
    const s3 = new S3Client({ region: "us-east-1" });
    const get = async (key) => {
      const r = await s3.send(new GetObjectCommand({ Bucket: event.bucket, Key: key }));
      return (await r.Body.transformToByteArray()).length;
    };
    await get(event.prefix + event.files[0]); // Establish connection (not included in measurement)
    for (const f of event.files) {
      const t0 = now();
      let status = 200, bytes = 0;
      try { bytes = await get(event.prefix + f); } catch (e) { status = e.$metadata?.httpStatusCode ?? 0; }
      const t = now() - t0;
      rows.push([f, status, +t.toFixed(1), +t.toFixed(1), "", "", bytes]);
    }
  } else {
    const url = (f, k) => `${event.base}/${event.prefix}${f}?k=${k}`;
    try { const w = await fetch(url(event.files[0], "warm-" + event.k)); await w.arrayBuffer(); } catch (e) {}
    for (const f of event.files) {
      const t0 = now();
      let status = 0, ttfb = 0, xc = "", pop = "", bytes = 0;
      try {
        const r = await fetch(url(f, event.k), { signal: AbortSignal.timeout(10000) });
        ttfb = now() - t0;
        status = r.status; xc = r.headers.get("x-cache") ?? ""; pop = r.headers.get("x-amz-cf-pop") ?? "";
        bytes = (await r.arrayBuffer()).byteLength;
      } catch (e) { status = -1; xc = String(e.name); }
      const t = now() - t0;
      rows.push([f, status, +ttfb.toFixed(1), +t.toFixed(1), xc, pop, bytes]);
    }
  }
  return { region: process.env.AWS_REGION, mode: event.mode, k: event.k, columns: ["file", "status", "ttfb_ms", "total_ms", "x_cache", "pop", "bytes"], rows };
};

Measurement Results

Units are in ms.

Direct Retrieval via S3 API

p50

p95

With a single S3 bucket, retrieval time increased the farther the region was from us-east-1, but with MRAP, all regions stayed within 28–38ms at p50. Retrieval from us-east-1 was nearly the same for both the single S3 bucket and MRAP.

Via CloudFront (No Edge Cache)

p50

p95

In ap-northeast-1, p50 dropped from 492ms to 38ms. This is shorter than Origin Shield from the previous article (p50 159.5ms), and is comparable to retrieving from MRAP directly via S3 API (p50 30.0ms).

Time for Updates to Propagate

We overwrote the same key in the us-east-1 bucket 5 times. We measured the time for the new content to reach the other 3 buckets, and the time until the new content could be read via MRAP from an Event Function in ap-northeast-1. Replication used the default settings without S3 Replication Time Control.

All 5 rounds took approximately 22 seconds. During this time, reading from MRAP in ap-northeast-1 returned the old content with HTTP 200.

Additional Cost Estimate

Using MRAP adds the following costs on top of a single S3 bucket configuration:

• PUT requests to replication destination buckets (priced at destination region rates)
• Cross-region data transfer ($0.02 per GB)
• Replica storage
• MRAP data routing ($0.0033 per GB)

We calculated based on the assumed configuration of the author page JSON targeted for production (1,200 files, approximately 1KB per file, all files overwritten once per day, 10,000 downloads per day). Assuming old versions from versioning are deleted, their storage costs are not included.

• Replication PUT: 1,200 requests/day × 30 days = 36,000 requests/month. At destination region rates (per 1,000 requests: Tokyo $0.0047, Oregon $0.005, Frankfurt $0.0054): $0.17 + $0.18 + $0.19 ≈ $0.54/month
• Cross-region data transfer: 1.2MB × 30 days × 3 regions ≈ 108MB → $0.002/month
• Replica storage: 1.2MB × 3 regions → less than $0.001/month
• MRAP data routing: 1KB × 10,000 times × 30 days = 300MB → $0.001/month

The total is approximately $0.54/month, with the majority being PUT requests to replication destinations.

https://aws.amazon.com/jp/s3/pricing/

Summary

By using MRAP, Lambda in regions far from S3 was able to achieve read performance nearly equivalent to S3 in the same region.

However, since the author page JSON targeted for production updates all 1,200 files every day, all files become subject to replication daily. Additionally, since introducing MRAP is expected to increase management complexity, we plan to continue using Origin Shield, which provides sufficient performance for the time being.

We would like to reconsider adopting MRAP in cases where S3 needs to be accessed directly without CloudFront for reasons such as authentication, or where large files are accessed at high frequency from Lambda or EC2 and cross-region transfer charges become a concern.