I measured the VPC endpoints required to run Epic Games' Lore on a private subnet without NAT

Introduction

Epic Games' version control system Lore allows you to self-host the server (loreserver). In a previous article, we deployed loreserver on ECS Fargate with S3 and DynamoDB. However, that configuration placed it in the default VPC's public subnet. In this article, we replace that with a private subnet without a NAT gateway and identify the minimum set of required VPC endpoints.

What is Lore

Lore is an open-source version control system that Epic Games released in June 2026. It primarily targets projects where source code and large binaries coexist, especially game and entertainment production. It supports self-hosting the server, and AWS S3 and DynamoDB can be chosen as storage backends.

Verification Environment

• Region: ap-northeast-1 (Tokyo)
• Terraform: 1.15.2
• Docker (buildx): used for building images
• loreserver: 0.8.5-nightly
• Authentication: using aws-vault profile

Target Audience

• Cloud engineers who want to self-host Lore for their own team
• Those who want to operate systems in a private subnet isolated from the internet
• Those who want to know the VPC endpoint configuration when running Fargate in a private subnet

References

• Lore official repository (GitHub)
• Lore official documentation
• AWS PrivateLink pricing
• Amazon ECR interface VPC endpoints (AWS official)

Background: Why Investigate VPC Endpoints

Although placing a NAT gateway would be the quick solution, we deliberately investigate VPC endpoints. There are two reasons, both rooted in the nature of Lore and real-world operational requirements.

The first reason is security requirements. Game studios handle highly valuable intellectual property such as source code and assets, so operations that isolate the infrastructure from the internet are widely demanded in the field. A NAT gateway opens a communication path not only to necessary AWS services but to the entire internet, which conflicts with this requirement. With VPC endpoints, you can limit destinations to only the necessary AWS services and keep communications within AWS's internal network.

The second reason is cost. Lore is a version control system that handles large binaries. It flows S3 in GB units during asset push and clone operations. NAT gateways charge 0.062 USD/GB for data processing, so with TB-scale asset traffic, this becomes a non-negligible amount. On the other hand, gateway-type endpoints for S3 and DynamoDB have no data processing charges. The more large data Lore handles, the more meaningful it becomes to route that data through the free gateway-type endpoints.

Therefore, in this article, we explore the minimum number of VPC endpoints required when running loreserver on Fargate without any route to the internet. We observe what breaks and where when even one is missing.

Configuration

The verification environment is a private subnet with no NAT gateway and no route to an internet gateway.

The overall configuration is as follows.

There are two types of VPC endpoints. Interface-type endpoints place an ENI in the subnet and relay to the target service on port 443. They incur hourly charges and data processing charges. Gateway-type endpoints simply add a route to the route table, can only be used for S3 and DynamoDB, and are free of charge.

The candidate endpoints are four interface-type endpoints: ecr.api/ecr.dkr/logs/sts, and two gateway-type endpoints: s3/dynamodb. Since whether sts is required is unknown, it is included in the candidates for actual measurement.

Verification Method: Judging by Startup Success Only

Verification is done by judging only whether the Fargate task starts up, without connecting a client. This method works without involving client push because loreserver performs existence-check accesses to S3 and DynamoDB at startup. If an endpoint is missing, these startup accesses fail.

This can be confirmed in the source code. In commit 28728f2e matching the running version (0.8.5-nightly), the AWS plugin initialization is as follows. When assembling the S3 client, ensure_bucket is called, and its build() issues an S3 bucket existence check (HeadBucket). Then the DynamoDB client calls ensure_table, issuing a table existence check (DescribeTable).

                .s3_with_path_style(plugin_config.s3_force_path_style)
                .ensure_bucket(&plugin_config.s3_bucket)
                .build()
                .await
                .map_err(|e| {
                    PluginError::from(PluginInitError {
                        plugin_name: plugin_name.to_string(),
                        message: format!("Failed to create S3 client: {e}"),
                    })
                })?;

                // Build DynamoDB client
                let dynamodb_client_builder = Box::pin(
                    AwsClientBuilder::builder()
                        // ...
                        .dynamodb()
                        .ensure_table(&plugin_config.dynamodb_fragments_table)
                        .ensure_table(&plugin_config.dynamodb_metadata_table);

                let dynamodb_client =
                    Box::pin(dynamodb_client_builder.build())
                        .await
                        .map_err(|e| {
                            PluginError::from(PluginInitError {
                                plugin_name: plugin_name.to_string(),
                                message: format!("Failed to create DynamoDB client: {e}"),
                            })
                        })?;

The fact that the S3 existence check comes before DynamoDB matches the later verification results. These are not lazy-loading operations on the first request, but synchronous operations at startup. If they fail, the server cannot start.

The specific verification steps are as follows.

1. Use the state with all candidate endpoints in place as the baseline
2. Remove only one endpoint, start the Fargate task, and record the stop reason (stoppedReason) from aws ecs describe-tasks and the CloudWatch logs
3. Restore it
4. By removing each one independently, determine the necessity of each endpoint

Verification Results

Baseline

When starting the task with all candidate endpoints in place, the task started even without an internet route, and the logs reached the following line.

Server is up, waiting for shutdown signal

The logs show lines where the AWS immutable store (S3 and DynamoDB), mutable store, and lock store are created in order. Even without an internet route, loreserver works if the necessary endpoints are in place.

Verification of the sts Endpoint

Even after removing the sts endpoint, the task started without issues. No STS-related errors appeared in the logs either. This is because the credentials for the ECS task role are obtained from the task metadata endpoint and do not directly call STS. Therefore, the sts endpoint is not required.

Verification of the logs Endpoint

When the logs endpoint was removed, the task failed to start, with the following stop reason.

ResourceInitializationError: failed to validate logger args: The task cannot find
the Amazon CloudWatch log group defined in the task definition. There is a connection
issue between the task and Amazon CloudWatch. Check your network configuration.

This failure occurs during the initialization of the awslogs driver, so it stops before the container starts. Therefore, no loreserver logs remain. The cause cannot be tracked in CloudWatch and can only be found in the ECS stop reason. When verifying the presence or absence of VPC endpoints, if you don't secure logs first, you won't be able to track the causes of other failures either.

Note that the stop reason message reads as if the log group cannot be found, but the actual issue is an inability to reach CloudWatch. The log group itself exists. Rather than taking the message at face value, it is better to verify the log group's existence with aws logs describe-log-groups.

Verification of ecr.api and ecr.dkr

There are two ECR endpoints related to image retrieval, and removing them causes different failures.

Removing ecr.api causes a failure when obtaining the authentication token.

ResourceInitializationError: unable to pull secrets or registry auth: ...
operation error ECR: GetAuthorizationToken ...
Post "https://api.ecr.ap-northeast-1.amazonaws.com/": dial tcp 3.112.x.x:443: i/o timeout

Removing ecr.dkr causes a failure when resolving the image manifest.

CannotPullContainerError: pull image manifest has been retried 7 time(s):
failed to resolve ref .../lorevpce-server:v1: ...
Head "https://<account ID>.dkr.ecr.ap-northeast-1.amazonaws.com/v2/.../manifests/v1":
dial tcp 54.250.x.x:443: i/o timeout

ecr.api is used for obtaining the authentication token (GetAuthorizationToken), and ecr.dkr is used for resolving the manifest. Since their roles are separate, both are required. Without private DNS, the registry name resolves to a public IP, and you can also observe the timeout occurring due to the lack of a route.

Verification of the s3 Endpoint

When the s3 gateway-type endpoint was removed, image retrieval failed. However, the point of failure is different from the ecr.dkr case.

CannotPullContainerError: ... failed to copy: httpReadSeeker: failed open: ...
dial tcp 52.219.x.x:443: i/o timeout

failed to copy is image layer retrieval. The destination IP 52.219.x is in S3's IP range. Since ECR image layers are stored in S3, even if manifest resolution (ecr.dkr) succeeds, the process stops at layer retrieval (S3).

There is actually one more reason why s3 is required. As mentioned in the source code above, loreserver calls S3's HeadBucket at startup. In this actual measurement, removing s3 causes failure at the image retrieval stage first, so the HeadBucket failure at startup itself cannot be observed. However, based on the source, S3 is required for both image retrieval and the startup-time access of the application.

Verification of the dynamodb Endpoint

When the dynamodb gateway-type endpoint was removed, unlike before, the container starts. This is because image retrieval and log output both succeed. After that, loreserver calls DynamoDB at startup, fails, and exits with exit code 1. The following appeared in the logs.

"span.name":"DynamoDbImpl::table_exists" ... "elapsed_ms":"18240" ...
error: ... ConnectorError { kind: Timeout, ... }
Failed to create DynamoDB client: AWS SDK error: ... Timeout

This Failed to create DynamoDB client is the same failure message from aws.rs quoted in the source code earlier. The runtime logs and the source code match. This is the basis for being able to determine whether an endpoint is necessary or not just by checking task startup success without connecting a client. Since loreserver calls DynamoDB at startup, if the endpoint is absent, it fails at this stage.

Starting with the Minimum Set

As a final check, we started the task with only sts removed, leaving the five endpoints ecr.api/ecr.dkr/logs/s3/dynamodb. All of the immutable/mutable/lock store creations succeeded, and it reached Server is up. This confirmed as a control check that startup is possible with these five endpoints and without sts.

Conclusion: Minimum Set of Required VPC Endpoints

The conclusion is as follows.

The dependencies specific to Lore are s3 and dynamodb. The dependencies common to Fargate in general are ecr.api/ecr.dkr/logs and s3, which also handles ECR layer retrieval.

Cost Comparison with NAT Gateway

Among the minimum set, the interface-type endpoints are ecr.api/ecr.dkr/logs, three in total. In ap-northeast-1, the charge is 0.014 USD/hour per endpoint per availability zone, and data processing is 0.01 USD/GB. The gateway-type s3 and dynamodb are free.

For comparison, a NAT gateway costs 0.062 USD/hour plus 0.062 USD/GB for data processing. For 24/7 continuous operation, three interface-type endpoints cost approximately 30 USD/month, while one NAT gateway costs approximately 45 USD/month. Looking at hourly charges alone, the difference is not large.

The difference appears in data processing. Lore flows large binaries through S3. This S3 traffic would be processed at 0.062 USD/GB with a NAT gateway, whereas with S3's gateway-type endpoint it flows for free. The more frequently assets are pushed in day-to-day operations, the greater this difference becomes. The motivation for choosing VPC endpoints lies in both the security aspect of isolating from the internet and the cost aspect of flowing large amounts of data for free.

Summary

When running loreserver on Fargate in a private subnet without a NAT gateway, the required VPC endpoints were five in total: three interface-type endpoints (ecr.api/ecr.dkr/logs) and two gateway-type endpoints (s3/dynamodb). The sts endpoint is not required.

A practical lesson learned from verification is the trap of the logs endpoint order. Without the logs endpoint, logs themselves cannot be sent, making it impossible to track the causes of other failures. When narrowing down endpoints, it is best to first secure logs before verifying the others.

Appendix: Full Implementation

Here are the Terraform configurations and image definitions used in this article. Please replace s3_bucket in local.toml with your own account ID. Since the storage (S3 bucket and 4 DynamoDB tables) and task role (including DescribeTable) are the same as in the previous article, we focus here on the network-related differences.

# Dedicated VPC for verification. Enable DNS support and hostnames
# to use private DNS for interface-type endpoints.
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = { Name = "${var.name_prefix}-vpc" }
}

# One private subnet. No internet gateway or NAT attached.
resource "aws_subnet" "private" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.subnet_cidr
  availability_zone       = var.az
  map_public_ip_on_launch = false

  tags = { Name = "${var.name_prefix}-private" }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "${var.name_prefix}-private" }
}

resource "aws_route_table_association" "private" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}

# Security group for tasks. Since isolation is ensured by routing, egress can be fully open.
resource "aws_security_group" "task" {
  name        = "${var.name_prefix}-task"
  description = "loreserver task; egress via endpoints only (no internet route)"
  vpc_id      = aws_vpc.main.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Security group for interface-type endpoints.
resource "aws_security_group" "endpoint" {
  name        = "${var.name_prefix}-endpoint"
  description = "Allow 443 from task SG to interface endpoints"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    security_groups = [aws_security_group.task.id]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Interface-type endpoints (individually togglable)
resource "aws_vpc_endpoint" "ecr_api" {
  count               = var.enable_ecr_api ? 1 : 0
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.endpoint.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "ecr_dkr" {
  count               = var.enable_ecr_dkr ? 1 : 0
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.endpoint.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "logs" {
  count               = var.enable_logs ? 1 : 0
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.logs"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.endpoint.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "sts" {
  count               = var.enable_sts ? 1 : 0
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.sts"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.endpoint.id]
  private_dns_enabled = true
}

# Gateway-type endpoints (free, associated with route table)
resource "aws_vpc_endpoint" "s3" {
  count             = var.enable_s3 ? 1 : 0
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

resource "aws_vpc_endpoint" "dynamodb" {
  count             = var.enable_dynamodb ? 1 : 0
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.dynamodb"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

variable "region" {
  type    = string
  default = "ap-northeast-1"
}
variable "az" {
  type    = string
  default = "ap-northeast-1a"
}
variable "name_prefix" {
  type    = string
  default = "lorevpce"
}
variable "vpc_cidr" {
  type    = string
  default = "10.20.0.0/16"
}
variable "subnet_cidr" {
  type    = string
  default = "10.20.1.0/24"
}
variable "image_tag" {
  type    = string
  default = "v1"
}
variable "task_cpu" {
  type    = number
  default = 512
}
variable "task_memory" {
  type    = number
  default = 1024
}

# Individual toggles for VPC endpoints. The baseline has all set to true.
# For incremental absence verification, set only the endpoint being checked to false using -var.
variable "enable_ecr_api" {
  type    = bool
  default = true
}
variable "enable_ecr_dkr" {
  type    = bool
  default = true
}
variable "enable_logs" {
  type    = bool
  default = true
}
variable "enable_sts" {
  type    = bool
  default = true
}
variable "enable_s3" {
  type    = bool
  default = true
}
variable "enable_dynamodb" {
  type    = bool
  default = true
}

resource "aws_ecs_service" "lore" {
  name            = "${var.name_prefix}-server"
  cluster         = aws_ecs_cluster.lore.id
  task_definition = aws_ecs_task_definition.lore.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = [aws_subnet.private.id]
    security_groups  = [aws_security_group.task.id]
    assign_public_ip = false
  }
}

FROM lore-server:base-amd64
COPY local.toml /etc/lore/config/local.toml

[immutable_store]
mode = "aws"
[mutable_store]
mode = "aws"
[lock_store]
mode = "aws"

[plugins.aws.immutable_store]
s3_bucket = "lorevpce-store-<account ID>"
s3_region = "ap-northeast-1"
s3_force_path_style = false
dynamodb_fragments_table = "lorevpce-fragments"
dynamodb_metadata_table = "lorevpce-metadata"
dynamodb_region = "ap-northeast-1"
timeout_millis = 120000

[plugins.aws.mutable_store]
dynamodb_table = "lorevpce-mutable"
dynamodb_region = "ap-northeast-1"
timeout_millis = 120000

[plugins.aws.lock_store]
dynamodb_table = "lorevpce-locks"
dynamodb_region = "ap-northeast-1"
timeout_millis = 120000