I measured the VPC endpoints required to run Epic Games' Lore on a private subnet without NAT

Introduction

Epic Games' version control system Lore allows self-hosting its server (loreserver). In a previous article, we deployed loreserver using ECS Fargate, S3, and DynamoDB. However, that configuration placed the server in a public subnet within the default VPC. In this article, we replace that setup with a private subnet without a NAT gateway and identify the minimum set of VPC endpoints required.

What is Lore

Lore is an open-source version control system released by Epic Games in June 2026. It primarily targets projects with a mix of source code and large binaries, particularly game and entertainment production. The server can be self-hosted, and AWS S3 and DynamoDB can be selected as storage backends.

Verification Environment

• Region: ap-northeast-1 (Tokyo)
• Terraform: 1.15.2
• Docker (buildx): used for image builds
• loreserver: 0.8.5-nightly
• Authentication: using aws-vault profile

Target Audience

• Cloud engineers who want to self-host Lore for their team
• Those who want to operate systems in a private subnet isolated from the internet
• Those who want to know the VPC endpoint configuration when running Fargate in a private subnet

References

• Lore official repository (GitHub)
• Lore official documentation
• AWS PrivateLink pricing
• Amazon ECR interface VPC endpoints (AWS official)

Background: Why Investigate VPC Endpoints

While adding a NAT gateway would be the straightforward solution, we deliberately investigate VPC endpoints. There are two reasons, both rooted in the nature of Lore and real-world operational requirements.

First, security requirements. Game studios handle high-value intellectual property such as source code and assets, so isolating infrastructure from the internet is widely required in practice. A NAT gateway opens a communication path not only to necessary AWS services but to the entire internet, which conflicts with this requirement. VPC endpoints allow restricting reachable destinations to only the necessary AWS services, keeping traffic within AWS's internal network.

Second, cost. Lore is a version control system that handles large binaries. Asset pushes and clones move data through S3 in GB increments. NAT gateways charge $0.062 USD/GB for data processing, which becomes non-negligible at TB-scale asset traffic. In contrast, gateway-type endpoints for S3 and DynamoDB have no data processing charges. The more large data Lore handles, the more meaningful it becomes to route that data through free gateway-type endpoints.

This article therefore explores what the minimum set of VPC endpoints is when running loreserver on Fargate without any internet route. We observe what breaks and where when each endpoint is missing.

Configuration

The verification environment is a private subnet with no NAT gateway and no route to an internet gateway.

The overall structure is as follows.

There are two types of VPC endpoints. Interface-type endpoints place an ENI in the subnet and relay to the target service on port 443. They incur hourly and data processing charges. Gateway-type endpoints simply add a route to the route table, are available only for S3 and DynamoDB, and are free of charge.

The endpoint candidates are four interface-type endpoints — ecr.api/ecr.dkr/logs/sts — and two gateway-type endpoints — s3/dynamodb. Since it is unclear whether sts is required, it is included as a candidate for empirical testing.

Verification Method: Judging by Startup Success Alone

Verification is done without connecting a client, judging solely by whether the Fargate task starts successfully. This method works without client push operations because loreserver accesses S3 and DynamoDB to confirm their existence at startup. If an endpoint is missing, this startup-time access fails.

This can be confirmed in the source code. In commit 28728f2e matching the running version (0.8.5-nightly), the AWS plugin initialization works as follows. When building the S3 client, ensure_bucket is called, and its build() issues an S3 bucket existence check (HeadBucket). Next, the DynamoDB client calls ensure_table, issuing a table existence check (DescribeTable).

                .s3_with_path_style(plugin_config.s3_force_path_style)
                .ensure_bucket(&plugin_config.s3_bucket)
                .build()
                .await
                .map_err(|e| {
                    PluginError::from(PluginInitError {
                        plugin_name: plugin_name.to_string(),
                        message: format!("Failed to create S3 client: {e}"),
                    })
                })?;

                // Build DynamoDB client
                let dynamodb_client_builder = Box::pin(
                    AwsClientBuilder::builder()
                        // ...
                        .dynamodb()
                        .ensure_table(&plugin_config.dynamodb_fragments_table)
                        .ensure_table(&plugin_config.dynamodb_metadata_table);

                let dynamodb_client =
                    Box::pin(dynamodb_client_builder.build())
                        .await
                        .map_err(|e| {
                            PluginError::from(PluginInitError {
                                plugin_name: plugin_name.to_string(),
                                message: format!("Failed to create DynamoDB client: {e}"),
                            })
                        })?;

The fact that the S3 existence check comes before DynamoDB aligns with the verification results later. These are not lazy operations on the first request but synchronous operations at startup. If they fail, the server cannot start.

The specific verification steps are as follows.

1. Use the state with all candidate endpoints present as the baseline
2. Remove one endpoint, start a Fargate task, and record the stop reason (stoppedReason) from aws ecs describe-tasks and the CloudWatch logs
3. Restore the removed endpoint
4. By removing each endpoint independently, determine the necessity of each one

Verification Results

Baseline

When starting the task with all candidate endpoints present, the task starts successfully even without an internet route, and the logs reach the following line.

Server is up, waiting for shutdown signal

The logs show lines for creating the AWS immutable store (S3 and DynamoDB), mutable store, and lock store in order. Even without an internet route, loreserver works if the necessary endpoints are in place.

Verification of the sts Endpoint

Even without the sts endpoint, the task started without any issues. No STS-related errors appear in the logs either. This is because credentials for the ECS task role are retrieved from the task metadata endpoint and do not directly call STS. Therefore, the sts endpoint is not required.

Verification of the logs Endpoint

When the logs endpoint is removed, the task fails to start, and the stop reason is as follows.

ResourceInitializationError: failed to validate logger args: The task cannot find
the Amazon CloudWatch log group defined in the task definition. There is a connection
issue between the task and Amazon CloudWatch. Check your network configuration.

This failure occurs during awslogs driver initialization, so it stops before the container starts. As a result, no loreserver logs are left. The cause cannot be traced in CloudWatch and can only be found in the ECS stop reason. When verifying the presence or absence of VPC endpoints, securing logs first is essential — otherwise the causes of other failures become untraceable as well.

Note that while the stop reason message reads as though the log group cannot be found, the actual issue is unreachability to CloudWatch. The log group itself exists. Rather than taking the message at face value, it is advisable to verify the log group's existence with aws logs describe-log-groups.

Verification of ecr.api and ecr.dkr

There are two ECR endpoints involved in image retrieval, and removing each causes a different failure.

Removing ecr.api causes failure when retrieving the authentication token.

ResourceInitializationError: unable to pull secrets or registry auth: ...
operation error ECR: GetAuthorizationToken ...
Post "https://api.ecr.ap-northeast-1.amazonaws.com/": dial tcp 3.112.x.x:443: i/o timeout

Removing ecr.dkr causes failure when resolving the image manifest.

CannotPullContainerError: pull image manifest has been retried 7 time(s):
failed to resolve ref .../lorevpce-server:v1: ...
Head "https://<account ID>.dkr.ecr.ap-northeast-1.amazonaws.com/v2/.../manifests/v1":
dial tcp 54.250.x.x:443: i/o timeout

ecr.api is used for authentication token retrieval (GetAuthorizationToken), and ecr.dkr is used for manifest resolution. Since they serve different roles, both are required. It is also apparent that without private DNS, the registry name resolves to a public IP, resulting in a timeout due to no available route.

Verification of the s3 Endpoint

Removing the s3 gateway-type endpoint causes failure during image retrieval. However, the failure occurs at a different point than with ecr.dkr.

CannotPullContainerError: ... failed to copy: httpReadSeeker: failed open: ...
dial tcp 52.219.x.x:443: i/o timeout

failed to copy refers to image layer retrieval. The destination IP 52.219.x is in the S3 IP range. ECR image layers are stored in S3, so even though manifest resolution (ecr.dkr) succeeds, layer retrieval (S3) fails.

There is actually a second reason why s3 is required. As mentioned in the source code earlier, loreserver calls S3's HeadBucket at startup. In the actual test, removing s3 caused failure at the image retrieval stage first, so the startup HeadBucket failure itself was not observed. However, based on the source, S3 is required for both image retrieval and the application's startup-time access.

Verification of the dynamodb Endpoint

Removing the dynamodb gateway-type endpoint differs from the previous cases in that the container does start. Image retrieval and log output both succeed. Then loreserver calls DynamoDB at startup, fails, and exits with exit code 1. The following appears in the logs.

"span.name":"DynamoDbImpl::table_exists" ... "elapsed_ms":"18240" ...
error: ... ConnectorError { kind: Timeout, ... }
Failed to create DynamoDB client: AWS SDK error: ... Timeout

This Failed to create DynamoDB client message matches the failure message in aws.rs quoted earlier. The runtime log and the source code are consistent. This is the basis for being able to determine endpoint necessity from task startup success alone without connecting a client. Since loreserver calls DynamoDB at startup, missing the endpoint causes it to fail at this point.

Starting with the Minimum Set

As a final check, we started the task with only sts removed, keeping the five endpoints ecr.api/ecr.dkr/logs/s3/dynamodb. Creation of the immutable/mutable/lock stores all succeeded, and Server is up was reached. This confirmed by contrast that the task starts with these five endpoints and without sts.

Conclusion: Minimum Set of Required VPC Endpoints

The conclusion is as follows.

The Lore-specific dependencies are s3 and dynamodb. The general Fargate dependencies are ecr.api/ecr.dkr/logs and s3, which also handles ECR layer retrieval.

Cost Comparison with NAT Gateway

Of the minimum set, the interface-type endpoints are the three: ecr.api/ecr.dkr/logs. In ap-northeast-1, each endpoint per availability zone costs $0.014 USD/hour, with data processing at $0.01 USD/GB. The gateway-type s3 and dynamodb endpoints are free.

For comparison, a NAT gateway costs $0.062 USD/hour plus $0.062 USD/GB for data processing. For 24/7 continuous operation, three interface-type endpoints cost approximately $30 USD/month, while one NAT gateway costs approximately $45 USD/month. Looking at hourly charges alone, the difference is not large.

The difference lies in data processing. Lore flows large binaries through S3. While a NAT gateway processes this S3 traffic at $0.062 USD/GB, an S3 gateway-type endpoint routes it for free. The more frequently assets are pushed in day-to-day operations, the more this difference matters. The motivation for choosing VPC endpoints lies in both the security aspect of isolation from the internet and the cost aspect of routing large volumes of data for free.

Summary

When running loreserver on Fargate in a private subnet without a NAT gateway, the required VPC endpoints were five in total: three interface-type endpoints — ecr.api/ecr.dkr/logs — and two gateway-type endpoints — s3/dynamodb. sts is not required.

A practical pitfall discovered during verification is the ordering trap with the logs endpoint. Without logs, log delivery itself fails, making it impossible to trace the causes of other failures. When narrowing down endpoints, it is advisable to secure logs first before verifying the others.

Appendix: Full Implementation

Below are the Terraform and image definitions used in this article. Replace the s3_bucket in local.toml with your own account ID. Since the storage (S3 bucket and 4 DynamoDB tables) and task role (including DescribeTable) are the same as in the previous article, we focus here on the network-related differences.

# VPC for verification only. Enable DNS support and hostnames
# to use private DNS for interface-type endpoints.
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = { Name = "${var.name_prefix}-vpc" }
}

# One private subnet. No internet gateway or NAT attached.
resource "aws_subnet" "private" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.subnet_cidr
  availability_zone       = var.az
  map_public_ip_on_launch = false

  tags = { Name = "${var.name_prefix}-private" }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "${var.name_prefix}-private" }
}

resource "aws_route_table_association" "private" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}

# Security group for tasks. Isolation is ensured by routing,
# so egress can be fully open.
resource "aws_security_group" "task" {
  name        = "${var.name_prefix}-task"
  description = "loreserver task; egress via endpoints only (no internet route)"
  vpc_id      = aws_vpc.main.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Security group for interface-type endpoints.
resource "aws_security_group" "endpoint" {
  name        = "${var.name_prefix}-endpoint"
  description = "Allow 443 from task SG to interface endpoints"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    security_groups = [aws_security_group.task.id]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Interface-type endpoints (individually toggled on/off)
resource "aws_vpc_endpoint" "ecr_api" {
  count               = var.enable_ecr_api ? 1 : 0
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.endpoint.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "ecr_dkr" {
  count               = var.enable_ecr_dkr ? 1 : 0
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.endpoint.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "logs" {
  count               = var.enable_logs ? 1 : 0
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.logs"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.endpoint.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "sts" {
  count               = var.enable_sts ? 1 : 0
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.sts"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.endpoint.id]
  private_dns_enabled = true
}

# Gateway-type endpoints (free, associated with route table)
resource "aws_vpc_endpoint" "s3" {
  count             = var.enable_s3 ? 1 : 0
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

resource "aws_vpc_endpoint" "dynamodb" {
  count             = var.enable_dynamodb ? 1 : 0
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.dynamodb"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

variable "region" {
  type    = string
  default = "ap-northeast-1"
}
variable "az" {
  type    = string
  default = "ap-northeast-1a"
}
variable "name_prefix" {
  type    = string
  default = "lorevpce"
}
variable "vpc_cidr" {
  type    = string
  default = "10.20.0.0/16"
}
variable "subnet_cidr" {
  type    = string
  default = "10.20.1.0/24"
}
variable "image_tag" {
  type    = string
  default = "v1"
}
variable "task_cpu" {
  type    = number
  default = 512
}
variable "task_memory" {
  type    = number
  default = 1024
}

# Individual toggles for VPC endpoints. Baseline is all true.
# For incremental removal testing, set only the target endpoint to false via -var.
variable "enable_ecr_api" {
  type    = bool
  default = true
}
variable "enable_ecr_dkr" {
  type    = bool
  default = true
}
variable "enable_logs" {
  type    = bool
  default = true
}
variable "enable_sts" {
  type    = bool
  default = true
}
variable "enable_s3" {
  type    = bool
  default = true
}
variable "enable_dynamodb" {
  type    = bool
  default = true
}

resource "aws_ecs_service" "lore" {
  name            = "${var.name_prefix}-server"
  cluster         = aws_ecs_cluster.lore.id
  task_definition = aws_ecs_task_definition.lore.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = [aws_subnet.private.id]
    security_groups  = [aws_security_group.task.id]
    assign_public_ip = false
  }
}

FROM lore-server:base-amd64
COPY local.toml /etc/lore/config/local.toml

[immutable_store]
mode = "aws"
[mutable_store]
mode = "aws"
[lock_store]
mode = "aws"

[plugins.aws.immutable_store]
s3_bucket = "lorevpce-store-<account ID>"
s3_region = "ap-northeast-1"
s3_force_path_style = false
dynamodb_fragments_table = "lorevpce-fragments"
dynamodb_metadata_table = "lorevpce-metadata"
dynamodb_region = "ap-northeast-1"
timeout_millis = 120000

[plugins.aws.mutable_store]
dynamodb_table = "lorevpce-mutable"
dynamodb_region = "ap-northeast-1"
timeout_millis = 120000

[plugins.aws.lock_store]
dynamodb_table = "lorevpce-locks"
dynamodb_region = "ap-northeast-1"
timeout_millis = 120000