I tried migrating the DevelopersIO infrastructure to an ECS configuration without public subnets

We have completed the migration to a public subnet-less configuration using Regional NAT Gateway and CloudFront VPC Origin, which had been under verification for the backend infrastructure of this blog (DevelopersIO), and have started production operation.

This article introduces the details of the network configuration with public subnets completely eliminated, verification of ARM architecture (Fargate/Graviton) adoption, and considerations for cost optimization.

1. Configuration Overview: Migration to Public Subnet-Less

Based on the architecture adopting a public subnet-less VPC that was verified in a previous article, we prepared each environment for production, development, and DR.

https://dev.classmethod.jp/articles/ecs-express-mode-public-subnet-less/

Overall Configuration

The biggest feature is the elimination of public subnets from within the VPC, with all compute resources placed in private subnets.

• Regional NAT Gateway: Supports multi-AZ redundancy while eliminating the need for public subnets previously required for NAT Gateway placement.
• ECS + Internal ELB: Containers and load balancers all operate within the private domain.
• CloudFront VPC Origin: Connects CloudFront to the Internal ALB, blocking direct access from the internet.

Development Environment: Utilizing ECS Express Mode

For development and test environments, we adopted ECS Express Mode, prioritizing cost efficiency and deployment speed above all else.

• Shared ELB: We applied a configuration that shares a single ELB across approximately 10 execution environments (services with Express Mode applied). This significantly reduces fixed costs compared to creating an ALB for each service.
• SSL certificate behavior: While HTTPS is mandatory to use CloudFront VPC Origin, our verification confirmed that Express Mode automatically provides a certificate for the "aws" domain. This eliminates the need to add DNS records (for ACM) and contributes to shortening the lead time for environment provisioning.

2. Performance Verification: Adoption of ARM Architecture and Introduction of Graviton4

When building the production environment, we conducted verification from both performance and cost perspectives to determine whether to adopt x86 or ARM architecture.

x86 vs ARM Comparison on Fargate

First, we compared costs and response performance in a Fargate environment (1 vCPU, 2GB memory, 730 hours/month operation) using ALB access logs.

When using ARM Fargate Spot, the difference in response time compared to x86 on-demand was minimal (approximately 0.01-0.02 seconds).
On the other hand, we found that costs could be significantly reduced, so we decided to aggressively adopt Spot and ARM architecture in the production environment.

EC2 Graviton Generation Benchmark

Based on the above policy, we also targeted the Graviton family for regular ECS (EC2 launch type) and conducted performance tests for each generation (Graviton2, 3, 4).

Conditions: 1 vCPU, 2GB memory, 1 instance/1 task

As a result of log analysis, we reached the following conclusions.

• Adoption of Graviton4 (m8gd): We confirmed that the latest generation has the lowest latency. Additionally, since the price difference between m8gd (with disk) and m8g (without disk) is minimal, we adopted m8gd as the primary choice for the production environment and configured the built-in instance store (NVMe SSD) to be mounted and utilized as /tmp and swap space.
• Non-adoption of Graviton2 (im4gn): We confirmed that response times were inferior compared to the m8g and m7g generations, so we decided to hold off on adoption for now.

3. Ensuring Availability in Production: Hybrid Configuration

In the production environment, we adopted a hybrid configuration of EC2 and Fargate to balance cost reduction and availability maintenance.

Countermeasures against inventory shortages:
To prepare for the risk of spot instance interruptions, we applied the following distribution settings.

• Multiple AZs: Task launching is permitted across all 3 AZs.
• Diversification of instance types: Multiple families such as m7gd, m8gd, and r8gd are configured as candidates.

4. Network Cost Optimization Verification

Adoption of Dedicated ELB

In the production environment, we placed a dedicated ELB (Internal ALB). Since Internal ALB does not incur IPv4 charges (public IPv4 address usage fees), we were able to suppress additional fixed costs even with a high-availability configuration supporting 3 AZs. This has made it easier to separate access logs and metrics, improving the efficiency of incident investigations.

NAT Gateway vs VPC Endpoints

We compared and examined whether to use Interface VPC Endpoints (VPCE) or NAT Gateway as the communication path to AWS services such as ECR.

1. In the case of VPCE:

• 3 services × 3 AZs × $75–$90/month fixed cost increase

2. In the case of NAT Gateway:

• Expected traffic to services such as ECR is on the order of tens of GB per month. The estimate showed that data processing fees would also be within a few dollars per month.

We determined that at our current traffic scale, using NAT Gateway is overwhelmingly cheaper than introducing VPCE. We have also designed the configuration with an eye toward future IPv6 DualStack support and use of Egress-Only Internet Gateway (reducing NAT Gateway communication costs).

5. Operational Improvements and DR Strategy

Shortening ECS Express Mode Deployment

We identified an issue where deployment with ECS Express Mode takes a long time to complete with default settings. We manually changed the CodeDeploy settings and conducted verification.

• Canary bake time: 3 minutes → 0 minutes
• Deployment bake time: 3 minutes → 0 minutes

We confirmed that this configuration change can shorten deployment time by up to 6 minutes.

https://dev.classmethod.jp/articles/ecs-experss-reduce-deployment-time/

DR Environment: App Runner Returns

App Runner, which had been used for frontend operations since 2023, was removed from the main environment due to the difficulty of obtaining access logs and WAF cost considerations. It is currently operating as a "DR (standby) system in the Virginia region" with the following configuration.

• App Runner: Maintains a standby environment for around $10 per month.
• ECR Replication: Automates image synchronization from Tokyo to Virginia.
• StepFunctions: Achieves Lambda-less App Runner deployment via SDK integration.
• CloudFront Failover: Automatic switching upon detection of 500-series errors.

This delivers extremely high cost performance as a BCP measure.

Summary

We migrated to the new infrastructure utilizing Regional NAT Gateway and ECS Express Mode and confirmed the following results.

• Configuration: Achieved a secure, public subnet-less VPC configuration in the production environment.
• Performance: Adopted the latest Graviton4 (m8gd), achieving both cost efficiency and high performance through ARM architecture.
• Cost: By leveraging spot instances and minimizing unnecessary VPCE and IPv4 usage, we kept cost increases to a minimum even in an environment adopting VPC and ECS.

As AWS's IPv6 support continues to expand, we look forward to reducing NAT Gateway costs and ultimately using a VPC network without NAT Gateways or VPC endpoints.