I verified connectivity to private resources within a VPC using the VPC Egress Connector of Lambda MicroVMs

Introduction

In Lambda MicroVMs, you can choose between the default INTERNET_EGRESS and VPC_EGRESS (which specifies a subnet and security group within a VPC) as the outbound communication (Egress) options from MicroVMs.

VPC_EGRESS is not a "closed network only" feature. If the route table of the subnet where the connector is placed has a default route to a NAT Gateway, you can reach both VPC resources and the internet. On the other hand, if placed in a subnet without a default route, you can limit communication to within the VPC only.

If you want to connect from a MicroVM to RDS, ElastiCache, internal APIs, etc. placed in a private subnet, use a VPC Egress connector. In this article, we prepare a simple HTTP server on EC2 as a substitute for the connection target, and verify whether it can be reached via the VPC Egress connector. Connection verification to RDS/ElastiCache is out of scope for this article.

This article verifies the following:

• VPC Egress connector creation steps and state transitions
• Reachability with a connector via NAT Gateway
• Reachability with a closed-network connector without a default route
• Access control using security groups
• ENI reality and lifecycle
• Behavior when specifying multiple connectors
• Cross-subnet communication
• DNS resolution
• Impact on MicroVMs when a connector is deleted

https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-networking.html

Preparing the Test Environment

VPC Configuration

The following configuration was created using CloudFormation.

• VPC: 10.0.0.0/16 (DNS Hostnames / DNS Support both enabled)
• NAT subnet (10.0.0.0/24, ap-northeast-1a): Internet connectivity via Regional NAT Gateway
• Isolated subnet (10.0.1.0/24, ap-northeast-1a): No default route in route table, VPC-internal communication only
• Target EC2: Within NAT subnet, simple HTTP server on port 80 (returns JSON response)

Security Group Design

In a VPC Egress connector, the connector-side SG controls outbound traffic from MicroVMs. On the target side, the SG is designed to allow inbound traffic with the connector SG as the source.

This design allows only traffic via the connector to be permitted on the target side.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Lambda MicroVMs VPC Egress Test - Regional NAT Gateway + Isolated Subnet'

Parameters:
  ProjectName:
    Type: String
    Default: 'microvms-egress-test'

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: '10.0.0.0/16'
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: Name
          Value: !Ref ProjectName

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-igw'

  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  NATGatewayEIP:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-eip'

  RegionalNATGateway:
    Type: AWS::EC2::NatGateway
    DependsOn: InternetGatewayAttachment
    Properties:
      AvailabilityMode: regional
      ConnectivityType: public
      VpcId: !Ref VPC
      AvailabilityZoneAddresses:
        - AvailabilityZone: !Select [0, !GetAZs '']
          AllocationIds:
            - !GetAtt NATGatewayEIP.AllocationId
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-regional-nat-gw'

  NATSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: '10.0.0.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-subnet-1a'

  NATRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-rt'

  NATRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref NATRouteTable
      DestinationCidrBlock: '0.0.0.0/0'
      NatGatewayId: !Ref RegionalNATGateway

  NATSubnetAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref NATSubnet
      RouteTableId: !Ref NATRouteTable

  IsolatedSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: '10.0.1.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-isolated-subnet-1a'

  IsolatedRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-isolated-rt'

  IsolatedSubnetAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref IsolatedSubnet
      RouteTableId: !Ref IsolatedRouteTable

  ConnectorSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for MicroVM VPC Egress Connector (NAT subnet)'
      VpcId: !Ref VPC
      SecurityGroupEgress:
        - IpProtocol: '-1'
          CidrIp: '0.0.0.0/0'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-connector-sg'

  ConnectorIsolatedSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for MicroVM VPC Egress Connector (isolated subnet)'
      VpcId: !Ref VPC
      SecurityGroupEgress:
        - IpProtocol: '-1'
          CidrIp: '10.0.0.0/16'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-connector-isolated-sg'

  TargetSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for target resource in VPC'
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !Ref ConnectorSG
          Description: 'From MicroVM connector (NAT)'
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !Ref ConnectorIsolatedSG
          Description: 'From MicroVM connector (isolated)'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-target-sg'

  TargetInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: '{{resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64}}'
      InstanceType: t4g.nano
      SubnetId: !Ref NATSubnet
      SecurityGroupIds:
        - !Ref TargetSG
      UserData:
        Fn::Base64: |
          #!/bin/bash
          dnf install -y python3
          cat > /home/ec2-user/server.py << 'EOF'
          import http.server, json, time, socket
          class H(http.server.BaseHTTPRequestHandler):
              def do_GET(self):
                  self.send_response(200)
                  self.send_header('Content-Type','application/json')
                  self.end_headers()
                  self.wfile.write(json.dumps({"source":"vpc-target","host":socket.gethostname(),"time":time.time()}).encode())
          http.server.HTTPServer(('0.0.0.0',80),H).serve_forever()
          EOF
          python3 /home/ec2-user/server.py &
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-target'

The stack was deployed with the following command.

aws cloudformation deploy \
  --stack-name microvms-egress-test \
  --template-file vpc.yaml \
  --region ap-northeast-1

Creating the NetworkConnectorOperatorRole

To create a VPC Egress connector, an IAM role is required for the Lambda infrastructure to create ENIs in the customer account's VPC.

The trust policy trusts lambda.amazonaws.com and restricts it to your own account using the aws:SourceAccount condition.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "XXXXXXXXXXXX"
        }
      }
    }
  ]
}

The permission policy allows creating, deleting, and tagging ENIs.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DeleteNetworkInterface",
        "ec2:CreateTags"
      ],
      "Resource": "*"
    }
  ]
}

aws iam create-role \
  --role-name NetworkConnectorOperatorRole \
  --assume-role-policy-document file://trust-policy.json

aws iam put-role-policy \
  --role-name NetworkConnectorOperatorRole \
  --policy-name NetworkConnectorENIPolicy \
  --policy-document file://eni-policy.json

Connector Creation and State Transitions

Creating the NAT-Routed Connector

Create the connector by specifying the NAT subnet and connector SG (Egress fully open).

aws lambda-core create-network-connector \
  --connector-name nat-egress-connector \
  --connector-type VPC_EGRESS \
  --vpc-config SubnetIds=subnet-XXXXXXXXXXXXXXXXX,SecurityGroupIds=sg-XXXXXXXXXXXXXXXXX \
  --operator-role-arn arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole \
  --region ap-northeast-1

{
    "NetworkConnector": {
        "ConnectorArn": "arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector",
        "ConnectorName": "nat-egress-connector",
        "ConnectorIdentifier": "nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        "ConnectorType": "VPC_EGRESS",
        "State": "PENDING",
        "VpcConfig": {
            "SubnetIds": ["subnet-XXXXXXXXXXXXXXXXX"],
            "SecurityGroupIds": ["sg-XXXXXXXXXXXXXXXXX"],
            "VpcId": "vpc-XXXXXXXXXXXXXXXXX"
        },
        "OperatorRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole"
    }
}

Creating the Isolated Connector

Specify the isolated subnet and connector SG (Egress restricted to VPC only).

aws lambda-core create-network-connector \
  --connector-name isolated-egress-connector \
  --connector-type VPC_EGRESS \
  --vpc-config SubnetIds=subnet-XXXXXXXXXXXXXXXXX,SecurityGroupIds=sg-XXXXXXXXXXXXXXXXX \
  --operator-role-arn arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole \
  --region ap-northeast-1

State Transition: PENDING → ACTIVE

In this verification, both connectors transitioned from PENDING to ACTIVE in approximately 4 to 5 minutes. They cannot be attached to a MicroVM for use until they become ACTIVE.

aws lambda-core get-network-connector \
  --identifier nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
  --region ap-northeast-1

MicroVM Launch and Connectivity Verification

Launching a MicroVM

Specify the connector ARN using the --egress-network-connectors parameter of run-microvm.

aws lambda-microvms run-microvm \
  --microvm-image-identifier arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:microvm-image:egress-test-app-v3 \
  --egress-network-connectors ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector \
  --region ap-northeast-1

The test MicroVM executes outbound HTTP requests from within using the /fetch?url=<URL> endpoint.

Verifying the NAT-Routed Connector

We confirmed that a MicroVM with the NAT-routed connector attached can reach both the VPC internal target and the internet.

Connecting to VPC internal target:

curl "https://<microvm-endpoint>/fetch?url=http://10.0.0.126"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668000.0}"}

Connecting to the internet:

curl "https://<microvm-endpoint>/fetch?url=http://checkip.amazonaws.com"

XX.XXX.XXX.XXX

The exit IP matched the NAT Gateway's EIP. In this configuration, since the default route of the subnet where the connector is placed points to the NAT Gateway, internet-bound traffic from MicroVMs goes through the NAT Gateway.

Verifying the Isolated Connector

With the isolated connector attached, the MicroVM could reach the VPC internal target but could not reach the internet.

Connecting to VPC internal target:

curl "https://<microvm-endpoint>/fetch?url=http://10.0.0.126"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668100.0}"}

Connecting to the internet:

curl "https://<microvm-endpoint>/fetch?url=http://checkip.amazonaws.com"

The connection timed out and could not be established.

The primary reason for being unable to reach the internet is that the isolated subnet's route table has no default route (0.0.0.0/0). The connector SG's Egress restriction (VPC only) functions as an additional layer of defense.

Cross-Subnet Communication

A MicroVM with the isolated connector (10.0.1.0/24) attached was able to reach the target EC2 in the NAT subnet (10.0.0.0/24). Via the VPC's local route (10.0.0.0/16 → local), resources in subnets different from the one where the connector is placed can also be reached.

VPC Internal DNS Resolution

We verified whether a MicroVM could connect to the target EC2 using its private DNS name.

curl "https://<microvm-endpoint>/fetch?url=http://ip-10-0-0-126.ap-northeast-1.compute.internal"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668200.0}"}

In this test environment, with the VPC's EnableDnsHostnames and EnableDnsSupport enabled, we confirmed that the MicroVM could resolve the EC2's private DNS name and establish a connection.

Filtering with Security Groups

We removed the inbound permission for the connector SG from the target EC2's SG (target-sg) and confirmed that connections from the MicroVM timed out.

When the same request was sent with the SG inbound rule removed, the connection timed out. Restoring the rule returned 200 OK again. This confirmed that SG-to-SG referencing using the connector SG as the source is working correctly.

ENI Reality

In this verification, ENIs were created by the Lambda infrastructure when the connector was created, and no increase or decrease in ENI count was observed with MicroVM start/stop.

Relationship Between MicroVM Scale-Out and ENI Count

We monitored the number of ENIs in the VPC using describe-network-interfaces while launching multiple MicroVMs.

No matter how many MicroVMs were launched, the number of ENIs in the customer VPC visible from the own account did not increase.

Reading ENI Lifecycle from CloudTrail Events

By tracking CreateNetworkInterface events in CloudTrail, the following behavior was observed in this verification.

• ENIs were created when the connector was created (when create-network-connector was executed)
• No ENI creation or deletion events were observed when MicroVMs were started or stopped
• DeleteNetworkInterface was called when the connector was deleted
• For each connector created in this test, 2 CreateNetworkInterface events were recorded
  • 1st: request-validation-session (an ENI presumed to be for permission validation based on the session name)
  • 2nd: An ENI presumed to be for production use

The main attributes of the CloudTrail events are as follows.

Similarity to Hyperplane ENIs

When trying to reference an ENI directly with describe-network-interfaces, the result was InvalidNetworkInterfaceID.NotFound. Since CloudTrail confirms ENI creation by the Lambda infrastructure, this suggests it is being treated as a cross-account ENI that is created on the customer VPC's subnet side but cannot be directly referenced from the own account.

This behavior observationally shares similar characteristics with the Hyperplane ENI used when connecting traditional Lambda functions to a VPC. It is presumed to have a structure where ENIs are shared per connector, aggregating Egress traffic from multiple MicroVMs.

Constraint Verification

Specifying Multiple VPC Egress Connectors Simultaneously

We attempted to launch a MicroVM by specifying multiple connector ARNs in --egress-network-connectors.

aws lambda-microvms run-microvm \
  --microvm-image-identifier arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:microvm-image:egress-test-app-v3 \
  --egress-network-connectors \
    ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector \
    ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:isolated-egress-connector \
  --region ap-northeast-1

The result was an InternalFailure. While the CLI allowed specifying multiple elements, the execution resulted in an InternalFailure, and launching a MicroVM with multiple connectors specified simultaneously did not succeed.

Impact on MicroVMs When a Connector is Deleted

We verified the behavior when a connector attached to a running MicroVM is deleted.

aws lambda-core delete-network-connector \
  --identifier nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
  --region ap-northeast-1

Summary

We confirmed that Lambda MicroVMs can connect to private resources within a VPC using VPC Egress connectors. In this verification, we used a simple HTTP server on EC2 as the target and confirmed reachability in two configurations: a connector via NAT Gateway and an isolated connector without a default route.

When a connector is placed in a subnet with a default route to a NAT Gateway, both the VPC internal target and the internet were reachable. On the other hand, when placed in an isolated subnet without a default route, the VPC internal target was reachable, but internet-bound communication timed out.

Using a connector requires preparing a NetworkConnectorOperatorRole and waiting until it becomes ACTIVE. In this verification, it took approximately 4 to 5 minutes from connector creation to becoming ACTIVE. Additionally, an already-created connector can be used by multiple MicroVMs, and at the scale of a few instances, the number of ENIs in the VPC visible from the own account did not increase.

For cases where you want to connect from Lambda MicroVMs to VPC resources such as databases or internal APIs on private subnets, VPC Egress connectors seem to be an effective option.

Reference Links

https://docs.aws.amazon.com/cli/latest/reference/lambda-core/create-network-connector.html

https://docs.aws.amazon.com/cli/latest/reference/lambda-microvms/run-microvm.html