I verified connectivity to private resources within a VPC using the VPC Egress Connector for Lambda MicroVMs

Introduction

In Lambda MicroVMs, you can choose between INTERNET_EGRESS as the default and VPC_EGRESS, which specifies a subnet and security group within a VPC, for outbound communication (Egress) from the MicroVM.

VPC_EGRESS is not a "closed network only" feature. If the route table of the subnet where the connector is placed has a default route to a NAT Gateway, both VPC internal resources and the internet can be reached. On the other hand, if placed in a subnet without a default route, a configuration limited to VPC internal communication is also possible.

When you want to connect from a MicroVM to RDS, ElastiCache, internal APIs, etc. placed in private subnets, use a VPC Egress connector. In this article, we prepare a simple HTTP server on EC2 as an alternative connection target, and verify whether it can be reached via the VPC Egress connector. Connection verification to RDS/ElastiCache is outside the scope of this article.

This article verifies the following:

• VPC Egress connector creation procedure and state transitions
• Reachability via a connector with NAT Gateway
• Reachability via a closed network connector without a default route
• Access control by security groups
• The reality of ENIs and their lifecycle
• Behavior when specifying multiple connectors
• Cross-subnet communication
• DNS resolution
• Impact on MicroVMs when a connector is deleted

https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-networking.html

Preparing the Test Environment

VPC Configuration

The following configuration was created with CloudFormation.

• VPC: 10.0.0.0/16 (both DNS Hostnames and DNS Support enabled)
• NAT subnet (10.0.0.0/24, ap-northeast-1a): Internet connectivity via Regional NAT Gateway
• Isolated subnet (10.0.1.0/24, ap-northeast-1a): No default route in route table, VPC internal communication only
• Target EC2: Inside NAT subnet, simple HTTP server on port 80 (returns JSON response)

Security Group Design

In a VPC Egress connector, the connector-side SG controls outbound traffic from the MicroVM. On the target side, the design allows inbound traffic with the connector SG as the source.

This design allows only traffic via the connector to be permitted on the target side.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Lambda MicroVMs VPC Egress Test - Regional NAT Gateway + Isolated Subnet'

Parameters:
  ProjectName:
    Type: String
    Default: 'microvms-egress-test'

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: '10.0.0.0/16'
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: Name
          Value: !Ref ProjectName

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-igw'

  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  NATGatewayEIP:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-eip'

  RegionalNATGateway:
    Type: AWS::EC2::NatGateway
    DependsOn: InternetGatewayAttachment
    Properties:
      AvailabilityMode: regional
      ConnectivityType: public
      VpcId: !Ref VPC
      AvailabilityZoneAddresses:
        - AvailabilityZone: !Select [0, !GetAZs '']
          AllocationIds:
            - !GetAtt NATGatewayEIP.AllocationId
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-regional-nat-gw'

  NATSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: '10.0.0.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-subnet-1a'

  NATRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-rt'

  NATRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref NATRouteTable
      DestinationCidrBlock: '0.0.0.0/0'
      NatGatewayId: !Ref RegionalNATGateway

  NATSubnetAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref NATSubnet
      RouteTableId: !Ref NATRouteTable

  IsolatedSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: '10.0.1.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-isolated-subnet-1a'

  IsolatedRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-isolated-rt'

  IsolatedSubnetAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref IsolatedSubnet
      RouteTableId: !Ref IsolatedRouteTable

  ConnectorSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for MicroVM VPC Egress Connector (NAT subnet)'
      VpcId: !Ref VPC
      SecurityGroupEgress:
        - IpProtocol: '-1'
          CidrIp: '0.0.0.0/0'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-connector-sg'

  ConnectorIsolatedSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for MicroVM VPC Egress Connector (isolated subnet)'
      VpcId: !Ref VPC
      SecurityGroupEgress:
        - IpProtocol: '-1'
          CidrIp: '10.0.0.0/16'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-connector-isolated-sg'

  TargetSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for target resource in VPC'
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !Ref ConnectorSG
          Description: 'From MicroVM connector (NAT)'
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !Ref ConnectorIsolatedSG
          Description: 'From MicroVM connector (isolated)'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-target-sg'

  TargetInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: '{{resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64}}'
      InstanceType: t4g.nano
      SubnetId: !Ref NATSubnet
      SecurityGroupIds:
        - !Ref TargetSG
      UserData:
        Fn::Base64: |
          #!/bin/bash
          dnf install -y python3
          cat > /home/ec2-user/server.py << 'EOF'
          import http.server, json, time, socket
          class H(http.server.BaseHTTPRequestHandler):
              def do_GET(self):
                  self.send_response(200)
                  self.send_header('Content-Type','application/json')
                  self.end_headers()
                  self.wfile.write(json.dumps({"source":"vpc-target","host":socket.gethostname(),"time":time.time()}).encode())
          http.server.HTTPServer(('0.0.0.0',80),H).serve_forever()
          EOF
          python3 /home/ec2-user/server.py &
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-target'

The stack was deployed with the following command.

aws cloudformation deploy \
  --stack-name microvms-egress-test \
  --template-file vpc.yaml \
  --region ap-northeast-1

Creating the NetworkConnectorOperatorRole

To create a VPC Egress connector, an IAM role is required for the Lambda infrastructure to create ENIs in the Customer account's VPC.

In the trust policy, trust lambda.amazonaws.com and restrict it to your own account using the aws:SourceAccount condition.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "XXXXXXXXXXXX"
        }
      }
    }
  ]
}

The permissions policy allows creating, deleting, and tagging ENIs.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DeleteNetworkInterface",
        "ec2:CreateTags"
      ],
      "Resource": "*"
    }
  ]
}

aws iam create-role \
  --role-name NetworkConnectorOperatorRole \
  --assume-role-policy-document file://trust-policy.json

aws iam put-role-policy \
  --role-name NetworkConnectorOperatorRole \
  --policy-name NetworkConnectorENIPolicy \
  --policy-document file://eni-policy.json

Connector Creation and State Transitions

Creating the NAT-Routed Connector

Create the connector specifying the NAT subnet and the connector SG (Egress fully open).

aws lambda-core create-network-connector \
  --connector-name nat-egress-connector \
  --connector-type VPC_EGRESS \
  --vpc-config SubnetIds=subnet-XXXXXXXXXXXXXXXXX,SecurityGroupIds=sg-XXXXXXXXXXXXXXXXX \
  --operator-role-arn arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole \
  --region ap-northeast-1

{
    "NetworkConnector": {
        "ConnectorArn": "arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector",
        "ConnectorName": "nat-egress-connector",
        "ConnectorIdentifier": "nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        "ConnectorType": "VPC_EGRESS",
        "State": "PENDING",
        "VpcConfig": {
            "SubnetIds": ["subnet-XXXXXXXXXXXXXXXXX"],
            "SecurityGroupIds": ["sg-XXXXXXXXXXXXXXXXX"],
            "VpcId": "vpc-XXXXXXXXXXXXXXXXX"
        },
        "OperatorRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole"
    }
}

Creating the Isolated Connector

Specify the isolated subnet and the connector SG (Egress restricted to VPC internal only).

aws lambda-core create-network-connector \
  --connector-name isolated-egress-connector \
  --connector-type VPC_EGRESS \
  --vpc-config SubnetIds=subnet-XXXXXXXXXXXXXXXXX,SecurityGroupIds=sg-XXXXXXXXXXXXXXXXX \
  --operator-role-arn arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole \
  --region ap-northeast-1

State Transition: PENDING → ACTIVE

In this verification, both connectors transitioned from PENDING to ACTIVE in approximately 4 to 5 minutes. Connectors cannot be attached to and used by MicroVMs until they become ACTIVE.

aws lambda-core get-network-connector \
  --identifier nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
  --region ap-northeast-1

Launching MicroVMs and Verifying Connectivity

Launching a MicroVM

Specify the connector ARN with the --egress-network-connectors parameter in run-microvm.

aws lambda-microvms run-microvm \
  --microvm-image-identifier arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:microvm-image:egress-test-app-v3 \
  --egress-network-connectors ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector \
  --region ap-northeast-1

The test MicroVM executes outbound HTTP requests from inside via the /fetch?url=<URL> endpoint.

Verifying the NAT-Routed Connector

Confirmed that from a MicroVM with the NAT-routed connector attached, both the VPC internal target and the internet can be reached.

Connecting to the VPC internal target:

curl "https://<microvm-endpoint>/fetch?url=http://10.0.0.126"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668000.0}"}

Connecting to the internet:

curl "https://<microvm-endpoint>/fetch?url=http://checkip.amazonaws.com"

XX.XXX.XXX.XXX

The egress IP matched the EIP of the NAT Gateway. In this configuration, since the default route in the subnet where the connector is placed points to the NAT Gateway, internet-bound traffic from the MicroVM goes through the NAT Gateway.

Verifying the Isolated Connector

With a MicroVM that has the isolated connector attached, the VPC internal target was reachable, but the internet was not.

Connecting to the VPC internal target:

curl "https://<microvm-endpoint>/fetch?url=http://10.0.0.126"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668100.0}"}

Connecting to the internet:

curl "https://<microvm-endpoint>/fetch?url=http://checkip.amazonaws.com"

The connection timed out and could not be established.

The primary reason the internet cannot be reached is that the isolated subnet's route table does not have a default route (0.0.0.0/0). The connector SG's Egress restriction (VPC internal only) functions as an additional layer of defense.

Cross-Subnet Communication

From a MicroVM with the isolated connector (10.0.1.0/24) attached, it was possible to reach the target EC2 in the NAT subnet (10.0.0.0/24). The VPC's local route (10.0.0.0/16 → local) allows resources in subnets different from the one where the connector is placed to be reached.

VPC Internal DNS Resolution

Verified whether the MicroVM can connect to the target EC2 using its private DNS name.

curl "https://<microvm-endpoint>/fetch?url=http://ip-10-0-0-126.ap-northeast-1.compute.internal"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668200.0}"}

In this test environment, with EnableDnsHostnames and EnableDnsSupport enabled for the VPC, it was confirmed that the MicroVM can resolve the EC2's private DNS name and connect to it.

Filtering by Security Groups

After removing the inbound allow rule for the connector SG from the target EC2's SG (target-sg), it was confirmed that the connection from the MicroVM timed out.

When the same request was sent with the inbound rule removed, the connection timed out. Restoring the rule caused 200 OK to be returned again. It was confirmed that SG-to-SG references with the connector SG as the source are working correctly.

The Reality of ENIs

In this verification, ENIs were created by the Lambda infrastructure when the connector was created, and no increase or decrease in the number of ENIs was observed when MicroVMs started or stopped.

Relationship Between MicroVM Scaling and ENI Count

The number of ENIs in the VPC was monitored with describe-network-interfaces while launching multiple MicroVMs.

No matter how many MicroVMs were launched, the number of ENIs in the Customer VPC visible from the own account did not increase.

Understanding the ENI Lifecycle from CloudTrail Events

Tracking CreateNetworkInterface events in CloudTrail, the following behavior was observed in this verification.

• ENIs were created when the connector was created (when create-network-connector was executed)
• No ENI creation or deletion events were confirmed when MicroVMs started or stopped
• DeleteNetworkInterface was called when the connector was deleted
• For each connector created this time, two CreateNetworkInterface events were recorded
  • 1st: request-validation-session (ENI presumed to be for permission validation based on the session name)
  • 2nd: ENI presumed to be for production use

The main attributes of CloudTrail events are as follows.

Similarity to Hyperplane ENIs

When directly referencing the ENI with describe-network-interfaces, the result was InvalidNetworkInterfaceID.NotFound. Since ENI creation by the Lambda infrastructure can be confirmed in CloudTrail, this suggests that while they are created on the Customer VPC's subnet side, they are treated as cross-account ENIs that cannot be directly referenced from the own account.

This behavior shares observationally similar characteristics to Hyperplane ENIs used when connecting traditional Lambda functions to a VPC. It is presumed to be a structure where ENIs are shared per connector, aggregating the Egress traffic of multiple MicroVMs.

Constraint Verification

Specifying Multiple VPC Egress Connectors Simultaneously

Attempted to launch a MicroVM by specifying multiple connector ARNs in --egress-network-connectors.

aws lambda-microvms run-microvm \
  --microvm-image-identifier arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:microvm-image:egress-test-app-v3 \
  --egress-network-connectors \
    ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector \
    ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:isolated-egress-connector \
  --region ap-northeast-1

The result was an InternalFailure. While multiple elements could be specified on the CLI, the execution result was InternalFailure, and launching a MicroVM with multiple connectors specified simultaneously did not succeed.

Impact on MicroVMs When a Connector is Deleted

Verified the behavior when a connector attached to a running MicroVM is deleted.

aws lambda-core delete-network-connector \
  --identifier nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
  --region ap-northeast-1

Summary

It was confirmed that Lambda MicroVMs can connect to private resources within a VPC using VPC Egress connectors. In this verification, using a simple HTTP server on EC2 as the target, reachability was confirmed in two configurations: a connector via NAT Gateway, and an isolated connector without a default route.

When a connector is placed in a subnet with a default route to a NAT Gateway, both VPC internal targets and the internet can be reached. On the other hand, when placed in an isolated subnet without a default route, the VPC internal target was reachable, but internet-bound communication timed out.

Using a connector requires preparing a NetworkConnectorOperatorRole and waiting until it becomes ACTIVE. In this verification, it took approximately 4 to 5 minutes from connector creation to becoming ACTIVE. Additionally, already-created connectors can be used by multiple MicroVMs, and when launching a few instances, the number of ENIs in the VPC visible from the own account did not increase.

VPC Egress connectors appear to be an effective option when connecting Lambda MicroVMs to VPC internal resources such as databases and internal APIs on private subnets.

Reference Links

https://docs.aws.amazon.com/cli/latest/reference/lambda-core/create-network-connector.html

https://docs.aws.amazon.com/cli/latest/reference/lambda-microvms/run-microvm.html