I Tried Verifying Connectivity to Private Resources in a VPC Using the VPC Egress Connector for Lambda MicroVMs

Introduction

With Lambda MicroVMs, for outbound communication (Egress) from a MicroVM, you can choose between the default INTERNET_EGRESS and VPC_EGRESS, which lets you specify subnets and security groups within a VPC.

VPC_EGRESS is not a feature exclusively for isolated networks. If the route table of the subnet where the connector is placed has a default route to a NAT Gateway, both resources in the VPC and the internet can be reached. On the other hand, if you place it in a subnet without a default route, you can also build a configuration limited to communication within the VPC.

If you want to connect from a MicroVM to RDS, ElastiCache, internal APIs, and so on placed in private subnets, use a VPC Egress connector. In this article, as a stand-in for such destinations, I prepare a simple HTTP server on EC2 and verify whether it can be reached via the VPC Egress connector. Connectivity verification to RDS/ElastiCache is outside the scope of this article.

This article verifies the following.

• The procedure for creating a VPC Egress connector and its state transitions
• Reachability with a connector going through a NAT Gateway
• Reachability with an isolated connector that has no default route
• Access control with security groups
• The reality and lifecycle of the ENI
• Behavior when multiple connectors are specified
• Cross-subnet communication
• DNS resolution
• Impact on the MicroVM when the connector is deleted

https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-networking.html

Preparing the Test Environment

VPC Configuration

I created the following configuration with CloudFormation.

• VPC: 10.0.0.0/16 (both DNS Hostnames and DNS Support enabled)
• NAT subnet (10.0.0.0/24, ap-northeast-1a): Has internet connectivity via a Regional NAT Gateway
• Isolated subnet (10.0.1.0/24, ap-northeast-1a): No default route in the route table, communication within the VPC only
• Target EC2: In the NAT subnet, a simple HTTP server on port 80 (returns a JSON response)

Security Group Design

With a VPC Egress connector, the connector-side SG controls outbound traffic from the MicroVM. On the target side, the SG is designed to allow inbound traffic with the connector SG as the source.

This design allows the target side to permit only traffic coming through the connector.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Lambda MicroVMs VPC Egress Test - Regional NAT Gateway + Isolated Subnet'

Parameters:
  ProjectName:
    Type: String
    Default: 'microvms-egress-test'

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: '10.0.0.0/16'
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: Name
          Value: !Ref ProjectName

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-igw'

  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  NATGatewayEIP:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-eip'

  RegionalNATGateway:
    Type: AWS::EC2::NatGateway
    DependsOn: InternetGatewayAttachment
    Properties:
      AvailabilityMode: regional
      ConnectivityType: public
      VpcId: !Ref VPC
      AvailabilityZoneAddresses:
        - AvailabilityZone: !Select [0, !GetAZs '']
          AllocationIds:
            - !GetAtt NATGatewayEIP.AllocationId
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-regional-nat-gw'

  NATSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: '10.0.0.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-subnet-1a'

  NATRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-nat-rt'

  NATRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref NATRouteTable
      DestinationCidrBlock: '0.0.0.0/0'
      NatGatewayId: !Ref RegionalNATGateway

  NATSubnetAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref NATSubnet
      RouteTableId: !Ref NATRouteTable

  IsolatedSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: '10.0.1.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-isolated-subnet-1a'

  IsolatedRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-isolated-rt'

  IsolatedSubnetAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref IsolatedSubnet
      RouteTableId: !Ref IsolatedRouteTable

  ConnectorSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for MicroVM VPC Egress Connector (NAT subnet)'
      VpcId: !Ref VPC
      SecurityGroupEgress:
        - IpProtocol: '-1'
          CidrIp: '0.0.0.0/0'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-connector-sg'

  ConnectorIsolatedSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for MicroVM VPC Egress Connector (isolated subnet)'
      VpcId: !Ref VPC
      SecurityGroupEgress:
        - IpProtocol: '-1'
          CidrIp: '10.0.0.0/16'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-connector-isolated-sg'

  TargetSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'SG for target resource in VPC'
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !Ref ConnectorSG
          Description: 'From MicroVM connector (NAT)'
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !Ref ConnectorIsolatedSG
          Description: 'From MicroVM connector (isolated)'
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-target-sg'

  TargetInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: '{{resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64}}'
      InstanceType: t4g.nano
      SubnetId: !Ref NATSubnet
      SecurityGroupIds:
        - !Ref TargetSG
      UserData:
        Fn::Base64: |
          #!/bin/bash
          dnf install -y python3
          cat > /home/ec2-user/server.py << 'EOF'
          import http.server, json, time, socket
          class H(http.server.BaseHTTPRequestHandler):
              def do_GET(self):
                  self.send_response(200)
                  self.send_header('Content-Type','application/json')
                  self.end_headers()
                  self.wfile.write(json.dumps({"source":"vpc-target","host":socket.gethostname(),"time":time.time()}).encode())
          http.server.HTTPServer(('0.0.0.0',80),H).serve_forever()
          EOF
          python3 /home/ec2-user/server.py &
      Tags:
        - Key: Name
          Value: !Sub '${ProjectName}-target'

I deployed the stack with the following command.

aws cloudformation deploy \
  --stack-name microvms-egress-test \
  --template-file vpc.yaml \
  --region ap-northeast-1

Creating the NetworkConnectorOperatorRole

To create a VPC Egress connector, you need an IAM role that allows the Lambda infrastructure to create ENIs in the customer account's VPC.

The trust policy trusts lambda.amazonaws.com and restricts it to your own account with the aws:SourceAccount condition.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "XXXXXXXXXXXX"
        }
      }
    }
  ]
}

The permissions policy allows creating, deleting, and tagging ENIs.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DeleteNetworkInterface",
        "ec2:CreateTags"
      ],
      "Resource": "*"
    }
  ]
}

aws iam create-role \
  --role-name NetworkConnectorOperatorRole \
  --assume-role-policy-document file://trust-policy.json

aws iam put-role-policy \
  --role-name NetworkConnectorOperatorRole \
  --policy-name NetworkConnectorENIPolicy \
  --policy-document file://eni-policy.json

Connector Creation and State Transitions

Creating the NAT-Routed Connector

Create the connector by specifying the NAT subnet and the connector SG (Egress fully open).

aws lambda-core create-network-connector \
  --connector-name nat-egress-connector \
  --connector-type VPC_EGRESS \
  --vpc-config SubnetIds=subnet-XXXXXXXXXXXXXXXXX,SecurityGroupIds=sg-XXXXXXXXXXXXXXXXX \
  --operator-role-arn arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole \
  --region ap-northeast-1

{
    "NetworkConnector": {
        "ConnectorArn": "arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector",
        "ConnectorName": "nat-egress-connector",
        "ConnectorIdentifier": "nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        "ConnectorType": "VPC_EGRESS",
        "State": "PENDING",
        "VpcConfig": {
            "SubnetIds": ["subnet-XXXXXXXXXXXXXXXXX"],
            "SecurityGroupIds": ["sg-XXXXXXXXXXXXXXXXX"],
            "VpcId": "vpc-XXXXXXXXXXXXXXXXX"
        },
        "OperatorRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole"
    }
}

Creating the Isolated Connector

Specify the isolated subnet and the connector SG (Egress limited to within the VPC).

aws lambda-core create-network-connector \
  --connector-name isolated-egress-connector \
  --connector-type VPC_EGRESS \
  --vpc-config SubnetIds=subnet-XXXXXXXXXXXXXXXXX,SecurityGroupIds=sg-XXXXXXXXXXXXXXXXX \
  --operator-role-arn arn:aws:iam::XXXXXXXXXXXX:role/NetworkConnectorOperatorRole \
  --region ap-northeast-1

State Transition: PENDING → ACTIVE

In this verification, both connectors transitioned from PENDING to ACTIVE in about 4 to 5 minutes. You cannot attach and use a connector with a MicroVM until it becomes ACTIVE.

aws lambda-core get-network-connector \
  --identifier nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
  --region ap-northeast-1

Launching the MicroVM and Verifying Connectivity

Launching the MicroVM

Specify the connector ARN with the --egress-network-connectors parameter of run-microvm.

aws lambda-microvms run-microvm \
  --microvm-image-identifier arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:microvm-image:egress-test-app-v3 \
  --egress-network-connectors ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector \
  --region ap-northeast-1

The test MicroVM executes outbound HTTP requests from the inside via the /fetch?url=<URL> endpoint.

Verifying the NAT-Routed Connector

I confirmed that a MicroVM with the NAT-routed connector attached can reach both the target in the VPC and the internet.

Connecting to the target in the VPC:

curl "https://<microvm-endpoint>/fetch?url=http://10.0.0.126"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668000.0}"}

Connecting to the internet:

curl "https://<microvm-endpoint>/fetch?url=http://checkip.amazonaws.com"

XX.XXX.XXX.XXX

The egress IP matched the NAT Gateway's EIP. In this configuration, because the default route of the subnet where the connector is placed points to the NAT Gateway, internet-bound traffic from the MicroVM goes through the NAT Gateway.

Verifying the Isolated Connector

With the MicroVM that had the isolated connector attached, the target in the VPC was reachable, but the internet was not.

Connecting to the target in the VPC:

curl "https://<microvm-endpoint>/fetch?url=http://10.0.0.126"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668100.0}"}

Connecting to the internet:

curl "https://<microvm-endpoint>/fetch?url=http://checkip.amazonaws.com"

It timed out and could not connect.

The main reason the internet is unreachable is that the isolated subnet's route table has no default route (0.0.0.0/0). The connector SG's Egress restriction (within the VPC only) functions as an additional layer of defense.

Cross-Subnet Communication

From the MicroVM with the isolated connector (10.0.1.0/24) attached, I was able to reach the target EC2 in the NAT subnet (10.0.0.0/24). Thanks to the VPC's local route (10.0.0.0/16 → local), resources in subnets different from the one where the connector is placed can also be reached.

DNS Resolution within the VPC

I checked whether the MicroVM can connect using the target EC2's private DNS name.

curl "https://<microvm-endpoint>/fetch?url=http://ip-10-0-0-126.ap-northeast-1.compute.internal"

{"status": 200, "body": "{\"source\": \"vpc-target\", \"host\": \"ip-10-0-0-126\", \"time\": 1750668200.0}"}

In this test environment, with the VPC's EnableDnsHostnames and EnableDnsSupport enabled, I confirmed that the MicroVM could resolve the EC2 private DNS name and connect.

Filtering with Security Groups

I removed the inbound permission for the connector SG from the target EC2's SG (target-sg) and confirmed that connections from the MicroVM time out.

When I sent the same request with the SG inbound rule removed, the connection timed out. When I restored the rule, 200 OK was returned again. This confirmed that SG-to-SG references with the connector SG as the source work correctly.

The Reality of the ENI

In this verification, the ENI was created on the Lambda infrastructure side when the connector was created, and no increase or decrease in the number of ENIs accompanying MicroVM start/stop was observed.

Relationship Between the Number of MicroVMs and ENIs

I launched multiple MicroVMs while monitoring the number of ENIs in the VPC with describe-network-interfaces.

No matter how many MicroVMs were launched, the number of ENIs in the Customer VPC visible from my own account did not increase.

ENI Lifecycle as Read from CloudTrail Events

When I tracked CreateNetworkInterface events in CloudTrail, I observed the following behavior in this verification.

• The ENI was created when the connector was created (when create-network-connector was run)
• No ENI creation or deletion events were observed when MicroVMs started or stopped
• DeleteNetworkInterface was called when the connector was deleted
• For each connector created this time, two CreateNetworkInterface calls were recorded
  • 1st: request-validation-session (an ENI presumed from the session name to be for permission validation)
  • 2nd: An ENI that appears to be for actual operation

The main attributes of the CloudTrail events are as follows.

Similarity to Hyperplane ENI

When I referenced the ENI directly with describe-network-interfaces, it returned InvalidNetworkInterfaceID.NotFound. Since CloudTrail confirms that the Lambda infrastructure creates the ENI, this suggests that it is treated as a cross-account ENI that is created on the Customer VPC's subnet side but cannot be referenced directly from your own account.

This behavior has characteristics that, as observed, are similar to the Hyperplane ENI used when connecting a conventional Lambda function to a VPC. It is presumed to be a structure in which an ENI is shared per connector and aggregates the Egress traffic of multiple MicroVMs.

Verifying Constraints

Specifying Multiple VPC Egress Connectors Simultaneously

I tried launching a MicroVM by specifying multiple connector ARNs in --egress-network-connectors.

aws lambda-microvms run-microvm \
  --microvm-image-identifier arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:microvm-image:egress-test-app-v3 \
  --egress-network-connectors \
    ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:nat-egress-connector \
    ConnectorArn=arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:network-connector:isolated-egress-connector \
  --region ap-northeast-1

The result was InternalFailure. Although the CLI accepted multiple elements, the execution resulted in InternalFailure, and launching a MicroVM with multiple connectors specified simultaneously did not succeed.

Impact on the MicroVM When a Connector Is Deleted

I checked the behavior when deleting a connector attached to a running MicroVM.

aws lambda-core delete-network-connector \
  --identifier nc-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
  --region ap-northeast-1

Summary

I confirmed that using a VPC Egress connector allows Lambda MicroVMs to connect to private resources in a VPC. In this verification, I used a simple HTTP server on EC2 as the target and confirmed reachability with two configurations: a connector going through a NAT Gateway and an isolated connector with no default route.

When the connector was placed in a subnet with a default route to a NAT Gateway, both the target in the VPC and the internet were reachable. On the other hand, when placed in an isolated subnet with no default route, the target in the VPC was reachable, but internet-bound communication timed out.

Using a connector requires preparing a NetworkConnectorOperatorRole and waiting until it becomes ACTIVE. In this verification, it took about 4 to 5 minutes from connector creation to ACTIVE. Also, a created connector can be used by multiple MicroVMs, and when launching a few of them, the number of ENIs in the VPC visible from my own account did not increase.

A VPC Egress connector looks like a viable option when you want to connect Lambda MicroVMs to resources in a VPC, such as databases and internal APIs in private subnets.

Reference Links

https://docs.aws.amazon.com/cli/latest/reference/lambda-core/create-network-connector.html

https://docs.aws.amazon.com/cli/latest/reference/lambda-microvms/run-microvm.html