I checked the reachable range of AWS services from Amazon MWAA Serverless tasks with and without a VPC
This page has been translated by machine translation. View original
Hello. This is Takeda from the Service Development Department.
PythonOperator and BashOperator are now available in Amazon MWAA Serverless. You can call AWS APIs with your own code from workflow tasks.
We confirmed the usage and snapshot behavior of code in a previous article.
During that verification, calling sts get-caller-identity via boto3 in a configuration without a VPC resulted in a ConnectTimeoutError. On the other hand, writing to S3 succeeded. Since VPC creation is optional for MWAA Serverless, we wanted to know which AWS services can be reached from tasks without a VPC, and what needs to be set up in a VPC to reach services that cannot be reached. We tested this by changing the configuration.
What can be reached without a VPC
The documentation states that tasks run without internet access by default, and if external connectivity is needed, a VPC with internet access should be passed via NetworkConfiguration.
First, we confirmed whether each service could be reached without a VPC. We prepared a Python task that calls each service's API via boto3 once with a connection timeout of 3 seconds and no retries, and a BashOperator task that outputs name resolution results using getent hosts. Since the execution role does not have permissions for the target services, receiving AccessDenied means the service was reached.
# netprobe.py (excerpt)
import boto3
from botocore.config import Config
CFG = Config(connect_timeout=3, read_timeout=5, retries={"max_attempts": 0})
def probe():
calls = {
"sts": lambda: boto3.client("sts", config=CFG).get_caller_identity(),
"s3": lambda: boto3.client("s3", config=CFG).list_buckets(),
"sqs": lambda: boto3.client("sqs", config=CFG).list_queues(),
"logs": lambda: boto3.client("logs", config=CFG).describe_log_groups(limit=1),
"dynamodb": lambda: boto3.client("dynamodb", config=CFG).list_tables(Limit=1),
}
results = {}
for name, fn in calls.items():
try:
fn()
results[name] = "ok"
except Exception as e:
results[name] = f"{e.__class__.__name__}"
print(results)
return results
The results are as follows.
| Destination | Result | Name Resolution |
|---|---|---|
STS (sts.ap-northeast-1.amazonaws.com) |
ConnectTimeoutError |
Public IP |
STS (global sts.amazonaws.com) |
ConnectTimeoutError |
Public IP |
| S3 | AccessDenied (reached) |
Public IP |
| SQS | ConnectTimeoutError |
Public IP |
| CloudWatch Logs | AccessDeniedException (reached) |
Private IP (10.x) |
| DynamoDB | ConnectTimeoutError |
Public IP |
Internet (checkip.amazonaws.com) |
Timeout | Public IP |
Only CloudWatch Logs resolved to a private IP, while S3 resolved to a public IP but was still reachable. The worker's /etc/resolv.conf showed nameserver 10.0.0.2 and search ap-northeast-1.compute.internal. This is the same configuration seen with Amazon-provided DNS in a VPC. This result can be explained by a configuration where the service management side has a gateway endpoint for S3 and an interface endpoint for CloudWatch Logs (with private DNS enabled).
If you cannot reach the necessary destinations, specify your own VPC in NetworkConfiguration and set up a route using a NAT gateway or VPC endpoints.
Passing your own VPC
Common requirements and routing options
The network requirements from the documentation are as follows.
The three things commonly required are:
- Two or more private subnets in separate AZs (must not have a route to an internet gateway)
- Security group (allow self-referencing inbound and allow all outbound traffic)
- Network ACL (the documentation's recommended example allows all inbound and outbound; the default ACL is in this state)
Choose one of the following routing options:
- Public routing with a NAT gateway (documentation recommends one per public subnet)
- Private routing with VPC endpoints for each service you use
The configuration we created uses a VPC of 10.0.0.0/16, two public subnets (for NAT) and two private subnets (10.0.10.0/24, 10.0.11.0/24).
The security group allows self-referencing inbound. For outbound, we kept the default allow-all rule created at the time of creation.
SG=$(aws ec2 create-security-group --group-name mwaa-sls-sg \
--description "mwaa serverless" --vpc-id vpc-xxxxxxxx \
--query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol -1 --source-group "$SG"
We create a gateway endpoint for S3 and an interface endpoint for STS. The interface endpoint has private DNS enabled and is associated with the two private subnets and the security group above.
# S3 (gateway endpoint)
aws ec2 create-vpc-endpoint --vpc-id vpc-xxxxxxxx \
--service-name com.amazonaws.ap-northeast-1.s3 --vpc-endpoint-type Gateway \
--route-table-ids rtb-private-a rtb-private-c
# STS (interface endpoint, private DNS enabled)
aws ec2 create-vpc-endpoint --vpc-id vpc-xxxxxxxx \
--service-name com.amazonaws.ap-northeast-1.sts --vpc-endpoint-type Interface \
--subnet-ids subnet-private-a subnet-private-c \
--security-group-ids "$SG" --private-dns-enabled
The endpoint policy is left as the default (full access) in this case. Be careful when restricting the S3 gateway endpoint policy to only your own buckets. The documentation's policy example states that access to prod-<region>-starport-layer-bucket (the bucket used for retrieving Amazon ECR image layers) is required. Restricting to only your own bucket may prevent the worker from retrieving container images.
Passing a VPC to the workflow
In update-workflow (or create-workflow), specify the created private subnets and security group in --network-configuration. The other workflow settings (definition, code, role) must be specified together each time.
aws mwaa-serverless update-workflow \
--workflow-arn arn:aws:airflow-serverless:ap-northeast-1:123456789012:workflow/net-probe-xxxxxxxxxx \
--definition-s3-location '{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"definitions/net.yaml"}' \
--code '{"S3Location":{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"code/netprobe.zip"}}' \
--role-arn arn:aws:iam::123456789012:role/mwaa-sls-exec \
--network-configuration '{"SecurityGroupIds":["sg-xxxxxxxx"],"SubnetIds":["subnet-private-a","subnet-private-c"]}'
Incidentally, an update-workflow specifying an empty array for NetworkConfiguration during cleanup resulted in a ValidationException. The method for reverting an existing workflow back to no VPC has not been confirmed.
Testing with different configurations
We ran the same workflow changing only the VPC-side configuration.
| Configuration | STS | S3 | SQS / DynamoDB | CloudWatch Logs API / Task Logs | Internet |
|---|---|---|---|---|---|
| No VPC | ✗ | ○ | ✗ | ○ / present | ✗ |
| Own VPC + NAT + STS endpoint | ○ (via endpoint) | ○ | ○ | ○ / present | ○ |
| Own VPC + NAT only | ○ (via NAT) | ○ | ○ | ○ / present | ○ |
| Own VPC + STS endpoint + S3 gateway only (no NAT) | ○ (via endpoint) | ○ | ✗ | ✗ / absent | ✗ |
| Above + CloudWatch Logs endpoint | ○ | ○ | ✗ | ○ / present | ✗ |
NAT + STS endpoint
In the first run with the VPC passed, the worker was running at 10.0.10.243, within the specified private subnet. sts.ap-northeast-1.amazonaws.com resolved to the endpoint ENI IPs (10.0.10.161 and 10.0.11.213). In this state, get_caller_identity succeeded. SQS, DynamoDB, and the internet were reached via NAT.
NAT only
After deleting the STS endpoint and re-running, sts.ap-northeast-1.amazonaws.com resolved to a public IP, and get_caller_identity succeeded. Even without an endpoint, NAT is sufficient to reach it.
Endpoints only (no NAT)
We removed the 0.0.0.0/0 route from the private route table, recreated the STS endpoint, and ran the workflow. The only routes were local and the S3 gateway.
Even with this configuration, the workflow completed with SUCCESS, STS was reached via the endpoint, and XCom values were returned. The documentation lists VPC endpoints for Apache Airflow in addition to the services used by the workflow as requirements for private routing. However, the endpoint service names for Serverless are not published, and status updates and XCom returns completed without creating them. They appear to be unnecessary at this time.
On the other hand, only this run did not create a task log stream in CloudWatch Logs. The run succeeded, but there were no logs. After adding a CloudWatch Logs interface endpoint and re-running, logs.ap-northeast-1.amazonaws.com resolved to the endpoint IP. In this case, the log stream was created. From these two results, we can determine that a route from your own VPC to CloudWatch Logs is needed to produce task logs.
The three things we prepared in the no-NAT configuration tested this time were: an S3 gateway endpoint, a CloudWatch Logs interface endpoint, and an endpoint for the service called from the code (STS). Since the workflow succeeds even without logs, the CloudWatch Logs endpoint is easy to overlook.
That requirement is not explicitly stated for CloudWatch Logs. The documentation for conventional MWAA (provisioned type) lists com.amazonaws.<region>.logs in the list of required endpoints. We interpret producing task logs as also being included under "services used by the workflow."
NAT or interface endpoints?
Once you decide to pass a VPC, whether to route through a NAT gateway or interface endpoints (PrivateLink) depends on the destinations and cost.
The decision order is as follows:
- If you can reach the necessary destinations without a VPC, don't pass a VPC (no additional cost)
- If internet access is needed, or if a destination does not support VPC endpoints, use a NAT gateway
- If destinations are fixed and all support VPC endpoints, use interface endpoints (include a CloudWatch Logs endpoint)
- If data processing volume is high, compare both hourly charges and data processing charges
Let's compare costs using Tokyo Region figures. Monthly estimates assume 730 hours and exclude data processing charges and standard data transfer charges.
| Route | Hourly charge | Data processing charge | Estimated monthly cost |
|---|---|---|---|
| Interface endpoint | 0.014 USD/hour/AZ | 0.01 USD/GB | ~20 USD / service (2 AZs) |
| NAT gateway | 0.062 USD/hour/unit | 0.062 USD/GB | ~90 USD (2 units), ~45 USD (1 unit) |
| S3 gateway endpoint | No charge | No charge | 0 USD |
Interface endpoints cost approximately 41 USD/month for 2 services (STS and CloudWatch Logs), and approximately 61 USD/month for 3 services. The break-even point versus the documentation's recommended 2-NAT configuration (~90 USD/month) falls between 4 and 5 services using interface endpoints. Compared to 1 NAT (~45 USD/month), the break-even is between 2 and 3 services, but 1 NAT differs in availability compared to endpoints across 2 AZs.
Data processing charges are lower for interface endpoints (0.01 USD/GB vs. 0.062 USD/GB), so the more traffic goes to supported services, the more advantageous interface endpoints become.
This time, following the documentation, we associated interface endpoints with two private subnets.
Summary
- Among the destinations tested this time, S3 and CloudWatch Logs were reachable without a VPC; STS, SQS, DynamoDB, and the internet could not be confirmed as reachable
- When a VPC is passed, workers run within the specified private subnets. STS was reachable both via NAT and via interface endpoint
- In private routing without NAT, task logs will not be retained without a CloudWatch Logs interface endpoint. This is easy to miss since the workflow itself succeeds
- Excluding data processing volume, interface endpoints across 2 AZs become roughly equivalent to 2 NAT gateways at 4 to 5 services. The break-even with 1 NAT is between 2 and 3 services, but availability differs
What we learned about reachability without a VPC this time is limited to the 5 services tested and the internet. Test reachability to the services called from your tasks first, then decide whether to set up your own VPC and routing.
