I Tried Fixing a Failed Deployment of a 3-AZ CloudFormation Stack (!GetAZs Doesn't Return All AZs)
This page has been translated by machine translation. View original
When I tried to deploy a 3-AZ VPC to the Tokyo region (ap-northeast-1) with CloudFormation, the following error occurred.
Template error: Fn::Select cannot select nonexistent value at index 2.
The template referenced the third item (index 2) of the AZ list obtained with !GetAZs.
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
Subnet3:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
CidrBlock: 10.0.2.0/24
AvailabilityZone: !Select [2, !GetAZs ''] # ← Error here
In the Tokyo region, !GetAZs should normally return three AZs, but it appeared to return only two. I had the opportunity to investigate the cause of this phenomenon, so I'd like to share it.
Comparison with the EC2 API
To be sure, I checked the available AZs in the Tokyo region with the EC2 API (AWS CLI).
$ aws ec2 describe-availability-zones --region ap-northeast-1 \
--query 'AvailabilityZones[*].ZoneName' --output table
-------------------------------
|DescribeAvailabilityZones |
+---------------------------+
| ap-northeast-1a |
| ap-northeast-1c |
| ap-northeast-1d |
+---------------------------+
Via the API, three AZs (1a, 1c, 1d) were returned without any problem.
There was a discrepancy: "the EC2 API shows three, but CloudFormation's !GetAZs can only get two."
Root Cause
The investigation revealed that the state of the default subnets was the influencing factor. You can check it with the following command.
$ aws ec2 describe-subnets \
--filters "Name=default-for-az,Values=true" \
--region ap-northeast-1 \
--query 'Subnets[*].[AvailabilityZone,SubnetId]' \
--output table
-----------------------------------------
| DescribeSubnets |
+------------------+-------------------+
| ap-northeast-1a | subnet-xxxxxxxx |
| ap-northeast-1c | subnet-yyyyyyyy |
+------------------+-------------------+
# ap-northeast-1d is missing
As the output shows, there was no default subnet in ap-northeast-1d.
This account had been in use since before the release of ap-northeast-1d in 2018, and it is possible that the subnet had been excluded because an AMI that did not support 1d was being used at that time.
Reference: A fourth Availability Zone (ap-northeast-1d) has been added to the Tokyo region! | DevelopersIO
The official AWS CloudFormation documentation contains the following description of the behavior of Fn::GetAZs.
The Fn::GetAZs function returns only Availability Zones that have a default subnet unless none of the Availability Zones has a default subnet; in that case, all Availability Zones are returned.
In summary, the behavior of !GetAZs is as follows.
- If at least one default subnet exists
→ Returns only the AZs in which a default subnet exists (this case) - If no default subnet exists at all
→ Returns all AZs in that region
In this case, default subnets existed only in some AZs (1a, 1c), so 1d was left out of the list.
Solutions
The AWS account where this problem occurred was a verification environment.
Since the default VPC might be used for hands-on sessions, EC2 Image Builder, and so on, I addressed it by adding the default subnet.
Solution 1: Create a default subnet
Create a default subnet in the missing AZ (in this case, ap-northeast-1d).
aws ec2 create-default-subnet --availability-zone ap-northeast-1d
This puts the account in a state where "every AZ has a default subnet," and !GetAZs will return three AZs. This is effective if you want to keep the existing default VPC.
Solution 2: Delete the default VPC
Completely delete the default VPC itself.
aws ec2 delete-vpc --vpc-id vpc-xxxxx
As per the specification above, all AZs are returned when "no default subnet exists at all," so with this approach too, !GetAZs will return three AZs.
Summary
CloudFormation's !GetAZs does not simply return "all AZs in the region"; its behavior changes depending on the presence of the default VPC and default subnets. If you cannot get the AZs displayed in the EC2 API or the VPC dashboard, suspect a missing default subnet.
Also, except for development and verification environments, it is recommended to consider deleting the default VPC, which is a risk factor, especially in production environments.
