I tried a two-stage automatic recovery of Nginx using EC2 application status checks and Step Functions
This page has been translated by machine translation. View original
Introduction
In August 2026, application status checks were made available for EC2. This is a feature that sends HTTP requests to monitored instances from within a VPC and determines the state of the application layer based on the response.
A previous article covered a configuration where Step Functions stops and starts EC2 in response to Instance Status Check failures.
In this article, we delegate HTTP response checking for Nginx to this check, and build a mechanism where Step Functions recovers in two stages when an anomaly is detected. The first stage is Nginx restart via SSM, and the second stage is EC2 Stop/Start. No Lambda functions are used — it is built using only AWS SDK integrations and JSONata.
Verification Details
The explanation covers the configuration, the progress until monitoring begins, what happens when Nginx is stopped, and what happens when it cannot be restarted.
Configuration
Application status checks are a feature released in August 2026. A separate article verifies everything from check creation to status transitions and the managed ENI mechanism.
When an HTTP GET is sent to the monitored instance and the check fails, it is recorded in the CloudWatch metric StatusCheckFailed_Application. A Step Functions execution is triggered via a CloudWatch alarm and EventBridge. The state machine first restarts Nginx via SSM, and only if that fails does it proceed to EC2 Stop/Start, then sends a notification to SNS.
The verification environment is consolidated into a single CloudFormation template. The resources created are: EC2 instance, application status check, CloudWatch alarm, SNS topic and email subscription, EventBridge rule, 36-state Step Functions state machine, CloudWatch Logs log group, and the IAM roles and security groups required for these. The EC2 instance uses Amazon Linux 2023 ARM64 t4g.nano, with Nginx installed via UserData.
Key points to note in the template are introduced along with their corresponding definitions.
Health Check Source and Destination
HealthCheckPaths specifies the health check source and destination. For the source, specify the subnet and security group where the managed ENI will be placed; for the destination, specify the subnet and security group of the instance to be monitored. Subnet specification is required for both Source and Destinations.
HealthCheckPaths:
- Source:
SecurityGroupId: !GetAtt AppStatusCheckSourceSG.GroupId
SubnetId: !Ref SubnetId
Destinations:
- SecurityGroupId: !GetAtt EC2SecurityGroup.GroupId
SubnetId: !Ref SubnetId
Specifying Security Group IDs
Security group IDs are retrieved using !GetAtt <LogicalID>.GroupId.
SecurityGroupId: !GetAtt AppStatusCheckSourceSG.GroupId
Inbound Rules on the Destination Side
The security group of the destination instance must allow port 80 from the source security group.
EC2SecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !Ref EC2SecurityGroup
IpProtocol: tcp
FromPort: 80
ToPort: 80
SourceSecurityGroupId: !GetAtt AppStatusCheckSourceSG.GroupId
Outputting the Application Status Check ID
!Ref ApplicationStatusCheck returns an ARN. Use !GetAtt ApplicationStatusCheck.ApplicationStatusCheckId where an ID is needed. In this template, the ID used in the association command is exported to Outputs.
ApplicationStatusCheckId:
Value: !GetAtt ApplicationStatusCheck.ApplicationStatusCheckId
State Machine Overall Timeout
A TimeoutSeconds was specified for the state machine. In this case, it is 1800 seconds. If the Stop/Start completion wait loop runs unexpectedly long, it will be aborted.
"TimeoutSeconds": 1800,
Since recovery is performed using SSM Run Command, the subnet where EC2 is placed must be able to reach the SSM endpoint. Installing Nginx via UserData also assumes reachability to the repository. Please prepare one of the following: a public subnet with a public IP, a NAT gateway, or VPC endpoints for SSM.
If the source specification is omitted, it becomes an AWS-managed network path, and AWS selects the source. In that case, the source security group is not included in the return value of describe-application-status-checks. The permission on the destination side must be written using CIDR rather than security groups.
The full template and deployment instructions are provided at the end of the article.
Once deployment is complete, approve the SNS subscription confirmation email that arrives at the notification address.
Next, associate the application status check with the instance. The check itself can be created with AWS::EC2::ApplicationStatusCheck. Since no resource type is provided for associating with an instance, the association is performed via API. The association target can be specified using either an instance ID or a tag. Here, the aws:cloudformation:stack-id tag automatically assigned by CloudFormation was used. Since its value is the stack ARN containing a UUID, even if the stack is recreated with the same name, the previous association will not match the new instance.
STACK_ID=$(aws cloudformation describe-stacks \
--stack-name ec2-app-recovery-test \
--query 'Stacks[0].StackId' \
--output text --region ap-northeast-1)
ASC_ID=$(aws cloudformation describe-stacks \
--stack-name ec2-app-recovery-test \
--query 'Stacks[0].Outputs[?OutputKey==`ApplicationStatusCheckId`].OutputValue' \
--output text --region ap-northeast-1)
aws ec2 associate-application-status-check \
--application-status-check-id $ASC_ID \
--target-tag-associations Key=aws:cloudformation:stack-id,Value=$STACK_ID \
--region ap-northeast-1
In the response, the association type was returned as EC2TAG.
{
"SuccessfulResults": [
{
"ApplicationStatusCheckId": "asc-xxxxxxxx",
"AssociationType": "EC2TAG",
"AssociationValue": "aws:cloudformation:stack-id=arn:aws:cloudformation:ap-northeast-1:xxxxxxxxxxxx:stack/ec2-app-recovery-test/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
],
"UnsuccessfulResults": []
}
When retrieving the list, the same association is registered as a tag.
{
"Associations": [
{
"ApplicationStatusCheckId": "asc-xxxxxxxx",
"AssociationType": "tag",
"Key": "aws:cloudformation:stack-id",
"Value": "arn:aws:cloudformation:ap-northeast-1:xxxxxxxxxxxx:stack/ec2-app-recovery-test/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
]
}
Start of Monitoring
The application status check status was initializing, and changed to ok after approximately 5 minutes. Times are in JST.
| Time | Overall Status |
|---|---|
| 08:03:55 | initializing |
| 08:09:00 | ok |
This waiting time is due to the initialization grace period. The default for InitializationGracePeriodSeconds is 300 seconds, with a valid range of 1–600 seconds. The above progress was measured with the default 300 seconds. After verification, the initialization grace period was changed to 60 seconds. The posted template specifies 60 seconds. The configuration values after the change are as follows, in order: initialization grace period, timeout, failure threshold, success threshold, and check interval.
$ aws ec2 describe-application-status-checks --application-status-check-ids asc-xxxxxxxx \
--query 'ApplicationStatusChecks[].[InitializationGracePeriodSeconds,Timeout,FailureThreshold,SuccessThreshold,Interval]'
60 6 2 2 60
The check interval is fixed at 60 seconds and cannot be changed (the Default settings table in the user guide states Check interval | 60 seconds (fixed; not configurable)). The only valid value for Interval in CloudFormation is also 60.
The response during normal operation was confirmed.
{
"ApplicationStatuses": {
"Instances": [
{
"InstanceId": "i-xxxxxxxxxxxxxxxxx1",
"AvailabilityZone": "apne1-az4",
"ApplicationStatus": {
"Status": "ok",
"StatusTimeStamp": "2026-09-04T23:11:55.674000+00:00",
"StatusSince": "2026-09-04T23:09:00+00:00",
"Details": [
{
"ApplicationStatusCheckId": "asc-xxxxxxxx",
"CheckUpdateTime": "2026-09-04T23:11:55.674000+00:00",
"Aggregation": "included",
"Status": "passed",
"StatusTimeStamp": "2026-09-04T23:11:55.674000+00:00",
"StatusSince": "2026-09-04T23:09:00+00:00",
"Reason": {
"Code": "ResponseCodeMatched",
"StatusCode": 200,
"Protocol": "HTTP"
}
}
]
}
}
]
}
}
The overall status was ok, the individual check result was passed, and the reason code was ResponseCodeMatched, indicating that the response code matched. Since the instance was replaced after this, the instance ID is different in subsequent verifications.
Nginx Stop
To trigger the first-stage recovery, Nginx was stopped using SSM Run Command.
aws ssm send-command \
--instance-ids i-xxxxxxxxxxxxxxxxx2 \
--document-name "AWS-RunShellScript" \
--parameters 'commands=["sudo systemctl stop nginx"]' \
--region ap-northeast-1
The application status changed to impaired.
{
"ApplicationStatuses": {
"Instances": [
{
"InstanceId": "i-xxxxxxxxxxxxxxxxx2",
"AvailabilityZone": "apne1-az4",
"ApplicationStatus": {
"Status": "impaired",
"StatusTimeStamp": "2026-09-04T23:28:07.456000+00:00",
"StatusSince": "2026-09-04T23:29:00+00:00",
"Details": [
{
"ApplicationStatusCheckId": "asc-xxxxxxxx",
"CheckUpdateTime": "2026-09-04T23:28:07.456000+00:00",
"Aggregation": "included",
"Status": "failed",
"StatusTimeStamp": "2026-09-04T23:28:07.456000+00:00",
"StatusSince": "2026-09-04T23:29:00+00:00",
"Reason": {
"Code": "ConnectionRefused",
"StatusCode": 0
}
}
]
}
}
]
}
}
The state where no process is listening appeared as reason code ConnectionRefused.
In the Step Functions execution launched from the alarm, 15 states were executed and the history had 53 events.
08:30:20.034 Init
08:30:20.034 CheckNginxInit
08:30:20.166 WaitCheckInit
08:30:25.218 GetCheckInitResult
08:30:25.433 EvalInitStatus
08:30:25.433 SSMRestart
08:30:25.576 WaitSSM
08:30:35.632 CheckSSM
08:30:35.743 EvalSSM
08:30:35.743 WaitAfterRestart
08:30:45.785 CheckNginxAfterRestart
08:30:45.939 WaitCheckAfterRestart
08:30:50.996 GetCheckAfterRestartResult
08:30:51.138 EvalStage1
08:30:51.138 NotifyStage1OK
Recovery was confirmed by running systemctl is-active nginx via SSM, and it ended at NotifyStage1OK. The execution graph also showed that only the first-stage states were traversed.
The time from fault injection to recovery completion was 4 minutes and 27 seconds.
| Time | Event |
|---|---|
| 08:26:24 | Nginx stopped via SSM |
| 08:29:00 | Application status becomes impaired (ConnectionRefused) |
| 08:30:19 | CloudWatch alarm transitions from OK to ALARM |
| 08:30:20 | Step Functions execution starts |
| 08:30:51 | Execution ends at NotifyStage1OK (execution time: 31 seconds) |
| 08:32:19 | Alarm transitions from ALARM to OK |
Looking at the breakdown, detection took about 2.5 minutes, and from the alarm transition to recovery completion was less than 1 minute.
State Where SSM Restart Fails
To trigger the second-stage recovery, Nginx was stopped and a process that accepts connections but returns no response was made to occupy port 80. In this state, Nginx cannot start, so the SSM restart fails. Since the process is not persisted, it disappears after Stop/Start.
Parameter file to occupy port 80
{
"commands": [
"sudo systemctl stop nginx",
"printf '%s\\n' 'import socket' 's=socket.socket()' 's.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)' 's.bind((\"0.0.0.0\",80))' 's.listen(5)' 'while True:' ' c,a=s.accept()' | sudo tee /tmp/hog.py",
"sudo setsid nohup python3 /tmp/hog.py >/tmp/hog.log 2>&1 < /dev/null &",
"sleep 3",
"sudo ss -ltnp | grep ':80' || echo 'NOT LISTENING'",
"systemctl is-active nginx || true"
]
}
The above parameters were saved as port80-hog-params.json and passed to Run Command.
aws ssm send-command \
--instance-ids i-xxxxxxxxxxxxxxxxx2 \
--document-name "AWS-RunShellScript" \
--parameters file://port80-hog-params.json \
--region ap-northeast-1
The end of standard output from the execution result is as follows. Port 80 is held by python3, and Nginx is stopped.
LISTEN 0 5 0.0.0.0:80 0.0.0.0:* users:(("python3",pid=26438,fd=3))
inactive
The application status is impaired as in the previous section, but the reason code was different. Since the TCP connection is established but no HTTP response is returned, instead of ConnectionRefused, it was ResponseTimeout.
{
"ApplicationStatuses": {
"Instances": [
{
"InstanceId": "i-xxxxxxxxxxxxxxxxx2",
"AvailabilityZone": "apne1-az4",
"ApplicationStatus": {
"Status": "impaired",
"StatusTimeStamp": "2026-09-04T23:35:09.514000+00:00",
"StatusSince": "2026-09-04T23:36:00+00:00",
"Details": [
{
"ApplicationStatusCheckId": "asc-xxxxxxxx",
"CheckUpdateTime": "2026-09-04T23:35:09.514000+00:00",
"Aggregation": "included",
"Status": "failed",
"StatusTimeStamp": "2026-09-04T23:35:09.514000+00:00",
"StatusSince": "2026-09-04T23:36:00+00:00",
"Reason": {
"Code": "ResponseTimeout",
"StatusCode": 0
}
}
]
}
}
]
}
}
In this execution, 23 states were executed and the history had 81 events.
08:37:20.068 Init
08:37:20.068 CheckNginxInit
08:37:20.245 WaitCheckInit
08:37:25.296 GetCheckInitResult
08:37:25.484 EvalInitStatus
08:37:25.484 SSMRestart
08:37:25.640 WaitSSM
08:37:35.733 CheckSSM
08:37:35.869 EvalSSM
08:37:35.869 Stage2Stop
08:37:36.401 WaitStop
08:37:51.455 CheckStop
08:37:51.694 EvalStop
08:37:51.694 StartInstance
08:37:52.743 WaitStart
08:38:07.798 CheckStart
08:38:08.067 EvalStart
08:38:08.067 WaitRecovery
08:38:38.117 CheckNginxRecovery
08:38:38.270 WaitCheckRecovery
08:38:43.320 GetCheckRecoveryResult
08:38:43.480 EvalRecovery
08:38:43.480 NotifyStage2OK
Since the SSM command status retrieved by CheckSSM was Failed, EvalSSM branched to Stage2Stop. After that, it completed at NotifyStage2OK after confirming the stop, starting the instance, and confirming recovery.
The state on the instance side was also confirmed. The startup time was updated to the time of Stop/Start, the process that had occupied port 80 was gone, and Nginx had reclaimed the listening port.
$ uptime -s
2026-09-04 23:37:59
$ systemctl is-active nginx
active
$ ss -ltnp | grep :80
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1978,fd=6),("nginx",pid=1976,fd=6),("nginx",pid=1975,fd=6))
LISTEN 0 511 [::]:80 [::]:* users:(("nginx",pid=1978,fd=7),("nginx",pid=1976,fd=7),("nginx",pid=1975,fd=7))
$ pgrep -af hog.py || echo NOHOG
NOHOG
The time from fault injection to recovery completion was 5 minutes and 29 seconds.
| Time | Event |
|---|---|
| 08:33:14 | Nginx stopped and port 80 occupied |
| 08:36:00 | Application status becomes impaired (ResponseTimeout) |
| 08:37:19 | CloudWatch alarm transitions from OK to ALARM |
| 08:37:20 | Step Functions execution starts |
| 08:37:35 | SSM restart command becomes Failed and branches to Stage2Stop |
| 08:37:51 | Stop confirmed and instance started |
| 08:38:43 | Execution ends at NotifyStage2OK (execution time: 83 seconds) |
The difference from the execution that ended at stage 1 is only the Stop/Start and the wait for its completion.
Tracking Instance Replacement
When associated by tag, we verified whether monitoring continues even after an instance is replaced. Changing the logical ID of the EC2 instance and running update-stack causes the instance to be replaced. The association API was not re-executed.
| Item | Result |
|---|---|
| Instance before replacement | terminated |
| Instance after replacement | running |
| Association | Only 1 entry as tag (aws:cloudformation:stack-id) |
| Application status of new instance immediately after replacement | initializing (then ok) |
The application status of the instance after replacement is as follows.
{
"ApplicationStatuses": {
"Instances": [
{
"InstanceId": "i-xxxxxxxxxxxxxxxxx2",
"AvailabilityZone": "apne1-az2",
"ApplicationStatus": {
"Status": "ok",
"StatusTimeStamp": "2026-09-05T08:00:29.426000+00:00",
"StatusSince": "2026-09-05T08:01:00+00:00",
"Details": [
{
"ApplicationStatusCheckId": "asc-xxxxxxxx",
"CheckUpdateTime": "2026-09-05T08:00:29.426000+00:00",
"Aggregation": "included",
"Status": "passed",
"StatusTimeStamp": "2026-09-05T08:00:29.426000+00:00",
"StatusSince": "2026-09-05T08:01:00+00:00",
"Reason": {
"Code": "ResponseCodeMatched",
"StatusCode": 200,
"Protocol": "HTTP"
}
}
]
}
}
]
}
}
Monitoring continued without any association operations.
Running Costs
The billable item is the managed ENI, at a unit price of 0.01 USD per hour per Availability Zone. The Pricing section of the user guide states the following:
An hourly charge of $0.01 for each managed elastic network interface (ENI), per Availability Zone.
Standard Amazon CloudWatch pricing applies to application status check metrics.
Source: Application status checks in the EC2 User Guide (referenced 2026-09-05). One managed ENI is created per combination of "source subnet × security group" and does not scale with the number of instances. The user guide explains that 200 instances in 2 subnets with 1 security group results in 2 ENIs, while the same 200 instances with 3 security groups results in up to 6 ENIs. With 1 ENI, that is 7.30 USD per month for 730 hours.
While this configuration is running, charges apply for the managed ENI, the alarm, and Step Functions state transitions when a failure occurs. The health check is executed by the EC2 feature itself, so nothing runs during normal operation.
| Item | Unit Price | Free Tier |
|---|---|---|
| Managed ENI | 0.01 USD / hour / ENI | None |
| Step Functions Standard state transitions | 0.000025 USD / transition | 4,000 transitions per month |
| CloudWatch metric alarm | 0.10 USD / alarm metric month | 10 alarm metrics per month |
| CloudWatch Logs Vended Logs ingestion (CloudWatch Logs destination) | 0.50 USD / GB (0–10 TB) | 5 GB per month |
The unit prices for state transitions and logs are values from the calculation examples shown on the AWS Step Functions pricing page and the Amazon CloudWatch pricing page. The calculation examples are both for US East, referenced on 2026-09-05. The free tier of 4,000 state transitions per month does not expire even after the 12 months of the AWS Free Tier ends.
With a single-ENI configuration, 7.30 USD per month is a fixed cost, and the alarm is 0 USD if the free tier remains. Even if recovery occurs a few times per month, the number of state transitions is 15–23 per execution, which stays within the free tier. Increasing the number of monitored instances does not change the ENI cost, but creating alarms per instance will add to that cost.
If you build health checks yourself with periodic Step Functions executions, costs are determined by the number of state transitions. Assuming 5 transitions per execution, that's 216,000 transitions per month at 1-minute intervals for 5.30 USD, or 21,600 transitions per month at 10-minute intervals for 0.44 USD. Longer intervals reduce costs, but you need to handle concurrent execution when periodic and recovery executions overlap, and execution history will also be mixed with normal-operation records. The decision was made to delegate the health check mechanism to application status checks.
The log group was output with Level: ALL and IncludeExecutionData: true.
| Execution | States Executed | Log Events | Ingested Bytes |
|---|---|---|---|
| Execution where Nginx had already recovered | 8 | 23 | 27,090 |
| Execution recovered in Stage 1 | 15 | 53 | 58,294 |
| Execution recovered in Stage 2 | 23 | 81 | 122,171 |
Since only recovery events remain in the execution history and log group, at this scale it stays within the Vended Logs free tier.
Teardown
When tearing down, first disassociate the tag-based association, then delete the stack.
aws ec2 disassociate-application-status-check \
--application-status-check-id $ASC_ID \
--target-tag-associations Key=aws:cloudformation:stack-id,Value=$STACK_ID \
--region ap-northeast-1
aws cloudformation delete-stack \
--stack-name ec2-app-recovery-test \
--region ap-northeast-1
The EC2 instance, application status check, and state machine were deleted, but deleting the security groups resulted in a DependencyViolation. This is because the managed ENI is using the security groups.
An error occurred (DependencyViolation) when calling the DeleteSecurityGroup operation:
resource sg-xxxxxxxxxxxxxxxxx has a dependent object
Even after deleting the application status check, the managed ENI deletion is asynchronous and takes time to complete. In this case, retrying 11 minutes after the stack deletion operation still resulted in DependencyViolation, and the stack became DELETE_FAILED. The security group could be deleted about 4 hours later, and running delete-stack afterward completed the stack deletion.
If you want to delete a stack that has become DELETE_FAILED first, delete it while retaining the security groups. After the ENI is gone, manually delete the security groups.
aws cloudformation delete-stack \
--stack-name ec2-app-recovery-test \
--retain-resources AppStatusCheckSourceSG \
--region ap-northeast-1
Summary
A multi-stage recovery scenario was built with Step Functions. The first stage attempts to restart Nginx via SSM, and only if that fails does it proceed to EC2 Stop/Start. Since the determination of whether the application is responding is delegated to the EC2 feature, neither health check implementation nor Lambda functions were written.
By combining SSM, procedures beyond recovery itself can be included in the same state machine. Additional investigation after receiving an alert, fault isolation based on retrieved information, and application restart can all be targets for automation.
For stateful environments where replacing instances with EC2 Auto Scaling is difficult and where application failures are being recovered by restarting services, please consider trying this as one option for automation.
Reference
Deployment
EXPRESS was specified as the CloudFormation deployment mode.
aws cloudformation create-stack \
--stack-name ec2-app-recovery-test \
--template-body file://ec2-app-recovery-test.yaml \
--parameters \
ParameterKey=NotificationEmail,ParameterValue=alerts@example.com \
ParameterKey=VpcId,ParameterValue=vpc-xxxxxxxxxxxxxxxxx \
ParameterKey=SubnetId,ParameterValue=subnet-xxxxxxxxxxxxxxxxx \
--capabilities CAPABILITY_IAM \
--deployment-config '{"Mode": "EXPRESS"}' \
--region ap-northeast-1
aws cloudformation wait stack-create-complete \
--stack-name ec2-app-recovery-test \
--region ap-northeast-1
Full Template
CloudFormation Full Template
AWSTemplateFormatVersion: '2010-09-09'
Description: 'EC2 Nginx + Application Status Check + Step Functions 2-Stage Recovery'
Parameters:
NotificationEmail:
Type: String
InstanceType:
Type: String
Default: t4g.nano
LatestAmiId:
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64
LogRetentionDays:
Type: Number
Default: 7
VpcId:
Type: AWS::EC2::VPC::Id
Description: VPC ID
SubnetId:
Type: AWS::EC2::Subnet::Id
Description: Subnet ID for EC2 and Application Status Check
Resources:
EC2SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Sub ${AWS::StackName} - EC2 SG
VpcId: !Ref VpcId
Tags:
- Key: Name
Value: !Sub ${AWS::StackName}-ec2-sg
AppStatusCheckSourceSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Sub ${AWS::StackName} - Source SG for the application status check ENI
VpcId: !Ref VpcId
Tags:
- Key: Name
Value: !Sub ${AWS::StackName}-asc-sg
EC2SecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !Ref EC2SecurityGroup
IpProtocol: tcp
FromPort: 80
ToPort: 80
SourceSecurityGroupId: !GetAtt AppStatusCheckSourceSG.GroupId
Description: Application status check from the managed ENI
EC2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles:
- !Ref EC2Role
NginxEC2Instance:
Type: AWS::EC2::Instance
Properties:
InstanceType: !Ref InstanceType
ImageId: !Ref LatestAmiId
IamInstanceProfile: !Ref EC2InstanceProfile
SubnetId: !Ref SubnetId
SecurityGroupIds:
- !Ref EC2SecurityGroup
UserData:
Fn::Base64: |
#!/bin/bash
dnf install -y nginx
echo ok > /usr/share/nginx/html/health
systemctl enable nginx
systemctl start nginx
Tags:
- Key: Name
Value: !Sub ${AWS::StackName}-nginx
ApplicationStatusCheck:
Type: AWS::EC2::ApplicationStatusCheck
Properties:
Protocol: http
Port: 80
Path: /
StatusCodeMatcher: '200'
InitializationGracePeriodSeconds: 60
HealthCheckPaths:
- Source:
SecurityGroupId: !GetAtt AppStatusCheckSourceSG.GroupId
SubnetId: !Ref SubnetId
Destinations:
- SecurityGroupId: !GetAtt EC2SecurityGroup.GroupId
SubnetId: !Ref SubnetId
Tags:
- Key: Name
Value: !Sub ${AWS::StackName}-asc
AppStatusCheckAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub ${AWS::StackName}-StatusCheckFailed-App
Namespace: AWS/EC2
MetricName: StatusCheckFailed_Application
Dimensions:
- Name: InstanceId
Value: !Ref NginxEC2Instance
Statistic: Maximum
Period: 60
EvaluationPeriods: 2
Threshold: 1
ComparisonOperator: GreaterThanOrEqualToThreshold
TreatMissingData: missing
AlertTopic:
Type: AWS::SNS::Topic
EmailSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref AlertTopic
Protocol: email
Endpoint: !Ref NotificationEmail
EventBridgeRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: InvokeStepFunctions
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: states:StartExecution
Resource: !GetAtt RecoveryStateMachine.Arn
RecoveryTriggerRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.cloudwatch
detail-type:
- CloudWatch Alarm State Change
detail:
alarmName:
- !Ref AppStatusCheckAlarm
state:
value:
- ALARM
State: ENABLED
Targets:
- Arn: !GetAtt RecoveryStateMachine.Arn
RoleArn: !GetAtt EventBridgeRole.Arn
Id: RecoveryStateMachineTarget
StepFunctionsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: states.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: EC2AndSSM
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ec2:StopInstances
- ec2:StartInstances
Resource: !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${NginxEC2Instance}
- Effect: Allow
Action:
- ec2:DescribeInstances
Resource: '*'
- Effect: Allow
Action: ssm:SendCommand
Resource:
- !Sub arn:aws:ssm:${AWS::Region}::document/AWS-RunShellScript
- !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${NginxEC2Instance}
- Effect: Allow
Action: ssm:GetCommandInvocation
Resource: '*'
- Effect: Allow
Action: sns:Publish
Resource: !Ref AlertTopic
- PolicyName: CloudWatchLogs
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogDelivery
- logs:GetLogDelivery
- logs:UpdateLogDelivery
- logs:DeleteLogDelivery
- logs:ListLogDeliveries
- logs:PutLogEvents
- logs:PutResourcePolicy
- logs:DescribeResourcePolicies
- logs:DescribeLogGroups
Resource: '*'
StepFunctionsLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub /aws/vendedlogs/states/${AWS::StackName}
RetentionInDays: !Ref LogRetentionDays
RecoveryStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
StateMachineType: STANDARD
RoleArn: !GetAtt StepFunctionsRole.Arn
LoggingConfiguration:
Level: ALL
IncludeExecutionData: true
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn: !GetAtt StepFunctionsLogGroup.Arn
DefinitionSubstitutions:
InstanceId: !Ref NginxEC2Instance
SnsTopicArn: !Ref AlertTopic
DefinitionString: |
{
"Comment": "2-Stage Recovery: SSM restart -> EC2 Stop/Start",
"QueryLanguage": "JSONata",
"TimeoutSeconds": 1800,
"StartAt": "Init",
"States": {
"Init": {
"Type": "Pass",
"Assign": {
"stage1Retry": 0,
"stopCheck": 0,
"startCheck": 0,
"recoveryCheck": 0,
"forceStop": false
},
"Next": "CheckNginxInit"
},
"CheckNginxInit": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:sendCommand",
"Arguments": {
"InstanceIds": ["${InstanceId}"],
"DocumentName": "AWS-RunShellScript",
"Parameters": {"commands": ["systemctl is-active nginx"]},
"TimeoutSeconds": 30
},
"Assign": {"checkCmdId": "{% $states.result.Command.CommandId %}"},
"Next": "WaitCheckInit",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "SSMRestart"}]
},
"WaitCheckInit": {
"Type": "Wait",
"Seconds": 5,
"Next": "GetCheckInitResult"
},
"GetCheckInitResult": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:getCommandInvocation",
"Arguments": {"CommandId": "{% $checkCmdId %}", "InstanceId": "${InstanceId}"},
"Assign": {
"nginxStatus": "{% $states.result.StandardOutputContent %}",
"checkExitCode": "{% $states.result.ResponseCode %}"
},
"Next": "EvalInitStatus",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "SSMRestart"}]
},
"EvalInitStatus": {
"Type": "Choice",
"Choices": [{"Condition": "{% $checkExitCode = 0 %}", "Next": "NotifyAlreadyOK"}],
"Default": "SSMRestart"
},
"SSMRestart": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:sendCommand",
"Arguments": {
"InstanceIds": ["${InstanceId}"],
"DocumentName": "AWS-RunShellScript",
"Parameters": {"commands": ["sudo systemctl restart nginx"]},
"TimeoutSeconds": 60
},
"Assign": {"cmdId": "{% $states.result.Command.CommandId %}"},
"Next": "WaitSSM",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Stage2Stop"}]
},
"WaitSSM": {
"Type": "Wait",
"Seconds": 10,
"Next": "CheckSSM"
},
"CheckSSM": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:getCommandInvocation",
"Arguments": {"CommandId": "{% $cmdId %}", "InstanceId": "${InstanceId}"},
"Assign": {"ssmStatus": "{% $states.result.Status %}"},
"Next": "EvalSSM",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Stage2Stop"}]
},
"EvalSSM": {
"Type": "Choice",
"Choices": [
{"Condition": "{% $ssmStatus = 'Success' %}", "Next": "WaitAfterRestart"},
{"Condition": "{% $ssmStatus = 'Failed' or $ssmStatus = 'TimedOut' %}", "Next": "Stage2Stop"}
],
"Default": "WaitSSM"
},
"WaitAfterRestart": {
"Type": "Wait",
"Seconds": 10,
"Next": "CheckNginxAfterRestart"
},
"CheckNginxAfterRestart": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:sendCommand",
"Arguments": {
"InstanceIds": ["${InstanceId}"],
"DocumentName": "AWS-RunShellScript",
"Parameters": {"commands": ["systemctl is-active nginx"]},
"TimeoutSeconds": 30
},
"Assign": {"checkCmdId": "{% $states.result.Command.CommandId %}"},
"Next": "WaitCheckAfterRestart",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Stage2Stop"}]
},
"WaitCheckAfterRestart": {
"Type": "Wait",
"Seconds": 5,
"Next": "GetCheckAfterRestartResult"
},
"GetCheckAfterRestartResult": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:getCommandInvocation",
"Arguments": {"CommandId": "{% $checkCmdId %}", "InstanceId": "${InstanceId}"},
"Assign": {
"checkExitCode": "{% $states.result.ResponseCode %}",
"stage1Retry": "{% $stage1Retry + 1 %}"
},
"Next": "EvalStage1",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Stage2Stop"}]
},
"EvalStage1": {
"Type": "Choice",
"Choices": [
{"Condition": "{% $checkExitCode = 0 %}", "Next": "NotifyStage1OK"},
{"Condition": "{% $stage1Retry >= 3 %}", "Next": "Stage2Stop"}
],
"Default": "WaitAfterRestart"
},
"Stage2Stop": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:stopInstances",
"Arguments": {"InstanceIds": ["${InstanceId}"], "Force": false},
"Assign": {"stopCheck": 0},
"Next": "WaitStop",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "NotifyError"}]
},
"ForceStop": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:stopInstances",
"Arguments": {"InstanceIds": ["${InstanceId}"], "Force": true},
"Assign": {"forceStop": true, "stopCheck": 0},
"Next": "WaitStop",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "NotifyError"}]
},
"WaitStop": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckStop"
},
"CheckStop": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {"InstanceIds": ["${InstanceId}"]},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
"stopCheck": "{% $stopCheck + 1 %}"
},
"Next": "EvalStop",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "NotifyError"}]
},
"EvalStop": {
"Type": "Choice",
"Choices": [
{"Condition": "{% $ec2State = 'stopped' %}", "Next": "StartInstance"},
{"Condition": "{% $stopCheck >= 40 and $forceStop = false %}", "Next": "ForceStop"},
{"Condition": "{% $stopCheck >= 40 and $forceStop = true %}", "Next": "NotifyStopFail"}
],
"Default": "WaitStop"
},
"StartInstance": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:startInstances",
"Arguments": {"InstanceIds": ["${InstanceId}"]},
"Assign": {"startCheck": 0},
"Next": "WaitStart",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "NotifyError"}]
},
"WaitStart": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckStart"
},
"CheckStart": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {"InstanceIds": ["${InstanceId}"]},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
"startCheck": "{% $startCheck + 1 %}"
},
"Next": "EvalStart",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "NotifyError"}]
},
"EvalStart": {
"Type": "Choice",
"Choices": [
{"Condition": "{% $ec2State = 'running' %}", "Next": "WaitRecovery"},
{"Condition": "{% $startCheck >= 20 %}", "Next": "NotifyFail"}
],
"Default": "WaitStart"
},
"WaitRecovery": {
"Type": "Wait",
"Seconds": 30,
"Next": "CheckNginxRecovery"
},
"CheckNginxRecovery": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:sendCommand",
"Arguments": {
"InstanceIds": ["${InstanceId}"],
"DocumentName": "AWS-RunShellScript",
"Parameters": {"commands": ["systemctl is-active nginx"]},
"TimeoutSeconds": 30
},
"Assign": {"checkCmdId": "{% $states.result.Command.CommandId %}"},
"Next": "WaitCheckRecovery",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "RetryRecoveryCheck"}]
},
"WaitCheckRecovery": {
"Type": "Wait",
"Seconds": 5,
"Next": "GetCheckRecoveryResult"
},
"GetCheckRecoveryResult": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:getCommandInvocation",
"Arguments": {"CommandId": "{% $checkCmdId %}", "InstanceId": "${InstanceId}"},
"Assign": {
"checkExitCode": "{% $states.result.ResponseCode %}",
"recoveryCheck": "{% $recoveryCheck + 1 %}"
},
"Next": "EvalRecovery",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "RetryRecoveryCheck"}]
},
"RetryRecoveryCheck": {
"Type": "Pass",
"Assign": {"recoveryCheck": "{% $recoveryCheck + 1 %}"},
"Next": "EvalRecoveryRetry"
},
"EvalRecoveryRetry": {
"Type": "Choice",
"Choices": [{"Condition": "{% $recoveryCheck >= 10 %}", "Next": "NotifyFail"}],
"Default": "WaitRecovery"
},
"EvalRecovery": {
"Type": "Choice",
"Choices": [
{"Condition": "{% $checkExitCode = 0 %}", "Next": "NotifyStage2OK"},
{"Condition": "{% $recoveryCheck >= 10 %}", "Next": "NotifyFail"}
],
"Default": "WaitRecovery"
},
"NotifyStage1OK": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[OK] Nginx recovered (SSM restart)",
"Message": "{% 'Recovered via SSM restart.\\nInstanceId: ${InstanceId}\\nRetry: ' & $string($stage1Retry) & '\\nExecutionId: ' & $states.context.Execution.Id %}"
},
"End": true
},
"NotifyStage2OK": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[OK] EC2 recovered (Stop/Start)",
"Message": "{% 'Recovered via EC2 Stop/Start.\\nInstanceId: ${InstanceId}\\nRecoveryCheck: ' & $string($recoveryCheck) & '\\nExecutionId: ' & $states.context.Execution.Id %}"
},
"End": true
},
"NotifyAlreadyOK": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[INFO] Self-recovered",
"Message": "{% 'Nginx is already healthy.\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id %}"
},
"End": true
},
"NotifyFail": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[CRITICAL] Recovery failed",
"Message": "{% 'Automatic recovery has failed. Manual intervention is required.\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id %}"
},
"End": true
},
"NotifyStopFail": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[CRITICAL] EC2 stop failed",
"Message": "{% 'Failed to stop the EC2 instance.\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id %}"
},
"End": true
},
"NotifyError": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[ERROR] Unexpected error",
"Message": "{% 'An error has occurred.\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id %}"
},
"End": true
}
}
}
Outputs:
InstanceId:
Value: !Ref NginxEC2Instance
ApplicationStatusCheckId:
Value: !GetAtt ApplicationStatusCheck.ApplicationStatusCheckId
AppStatusCheckSourceSGId:
Value: !Ref AppStatusCheckSourceSG
SNSTopicArn:
Value: !Ref AlertTopic
StateMachineArn:
Value: !GetAtt RecoveryStateMachine.Arn
StepFunctionsLogGroupName:
Value: !Ref StepFunctionsLogGroup
