I tried automatically recovering EC2 instances that failed Status Checks using Step Functions without Lambda
This page has been translated by machine translation. View original
Introduction
EC2 Auto Recovery attempts to restore availability when system status checks fail due to underlying infrastructure issues. However, it does not cover cases where only the instance status check fails, and when Auto Recovery cannot complete the restoration, manual actions such as Stop/Start are required.
In this article, I automated this manual response using Step Functions triggered by a CloudWatch alarm. The Step Functions state machine handles EC2 API calls, waiting for stop/start completion, re-checking Status Check results, and conditional branching. I used FIS to disable a NIC and reproduce an Instance Status Check failure, then verified the recovery flow through the state machine definition and execution history.
What Was Tested
I will explain in order: the overall architecture, the state machine definition, and the actual behavior during fault injection with FIS.
Architecture
The EC2 StatusCheckFailed_Instance metric is monitored at 1-minute intervals, and the alarm transitions to ALARM when the threshold is exceeded for 2 consecutive data points. An EventBridge rule that picks up this state change triggers the state machine. The state machine then proceeds to stop and start the EC2 instance via API, and finally sends a notification via SNS.
Inside the State Machine
There are two reasons why Lambda functions are not needed: AWS SDK integration and JSONata.
AWS SDK integration is a mechanism that allows you to call AWS APIs directly by simply writing the ARN in the Resource field of a Task state. By specifying arn:aws:states:::aws-sdk:ec2:describeInstances, you can retrieve EC2 instance information. In this state machine, the only calls made are to retrieve instance state and Status Check results, stop/start the instance, and send SNS notifications.
The other feature, JSONata, is used for state management and condition evaluation. When QueryLanguage is set to JSONata, the way state input/output is written changes. Only two fields are used: Arguments and Output. Choice conditions are written as JSONata expressions in Condition, and the Variable and comparison operator fields used in JSONPath cannot be used. Counters such as loop counts can be carried over by updating variables with Assign.
A stop-waiting loop can be written with three states: Wait, Task, and Choice. The following WaitForStop loop definition omits error Catch handling.
"WaitForStop": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckStopStatus"
},
"CheckStopStatus": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
"stopCheckCount": "{% $stopCheckCount + 1 %}"
},
"Next": "EvaluateStopStatus"
},
"EvaluateStopStatus": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $ec2State = 'stopped' %}",
"Next": "StartInstance"
},
{
"Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = false %}",
"Next": "ForceStopInstance"
},
{
"Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = true %}",
"Next": "StopTimeout"
}
],
"Default": "WaitForStop"
}
It waits 15 seconds, calls describeInstances again, and if the state is stopped it proceeds to start; otherwise it returns to waiting. The maximum check count is 40, so the design waits approximately 10 minutes for a normal stop before switching to a force stop. If the count reaches 40 again after a force stop, it notifies as a timeout. The maximum for waiting for running after startup is 20 times (approximately 5 minutes), and the maximum for waiting for Status Check recovery is 40 times (approximately 10 minutes).
Fault Injection and Recovery Results
To bring down the Status Check, I disabled the NIC via SSM from an FIS experiment template.
Actions:
DisableNic:
ActionId: aws:ssm:send-command
Parameters:
documentArn: !Sub arn:aws:ssm:${AWS::Region}::document/AWS-RunShellScript
documentParameters: '{"commands":["sudo ip link set ens5 down"]}'
duration: PT1M
Targets:
Instances: TargetEC2
StopConditions:
- Source: none
The NIC name ens5 is the value confirmed on a t4g.nano running Amazon Linux 2023 (ARM64). Device names vary by environment, so please verify on the target instance before running.
After the NIC was brought down, the alarm transitioned to ALARM and EventBridge started the state machine. The event passed by EventBridge contains the reason that the transition occurred because two data points at 1-minute intervals were met.
{
"detail-type": "CloudWatch Alarm State Change",
"source": "aws.cloudwatch",
"time": "2026-09-04T17:44:13Z",
"detail": {
"alarmName": "ec2-recovery-test-StatusCheckFailed-Instance",
"state": {
"value": "ALARM",
"reason": "Threshold Crossed: 2 datapoints [1.0 (04/09/26 17:43:00), 1.0 (04/09/26 17:42:00)] were greater than or equal to the threshold (1.0).",
"timestamp": "2026-09-04T17:44:13.159+0000"
},
"previousState": {
"value": "OK",
"reason": "Threshold Crossed: 1 datapoint [0.0 (04/09/26 17:34:00)] was not greater than or equal to the threshold (1.0).",
"timestamp": "2026-09-04T17:35:13.157+0000"
}
}
}
The event time is in UTC, which corresponds to 02:44:13 JST.
The FIS experiment itself completed in 73 seconds, but the disabled NIC remained as-is. Even at 02:44:13 when the state machine started, the Instance status was still impaired and the Status Check had not recovered.
I arranged the states obtained from the state machine execution history in chronological order. Times are in JST. state indicates the instance state from describeInstances. inst and sys are the Instance / System statuses from describeInstanceStatus.
02:44:13.731 GetEC2StatusCheck state=running inst=impaired sys=ok
02:44:29.428 CheckStopStatus state=stopping
02:44:44.743 CheckStopStatus state=stopped
02:45:01.252 CheckStartStatus state=running
02:45:16.453 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:45:31.693 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:45:46.903 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:46:02.113 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:46:17.327 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:46:32.541 CheckRecoveryStatus state=running inst=initializing sys=ok
02:46:47.755 CheckRecoveryStatus state=running inst=initializing sys=ok
02:47:03.062 CheckRecoveryStatus state=running inst=initializing sys=ok
02:47:18.254 CheckRecoveryStatus state=running inst=ok sys=ok
Since the initial GetEC2StatusCheck showed Instance as impaired and System as ok, EvaluateState branched to StopInstance. CheckStopStatus was satisfied in 2 iterations, CheckStartStatus in 1 iteration, and CheckRecoveryStatus in 9 iterations.
After startup, System recovered to ok first at 02:46:32, and Instance became ok approximately 46 seconds later at 02:47:18. The approximately 137 seconds of recovery waiting corresponds to waiting for the Instance side to finish initializing.
The execution status was SUCCEEDED. Of the total 139 events in the execution history, state entry events totaled 43: Task 17 times, Choice 13 times, Wait 12 times, and Pass 1 time. In this execution, the ForceStopInstance, StopTimeout, and RecoveryFailed paths were not taken. AlreadyRecovered, EscalateSystem, EscalateTerminated, and NotifyUnexpectedError were also not taken.
| Segment | Start → End (JST) | Duration |
|---|---|---|
| FIS experiment (NIC disabled) | 02:41:22.592 → 02:42:35.145 | Approximately 73 seconds |
| Alarm detection | 02:41:22.592 → 02:44:13.159 | Approximately 171 seconds |
| Step Functions execution | 02:44:13.337 → 02:47:18.507 | 185 seconds |
| Of which, waiting for Stop completion | 02:44:14.195 → 02:44:44.743 | Approximately 31 seconds |
| Of which, waiting for running after Start | 02:44:45.804 → 02:45:01.252 | Approximately 16 seconds |
| Of which, waiting for Status Check recovery | 02:45:01.252 → 02:47:18.254 | Approximately 137 seconds |
| From fault injection to recovery completion | 02:41:22.592 → 02:47:18.507 | Approximately 6 minutes |
The total time from fault injection to recovery completion was approximately 6 minutes.
Monthly Running Costs
Running costs were calculated using Tokyo Region pricing.
| Item | Unit Price (Asia Pacific - Tokyo) | Free Tier |
|---|---|---|
| Step Functions Standard state transitions | 0.000025 USD / transition | 4,000 transitions/month |
| CloudWatch metric alarm (standard resolution) | 0.10 USD / alarm metric/month | 10 alarm metrics/month |
| CloudWatch Logs Vended Logs ingestion (Standard) | 0.76 USD / GB (first 10 TB) | 5 GB/month |
| SNS email notifications | 2.00 USD / 100,000 messages | 1,000 messages/month |
Unit prices are from the Step Functions, CloudWatch, and Amazon SNS pricing pages.
Not included in the estimate are the last two rows in the table—state machine log output and SNS notifications—as well as FIS experiment execution costs and the EC2 instance itself.
Assuming recovery occurs once per month, costs were calculated at list price. Since the state transitions counted from the execution results were 43 per execution, that is 43 transitions × 0.000025 USD = 0.001075 USD. The alarm costs 0.10 USD per alarm metric. The total comes to approximately 0.101 USD per month.
Since state transitions and alarms each have free tiers, a workload of this scale will fit within the free tier if allowance remains.
Summary
I was able to automatically recover an EC2 instance that failed a Status Check using Step Functions without Lambda.
By not using Lambda functions, there is no need to keep up with Lambda runtime updates or manage per-function deployments and configurations. Additionally, since not only the execution results of each state but also waits and branches are recorded in the Step Functions execution history, it becomes easier to check the progress of recovery and identify where failures occurred.
If you have been manually handling EC2 instance restarts in your operations, please consider this as one option for automating that response.
Reference Materials
The complete configuration for this setup is compiled into a CloudFormation template.
CloudFormation Template (Full)
AWSTemplateFormatVersion: '2010-09-09'
Description: 'EC2 Stop/Start Auto Recovery Test Environment'
Parameters:
ProjectName:
Type: String
Default: ec2-recovery-test
NotificationEmail:
Type: String
Default: alerts@example.com
InstanceType:
Type: String
Default: t4g.nano
LogRetentionDays:
Type: Number
Default: 7
LatestAmiId:
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64
Resources:
# ============================================================
# EC2 Security Group (uses default VPC)
# ============================================================
EC2SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Sub ${ProjectName} - Outbound only for SSM
Tags:
- Key: Name
Value: !Sub ${ProjectName}-sg
# ============================================================
# EC2 IAM Role and Instance Profile
# ============================================================
EC2Role:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ProjectName}-ec2-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
InstanceProfileName: !Sub ${ProjectName}-ec2-profile
Roles:
- !Ref EC2Role
# ============================================================
# EC2 Instance
# ============================================================
TestEC2Instance:
Type: AWS::EC2::Instance
Properties:
InstanceType: !Ref InstanceType
ImageId: !Ref LatestAmiId
IamInstanceProfile: !Ref EC2InstanceProfile
SecurityGroupIds:
- !Ref EC2SecurityGroup
Tags:
- Key: Name
Value: !Sub ${ProjectName}-test-ec2
# ============================================================
# SNS Topic and Subscription
# ============================================================
AlertTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub ${ProjectName}-alerts
EmailSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref AlertTopic
Protocol: email
Endpoint: !Ref NotificationEmail
# ============================================================
# CloudWatch Alarm
# ============================================================
StatusCheckAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub ${ProjectName}-StatusCheckFailed-Instance
AlarmDescription: Alarm when StatusCheckFailed_Instance is >= 1
Namespace: AWS/EC2
MetricName: StatusCheckFailed_Instance
Dimensions:
- Name: InstanceId
Value: !Ref TestEC2Instance
Statistic: Maximum
Period: 60
EvaluationPeriods: 2
Threshold: 1
ComparisonOperator: GreaterThanOrEqualToThreshold
TreatMissingData: missing
# ============================================================
# EventBridge IAM Role
# ============================================================
EventBridgeRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ProjectName}-eventbridge-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: InvokeStepFunctions
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: states:StartExecution
Resource: !GetAtt RecoveryStateMachine.Arn
# ============================================================
# EventBridge Rule
# ============================================================
RecoveryTriggerRule:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${ProjectName}-recovery-trigger
Description: Trigger Step Functions when StatusCheckFailed alarm goes to ALARM
EventPattern:
source:
- aws.cloudwatch
detail-type:
- CloudWatch Alarm State Change
detail:
alarmName:
- !Ref StatusCheckAlarm
state:
value:
- ALARM
State: ENABLED
Targets:
- Arn: !GetAtt RecoveryStateMachine.Arn
RoleArn: !GetAtt EventBridgeRole.Arn
Id: RecoveryStateMachineTarget
# ============================================================
# Step Functions IAM Role
# ============================================================
StepFunctionsRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ProjectName}-stepfunctions-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: states.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: EC2Operations
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ec2:StopInstances
- ec2:StartInstances
Resource: !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${TestEC2Instance}
- Effect: Allow
Action:
- ec2:DescribeInstances
- ec2:DescribeInstanceStatus
Resource: '*'
- PolicyName: SNSPublish
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: sns:Publish
Resource: !Ref AlertTopic
- PolicyName: CloudWatchLogs
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogDelivery
- logs:GetLogDelivery
- logs:UpdateLogDelivery
- logs:DeleteLogDelivery
- logs:ListLogDeliveries
- logs:PutLogEvents
- logs:PutResourcePolicy
- logs:DescribeResourcePolicies
- logs:DescribeLogGroups
Resource: '*'
# ============================================================
# Step Functions Log Group
# ============================================================
StepFunctionsLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub /aws/vendedlogs/states/${ProjectName}-ec2-recovery
RetentionInDays: !Ref LogRetentionDays
# ============================================================
# Step Functions State Machine
# ============================================================
RecoveryStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
StateMachineName: !Sub ${ProjectName}-ec2-recovery
StateMachineType: STANDARD
RoleArn: !GetAtt StepFunctionsRole.Arn
LoggingConfiguration:
Level: ALL
IncludeExecutionData: true
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn: !GetAtt StepFunctionsLogGroup.Arn
DefinitionSubstitutions:
InstanceId: !Ref TestEC2Instance
SnsTopicArn: !Ref AlertTopic
DefinitionString: |
{
"Comment": "EC2 Stop/Start Auto Recovery Workflow",
"QueryLanguage": "JSONata",
"TimeoutSeconds": 2400,
"StartAt": "InitializeVariables",
"States": {
"InitializeVariables": {
"Type": "Pass",
"Assign": {
"stopCheckCount": 0,
"startCheckCount": 0,
"recoveryCheckCount": 0,
"forceStopAttempted": false,
"errorInfo": null
},
"Next": "GetEC2State"
},
"GetEC2State": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}"
},
"Next": "GetEC2StatusCheck",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"GetEC2StatusCheck": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstanceStatus",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"instanceStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].InstanceStatus.Status : 'unknown' %}",
"systemStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].SystemStatus.Status : 'unknown' %}"
},
"Next": "EvaluateState",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"EvaluateState": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $systemStatus = 'impaired' %}",
"Next": "EscalateSystem"
},
{
"Condition": "{% $instanceStatus = 'ok' and $systemStatus = 'ok' %}",
"Next": "AlreadyRecovered"
},
{
"Condition": "{% $ec2State = 'terminated' or $ec2State = 'shutting-down' %}",
"Next": "EscalateTerminated"
},
{
"Condition": "{% $ec2State = 'stopping' %}",
"Next": "WaitForStop"
},
{
"Condition": "{% $ec2State = 'stopped' %}",
"Next": "StartInstance"
},
{
"Condition": "{% $ec2State = 'pending' %}",
"Next": "WaitForStart"
},
{
"Condition": "{% $ec2State = 'running' and $instanceStatus = 'impaired' and $systemStatus = 'ok' %}",
"Next": "StopInstance"
}
],
"Default": "StopInstance"
},
"StopInstance": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:stopInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"],
"Force": false
},
"Assign": {
"stopCheckCount": 0
},
"Next": "WaitForStop",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"ForceStopInstance": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:stopInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"],
"Force": true
},
"Assign": {
"forceStopAttempted": true,
"stopCheckCount": 0
},
"Next": "WaitForStop",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"WaitForStop": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckStopStatus"
},
"CheckStopStatus": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
"stopCheckCount": "{% $stopCheckCount + 1 %}"
},
"Next": "EvaluateStopStatus",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"EvaluateStopStatus": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $ec2State = 'stopped' %}",
"Next": "StartInstance"
},
{
"Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = false %}",
"Next": "ForceStopInstance"
},
{
"Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = true %}",
"Next": "StopTimeout"
}
],
"Default": "WaitForStop"
},
"StartInstance": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:startInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"startCheckCount": 0
},
"Next": "WaitForStart",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"WaitForStart": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckStartStatus"
},
"CheckStartStatus": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
"startCheckCount": "{% $startCheckCount + 1 %}"
},
"Next": "EvaluateStartStatus",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"EvaluateStartStatus": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $ec2State = 'running' %}",
"Next": "WaitForRecovery"
},
{
"Condition": "{% $startCheckCount >= 20 %}",
"Next": "RecoveryFailed"
}
],
"Default": "WaitForStart"
},
"WaitForRecovery": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckRecoveryStatus"
},
"CheckRecoveryStatus": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstanceStatus",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"instanceStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].InstanceStatus.Status : 'unknown' %}",
"systemStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].SystemStatus.Status : 'unknown' %}",
"recoveryCheckCount": "{% $recoveryCheckCount + 1 %}"
},
"Next": "EvaluateRecoveryStatus",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"EvaluateRecoveryStatus": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $instanceStatus = 'ok' and $systemStatus = 'ok' %}",
"Next": "RecoverySuccess"
},
{
"Condition": "{% $recoveryCheckCount >= 40 %}",
"Next": "RecoveryFailed"
}
],
"Default": "WaitForRecovery"
},
"RecoverySuccess": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] EC2 Auto Recovery Succeeded",
"Message": "{% 'EC2 Stop/Start auto recovery has completed.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
},
"End": true
},
"RecoveryFailed": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] EC2 Auto Recovery Failed",
"Message": "{% 'EC2 Stop/Start auto recovery timed out. Manual intervention is required.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
},
"End": true
},
"StopTimeout": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] EC2 Stop Failed",
"Message": "{% 'EC2 instance stop timed out (failed even after force stop). Manual intervention is required.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State %}"
},
"End": true
},
"AlreadyRecovered": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] Natural Recovery Confirmed",
"Message": "{% 'EC2 instance is already in a healthy state (natural recovery).\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
},
"End": true
},
"EscalateSystem": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] System Failure",
"Message": "{% 'SystemStatus is impaired. Recovery via Stop/Start may not be possible.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
},
"End": true
},
"EscalateTerminated": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] Instance Terminated",
"Message": "{% 'EC2 instance is in terminated or shutting-down state. Recovery is not possible.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nState: ' & $ec2State %}"
},
"End": true
},
"NotifyUnexpectedError": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] Unexpected Error",
"Message": "{% 'An unexpected error occurred during Step Functions execution.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nError: ' & $string($errorInfo) %}"
},
"End": true
}
}
}
# ============================================================
# FIS IAM Role
# ============================================================
FISRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ProjectName}-fis-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: fis.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: FISSSMPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ssm:SendCommand
- ssm:GetCommandInvocation
- ssm:ListCommands
- ssm:ListCommandInvocations
- ssm:CancelCommand
Resource: '*'
- Effect: Allow
Action:
- ec2:DescribeInstances
- ec2:DescribeInstanceStatus
Resource: '*'
# ============================================================
# FIS Experiment Template
# ============================================================
NicDisableExperiment:
Type: AWS::FIS::ExperimentTemplate
Properties:
Description: NIC disable to trigger StatusCheckFailed_Instance
Actions:
DisableNic:
ActionId: aws:ssm:send-command
Parameters:
documentArn: !Sub arn:aws:ssm:${AWS::Region}::document/AWS-RunShellScript
documentParameters: '{"commands":["sudo ip link set ens5 down"]}'
duration: PT1M
Targets:
Instances: TargetEC2
Targets:
TargetEC2:
ResourceType: aws:ec2:instance
ResourceArns:
- !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${TestEC2Instance}
SelectionMode: ALL
StopConditions:
- Source: none
RoleArn: !GetAtt FISRole.Arn
Tags:
Name: !Sub ${ProjectName}-nic-disable
Outputs:
EC2InstanceId:
Description: EC2 Instance ID
Value: !Ref TestEC2Instance
EC2InstancePublicDnsName:
Description: EC2 Instance Public DNS Name
Value: !GetAtt TestEC2Instance.PublicDnsName
SecurityGroupId:
Description: Security Group ID
Value: !Ref EC2SecurityGroup
SNSTopicArn:
Description: SNS Topic ARN
Value: !Ref AlertTopic
CloudWatchAlarmName:
Description: CloudWatch Alarm Name
Value: !Ref StatusCheckAlarm
EventBridgeRuleName:
Description: EventBridge Rule Name
Value: !Ref RecoveryTriggerRule
StepFunctionsStateMachineArn:
Description: Step Functions State Machine ARN
Value: !GetAtt RecoveryStateMachine.Arn
StepFunctionsLogGroupName:
Description: Step Functions Log Group Name
Value: !Ref StepFunctionsLogGroup
FISExperimentTemplateId:
Description: FIS Experiment Template ID
Value: !Ref NicDisableExperiment
Save the above template as ec2-recovery-test.yaml. Deploy it with the following commands, inject a fault with FIS, and finally delete the stack. The prerequisites before execution are as follows.
- After deployment, a confirmation email will be sent to the notification address, so approve the SNS subscription
- Since the template does not specify a subnet, run it in an account that has a default VPC
- FIS fault injection is executed via SSM, so the instance needs to be able to reach SSM
- The FIS target is only the test instance created by this template. Since this is an operation that brings down the NIC, run it in a test account
- Use the latest version of AWS CLI (version at time of verification: aws-cli/2.36.38)
# Deploy
aws cloudformation create-stack \
--stack-name ec2-recovery-test \
--template-body file://ec2-recovery-test.yaml \
--parameters ParameterKey=NotificationEmail,ParameterValue=<notification destination email address> \
--capabilities CAPABILITY_NAMED_IAM \
--deployment-config '{"Mode": "EXPRESS"}' \
--region ap-northeast-1
aws cloudformation wait stack-create-complete \
--stack-name ec2-recovery-test \
--region ap-northeast-1
# Disable NIC with FIS to trigger StatusCheckFailed_Instance
aws fis start-experiment \
--experiment-template-id <output value of FISExperimentTemplateId> \
--region ap-northeast-1
# To test only the integration from EventBridge to Step Functions
aws cloudwatch set-alarm-state \
--alarm-name ec2-recovery-test-StatusCheckFailed-Instance \
--state-value ALARM \
--state-reason "Manual test trigger" \
--region ap-northeast-1
# Teardown
aws cloudformation delete-stack \
--stack-name ec2-recovery-test \
--region ap-northeast-1
