I Tried Automatically Recovering EC2 Instances That Failed Status Checks Using Step Functions Without Lambda
This page has been translated by machine translation. View original
Introduction
EC2 Auto Recovery attempts to restore availability when a system status check fails because of an underlying problem. On the other hand, it does not cover cases where only the instance status check fails, and when Auto Recovery does not succeed in recovering the instance, manual actions such as Stop/Start are still required.
In this article, I automated this manual response with Step Functions triggered from a CloudWatch alarm. In Step Functions, I defined EC2 API calls, waiting for stop and start to complete, rechecking the Status Check, and conditional branching. I reproduced an Instance Status Check failure by disabling the NIC with FIS, and verified the recovery flow from the state machine definition and execution history.
Verification Details
I will explain the overall configuration, the state machine definition, and the actual behavior when the failure was reproduced with FIS, in that order.
Configuration
The StatusCheckFailed_Instance metric of EC2 is monitored at 1-minute intervals, and when two consecutive datapoints are at or above the threshold, the alarm transitions to ALARM. An EventBridge rule that catches this state change starts the state machine. From there, the state machine proceeds with stop and start via the EC2 API, and finally notifies the result via SNS.
Inside the State Machine
There are two reasons why no Lambda function needs to be written: AWS SDK integrations and JSONata.
AWS SDK integration is a mechanism that lets you call AWS APIs directly just by writing an ARN in the Resource of a Task state. By specifying arn:aws:states:::aws-sdk:ec2:describeInstances, you can retrieve EC2 instance information. The only things this state machine calls are retrieving the instance state and Status Check, stopping and starting, and notifying SNS.
The other, JSONata, is used for holding state and evaluating conditions. When QueryLanguage is set to JSONata, the way state input and output are written changes. The two fields used are Arguments and Output. Choice conditions are written as JSONata expressions in Condition, and the Variable and comparison operator fields used with JSONPath cannot be used. Counters such as loop counts can be carried around by updating variables with Assign.
The stop-wait loop can be written with three states: Wait, Task, and Choice. The definition of the WaitForStop loop shown below omits the Catch for errors.
"WaitForStop": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckStopStatus"
},
"CheckStopStatus": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
"stopCheckCount": "{% $stopCheckCount + 1 %}"
},
"Next": "EvaluateStopStatus"
},
"EvaluateStopStatus": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $ec2State = 'stopped' %}",
"Next": "StartInstance"
},
{
"Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = false %}",
"Next": "ForceStopInstance"
},
{
"Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = true %}",
"Next": "StopTimeout"
}
],
"Default": "WaitForStop"
}
It waits 15 seconds and calls describeInstances again; if the state is stopped, it proceeds to start, and otherwise it returns to waiting. The upper limit on check count is 40, so the design waits about 10 minutes for a normal stop and then switches to a stop with Force specified. If the count reaches 40 again after the Force stop, it notifies as a timeout. The upper limit for waiting for running after start is 20 checks (about 5 minutes), and the upper limit for waiting for Status Check recovery is 40 checks (about 10 minutes).
Fault Injection and Recovery Results
To make the Status Check fail, I disabled the NIC via SSM from a FIS experiment template.
Actions:
DisableNic:
ActionId: aws:ssm:send-command
Parameters:
documentArn: !Sub arn:aws:ssm:${AWS::Region}::document/AWS-RunShellScript
documentParameters: '{"commands":["sudo ip link set ens5 down"]}'
duration: PT1M
Targets:
Instances: TargetEC2
StopConditions:
- Source: none
The NIC name ens5 is the value I confirmed on a t4g.nano running Amazon Linux 2023 (ARM64). Device names differ by environment, so please check on the target instance before running.
After the NIC was brought down, the alarm transitioned to ALARM, and EventBridge started the state machine. The event passed by EventBridge includes the reason that the transition occurred once two datapoints at 1-minute intervals were in place.
{
"detail-type": "CloudWatch Alarm State Change",
"source": "aws.cloudwatch",
"time": "2026-09-04T17:44:13Z",
"detail": {
"alarmName": "ec2-recovery-test-StatusCheckFailed-Instance",
"state": {
"value": "ALARM",
"reason": "Threshold Crossed: 2 datapoints [1.0 (04/09/26 17:43:00), 1.0 (04/09/26 17:42:00)] were greater than or equal to the threshold (1.0).",
"timestamp": "2026-09-04T17:44:13.159+0000"
},
"previousState": {
"value": "OK",
"reason": "Threshold Crossed: 1 datapoint [0.0 (04/09/26 17:34:00)] was not greater than or equal to the threshold (1.0).",
"timestamp": "2026-09-04T17:35:13.157+0000"
}
}
}
The event time is in UTC, which corresponds to 02:44:13 in JST.
The FIS experiment itself completed in 73 seconds, but the disabled NIC remained as it was. Even at 02:44:13 when the state machine started, the Instance status was impaired, and the Status Check was still failing.
From the state machine's execution history, I lined up the state each state retrieved in chronological order. Times are in JST. state indicates the instance state from describeInstances. inst and sys are the Instance / System statuses from describeInstanceStatus.
02:44:13.731 GetEC2StatusCheck state=running inst=impaired sys=ok
02:44:29.428 CheckStopStatus state=stopping
02:44:44.743 CheckStopStatus state=stopped
02:45:01.252 CheckStartStatus state=running
02:45:16.453 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:45:31.693 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:45:46.903 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:46:02.113 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:46:17.327 CheckRecoveryStatus state=running inst=initializing sys=initializing
02:46:32.541 CheckRecoveryStatus state=running inst=initializing sys=ok
02:46:47.755 CheckRecoveryStatus state=running inst=initializing sys=ok
02:47:03.062 CheckRecoveryStatus state=running inst=initializing sys=ok
02:47:18.254 CheckRecoveryStatus state=running inst=ok sys=ok
Because the first GetEC2StatusCheck showed Instance as impaired and System as ok, EvaluateState branched to StopInstance. The condition was satisfied after 2 runs of CheckStopStatus, 1 run of CheckStartStatus, and 9 runs of CheckRecoveryStatus.
After the start, System returned to ok first at 02:46:32, and Instance returned to ok about 46 seconds later at 02:47:18. The roughly 137 seconds spent waiting for recovery corresponds to waiting for the Instance side to initialize.
The execution status was SUCCEEDED. Of the 139 events in the execution history in total, the events that entered a state numbered 43 in total: 17 Task, 13 Choice, 12 Wait, and 1 Pass. In this execution, the ForceStopInstance, StopTimeout, and RecoveryFailed paths were not taken. AlreadyRecovered, EscalateSystem, EscalateTerminated, and NotifyUnexpectedError were likewise not taken.
| Interval | Start → End (JST) | Duration |
|---|---|---|
| FIS experiment (NIC disabled) | 02:41:22.592 → 02:42:35.145 | about 73 seconds |
| Alarm detection | 02:41:22.592 → 02:44:13.159 | about 171 seconds |
| Step Functions execution | 02:44:13.337 → 02:47:18.507 | 185 seconds |
| of which waiting for Stop to complete | 02:44:14.195 → 02:44:44.743 | about 31 seconds |
| of which waiting for running after Start | 02:44:45.804 → 02:45:01.252 | about 16 seconds |
| of which waiting for Status Check recovery | 02:45:01.252 → 02:47:18.254 | about 137 seconds |
| From fault injection to recovery completion | 02:41:22.592 → 02:47:18.507 | about 6 minutes |
It took about 6 minutes from fault injection to recovery completion.
Monthly Running Cost
Running costs were calculated using Tokyo region unit prices.
| Item | Unit price (Asia Pacific (Tokyo)) | Free tier |
|---|---|---|
| Step Functions Standard state transitions | 0.000025 USD / transition | 4,000 transitions per month |
| CloudWatch metric alarms (standard resolution) | 0.10 USD / alarm metric month | 10 alarm metrics per month |
| CloudWatch Logs Vended Logs ingestion (Standard) | 0.76 USD / GB (first 10 TB) | 5 GB per month |
| SNS email notifications | 2.00 USD / 100,000 emails | 1,000 emails per month |
The unit prices are the values from the pricing pages for Step Functions, CloudWatch, and Amazon SNS.
The estimate does not include the state machine log output and SNS notifications corresponding to the bottom two rows of the table, the FIS experiment execution fee, or the fee for the EC2 instance itself.
As a premise, I calculated the cost at list price assuming recovery happens once a month. The number of state transitions counted in the execution result is 43 per execution, so 43 × 0.000025 USD = 0.001075 USD. The alarm costs 0.10 USD per alarm metric. The total came to about 0.101 USD per month.
State transitions and alarms each have a free tier, so if the tier remains, this scale falls within it.
Summary
I was able to automatically recover an EC2 instance that failed its Status Check using Step Functions without Lambda.
Because no Lambda functions are used, there is no need to keep up with Lambda runtime updates or to manage per-function deployment and configuration. In addition, because the execution results of each state, along with waits and branches, remain in the Step Functions execution history, it became easier to check the progress of recovery and where a failure occurred.
If you have been restarting EC2 instances manually, please try this as one option for automating that response.
References
The entire configuration used this time is compiled into a CloudFormation template.
Full CloudFormation template
AWSTemplateFormatVersion: '2010-09-09'
Description: 'EC2 Stop/Start Auto Recovery Test Environment'
Parameters:
ProjectName:
Type: String
Default: ec2-recovery-test
NotificationEmail:
Type: String
Default: alerts@example.com
InstanceType:
Type: String
Default: t4g.nano
LogRetentionDays:
Type: Number
Default: 7
LatestAmiId:
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64
Resources:
# ============================================================
# EC2 Security Group (uses default VPC)
# ============================================================
EC2SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Sub ${ProjectName} - Outbound only for SSM
Tags:
- Key: Name
Value: !Sub ${ProjectName}-sg
# ============================================================
# EC2 IAM Role and Instance Profile
# ============================================================
EC2Role:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ProjectName}-ec2-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
InstanceProfileName: !Sub ${ProjectName}-ec2-profile
Roles:
- !Ref EC2Role
# ============================================================
# EC2 Instance
# ============================================================
TestEC2Instance:
Type: AWS::EC2::Instance
Properties:
InstanceType: !Ref InstanceType
ImageId: !Ref LatestAmiId
IamInstanceProfile: !Ref EC2InstanceProfile
SecurityGroupIds:
- !Ref EC2SecurityGroup
Tags:
- Key: Name
Value: !Sub ${ProjectName}-test-ec2
# ============================================================
# SNS Topic and Subscription
# ============================================================
AlertTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub ${ProjectName}-alerts
EmailSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref AlertTopic
Protocol: email
Endpoint: !Ref NotificationEmail
# ============================================================
# CloudWatch Alarm
# ============================================================
StatusCheckAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub ${ProjectName}-StatusCheckFailed-Instance
AlarmDescription: Alarm when StatusCheckFailed_Instance is >= 1
Namespace: AWS/EC2
MetricName: StatusCheckFailed_Instance
Dimensions:
- Name: InstanceId
Value: !Ref TestEC2Instance
Statistic: Maximum
Period: 60
EvaluationPeriods: 2
Threshold: 1
ComparisonOperator: GreaterThanOrEqualToThreshold
TreatMissingData: missing
# ============================================================
# EventBridge IAM Role
# ============================================================
EventBridgeRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ProjectName}-eventbridge-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: InvokeStepFunctions
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: states:StartExecution
Resource: !GetAtt RecoveryStateMachine.Arn
# ============================================================
# EventBridge Rule
# ============================================================
RecoveryTriggerRule:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${ProjectName}-recovery-trigger
Description: Trigger Step Functions when StatusCheckFailed alarm goes to ALARM
EventPattern:
source:
- aws.cloudwatch
detail-type:
- CloudWatch Alarm State Change
detail:
alarmName:
- !Ref StatusCheckAlarm
state:
value:
- ALARM
State: ENABLED
Targets:
- Arn: !GetAtt RecoveryStateMachine.Arn
RoleArn: !GetAtt EventBridgeRole.Arn
Id: RecoveryStateMachineTarget
# ============================================================
# Step Functions IAM Role
# ============================================================
StepFunctionsRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ProjectName}-stepfunctions-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: states.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: EC2Operations
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ec2:StopInstances
- ec2:StartInstances
Resource: !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${TestEC2Instance}
- Effect: Allow
Action:
- ec2:DescribeInstances
- ec2:DescribeInstanceStatus
Resource: '*'
- PolicyName: SNSPublish
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: sns:Publish
Resource: !Ref AlertTopic
- PolicyName: CloudWatchLogs
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogDelivery
- logs:GetLogDelivery
- logs:UpdateLogDelivery
- logs:DeleteLogDelivery
- logs:ListLogDeliveries
- logs:PutLogEvents
- logs:PutResourcePolicy
- logs:DescribeResourcePolicies
- logs:DescribeLogGroups
Resource: '*'
# ============================================================
# Step Functions Log Group
# ============================================================
StepFunctionsLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub /aws/vendedlogs/states/${ProjectName}-ec2-recovery
RetentionInDays: !Ref LogRetentionDays
# ============================================================
# Step Functions State Machine
# ============================================================
RecoveryStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
StateMachineName: !Sub ${ProjectName}-ec2-recovery
StateMachineType: STANDARD
RoleArn: !GetAtt StepFunctionsRole.Arn
LoggingConfiguration:
Level: ALL
IncludeExecutionData: true
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn: !GetAtt StepFunctionsLogGroup.Arn
DefinitionSubstitutions:
InstanceId: !Ref TestEC2Instance
SnsTopicArn: !Ref AlertTopic
DefinitionString: |
{
"Comment": "EC2 Stop/Start Auto Recovery Workflow",
"QueryLanguage": "JSONata",
"TimeoutSeconds": 2400,
"StartAt": "InitializeVariables",
"States": {
"InitializeVariables": {
"Type": "Pass",
"Assign": {
"stopCheckCount": 0,
"startCheckCount": 0,
"recoveryCheckCount": 0,
"forceStopAttempted": false,
"errorInfo": null
},
"Next": "GetEC2State"
},
"GetEC2State": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}"
},
"Next": "GetEC2StatusCheck",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"GetEC2StatusCheck": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstanceStatus",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"instanceStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].InstanceStatus.Status : 'unknown' %}",
"systemStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].SystemStatus.Status : 'unknown' %}"
},
"Next": "EvaluateState",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"EvaluateState": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $systemStatus = 'impaired' %}",
"Next": "EscalateSystem"
},
{
"Condition": "{% $instanceStatus = 'ok' and $systemStatus = 'ok' %}",
"Next": "AlreadyRecovered"
},
{
"Condition": "{% $ec2State = 'terminated' or $ec2State = 'shutting-down' %}",
"Next": "EscalateTerminated"
},
{
"Condition": "{% $ec2State = 'stopping' %}",
"Next": "WaitForStop"
},
{
"Condition": "{% $ec2State = 'stopped' %}",
"Next": "StartInstance"
},
{
"Condition": "{% $ec2State = 'pending' %}",
"Next": "WaitForStart"
},
{
"Condition": "{% $ec2State = 'running' and $instanceStatus = 'impaired' and $systemStatus = 'ok' %}",
"Next": "StopInstance"
}
],
"Default": "StopInstance"
},
"StopInstance": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:stopInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"],
"Force": false
},
"Assign": {
"stopCheckCount": 0
},
"Next": "WaitForStop",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"ForceStopInstance": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:stopInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"],
"Force": true
},
"Assign": {
"forceStopAttempted": true,
"stopCheckCount": 0
},
"Next": "WaitForStop",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"WaitForStop": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckStopStatus"
},
"CheckStopStatus": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
"stopCheckCount": "{% $stopCheckCount + 1 %}"
},
"Next": "EvaluateStopStatus",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"EvaluateStopStatus": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $ec2State = 'stopped' %}",
"Next": "StartInstance"
},
{
"Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = false %}",
"Next": "ForceStopInstance"
},
{
"Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = true %}",
"Next": "StopTimeout"
}
],
"Default": "WaitForStop"
},
"StartInstance": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:startInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"startCheckCount": 0
},
"Next": "WaitForStart",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"WaitForStart": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckStartStatus"
},
"CheckStartStatus": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
"startCheckCount": "{% $startCheckCount + 1 %}"
},
"Next": "EvaluateStartStatus",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"EvaluateStartStatus": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $ec2State = 'running' %}",
"Next": "WaitForRecovery"
},
{
"Condition": "{% $startCheckCount >= 20 %}",
"Next": "RecoveryFailed"
}
],
"Default": "WaitForStart"
},
"WaitForRecovery": {
"Type": "Wait",
"Seconds": 15,
"Next": "CheckRecoveryStatus"
},
"CheckRecoveryStatus": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ec2:describeInstanceStatus",
"Arguments": {
"InstanceIds": ["${InstanceId}"]
},
"Assign": {
"instanceStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].InstanceStatus.Status : 'unknown' %}",
"systemStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].SystemStatus.Status : 'unknown' %}",
"recoveryCheckCount": "{% $recoveryCheckCount + 1 %}"
},
"Next": "EvaluateRecoveryStatus",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "NotifyUnexpectedError",
"Output": {
"errorInfo": "{% $states.errorOutput %}"
}
}
]
},
"EvaluateRecoveryStatus": {
"Type": "Choice",
"Choices": [
{
"Condition": "{% $instanceStatus = 'ok' and $systemStatus = 'ok' %}",
"Next": "RecoverySuccess"
},
{
"Condition": "{% $recoveryCheckCount >= 40 %}",
"Next": "RecoveryFailed"
}
],
"Default": "WaitForRecovery"
},
"RecoverySuccess": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] EC2自動復旧成功",
"Message": "{% 'EC2 Stop/Start による自動復旧が完了しました。\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
},
"End": true
},
"RecoveryFailed": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] EC2自動復旧失敗",
"Message": "{% 'EC2 Stop/Start による自動復旧がタイムアウトしました。手動確認が必要です。\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
},
"End": true
},
"StopTimeout": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] EC2停止失敗",
"Message": "{% 'EC2 インスタンスの停止がタイムアウトしました（Force停止後も失敗）。手動確認が必要です。\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State %}"
},
"End": true
},
"AlreadyRecovered": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] 自然復旧確認",
"Message": "{% 'EC2 インスタンスは既に正常状態です（自然復旧）。\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
},
"End": true
},
"EscalateSystem": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] System障害",
"Message": "{% 'SystemStatus が impaired です。Stop/Start では復旧できない可能性があります。\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
},
"End": true
},
"EscalateTerminated": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] インスタンス終了",
"Message": "{% 'EC2 インスタンスが terminated または shutting-down 状態です。復旧不可能です。\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nState: ' & $ec2State %}"
},
"End": true
},
"NotifyUnexpectedError": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:sns:publish",
"Arguments": {
"TopicArn": "${SnsTopicArn}",
"Subject": "[TEST] 予期せぬエラー",
"Message": "{% 'Step Functions 実行中に予期せぬエラーが発生しました。\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nError: ' & $string($errorInfo) %}"
},
"End": true
}
}
}
# ============================================================
# FIS IAM Role
# ============================================================
FISRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ProjectName}-fis-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: fis.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: FISSSMPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ssm:SendCommand
- ssm:GetCommandInvocation
- ssm:ListCommands
- ssm:ListCommandInvocations
- ssm:CancelCommand
Resource: '*'
- Effect: Allow
Action:
- ec2:DescribeInstances
- ec2:DescribeInstanceStatus
Resource: '*'
# ============================================================
# FIS Experiment Template
# ============================================================
NicDisableExperiment:
Type: AWS::FIS::ExperimentTemplate
Properties:
Description: NIC disable to trigger StatusCheckFailed_Instance
Actions:
DisableNic:
ActionId: aws:ssm:send-command
Parameters:
documentArn: !Sub arn:aws:ssm:${AWS::Region}::document/AWS-RunShellScript
documentParameters: '{"commands":["sudo ip link set ens5 down"]}'
duration: PT1M
Targets:
Instances: TargetEC2
Targets:
TargetEC2:
ResourceType: aws:ec2:instance
ResourceArns:
- !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${TestEC2Instance}
SelectionMode: ALL
StopConditions:
- Source: none
RoleArn: !GetAtt FISRole.Arn
Tags:
Name: !Sub ${ProjectName}-nic-disable
Outputs:
EC2InstanceId:
Description: EC2 Instance ID
Value: !Ref TestEC2Instance
EC2InstancePublicDnsName:
Description: EC2 Instance Public DNS Name
Value: !GetAtt TestEC2Instance.PublicDnsName
SecurityGroupId:
Description: Security Group ID
Value: !Ref EC2SecurityGroup
SNSTopicArn:
Description: SNS Topic ARN
Value: !Ref AlertTopic
CloudWatchAlarmName:
Description: CloudWatch Alarm Name
Value: !Ref StatusCheckAlarm
EventBridgeRuleName:
Description: EventBridge Rule Name
Value: !Ref RecoveryTriggerRule
StepFunctionsStateMachineArn:
Description: Step Functions State Machine ARN
Value: !GetAtt RecoveryStateMachine.Arn
StepFunctionsLogGroupName:
Description: Step Functions Log Group Name
Value: !Ref StepFunctionsLogGroup
FISExperimentTemplateId:
Description: FIS Experiment Template ID
Value: !Ref NicDisableExperiment
Save the template above as ec2-recovery-test.yaml. Deploy it with the following commands, inject a failure with FIS, and finally delete the stack. The prerequisites before running are as follows.
- After deployment, a confirmation email will arrive at the notification address, so approve the SNS subscription
- The template does not specify a subnet, so run it in an account that has a default VPC
- The FIS fault injection is executed via SSM, so the instance must be able to reach SSM
- The FIS target is only the test instance created by this template. Because the operation brings down the NIC, run it in a test account
- Use the latest AWS CLI (the version used during verification was aws-cli/2.36.38)
# デプロイ
aws cloudformation create-stack \
--stack-name ec2-recovery-test \
--template-body file://ec2-recovery-test.yaml \
--parameters ParameterKey=NotificationEmail,ParameterValue=<通知先メールアドレス> \
--capabilities CAPABILITY_NAMED_IAM \
--deployment-config '{"Mode": "EXPRESS"}' \
--region ap-northeast-1
aws cloudformation wait stack-create-complete \
--stack-name ec2-recovery-test \
--region ap-northeast-1
# FIS で NIC を無効化して StatusCheckFailed_Instance を発生させる
aws fis start-experiment \
--experiment-template-id <FISExperimentTemplateId の出力値> \
--region ap-northeast-1
# EventBridge から Step Functions までの連携だけを試す場合
aws cloudwatch set-alarm-state \
--alarm-name ec2-recovery-test-StatusCheckFailed-Instance \
--state-value ALARM \
--state-reason "Manual test trigger" \
--region ap-northeast-1
# 撤去
aws cloudformation delete-stack \
--stack-name ec2-recovery-test \
--region ap-northeast-1
