developersIO
I tried automatically recovering EC2 instances that failed Status Checks using Step Functions without Lambda

I tried automatically recovering EC2 instances that failed Status Checks using Step Functions without Lambda

I created a mechanism to automate EC2 Stop/Start and recovery confirmation manually without using Lambda, for cases where EC2 Auto Recovery is not applicable or recovery does not occur. Using CloudWatch Alarms, EventBridge, Step Functions, and SNS, I reproduced a failure by disabling the NIC with FIS, and confirmed the behavior from detection through recovery completion.
AWS Step FunctionsAWS SDKAmazon EC2Amazon CloudWatchAWS
2026.09.05

This page has been translated by machine translation. View original

Introduction

EC2 Auto Recovery attempts to restore availability when system status checks fail due to underlying infrastructure issues. However, it does not cover cases where only instance status checks fail, and manual intervention such as Stop/Start is required when Auto Recovery cannot complete the recovery.

In this article, I automated this manual intervention using Step Functions triggered by a CloudWatch alarm. The Step Functions state machine handles EC2 API calls, waiting for stop/start completion, re-checking Status Check results, and conditional branching. I used FIS to disable a NIC and reproduce an Instance Status Check failure, then verified the recovery flow through the state machine definition and execution history.

What I Tested

I will explain in the following order: the overall architecture, the state machine definition, and the actual behavior during fault injection with FIS.

Architecture

The EC2 StatusCheckFailed_Instance metric is monitored at 1-minute intervals, and the alarm transitions to ALARM when the threshold is exceeded for 2 consecutive data points. An EventBridge rule that detects this state change triggers the state machine. The state machine then proceeds to stop and start the EC2 instance via API calls, and finally sends a notification via SNS.

Architecture where EC2 Status Check failure is detected and Step Functions performs Stop/Start and waits for Status Check recovery before sending an SNS notification

Inside the State Machine

There are two reasons why Lambda functions are not needed: AWS SDK integration and JSONata.

AWS SDK integration is a mechanism that allows you to call AWS APIs directly by simply writing an ARN in the Resource field of a Task state. By specifying arn:aws:states:::aws-sdk:ec2:describeInstances, you can retrieve EC2 instance information. The only calls this state machine makes are to retrieve instance state and Status Check results, stop/start the instance, and send notifications to SNS.

https://docs.aws.amazon.com/step-functions/latest/dg/supported-services-awssdk.html

The other feature, JSONata, is used for state retention and conditional evaluation. When QueryLanguage is set to JSONata, the way input/output is written for states changes. The two fields used are Arguments and Output. Conditions in Choice states are written as JSONata expressions in Condition, and the Variable and comparison operator fields used in JSONPath cannot be used. Counters such as loop counts can be carried over by updating variables with Assign.

https://docs.aws.amazon.com/step-functions/latest/dg/transforming-data.html

A stop-waiting loop can be written with three states: Wait, Task, and Choice. The following WaitForStop loop definition omits error Catch handling.

"WaitForStop": {
  "Type": "Wait",
  "Seconds": 15,
  "Next": "CheckStopStatus"
},
"CheckStopStatus": {
  "Type": "Task",
  "Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
  "Arguments": {
    "InstanceIds": ["${InstanceId}"]
  },
  "Assign": {
    "ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
    "stopCheckCount": "{% $stopCheckCount + 1 %}"
  },
  "Next": "EvaluateStopStatus"
},
"EvaluateStopStatus": {
  "Type": "Choice",
  "Choices": [
    {
      "Condition": "{% $ec2State = 'stopped' %}",
      "Next": "StartInstance"
    },
    {
      "Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = false %}",
      "Next": "ForceStopInstance"
    },
    {
      "Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = true %}",
      "Next": "StopTimeout"
    }
  ],
  "Default": "WaitForStop"
}

It waits 15 seconds, calls describeInstances again, and if the state is stopped, proceeds to start; otherwise, it returns to waiting. The maximum check count is 40, so the design waits approximately 10 minutes for a normal stop before switching to a Force stop. If the count reaches 40 even after a Force stop, it is treated as a timeout and a notification is sent. The maximum wait for running after startup is 20 times (approximately 5 minutes), and the maximum wait for Status Check recovery is 40 times (approximately 10 minutes).

Fault Injection and Recovery Results

To bring down the Status Check, I disabled a NIC via SSM from a FIS experiment template.

Actions:
  DisableNic:
    ActionId: aws:ssm:send-command
    Parameters:
      documentArn: !Sub arn:aws:ssm:${AWS::Region}::document/AWS-RunShellScript
      documentParameters: '{"commands":["sudo ip link set ens5 down"]}'
      duration: PT1M
    Targets:
      Instances: TargetEC2
StopConditions:
  - Source: none

The NIC name ens5 is the value confirmed on a t4g.nano running Amazon Linux 2023 (ARM64). Device names vary by environment, so please verify on the target instance before running.

After the NIC was brought down, the alarm transitioned to ALARM and EventBridge triggered the state machine. The event passed by EventBridge included the reason that the transition occurred because two data points at 1-minute intervals were accumulated.

{
  "detail-type": "CloudWatch Alarm State Change",
  "source": "aws.cloudwatch",
  "time": "2026-09-04T17:44:13Z",
  "detail": {
    "alarmName": "ec2-recovery-test-StatusCheckFailed-Instance",
    "state": {
      "value": "ALARM",
      "reason": "Threshold Crossed: 2 datapoints [1.0 (04/09/26 17:43:00), 1.0 (04/09/26 17:42:00)] were greater than or equal to the threshold (1.0).",
      "timestamp": "2026-09-04T17:44:13.159+0000"
    },
    "previousState": {
      "value": "OK",
      "reason": "Threshold Crossed: 1 datapoint [0.0 (04/09/26 17:34:00)] was not greater than or equal to the threshold (1.0).",
      "timestamp": "2026-09-04T17:35:13.157+0000"
    }
  }
}

The event time is in UTC, which corresponds to 02:44:13 JST.

EC2 console showing CloudWatch alarm transitioning to ALARM with two data points exceeding the threshold displayed

The FIS experiment itself completed in 73 seconds, but the disabled NIC remained as-is. Even at 02:44:13 when the state machine started, the Instance status was impaired and the Status Check was still failing.

I arranged the states retrieved by each step in the state machine execution history in chronological order. Times are in JST. state indicates the instance state from describeInstances. inst and sys are the Instance / System statuses from describeInstanceStatus.

02:44:13.731  GetEC2StatusCheck    state=running   inst=impaired      sys=ok
02:44:29.428  CheckStopStatus      state=stopping
02:44:44.743  CheckStopStatus      state=stopped
02:45:01.252  CheckStartStatus     state=running
02:45:16.453  CheckRecoveryStatus  state=running   inst=initializing  sys=initializing
02:45:31.693  CheckRecoveryStatus  state=running   inst=initializing  sys=initializing
02:45:46.903  CheckRecoveryStatus  state=running   inst=initializing  sys=initializing
02:46:02.113  CheckRecoveryStatus  state=running   inst=initializing  sys=initializing
02:46:17.327  CheckRecoveryStatus  state=running   inst=initializing  sys=initializing
02:46:32.541  CheckRecoveryStatus  state=running   inst=initializing  sys=ok
02:46:47.755  CheckRecoveryStatus  state=running   inst=initializing  sys=ok
02:47:03.062  CheckRecoveryStatus  state=running   inst=initializing  sys=ok
02:47:18.254  CheckRecoveryStatus  state=running   inst=ok            sys=ok

Since the initial GetEC2StatusCheck showed Instance as impaired and System as ok, EvaluateState branched to StopInstance. CheckStopStatus was satisfied in 2 iterations, CheckStartStatus in 1, and CheckRecoveryStatus in 9.

After startup, System returned to ok first at 02:46:32, and Instance became ok approximately 46 seconds later at 02:47:18. The approximately 137 seconds of recovery waiting corresponds to waiting for the Instance side initialization.

The execution status was SUCCEEDED. Of the 139 total events in the execution history, events that entered a state were 17 Task, 13 Choice, 12 Wait, and 1 Pass, for a total of 43. In this execution, the ForceStopInstance, StopTimeout, and RecoveryFailed paths were not taken. AlreadyRecovered, EscalateSystem, EscalateTerminated, and NotifyUnexpectedError were also not taken.

Interval Start → End (JST) Duration
FIS experiment (NIC disable) 02:41:22.592 → 02:42:35.145 Approximately 73 seconds
Alarm detection 02:41:22.592 → 02:44:13.159 Approximately 171 seconds
Step Functions execution 02:44:13.337 → 02:47:18.507 185 seconds
of which: waiting for Stop completion 02:44:14.195 → 02:44:44.743 Approximately 31 seconds
of which: waiting for running after Start 02:44:45.804 → 02:45:01.252 Approximately 16 seconds
of which: waiting for Status Check recovery 02:45:01.252 → 02:47:18.254 Approximately 137 seconds
From fault injection to recovery completion 02:41:22.592 → 02:47:18.507 Approximately 6 minutes

The time from fault injection to recovery completion was approximately 6 minutes.

Monthly Maintenance Cost

Maintenance costs were calculated using Tokyo Region unit prices.

Item Unit Price (Asia Pacific - Tokyo) Free Tier
Step Functions Standard state transitions 0.000025 USD / transition 4,000 transitions per month
CloudWatch metric alarm (standard resolution) 0.10 USD / alarm metric per month 10 alarm metrics per month
CloudWatch Logs Vended Logs ingestion (Standard) 0.76 USD / GB (first 10 TB) 5 GB per month
SNS Email notifications 2.00 USD / 100,000 messages 1,000 messages per month

Unit prices are from the pricing pages for Step Functions, CloudWatch, and Amazon SNS.

Not included in the estimate are the last two rows of the table: state machine log output, SNS notifications, FIS experiment execution costs, and the EC2 instance itself.

The calculation assumes recovery occurs once per month at list prices. Since the number of state transitions counted in the execution results is 43 per execution, the cost is 43 × 0.000025 USD = 0.001075 USD. The alarm costs 0.10 USD per alarm metric. The total comes to approximately 0.101 USD per month.

Since both state transitions and alarms each have free tiers, this scale of usage will fit within the free tier if the allowance remains.

Summary

I was able to automatically recover an EC2 instance with a failed Status Check using Step Functions without Lambda.

By not using Lambda functions, it eliminates the need to keep up with Lambda runtime updates and manage deployment and configuration per function. Additionally, since not only the execution results of each state but also waits and branches are recorded in the Step Functions execution history, it becomes easier to check the progress of recovery and identify where failures occurred.

If you have been manually handling EC2 instance restarts, please consider this as one option for automating that process.

References

The entire configuration for this setup is compiled into a CloudFormation template.

CloudFormation template full text
AWSTemplateFormatVersion: '2010-09-09'
Description: 'EC2 Stop/Start Auto Recovery Test Environment'

Parameters:
  ProjectName:
    Type: String
    Default: ec2-recovery-test
  NotificationEmail:
    Type: String
    Default: alerts@example.com
  InstanceType:
    Type: String
    Default: t4g.nano
  LogRetentionDays:
    Type: Number
    Default: 7
  LatestAmiId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64

Resources:
  # ============================================================
  # EC2 Security Group (uses default VPC)
  # ============================================================
  EC2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub ${ProjectName} - Outbound only for SSM
      Tags:
        - Key: Name
          Value: !Sub ${ProjectName}-sg

  # ============================================================
  # EC2 IAM Role and Instance Profile
  # ============================================================
  EC2Role:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ${ProjectName}-ec2-role
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

  EC2InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      InstanceProfileName: !Sub ${ProjectName}-ec2-profile
      Roles:
        - !Ref EC2Role

  # ============================================================
  # EC2 Instance
  # ============================================================
  TestEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
      IamInstanceProfile: !Ref EC2InstanceProfile
      SecurityGroupIds:
        - !Ref EC2SecurityGroup
      Tags:
        - Key: Name
          Value: !Sub ${ProjectName}-test-ec2

  # ============================================================
  # SNS Topic and Subscription
  # ============================================================
  AlertTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: !Sub ${ProjectName}-alerts

  EmailSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref AlertTopic
      Protocol: email
      Endpoint: !Ref NotificationEmail

  # ============================================================
  # CloudWatch Alarm
  # ============================================================
  StatusCheckAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub ${ProjectName}-StatusCheckFailed-Instance
      AlarmDescription: Alarm when StatusCheckFailed_Instance is >= 1
      Namespace: AWS/EC2
      MetricName: StatusCheckFailed_Instance
      Dimensions:
        - Name: InstanceId
          Value: !Ref TestEC2Instance
      Statistic: Maximum
      Period: 60
      EvaluationPeriods: 2
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: missing

  # ============================================================
  # EventBridge IAM Role
  # ============================================================
  EventBridgeRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ${ProjectName}-eventbridge-role
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: InvokeStepFunctions
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: states:StartExecution
                Resource: !GetAtt RecoveryStateMachine.Arn

  # ============================================================
  # EventBridge Rule
  # ============================================================
  RecoveryTriggerRule:
    Type: AWS::Events::Rule
    Properties:
      Name: !Sub ${ProjectName}-recovery-trigger
      Description: Trigger Step Functions when StatusCheckFailed alarm goes to ALARM
      EventPattern:
        source:
          - aws.cloudwatch
        detail-type:
          - CloudWatch Alarm State Change
        detail:
          alarmName:
            - !Ref StatusCheckAlarm
          state:
            value:
              - ALARM
      State: ENABLED
      Targets:
        - Arn: !GetAtt RecoveryStateMachine.Arn
          RoleArn: !GetAtt EventBridgeRole.Arn
          Id: RecoveryStateMachineTarget

  # ============================================================
  # Step Functions IAM Role
  # ============================================================
  StepFunctionsRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ${ProjectName}-stepfunctions-role
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: states.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: EC2Operations
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - ec2:StopInstances
                  - ec2:StartInstances
                Resource: !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${TestEC2Instance}
              - Effect: Allow
                Action:
                  - ec2:DescribeInstances
                  - ec2:DescribeInstanceStatus
                Resource: '*'
        - PolicyName: SNSPublish
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: sns:Publish
                Resource: !Ref AlertTopic
        - PolicyName: CloudWatchLogs
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogDelivery
                  - logs:GetLogDelivery
                  - logs:UpdateLogDelivery
                  - logs:DeleteLogDelivery
                  - logs:ListLogDeliveries
                  - logs:PutLogEvents
                  - logs:PutResourcePolicy
                  - logs:DescribeResourcePolicies
                  - logs:DescribeLogGroups
                Resource: '*'

  # ============================================================
  # Step Functions Log Group
  # ============================================================
  StepFunctionsLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/vendedlogs/states/${ProjectName}-ec2-recovery
      RetentionInDays: !Ref LogRetentionDays

  # ============================================================
  # Step Functions State Machine
  # ============================================================
  RecoveryStateMachine:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      StateMachineName: !Sub ${ProjectName}-ec2-recovery
      StateMachineType: STANDARD
      RoleArn: !GetAtt StepFunctionsRole.Arn
      LoggingConfiguration:
        Level: ALL
        IncludeExecutionData: true
        Destinations:
          - CloudWatchLogsLogGroup:
              LogGroupArn: !GetAtt StepFunctionsLogGroup.Arn
      DefinitionSubstitutions:
        InstanceId: !Ref TestEC2Instance
        SnsTopicArn: !Ref AlertTopic
      DefinitionString: |
        {
          "Comment": "EC2 Stop/Start Auto Recovery Workflow",
          "QueryLanguage": "JSONata",
          "TimeoutSeconds": 2400,
          "StartAt": "InitializeVariables",
          "States": {
            "InitializeVariables": {
              "Type": "Pass",
              "Assign": {
                "stopCheckCount": 0,
                "startCheckCount": 0,
                "recoveryCheckCount": 0,
                "forceStopAttempted": false,
                "errorInfo": null
              },
              "Next": "GetEC2State"
            },
            "GetEC2State": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
              "Arguments": {
                "InstanceIds": ["${InstanceId}"]
              },
              "Assign": {
                "ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}"
              },
              "Next": "GetEC2StatusCheck",
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "Next": "NotifyUnexpectedError",
                  "Output": {
                    "errorInfo": "{% $states.errorOutput %}"
                  }
                }
              ]
            },
            "GetEC2StatusCheck": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:ec2:describeInstanceStatus",
              "Arguments": {
                "InstanceIds": ["${InstanceId}"]
              },
              "Assign": {
                "instanceStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].InstanceStatus.Status : 'unknown' %}",
                "systemStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].SystemStatus.Status : 'unknown' %}"
              },
              "Next": "EvaluateState",
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "Next": "NotifyUnexpectedError",
                  "Output": {
                    "errorInfo": "{% $states.errorOutput %}"
                  }
                }
              ]
            },
            "EvaluateState": {
              "Type": "Choice",
              "Choices": [
                {
                  "Condition": "{% $systemStatus = 'impaired' %}",
                  "Next": "EscalateSystem"
                },
                {
                  "Condition": "{% $instanceStatus = 'ok' and $systemStatus = 'ok' %}",
                  "Next": "AlreadyRecovered"
                },
                {
                  "Condition": "{% $ec2State = 'terminated' or $ec2State = 'shutting-down' %}",
                  "Next": "EscalateTerminated"
                },
                {
                  "Condition": "{% $ec2State = 'stopping' %}",
                  "Next": "WaitForStop"
                },
                {
                  "Condition": "{% $ec2State = 'stopped' %}",
                  "Next": "StartInstance"
                },
                {
                  "Condition": "{% $ec2State = 'pending' %}",
                  "Next": "WaitForStart"
                },
                {
                  "Condition": "{% $ec2State = 'running' and $instanceStatus = 'impaired' and $systemStatus = 'ok' %}",
                  "Next": "StopInstance"
                }
              ],
              "Default": "StopInstance"
            },
            "StopInstance": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:ec2:stopInstances",
              "Arguments": {
                "InstanceIds": ["${InstanceId}"],
                "Force": false
              },
              "Assign": {
                "stopCheckCount": 0
              },
              "Next": "WaitForStop",
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "Next": "NotifyUnexpectedError",
                  "Output": {
                    "errorInfo": "{% $states.errorOutput %}"
                  }
                }
              ]
            },
            "ForceStopInstance": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:ec2:stopInstances",
              "Arguments": {
                "InstanceIds": ["${InstanceId}"],
                "Force": true
              },
              "Assign": {
                "forceStopAttempted": true,
                "stopCheckCount": 0
              },
              "Next": "WaitForStop",
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "Next": "NotifyUnexpectedError",
                  "Output": {
                    "errorInfo": "{% $states.errorOutput %}"
                  }
                }
              ]
            },
            "WaitForStop": {
              "Type": "Wait",
              "Seconds": 15,
              "Next": "CheckStopStatus"
            },
            "CheckStopStatus": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
              "Arguments": {
                "InstanceIds": ["${InstanceId}"]
              },
              "Assign": {
                "ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
                "stopCheckCount": "{% $stopCheckCount + 1 %}"
              },
              "Next": "EvaluateStopStatus",
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "Next": "NotifyUnexpectedError",
                  "Output": {
                    "errorInfo": "{% $states.errorOutput %}"
                  }
                }
              ]
            },
            "EvaluateStopStatus": {
              "Type": "Choice",
              "Choices": [
                {
                  "Condition": "{% $ec2State = 'stopped' %}",
                  "Next": "StartInstance"
                },
                {
                  "Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = false %}",
                  "Next": "ForceStopInstance"
                },
                {
                  "Condition": "{% $stopCheckCount >= 40 and $forceStopAttempted = true %}",
                  "Next": "StopTimeout"
                }
              ],
              "Default": "WaitForStop"
            },
            "StartInstance": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:ec2:startInstances",
              "Arguments": {
                "InstanceIds": ["${InstanceId}"]
              },
              "Assign": {
                "startCheckCount": 0
              },
              "Next": "WaitForStart",
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "Next": "NotifyUnexpectedError",
                  "Output": {
                    "errorInfo": "{% $states.errorOutput %}"
                  }
                }
              ]
            },
            "WaitForStart": {
              "Type": "Wait",
              "Seconds": 15,
              "Next": "CheckStartStatus"
            },
            "CheckStartStatus": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:ec2:describeInstances",
              "Arguments": {
                "InstanceIds": ["${InstanceId}"]
              },
              "Assign": {
                "ec2State": "{% $states.result.Reservations[0].Instances[0].State.Name %}",
                "startCheckCount": "{% $startCheckCount + 1 %}"
              },
              "Next": "EvaluateStartStatus",
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "Next": "NotifyUnexpectedError",
                  "Output": {
                    "errorInfo": "{% $states.errorOutput %}"
                  }
                }
              ]
            },
            "EvaluateStartStatus": {
              "Type": "Choice",
              "Choices": [
                {
                  "Condition": "{% $ec2State = 'running' %}",
                  "Next": "WaitForRecovery"
                },
                {
                  "Condition": "{% $startCheckCount >= 20 %}",
                  "Next": "RecoveryFailed"
                }
              ],
              "Default": "WaitForStart"
            },
            "WaitForRecovery": {
              "Type": "Wait",
              "Seconds": 15,
              "Next": "CheckRecoveryStatus"
            },
            "CheckRecoveryStatus": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:ec2:describeInstanceStatus",
              "Arguments": {
                "InstanceIds": ["${InstanceId}"]
              },
              "Assign": {
                "instanceStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].InstanceStatus.Status : 'unknown' %}",
                "systemStatus": "{% $count($states.result.InstanceStatuses) > 0 ? $states.result.InstanceStatuses[0].SystemStatus.Status : 'unknown' %}",
                "recoveryCheckCount": "{% $recoveryCheckCount + 1 %}"
              },
              "Next": "EvaluateRecoveryStatus",
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "Next": "NotifyUnexpectedError",
                  "Output": {
                    "errorInfo": "{% $states.errorOutput %}"
                  }
                }
              ]
            },
            "EvaluateRecoveryStatus": {
              "Type": "Choice",
              "Choices": [
                {
                  "Condition": "{% $instanceStatus = 'ok' and $systemStatus = 'ok' %}",
                  "Next": "RecoverySuccess"
                },
                {
                  "Condition": "{% $recoveryCheckCount >= 40 %}",
                  "Next": "RecoveryFailed"
                }
              ],
              "Default": "WaitForRecovery"
            },
            "RecoverySuccess": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:sns:publish",
              "Arguments": {
                "TopicArn": "${SnsTopicArn}",
                "Subject": "[TEST] EC2 Auto Recovery Successful",
                "Message": "{% 'EC2 Stop/Start auto recovery has completed.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
              },
              "End": true
            },
            "RecoveryFailed": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:sns:publish",
              "Arguments": {
                "TopicArn": "${SnsTopicArn}",
                "Subject": "[TEST] EC2 Auto Recovery Failed",
                "Message": "{% 'EC2 Stop/Start auto recovery timed out. Manual intervention required.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
              },
              "End": true
            },
            "StopTimeout": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:sns:publish",
              "Arguments": {
                "TopicArn": "${SnsTopicArn}",
                "Subject": "[TEST] EC2 Stop Failed",
                "Message": "{% 'EC2 instance stop timed out (also failed after Force stop). Manual intervention required.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nFinal State: ' & $ec2State %}"
              },
              "End": true
            },
            "AlreadyRecovered": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:sns:publish",
              "Arguments": {
                "TopicArn": "${SnsTopicArn}",
                "Subject": "[TEST] Natural Recovery Confirmed",
                "Message": "{% 'EC2 instance is already in a healthy state (natural recovery).\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
              },
              "End": true
            },
            "EscalateSystem": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:sns:publish",
              "Arguments": {
                "TopicArn": "${SnsTopicArn}",
                "Subject": "[TEST] System Failure",
                "Message": "{% 'SystemStatus is impaired. Recovery via Stop/Start may not be possible.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nInstance Status: ' & $instanceStatus & '\\nSystem Status: ' & $systemStatus %}"
              },
              "End": true
            },
            "EscalateTerminated": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:sns:publish",
              "Arguments": {
                "TopicArn": "${SnsTopicArn}",
                "Subject": "[TEST] Instance Terminated",
                "Message": "{% 'EC2 instance is in terminated or shutting-down state. Recovery is not possible.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nState: ' & $ec2State %}"
              },
              "End": true
            },
            "NotifyUnexpectedError": {
              "Type": "Task",
              "Resource": "arn:aws:states:::aws-sdk:sns:publish",
              "Arguments": {
                "TopicArn": "${SnsTopicArn}",
                "Subject": "[TEST] Unexpected Error",
                "Message": "{% 'An unexpected error occurred during Step Functions execution.\\n\\nInstanceId: ${InstanceId}\\nExecutionId: ' & $states.context.Execution.Id & '\\nError: ' & $string($errorInfo) %}"
              },
              "End": true
            }
          }
        }

  # ============================================================
  # FIS IAM Role
  # ============================================================
  FISRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ${ProjectName}-fis-role
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: fis.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: FISSSMPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - ssm:SendCommand
                  - ssm:GetCommandInvocation
                  - ssm:ListCommands
                  - ssm:ListCommandInvocations
                  - ssm:CancelCommand
                Resource: '*'
              - Effect: Allow
                Action:
                  - ec2:DescribeInstances
                  - ec2:DescribeInstanceStatus
                Resource: '*'

  # ============================================================
  # FIS Experiment Template
  # ============================================================
  NicDisableExperiment:
    Type: AWS::FIS::ExperimentTemplate
    Properties:
      Description: NIC disable to trigger StatusCheckFailed_Instance
      Actions:
        DisableNic:
          ActionId: aws:ssm:send-command
          Parameters:
            documentArn: !Sub arn:aws:ssm:${AWS::Region}::document/AWS-RunShellScript
            documentParameters: '{"commands":["sudo ip link set ens5 down"]}'
            duration: PT1M
          Targets:
            Instances: TargetEC2
      Targets:
        TargetEC2:
          ResourceType: aws:ec2:instance
          ResourceArns:
            - !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${TestEC2Instance}
          SelectionMode: ALL
      StopConditions:
        - Source: none
      RoleArn: !GetAtt FISRole.Arn
      Tags:
        Name: !Sub ${ProjectName}-nic-disable

Outputs:
  EC2InstanceId:
    Description: EC2 Instance ID
    Value: !Ref TestEC2Instance
  EC2InstancePublicDnsName:
    Description: EC2 Instance Public DNS Name
    Value: !GetAtt TestEC2Instance.PublicDnsName
  SecurityGroupId:
    Description: Security Group ID
    Value: !Ref EC2SecurityGroup
  SNSTopicArn:
    Description: SNS Topic ARN
    Value: !Ref AlertTopic
  CloudWatchAlarmName:
    Description: CloudWatch Alarm Name
    Value: !Ref StatusCheckAlarm
  EventBridgeRuleName:
    Description: EventBridge Rule Name
    Value: !Ref RecoveryTriggerRule
  StepFunctionsStateMachineArn:
    Description: Step Functions State Machine ARN
    Value: !GetAtt RecoveryStateMachine.Arn
  StepFunctionsLogGroupName:
    Description: Step Functions Log Group Name
    Value: !Ref StepFunctionsLogGroup
  FISExperimentTemplateId:
    Description: FIS Experiment Template ID
    Value: !Ref NicDisableExperiment

Save the above template as ec2-recovery-test.yaml. Deploy it with the following commands, inject a fault with FIS, and finally delete the stack. There are 4 prerequisites before execution.

  • After deployment, a confirmation email will be sent to the notification address, so approve the SNS subscription
  • Since the template does not specify a subnet, run it on an account that has a default VPC
  • FIS fault injection is executed via SSM, so the instance needs to be able to reach SSM
  • The FIS target is only the test instance created by this template. Since this is an operation that brings down the NIC, run it on a test account
  • Use the latest version of AWS CLI (version at time of verification: aws-cli/2.36.38)
# Deploy
aws cloudformation create-stack \
  --stack-name ec2-recovery-test \
  --template-body file://ec2-recovery-test.yaml \
  --parameters ParameterKey=NotificationEmail,ParameterValue=<notification email address> \
  --capabilities CAPABILITY_NAMED_IAM \
  --deployment-config '{"mode": "EXPRESS"}' \
  --region ap-northeast-1

aws cloudformation wait stack-create-complete \
  --stack-name ec2-recovery-test \
  --region ap-northeast-1

# Disable NIC with FIS to trigger StatusCheckFailed_Instance
aws fis start-experiment \
  --experiment-template-id <FISExperimentTemplateId output value> \
  --region ap-northeast-1

# To test only the integration from EventBridge to Step Functions
aws cloudwatch set-alarm-state \
  --alarm-name ec2-recovery-test-StatusCheckFailed-Instance \
  --state-value ALARM \
  --state-reason "Manual test trigger" \
  --region ap-northeast-1

# Teardown
aws cloudformation delete-stack \
  --stack-name ec2-recovery-test \
  --region ap-northeast-1

Share this article

AWSのお困り事はクラスメソッドへ