State Manager の関連付け実行時、ステータスが “Failed” 、詳細ステータスが “LimitExceeded” になっている際の対処法

2024.05.01

困っていること

AWS Systems Manager の一機能である State Manager を利用しています。
State Manager の関連付けを実行した際に、一部の対象マネージドノードでは、ステータスが "Failed" 、詳細ステータスが "LimitExceeded" になっている為、実行できませんでした。原因と対象法を教えてください。

どう対応すればいいの?

原因

詳細ステータスが "LimitExceeded" の場合、State Manager が以下の上限に抵触している可能性が考えられます。

  • 1 マネージドノードを対象とする関連付けの最大数 = 20

ステートマネージャー 単一のマネージドノードをターゲットとする関連付けの最大数 20

対処法

1 マネージドノードを対象とする関連付けの最大数の数を 20 より少なくすることで、事象が解消するかご確認ください。
一例として、内容が重複していそうな関連付け( AWS-ConfigureAWSPackage ドキュメントでの AWSCodeDeployAgent パッケージ導入のものなど )を整理することが可能かどうかなどをご確認ください。

関連付け数と情報の確認方法

AWS CLI コマンドを利用すると各種情報の取得が可能です。


対象マネージドノードに関連付けされている数の確認

$ aws ssm describe-instance-associations-status --instance-id < インスタンス ID >  | jq '.InstanceAssociationStatusInfos | length'

※ 実行例

$ aws ssm describe-instance-associations-status --instance-id i-xxxxxxx1234567  | jq '.InstanceAssociationStatusInfos | length'
26


対象マネージドノードの関連付けされている詳細情報の確認

$ aws ssm describe-instance-associations-status --instance-id < インスタンス ID >

※ 実行例

$ aws ssm describe-instance-associations-status --instance-id i-xxxxxxx1234567
{
    "InstanceAssociationStatusInfos": [
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-GatherSoftwareInventory",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2024-03-26T04:16:37.068000+09:00",
            "Status": "Failed",
            "DetailedStatus": "LimitExceeded",
            "ExecutionSummary": "1 out of 1 plugin processed, 1 success, 0 failed, 0 timedout, 0 skipped. ",
            "AssociationName": "InspectorInventoryCollection-do-not-delete"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-GatherSoftwareInventory",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "Status": "Failed",
            "DetailedStatus": "LimitExceeded",
            "AssociationName": "prd-Inventory"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:02.578000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AmazonInspector2-ConfigureInspectorSsmPlugin",
            "DocumentVersion": "2",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2024-03-30T07:01:48.218000+09:00",
            "Status": "Skipped",
            "DetailedStatus": "InvalidPlatform",
            "AssociationName": "InspectorDistributor-do-not-delete"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:02.822000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:33.151000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:02.646000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-UpdateSSMAgent",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2024-03-21T17:53:25.444000+09:00",
            "Status": "Success",
            "DetailedStatus": "Success",
            "AssociationName": "SystemAssociationForSsmAgentUpdate"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:33.359000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:33.450000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:06.332000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:33:56.070000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:11.119000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:33:57.237000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:33:54.856000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:09.964000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:33.400000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:33:59.347000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ubuntu_os_prd"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AmazonInspector2-InvokeInspectorSsmPlugin",
            "DocumentVersion": "6",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2024-03-30T13:01:19.641000+09:00",
            "Status": "Skipped",
            "DetailedStatus": "InvalidPlatform",
            "AssociationName": "InvokeInspectorSsmPlugin-do-not-delete"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AmazonInspector2-InvokeInspectorSsmPluginLinux",
            "DocumentVersion": "3",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2024-03-30T15:48:34.899000+09:00",
            "Status": "Success",
            "DetailedStatus": "Success",
            "AssociationName": "InvokeInspectorLinuxSsmPlugin-do-not-delete"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:33:55.235000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:09.938000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:14.141000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:05.972000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AWS-ConfigureAWSPackage",
            "DocumentVersion": "1",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2023-03-01T11:34:12.791000+09:00",
            "Status": "Failed",
            "DetailedStatus": "Failed",
            "AssociationName": "CodeDeployDG-ec2"
        },
        {
            "AssociationId": "12345678-xxxx-aaaa-bbbb-xxxxaaaabbbb",
            "Name": "AmazonInspector2-ConfigureInspectorSsmPluginLinux",
            "DocumentVersion": "2",
            "AssociationVersion": "1",
            "InstanceId": "i-xxxxxxx1234567",
            "ExecutionDate": "2024-03-30T14:48:35.503000+09:00",
            "Status": "Success",
            "DetailedStatus": "Success",
            "AssociationName": "InspectorLinuxDistributor-do-not-delete"
        }
    ]
}

参考資料