![[Trivia] A story about DeleteCertificate getting stuck in an infinite loop because the resource deletion order in ECS Express Mode was wrong](https://images.ctfassets.net/ct0aopd36mqt/wp-thumbnail-6d4fdf3578638cab4e6e4655b3d692aa/15a803840d72679e6593006354d5766b/amazon-elastic-container-service?w=3840&fm=webp)
[Trivia] A story about DeleteCertificate getting stuck in an infinite loop because the resource deletion order in ECS Express Mode was wrong
This page has been translated by machine translation. View original
Introduction
Hello everyone, I'm Akaike.
The other day, while browsing CloudTrail on a verification account, I noticed that DeleteCertificate events were being recorded continuously every few seconds.
Since I had no recollection of trying to delete a certificate myself, I decided to investigate the cause.
This article summarizes the investigation process, the root cause, and the resolution.
Summary First
Let me summarize the conclusion upfront.
- Situation
acm:DeleteCertificatewas being recorded in CloudTrail every few seconds withAccessDenied- The caller was Amazon ECS (Express Gateway Services) managed role
ecs-express-infrastructure-role
- Root Cause
- The target ACM certificate no longer existed, so the tag condition in the managed policy could not be evaluated, resulting in
AccessDenied - ECS could not determine whether the deletion had succeeded, so it kept retrying the same deletion operation
- The target ACM certificate no longer existed, so the tag condition in the managed policy could not be evaluated, resulting in
- Resolution
- By explicitly granting delete permission for the certificate in question, it started returning
ResourceNotFoundException, which broke the retry loop
- By explicitly granting delete permission for the certificate in question, it started returning
The flow of the process is illustrated in the diagram below.
What Happened
DeleteCertificate Keeps Piling Up in CloudTrail
First, let's check the CloudTrail event history. As shown below, DeleteCertificate was being recorded in large numbers at short intervals.
The contents of the event record were as follows.
{
"eventVersion": "1.11",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAXXXXXXXXXXXXXXXXX:ECSGateway",
"arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/ecs-express-infrastructure-role/ECSGateway",
"accountId": "XXXXXXXXXXXX",
"accessKeyId": "ASIAXXXXXXXXXXXXXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAXXXXXXXXXXXXXXXXX",
"arn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs-express-infrastructure-role",
"accountId": "XXXXXXXXXXXX",
"userName": "ecs-express-infrastructure-role"
},
"attributes": {
"creationDate": "2026-08-13T02:58:19Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "ecs.amazonaws.com"
},
"eventTime": "2026-08-13T02:58:19Z",
"eventSource": "acm.amazonaws.com",
"eventName": "DeleteCertificate",
"awsRegion": "ap-northeast-1",
"sourceIPAddress": "ecs.amazonaws.com",
"userAgent": "ecs.amazonaws.com",
"errorCode": "AccessDenied",
"errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/ecs-express-infrastructure-role/ECSGateway is not authorized to perform: acm:DeleteCertificate on resource: arn:aws:acm:ap-northeast-1:XXXXXXXXXXXX:certificate/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX because no identity-based policy allows the acm:DeleteCertificate action",
"requestParameters": null,
"responseElements": null,
"requestID": "5fafc5db-e2ad-4e94-b9ca-7b9fe3a15097",
"eventID": "d812a591-0b6b-4517-b55b-9a14a29191c6",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXXXXXXXXXX",
"eventCategory": "Management"
}
About the Cause
The Calling Role and Policy
Several important points can be read from the event record above.
userIdentity.arnisecs-express-infrastructure-role/ECSGateway, meaning the caller is not a user but an ECS managed roleinvokedByisecs.amazonaws.com, indicating this is being called automatically by the ECS serviceerrorCodeisAccessDenied, meaning the operation is failing becauseacm:DeleteCertificateis not permitted
ecs-express-infrastructure-role is the role used by ECS Express Gateway Services (a mechanism where ECS managed constructs and manages surrounding resources such as load balancers and ACM certificates). Let's check the contents of the managed policy AmazonECSInfrastructureRoleforExpressGatewayServices attached to this role.
Permissions related to certificates are defined in the CertificateOperations statement of the following policy.
Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ServiceLinkedRoleCreateOperations",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "*",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": [
"ecs.application-autoscaling.amazonaws.com",
"elasticloadbalancing.amazonaws.com"
]
}
}
},
{
"Sid": "ELBOperations",
"Effect": "Allow",
"Action": [
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:ModifyRule",
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:RemoveListenerCertificates",
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:DeleteRule",
"elasticloadbalancing:DeleteListener"
],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*",
"arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*/*",
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceTag/AmazonECSManaged": "true"
}
}
},
{
"Sid": "TagOnCreateELBResources",
"Effect": "Allow",
"Action": "elasticloadbalancing:AddTags",
"Resource": [
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*",
"arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*/*",
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
],
"Condition": {
"StringEquals": {
"elasticloadbalancing:CreateAction": [
"CreateLoadBalancer",
"CreateListener",
"CreateRule",
"CreateTargetGroup"
]
}
}
},
{
"Sid": "BlanketAllowCreateSecurityGroupsInVPCs",
"Effect": "Allow",
"Action": "ec2:CreateSecurityGroup",
"Resource": "arn:aws:ec2:*:*:vpc/*"
},
{
"Sid": "CreateSecurityGroupResourcesWithTags",
"Effect": "Allow",
"Action": [
"ec2:CreateSecurityGroup",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress"
],
"Resource": [
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:security-group-rule/*",
"arn:aws:ec2:*:*:vpc/*"
],
"Condition": {
"StringEquals": {
"aws:RequestTag/AmazonECSManaged": "true"
}
}
},
{
"Sid": "ModifySecurityGroupOperations",
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:DeleteSecurityGroup",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress"
],
"Resource": [
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:vpc/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceTag/AmazonECSManaged": "true"
}
}
},
{
"Sid": "TagOnCreateEC2Resources",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": [
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:security-group-rule/*"
],
"Condition": {
"StringEquals": {
"ec2:CreateAction": [
"CreateSecurityGroup",
"AuthorizeSecurityGroupIngress",
"AuthorizeSecurityGroupEgress"
]
}
}
},
{
"Sid": "CertificateOperations",
"Effect": "Allow",
"Action": [
"acm:RequestCertificate",
"acm:AddTagsToCertificate",
"acm:DeleteCertificate",
"acm:DescribeCertificate"
],
"Resource": [
"arn:aws:acm:*:*:certificate/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceTag/AmazonECSManaged": "true"
}
}
},
{
"Sid": "ApplicationAutoscalingCreateOperations",
"Effect": "Allow",
"Action": [
"application-autoscaling:RegisterScalableTarget",
"application-autoscaling:TagResource",
"application-autoscaling:DeregisterScalableTarget"
],
"Resource": [
"arn:aws:application-autoscaling:*:*:scalable-target/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceTag/AmazonECSManaged": "true"
}
}
},
{
"Sid": "ApplicationAutoscalingPolicyOperations",
"Effect": "Allow",
"Action": [
"application-autoscaling:PutScalingPolicy",
"application-autoscaling:DeleteScalingPolicy"
],
"Resource": [
"arn:aws:application-autoscaling:*:*:scalable-target/*"
],
"Condition": {
"StringEquals": {
"application-autoscaling:service-namespace": "ecs"
}
}
},
{
"Sid": "ApplicationAutoscalingReadOperations",
"Effect": "Allow",
"Action": [
"application-autoscaling:DescribeScalableTargets",
"application-autoscaling:DescribeScalingPolicies",
"application-autoscaling:DescribeScalingActivities"
],
"Resource": [
"arn:aws:application-autoscaling:*:*:scalable-target/*"
]
},
{
"Sid": "CloudWatchAlarmCreateOperations",
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricAlarm",
"cloudwatch:TagResource"
],
"Resource": [
"arn:aws:cloudwatch:*:*:alarm:*"
],
"Condition": {
"StringEquals": {
"aws:RequestTag/AmazonECSManaged": "true"
}
}
},
{
"Sid": "CloudWatchAlarmOperations",
"Effect": "Allow",
"Action": [
"cloudwatch:DeleteAlarms",
"cloudwatch:DescribeAlarms"
],
"Resource": [
"arn:aws:cloudwatch:*:*:alarm:*"
],
"Condition": {
"StringEquals": {
"aws:ResourceTag/AmazonECSManaged": "true"
}
}
},
{
"Sid": "ELBReadOperations",
"Effect": "Allow",
"Action": [
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeRules"
],
"Resource": "*"
},
{
"Sid": "VPCReadOperations",
"Effect": "Allow",
"Action": [
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeRouteTables",
"ec2:DescribeVpcs"
],
"Resource": "*"
},
{
"Sid": "CloudWatchLogsCreateOperations",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:TagResource"
],
"Resource": "arn:aws:logs:*:*:log-group:*",
"Condition": {
"StringEquals": {
"aws:RequestTag/AmazonECSManaged": "true"
}
}
},
{
"Sid": "CloudWatchLogsReadOperations",
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups"
],
"Resource": "*"
}
]
}
Why Does It Loop on AccessDenied?
The key point is the Condition in the CertificateOperations statement. While acm:DeleteCertificate itself is allowed, there is a tag condition (StringEquals) that restricts it to resources where aws:ResourceTag/AmazonECSManaged is true.
{
"Sid": "CertificateOperations",
"Effect": "Allow",
"Action": [
"acm:RequestCertificate",
"acm:AddTagsToCertificate",
"acm:DeleteCertificate",
"acm:DescribeCertificate"
],
"Resource": [
"arn:aws:acm:*:*:certificate/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceTag/AmazonECSManaged": "true"
}
}
}
Upon investigation, the target certificate had already been deleted and no longer existed in ACM.
When a resource does not exist, its tag (aws:ResourceTag/AmazonECSManaged) cannot be evaluated, so the StringEquals condition is not satisfied, the Allow is not applied, and AccessDenied is returned.
As a result, AccessDenied occurs at the IAM permission evaluation stage, so ECS cannot determine that the ACM deletion has completed, and it keeps retrying the same deletion operation.
DeleteCertificate Executed 4.08 Million Times
Counting how frequently it was being called, it was re-executed every 5 seconds, at a rate of 12 times per minute (60 seconds ÷ 5 seconds).
And since I had been testing ECS Express Mode as described in the blog post below, this had likely been occurring continuously since around that time.
From December 20, 2025 to the day this issue was resolved (August 13, 2026) is 236 days. By simple calculation:
236 days × 24 hours × 60 minutes × 12 times/minute = 4,078,080 times
That means approximately 4.08 million unnecessary DeleteCertificate calls were made.
Sorry ECS, sorry ACM…
Why Did This Situation Occur in the First Place? (Speculation)
I believe the root cause of this issue was likely the order in which resources were deleted.
With ECS Express Mode, surrounding resources such as ACM certificates, load balancers, and security groups are created and managed by ECS.
These are tagged with AmazonECSManaged=true, which pairs with the tag conditions in the policy above, allowing ECS itself to manage their lifecycle.
Therefore, the suspected scenario is that during cleanup after testing, the resources created by ECS (ACM certificates) were manually deleted before deleting the ECS Express instance itself.
(This is purely speculation though…)
About the Resolution
Granting Permissions
Now that the cause is understood, let's explicitly grant acm:DeleteCertificate permission for the certificate in question.
This time, I'll use the AWSCertificateManagerFullAccess policy.
After granting the permissions, let's check the CloudTrail events again.
The errorCode that had previously been AccessDenied had changed to ResourceNotFoundException.
ECS can now recognize that the certificate no longer exists, and can treat the deletion process as "complete."
{
"eventVersion": "1.11",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAXXXXXXXXXXXXXXXXX:ECSGateway",
"arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/ecs-express-infrastructure-role/ECSGateway",
"accountId": "XXXXXXXXXXXX",
"accessKeyId": "ASIAXXXXXXXXXXXXXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAXXXXXXXXXXXXXXXXX",
"arn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs-express-infrastructure-role",
"accountId": "XXXXXXXXXXXX",
"userName": "ecs-express-infrastructure-role"
},
"attributes": {
"creationDate": "2026-08-13T03:01:53Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "ecs.amazonaws.com"
},
"eventTime": "2026-08-13T03:01:53Z",
"eventSource": "acm.amazonaws.com",
"eventName": "DeleteCertificate",
"awsRegion": "ap-northeast-1",
"sourceIPAddress": "ecs.amazonaws.com",
"userAgent": "ecs.amazonaws.com",
"errorCode": "ResourceNotFoundException",
"errorMessage": "Could not find certificate arn:aws:acm:ap-northeast-1:XXXXXXXXXXXX:certificate/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.",
"requestParameters": {
"certificateArn": "arn:aws:acm:ap-northeast-1:XXXXXXXXXXXX:certificate/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
},
"responseElements": null,
"requestID": "518d1a27-a7b5-40ea-94bb-f3bde432c1ea",
"eventID": "e76729c7-cd69-4193-8d16-5b37543435f3",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXXXXXXXXXX",
"eventCategory": "Management"
}
Once the certificate cleanup is done, the cleanup of the remaining surrounding resources also proceeds.
Next is the deletion of the security group.
DeleteSecurityGroup is also returning Client.InvalidGroup.NotFound (the target no longer exists), and this has also completed without issue.
{
"eventVersion": "1.11",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAXXXXXXXXXXXXXXXXX:ECSGateway",
"arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/ecs-express-infrastructure-role/ECSGateway",
"accountId": "XXXXXXXXXXXX",
"accessKeyId": "ASIAXXXXXXXXXXXXXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAXXXXXXXXXXXXXXXXX",
"arn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs-express-infrastructure-role",
"accountId": "XXXXXXXXXXXX",
"userName": "ecs-express-infrastructure-role"
},
"attributes": {
"creationDate": "2026-08-13T03:01:53Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "ecs.amazonaws.com"
},
"eventTime": "2026-08-13T03:01:55Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "DeleteSecurityGroup",
"awsRegion": "ap-northeast-1",
"sourceIPAddress": "ecs.amazonaws.com",
"userAgent": "ecs.amazonaws.com",
"errorCode": "Client.InvalidGroup.NotFound",
"errorMessage": "The security group 'sg-XXXXXXXXXXXXXXXXX' does not exist",
"requestParameters": {
"groupId": "sg-XXXXXXXXXXXXXXXXX"
},
"responseElements": null,
"requestID": "af2dc28e-c6a3-4f2c-bf5a-c6832d8bcaf5",
"eventID": "7ea172fa-1d6e-4374-a6a3-87fcca663c7d",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXXXXXXXXXX",
"eventCategory": "Management"
}
With the entire cleanup process now complete, the DeleteCertificate retry loop has stopped…
Closing
That's it — I tracked down the cause of DeleteCertificate events being recorded endlessly in CloudTrail.
This time it happened to be ECS, but since AWS managed features automatically create and delete resources behind the scenes, it wouldn't be surprising if a similar idle loop occurred with other services.
Going forward, I'll make a point of occasionally browsing CloudTrail even in verification accounts, to check for any unusual event history. (Note to self)
