Risk Management: Actual Case in which an AWS Access Key was Accidentally Exposed
【実録】アクセスキー流出、攻撃者のとった行動とその対策 ｜ Developers.IO
https://dev.classmethod.jp/cloud/aws/accesskey-leak/
Keeping your AWS credentials secure is essential in order to safeguard your AWS account. I would like to write about the implications of having your access key out in a public repository by referring to an actual case where this happened to one of our customers.
Moreover, before a malicious attack occurs, it is important to understand what you can do to prevent potential consequences of having an access key compromised. I would like to introduce a few techniques designed to mitigate these unnecessary losses.
Incident Overview
The customer was using Developer Poweruser IAM which is a managed policy; PowerUserAccess for development purposes. During some process, the credential was inadvertently pushed to a public GitHub repository. The access key which was exposed had permission to create, delete, modify across a wide range of resources.
Unfortunately, the key was used by someone who intended to abuse the AWS account for cryptocurrency mining. In the end, the incident was resolved thanks to a prompt reaction. Some of the resources which were created by hackers using the leaked access key needed to be deleted instantaneously.
The events which occurred are as follows. Times are an approximation.
| NO | TIME | EVENT |
| 1 | 00:00 | Access key which contains poweruser permission is accidently pushed to a public repository on GitHub |
| 2 | 00:10 | Malicious hacking is performed by using the uploaded access key |
| 3 | 00:30 | AWS detects the incident and notifies us. Our team contacted customer via phone and investigated with customer |
| 4 | 01:00 | Deletion of all resources that were created by the exposed access key |
| 5 | xx:xx | Rotate access key(s) of each IAM user(s) |
| 6 | xx:xx | Rotate password(s) |
Further details of each event are as follows.
Fraudulent activities performed by using the stolen key were done at the level of API operation. Moreover, the sequences appear to be scripted.
CreateAccessKey- Created a new access key
DeleteAccessKey- Deleted leaked access key
CreateUser- Created IAM user (fails due to insufficient permissions; Error code: AccessDenied)
CreateKeyPair- Create a key pair using a new access key
RunInstances- Launch instances across all regions
- 20 instances (the default instance limit) are launched in each region
- Large computing size instances were prioritized, such as m5.24xlarge
- Launch instances across all regions
Once new instances had been launched, cryptocurrency mining was performed, which targeted specific currency utilizing CPU and was highly anonymous. In this case, "Monero" was running.
GuardDuty was enabled in each region, so a lot of high severity warnings were given. The detection was categorized as CryptoCurrency:EC2/BitcoinTool.B that each instance communicated to the mining pool through oversea VPS.
After tracking the incident overview, the following operations were advisable to the customer in order to minimize the implications.
- Delete relevant data from GitHub
- Delete access key of targeted IAM user
- By CloudTrail, etc., check for unauthorized use (all regions)
- Delete unexpectedly created resources (when bad intention use is confirmed)
- Rotate access keys for other IAM users
- Rotate passwords for all IAM users
It took about 1 hour to delete the detected resources since the launch of the instances. The AWS usage charge of around $500 was incurred within one hour from the occurrence. Since the initial response was early, the amount of damage was curtailed.
Suggestions for risk mitigation
git-secrets
The main reason this happened was that the developer inadvertently pushed data that contained an AWS access key credential to GitHub’s public repo. Git-secrets is a suitable tool to prevent credentials from being mixed into a public Git repo.
https://git-secret.io/
https://github.com/awslabs/git-secrets
Grant minimum permissions
Giving developers a certain level of permission is necessary. However, I recommend to carefully review whether or not all developers need to be granted PowerUserAccess or Administrator privileges. Imagine what the cost would be if the credential which is validated in the Administrator level of permission experienced any undesirable occurrence.
IAM policy should be minimized and given to each user with only the required permissions.
Enable GuardDuty in all regions
GuardDuty is a service which detects suspicious network communication in order to lessen damage through prompt initial action against threats.
I would like to emphasize here that GuardDuty must be enabled in each region. If you have activated service in only one region, suspicious activities in other regions may go completely unnoticed. It is worth considering the service cost for reducing the negative effects of threats.
(Japanese only)
Amazon GuardDutyによる疑わしいネットワーク通信の検知と初動対応の振り返り
【全員必須】GuardDutyがコスパ最強の脅威検知サービスであることを証明してみた
一発でGuardDutyを全リージョン有効化して通知設定するテンプレート作った
Set Alarm at Budget
Budget is a service for monitoring the cost of your AWS account. You can set the alarm to notify you based on configured conditions. It can detect when, and even before, your usage fee skyrockets.
Classmethod (Europe)
By signing up with our AWS total support service, "Classmethod Members", customers can take advantage of the discounts of AWS charges as well as our high-quality technical support, which includes a prompt call in case of such security incident. We also provide end-to-end solutions which include consultation of IAM management.
(Japanese only) Classmethod Members
Please do not hesitate to contact us if you have any issues concerning AWS. Our well-qualified engineers are always happy to help you!
Conclusion
Security incidents caused by credential exposure without intention can happen at any time, anywhere.
Today I have reviewed one actual incident case and pointed out ways that can prevent unexpected security issues from occurring. The key is to manage your credentials properly.
