Introduction
S3 Multipart Upload is a feature of Amazon S3 that allows you to upload large objects/files in multiple smaller parts. If these parts are successfully uploaded, S3 will assemble these parts to create a complete object. However, if the multipart upload is not completed successfully for any reason, the parts remain as incomplete multipart uploads in the S3 bucket. You incur costs for storing incomplete multipart uploads. While these uploads are essentially unnecessary objects, you still pay for the stored parts in S3. To minimize this cost, you can configure an S3 Lifecycle rule to automatically delete incomplete multipart uploads after a specified number of days.
How to find the S3 buckets containing incomplete multipart uploads
Amazon S3 Storage Lens can be used as one of the methods to discover the S3 buckets with incomplete multipart uploads. You can use the "default-account-dashboard" default dashboard to identify the S3 buckets.
- Sign in to the AWS Management Console and navigate to the S3 Console.
- Choose Dashboards under Storage Lens and open default-account-dashboard.
- Scroll down to the Top N overview section in the overview tab.
- Choose Incomplete multipart upload bytes as the Metric.
- Now you can view the list of top N buckets containing incomplete multipart uploads with their size in bytes.
Set S3 lifecycle to delete incomplete multipart uploads
We have obtained the list of S3 buckets with incomplete multipart uploads from S3 Storage Lens. Next, configure a lifecycle rule to delete the incomplete multipart uploads in these buckets.
- Choose the S3 bucket to which you want to configure a lifecycle rule in the Buckets list.
- Choose the Create lifecycle rule in the Management tab.
- Give a name for the lifecycle and choose Apply to all objects in the bucket as the scope of the lifecycle rule.
- Select Delete expired object delete markers or incomplete multipart uploads an action for the lifecycle rule.
- Check the Delete incomplete multipart uploads checkbox, enter the Number of days after which the cleanup of incomplete multipart uploads should occur, and create the lifecycle rule.
Conclusion
With this S3 lifecycle rule, the incomplete multipart uploads in the S3 bucket will be deleted after the specified number of days minimizing the storage costs associated with incomplete multipart uploads.
Reference
Configuring a bucket lifecycle configuration to delete incomplete multipart uploads
Discovering and Deleting Incomplete Multipart Uploads to Lower Amazon S3 Costs
Setting lifecycle configuration on a bucket
6
