I checked the ETag and SHA-256 of a file uploaded to S3 using multipart upload

Introduction

Sometimes you want to verify that a file stored in Amazon S3 is identical to the file that was uploaded. In such cases, you may want to compare the ETag or ChecksumSHA256 with a hash value calculated locally.

However, for objects uploaded via multipart upload, neither value is necessarily the hash of the entire file. When actually storing a large file in S3, the ETag and ChecksumSHA256 retrieved via head-object both had a suffix indicating the number of parts.

In this article, I saved the same 20 MiB synthetic file to S3 using both a single PUT and a multipart upload, and compared the results. I also recalculated the SHA-256 of the retrieved objects to confirm they were identical to the uploaded source file.

Target Audience

• Those storing large files in S3
• Those who want to use S3's ETag or ChecksumSHA256 for integrity verification
• Those considering what evidence to retain after a multipart upload

Verification Environment

The default values for multipart_threshold and multipart_chunksize in the AWS CLI aws s3 command are 8 MB. I did not override these settings, and verified under the condition where a 20 MiB file is split into 3 parts.

References

• AWS: Checking object integrity for data uploads in Amazon S3
• AWS: Uploading and copying objects using multipart upload
• AWS CLI: S3 Configuration
• AWS CLI: head-object
• AWS CLI: S3 FAQ

Verification Method

I saved one input file to separate keys using a single PUT via put-object and a multipart upload via s3 cp. I specified --checksum-algorithm SHA256 for both.

Creating the Synthetic File

I created a file with all bytes set to 0x00 for verification purposes.

fsutil file createnew .\source-20m.bin 20971520
Get-FileHash .\source-20m.bin -Algorithm SHA256
Get-FileHash .\source-20m.bin -Algorithm MD5

The calculation results are as follows.

MD5 is not used for security verification. In this case, I calculated it to compare the ETag of a single PUT using SSE-S3 with the ETag calculation of a multipart upload.

Uploading the Same File in Two Ways

aws s3api put-object `
  --bucket <bucket-name> `
  --key article-evidence/singlepart-20m.bin `
  --body .\source-20m.bin `
  --checksum-algorithm SHA256

aws s3 cp `
  .\source-20m.bin `
  s3://<bucket-name>/article-evidence/multipart-20m.bin `
  --checksum-algorithm SHA256

After uploading, I retrieved the object attributes with --checksum-mode ENABLED.

aws s3api head-object `
  --bucket <bucket-name> `
  --key article-evidence/multipart-20m.bin `
  --checksum-mode ENABLED

Results

Even with files of the same content, the results differed depending on the upload method.

The ChecksumSHA256 for the single PUT is the SHA-256 of the entire file expressed in Base64. Since the local Get-FileHash displays the same digest in hexadecimal, the appearance differs.

The ETag for the single PUT matched the local MD5 under the SSE-S3 conditions of this verification. However, ETag cannot generally be treated as an MD5 value. Its behavior varies depending on conditions such as the encryption method and upload method.

On the multipart side, ChecksumType became COMPOSITE. The -3 suffix appended to ChecksumSHA256 and ETag indicates that this object consists of 3 parts. Neither matches the full-file SHA-256 or MD5 calculated locally.

Recalculating the COMPOSITE Checksum

I reproduced the value returned by S3 from the digests of each part split into 8 MiB, 8 MiB, and 4 MiB. Rather than concatenating as strings, I concatenated the byte sequences of each digest and then computed the hash once more.

ETag = Hex(MD5(MD5(part1) || MD5(part2) || MD5(part3))) + "-3"

ChecksumSHA256 = Base64(
  SHA256(SHA256(part1) || SHA256(part2) || SHA256(part3))
) + "-3"

The recalculated results matched the values retrieved by head-object.

In this COMPOSITE checksum, the input to the final SHA-256 was 96 bytes, consisting of 3 SHA-256 digests concatenated together.

Verifying the Full-File SHA-256

The multipart ChecksumSHA256 is a value combining the SHA-256 of each part. It cannot be used to directly compare against a full-file SHA-256 provided by the distributor with the head-object value.

In this verification, I retrieved both objects using get-object --checksum-mode ENABLED. The AWS CLI verifies the S3 checksum upon retrieval. I then recalculated the full-file SHA-256 of the retrieved files locally.

aws s3api get-object `
  --bucket <bucket-name> `
  --key article-evidence/multipart-20m.bin `
  --checksum-mode ENABLED `
  .\downloaded-multipart-20m.bin

Get-FileHash .\downloaded-multipart-20m.bin -Algorithm SHA256

Files retrieved from both the single PUT and the multipart upload had the same full SHA-256 as the upload source. On the other hand, the multipart ChecksumSHA256 did not match the full SHA-256.

Comparison with AWS Official Specifications

With multipart upload, each part can be sent independently, and only failed parts need to be retransmitted. S3 verifies the checksum of each part upon receipt, and upon completion calculates the COMPOSITE checksum from the stored per-part checksums.

The official S3 documentation explains that calculating from per-part checksums reduces the computational cost of checksum calculation. It also states that CRC-64/NVME, CRC-32, and CRC-32C are the only algorithms capable of computing a full-file checksum during multipart upload. CRC-based checksums can derive the full-file value from per-part values, but SHA-256 and MD5 cannot perform the same computation. Therefore, a full SHA-256 cannot be recovered from a COMPOSITE checksum. If a full SHA-256 is needed, it must be recalculated from the entire file.

The same documentation also explains that replicating an object stored via multipart upload using a single copy operation may change the checksum even if the data is the same.

Discussion

Design Trade-offs

The reason S3 uses COMPOSITE checksums is likely to reduce the computational cost of checksum calculation at completion while maintaining the independence of multipart uploads. As objects grow larger, the difference between processing the entire file and processing the stored per-part checksums is expected to grow as well.

As a trade-off, COMPOSITE checksums reflect not only the file's content but also how the parts were divided. This can be understood as a design that prioritizes the efficiency of multipart transfer and integrity verification over identifying file content alone.

Impact on File Identification

Given this design, it is advisable to avoid using ETag or COMPOSITE checksums as file-unique identifiers. The values change when the upload tool, part size, or copy method changes, which could cause deduplication or artifact comparison to treat identical content as different files. When a SHA-256 from the distributor is available, it is appropriate to verify it before uploading and store that value separately.

Based on the above, it is advisable to manage the full SHA-256, S3 checksum, and VersionId separately according to their intended use.

Summary

I saved the same 20 MiB file to S3 using both a single PUT and a multipart upload, and compared the ETag and SHA-256 checksum. With multipart upload, both values reflected the part configuration and did not match the hash of the entire file. When verifying against a distributor's SHA-256, it is necessary to check ChecksumType and treat the full SHA-256, VersionId, size, and S3 checksum as separate pieces of evidence. I hope this article serves as a useful reference for integrity verification when storing large files in S3.

Appendix

The code used to recalculate the COMPOSITE checksum is provided below.

from __future__ import annotations

import argparse
import base64
import hashlib
import json
from pathlib import Path

def calculate(path: Path, part_size: int) -> dict[str, object]:
    full_md5 = hashlib.md5()
    full_sha256 = hashlib.sha256()
    part_md5_digests: list[bytes] = []
    part_sha256_digests: list[bytes] = []

    with path.open("rb") as source:
        while chunk := source.read(part_size):
            full_md5.update(chunk)
            full_sha256.update(chunk)
            part_md5_digests.append(hashlib.md5(chunk).digest())
            part_sha256_digests.append(hashlib.sha256(chunk).digest())

    part_count = len(part_md5_digests)
    composite_md5 = hashlib.md5(b"".join(part_md5_digests)).hexdigest()
    composite_sha256 = hashlib.sha256(b"".join(part_sha256_digests)).digest()

    return {
        "bytes": path.stat().st_size,
        "part_size": part_size,
        "part_count": part_count,
        "full_md5_hex": full_md5.hexdigest(),
        "full_sha256_hex": full_sha256.hexdigest(),
        "full_sha256_base64": base64.b64encode(full_sha256.digest()).decode("ascii"),
        "multipart_etag": f"{composite_md5}-{part_count}",
        "composite_sha256": (
            f"{base64.b64encode(composite_sha256).decode('ascii')}-{part_count}"
        ),
    }

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("path", type=Path)
    parser.add_argument("--part-size", type=int, default=8 * 1024 * 1024)
    args = parser.parse_args()
    print(json.dumps(calculate(args.path, args.part_size), indent=2))

if __name__ == "__main__":
    main()