Preparing LLM training data: I tried placing a 100GB slice of FineWeb on S3

Introduction

When you start learning about LLM training, the first thing you run into is "how do I prepare training data and where do I put it?" As a first step, I tried taking a portion of FineWeb (derived from Common Crawl, approximately 4TB), a well-known LLM training dataset, and placing it in my own S3 for use.

Online tutorials mostly focus on the HuggingFace side with things like load_dataset(...) or snapshot_download(...). But what I wanted to do was the AWS side — "place a fixed slice in S3 in a reproducible way" — and I couldn't find much information on that.

These are the two goals for this time.

• Place a fixed ~100GB slice from FineWeb in an S3 bucket
• Make it possible for others (or yourself) six months later to verify the same byte sequence using a manifest

The core of the transfer itself fits in a short Python script. However, I ran into several issues getting it to actually complete on AWS, so I'm documenting those as well.

Verification Environment

• A test AWS account (Tokyo region ap-northeast-1, with admin privileges)
• AWS authentication using temporary credentials (something like SSO / assume-role. No static access key file)
• SkyPilot (for spinning up EC2 in one shot. Setup in the next section)
• Python 3.12
• Terraform v1.13 (for managing the S3 bucket)
• Local machine is macOS

Setup: Installing SkyPilot and Connecting to AWS

This is the first hurdle. Install SkyPilot and get it into a state where it can access AWS.

I used uv for installation (SkyPilot supports Python 3.9–3.13). The pattern for installing per project looks like this.

uv venv --seed --python 3.12
uv pip install 'skypilot[aws]'

If you want to use it machine-wide, you can also do uv tool install 'skypilot[aws]' (no venv needed in that case). [aws] specifies installing only AWS-related dependencies.

Next, the AWS credentials. SkyPilot doesn't have its own authentication file; it uses the same credential chain as the AWS CLI. So as long as the AWS CLI is working, you're good (for access keys use aws configure, for SSO use aws sso login).

You can check the connection with sky check. This is the first hurdle — if it doesn't pass, all subsequent sky launch commands will fail.

uv run sky check aws

If successful, you'll see this.

🎉 Enabled infra 🎉
  AWS [compute, storage]

Once this display appears, you can use AWS (compute = EC2, storage = S3) from SkyPilot. Confirm this before moving on.

Reference: SkyPilot Installation

Overall Design

The pipeline is just this.

• Nothing is downloaded locally. The hf:// read stream and s3:// write stream are connected with fsspec, and bytes are simply passed through.
• The transfer runs on an EC2 instance launched by SkyPilot, not on a laptop (to avoid bandwidth and sleep issues).
• The manifest is generated via a separate path, from S3 after the upload. This makes it an independent artifact that "can be regenerated without re-accessing HuggingFace as long as the bucket exists."

The slice is defined as "the first N files from one dump, sorted by filename." If you specify the same dump and the same count, the same set of files will be retrieved by anyone (as long as the dataset revision is the same). One thing to note is that this selection method alone ensures the "files retrieved" are consistent, but if the contents of main are updated in the future, they won't be the same. For strict reproducibility, we record the revision and hash in the manifest (described later) so that identity can be verified afterward.

Folder Structure

Auto-generated files (.venv/, .terraform/, *.tfstate) are omitted.

test/
├── .sky.yaml                 # SkyPilot project config (auth method)
├── fineweb.yaml              # SkyPilot task (instance + run steps)
├── stream_fineweb_slice.py   # hf:// -> s3:// streaming
├── build_manifest.py         # build the manifest after upload
└── infra/
    └── main.tf               # S3 bucket (Terraform)

Creating the S3 Bucket with Terraform

The bucket is the only persistent resource in this project (EC2 instances are deleted after use). We manage it with Terraform so it can be cleanly removed later. Two key points.

• S3 bucket names must be globally unique. So we append a random suffix to a prefix like fineweb-mirror. This way, no account-specific information remains in the committed code.
• Setting force_destroy = true allows the bucket to be destroyed with terraform destroy even when it contains 100GB of data. A pragmatic choice for a test environment.

# infra/main.tf
terraform {
  required_providers {
    aws    = { source = "hashicorp/aws", version = "~> 5.0" }
    random = { source = "hashicorp/random", version = "~> 3.0" }
  }
}

provider "aws" {
  region = var.region
}

variable "region" {
  default = "ap-northeast-1"
}

variable "bucket_prefix" {
  default = "fineweb-mirror"
}

# S3 bucket names are globally unique; a random suffix avoids collisions
resource "random_id" "suffix" {
  byte_length = 4 # -> 8 hex chars, e.g. "a3f9c1d0"
}

resource "aws_s3_bucket" "fineweb" {
  bucket = "${var.bucket_prefix}-${random_id.suffix.hex}"

  # allow destroy even when the bucket still holds ~100GB (test env)
  force_destroy = true
}

output "bucket_name" {
  value = aws_s3_bucket.fineweb.bucket
}

cd test/infra
terraform init
terraform apply                # create the bucket
terraform output bucket_name   # pass this name to sky launch

SkyPilot Task and the Two Scripts

The transfer body is stream_fineweb_slice.py. It simply reads from hf:// and streams to s3://.

# stream_fineweb_slice.py
import argparse
import concurrent.futures
import os
import shutil
import sys

import fsspec
from huggingface_hub import HfFileSystem

HF_REPO = "datasets/HuggingFaceFW/fineweb"

def list_slice(dump: str, count: int) -> list[str]:
    """Return the first `count` parquet paths in `dump`, name-sorted (deterministic)."""
    hf = HfFileSystem()
    all_files = hf.glob(f"{HF_REPO}/data/{dump}/*.parquet")
    if not all_files:
        sys.exit(f"No parquet files found for dump {dump!r}")
    chosen = sorted(all_files)[:count]
    print(f"Selected {len(chosen)} / {len(all_files)} files from {dump}")
    return chosen

def stream_one(hf_path: str, bucket: str, dump: str, region: str) -> str:
    """Stream one file hf:// -> s3:// without touching local disk."""
    filename = hf_path.split("/")[-1]
    s3_key = f"fineweb/{dump}/{filename}"
    src = f"hf://{hf_path}"
    dst = f"s3://{bucket}/{s3_key}"

    # region in client_kwargs is REQUIRED (see below). Without it s3fs writes to
    # the wrong regional endpoint and PutObject fails: "No AWSAccessKey was presented".
    with fsspec.open(src, "rb") as fin, \
            fsspec.open(dst, "wb", client_kwargs={"region_name": region}) as fout:
        shutil.copyfileobj(fin, fout, length=32 * 1024 * 1024)  # 32MB chunks
    print(f"  uploaded {s3_key}")
    return s3_key

def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--dump", required=True)
    p.add_argument("--count", type=int, default=50)
    p.add_argument("--bucket", required=True)
    p.add_argument("--workers", type=int, default=8)
    p.add_argument("--region", default=os.environ.get("AWS_DEFAULT_REGION", "ap-northeast-1"))
    args = p.parse_args()

    files = list_slice(args.dump, args.count)

    # network I/O -> threads (not processes) overlap the waiting
    with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
        futures = [pool.submit(stream_one, f, args.bucket, args.dump, args.region) for f in files]
        for fut in concurrent.futures.as_completed(futures):
            fut.result()

    print(f"Done: {len(files)} files -> s3://{args.bucket}/fineweb/{args.dump}/")

if __name__ == "__main__":
    main()

The manifest is built separately from the transfer, by looking only at the S3 side after the upload. Each line pins one file with size + ETag + row count, so readers can check "do I really have the same byte sequence?" This is the practical deliverable of this article.

# build_manifest.py
import argparse
import json
import os

import boto3
import pyarrow.parquet as pq
import fsspec

def build(bucket, dump, revision, pulled_at, region):
    s3 = boto3.client("s3", region_name=region)
    paginator = s3.get_paginator("list_objects_v2")
    entries = []

    for page in paginator.paginate(Bucket=bucket, Prefix=f"fineweb/{dump}/"):
        for obj in page.get("Contents", []):
            key = obj["Key"]
            if not key.endswith(".parquet"):
                continue

            # read only the parquet footer for the row count (not the whole file)
            with fsspec.open(f"s3://{bucket}/{key}", "rb",
                             client_kwargs={"region_name": region}) as f:
                rows = pq.read_metadata(f).num_rows

            entries.append({
                "path": key,
                "size_bytes": obj["Size"],
                "etag": obj["ETag"].strip('"'),  # multipart ETags carry a "-N" suffix
                "rows": rows,
                "hf_revision": revision,
                "pulled_at": pulled_at,
            })
            print(f"  manifested {key} ({rows} rows)")

    entries.sort(key=lambda e: e["path"])
    return entries

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--dump", required=True)
    p.add_argument("--bucket", required=True)
    p.add_argument("--revision", default="main")
    p.add_argument("--pulled-at", default="unknown")
    p.add_argument("--region", default=os.environ.get("AWS_DEFAULT_REGION", "ap-northeast-1"))
    args = p.parse_args()

    entries = build(args.bucket, args.dump, args.revision, args.pulled_at, args.region)
    if not entries:
        raise SystemExit(f"No parquet objects under fineweb/{args.dump}/")

    body = "\n".join(json.dumps(e) for e in entries) + "\n"
    manifest_key = f"manifests/{args.dump}-{len(entries)}files-v1.jsonl"

    boto3.client("s3", region_name=args.region).put_object(
        Bucket=args.bucket, Key=manifest_key,
        Body=body.encode("utf-8"), ContentType="application/x-ndjson",
    )
    print(f"Wrote manifest: s3://{args.bucket}/{manifest_key} ({len(entries)} entries)")

if __name__ == "__main__":
    main()

The generated manifest is a jsonl file with one file per line.

{"path": "fineweb/CC-MAIN-2024-10/000_00000.parquet", "size_bytes": 2147483648, "etag": "a4f3...-21", "rows": 973991, "hf_revision": "main", "pulled_at": "2026-07-25T09:12:33Z"}

The SkyPilot task definition that bundles these is fineweb.yaml. The bucket name and HF token are passed via --env at launch time, so the file itself is safe to publish as-is.

# fineweb.yaml
resources:
  cloud: aws
  region: ap-northeast-1
  instance_type: c7g.2xlarge   # 16GB RAM (see the memory-shortage section below)
  use_spot: false              # on-demand (see the spot section below)
  disk_size: 30                # boot disk only; nothing lands on local disk

envs:
  HF_TOKEN: null                     # supplied via --env at launch
  S3_BUCKET: null                    # supplied via --env at launch
  CC_DUMP: CC-MAIN-2024-10
  FILE_COUNT: 50                     # ~2.0 GB/file measured, so 50 x 2.0 = ~100 GB
  AWS_DEFAULT_REGION: ap-northeast-1 # must match the bucket region (see below)

workdir: .

setup: |
  pip install 'huggingface_hub[hf_xet]>=1.5' 's3fs' 'pyarrow' 'boto3'

run: |
  set -e
  python stream_fineweb_slice.py --dump "$CC_DUMP" --count "$FILE_COUNT" --bucket "$S3_BUCKET" --workers 4
  python build_manifest.py --dump "$CC_DUMP" --bucket "$S3_BUCKET" \
      --revision main --pulled-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

Launching is just this. -i 5 --down is the setting for "automatically shut down the instance after 5 minutes of idle."

cd test
uv run sky launch -c fw-pull -i 5 --down fineweb.yaml \
    --env HF_TOKEN=none \
    --env S3_BUCKET=$(terraform -chdir=infra output -raw bucket_name) \
    --env FILE_COUNT=50

(FineWeb is public so it can be read without a token. The script doesn't use HF_TOKEN either. The HF_TOKEN=none above is just passing a dummy value because the YAML requires the env var to be set. Only pass a real token when working with gated datasets that require one.)

Try It with One File First

Before running the above command with 50 files, I ran it with FILE_COUNT=1 to transfer just one file. With a single file, it finishes in a short time for a few cents, making it a cheap way to verify that the pipeline works.

From this single-file test, I learned mainly two things.

• The whole flow works: read → S3 write → manifest generation
• One file is approximately 2.0GB (I used this number to decide the file count for the full run)

By the way, among the pitfalls described next, the authentication-related ones (1–3) were mostly surfaced during this single-file test. The ones affecting parallelism and execution time (4 and 5) only appeared when I scaled up to 50 files. The approach was to run small first, iron out problems early, then move to the full run.

5 Pitfalls I Hit Before Getting It to Work

The above covers the design. However, use_spot: false, c7g.2xlarge, and AWS_DEFAULT_REGION in the YAML above were not values I knew from the start — they are all values I arrived at after hitting problems. Listed in the order they occurred.

The individual fixes are as shown in the table. Here I'll deep-dive into the 3 that are most useful to understand for broader application.

(2) An IAM Role Doesn't "Have Permissions" — It "Distributes Temporary Credentials"

When I first saw the error No AWSAccessKey was presented, I thought "the role has S3 permissions, what is it talking about?" But that was a misunderstanding.

AWS API requests always require signing (SigV4). There is no implicit allowance of "this instance has permissions so let it through." What an IAM role does is distribute temporary credentials via the EC2 metadata service (IMDS, 169.254.169.254). The SDK (boto3 or s3fs) retrieves them and uses them to sign requests.

1. Role is attached → AWS places temporary credentials on IMDS
2. SDK retrieves credentials from IMDS
3. SDK signs the request with those credentials (this is where the "access key" is used)
4. S3 verifies the signature → only then checks the role's permissions

No AWSAccessKey was presented meant "the request was not signed" = step 3 did not happen. S3 checks authentication (signing) before authorization (permissions). So even if the role has S3FullAccess attached, an unsigned request is rejected before permissions are even checked.

The cause of (2) was that SkyPilot's default was to upload the local ~/.aws files. Temporary credentials like SSO or assume-role are often passed as environment variables, so there are no static authentication files to upload. As a result, nothing was passed to the instance, and no role was attached to IMDS by default. Switching to SERVICE_ACCOUNT caused SkyPilot to attach its own IAM role (skypilot-v1) to the instance, which resolved it.

Here is the .sky.yaml with that setting. Placing it in the project (test/) causes it to be automatically read when sky is run from that directory (taking priority over the global ~/.sky/config.yaml). The advantage is that "what was done" is fully contained within the repository. Note that whether the project's .sky.yaml is automatically read depends on the SkyPilot version. If it isn't being picked up, place the same content in ~/.sky/config.yaml or specify it explicitly with SKYPILOT_CONFIG.

# .sky.yaml
aws:
  remote_identity: SERVICE_ACCOUNT

(3) s3fs Passes for "Reads" but Requires Region for "Writes"

(3) was the most troublesome. The boto3 STS check passed. s3fs ls (bucket listing) also passed. But only s3fs writes were failing with No AWSAccessKey. Same process, same credentials.

The cause was the region. ListBuckets uses a global endpoint, so it passes. On the other hand, PutObject requires the region-specific endpoint of the bucket. Without specifying a region, s3fs was trying to write to the wrong endpoint, the signature was mismatched, and that error appeared.

The fix is simply to explicitly specify the region. However, since S3 is accessed from multiple places in the script, I used a two-pronged approach: passing the environment variable AWS_DEFAULT_REGION once in the YAML, and also explicitly specifying region_name in the code.

I'll keep this debugging ladder here since it's useful in other situations too.

# 1. Is the credential chain (IMDS/role) working at all?
python -c "import boto3; print(boto3.client('sts').get_caller_identity())"
# 2. Is an empty AWS_* env var breaking the chain?
env | grep -i aws
# 3. s3fs READ auth test
python -c "import s3fs; print(s3fs.S3FileSystem().ls(''))"
# 4. test WRITE separately (read working != write working)

(5) Even "Streaming Without Disk" Still Consumes RAM

(5) was a trap caused by an assumption. I thought "since it doesn't write to local disk, it's lightweight" and ran 8 parallel workers on a c7g.large (4GB RAM), and Ray killed the job due to out of memory.

Streaming still requires memory. hf_xet (HuggingFace's transfer layer) buffers chunks on the read side, and s3fs buffers multipart parts on the write side. In practice, each file buffers up to about 2GB on both ends combined. With 8 parallel workers, that's several times that, which exceeds the 4GB on that machine. The single-file test had passed simply because the parallelism was 1.

The fix was both increasing memory (c7g.2xlarge, 16GB) and reducing parallelism (--workers 4). 4 parallel × ~2GB = ~8GB fits comfortably within 16GB. The lesson was that even with "streaming, no disk," you need to size your instance according to the degree of parallelism.

Results

With the final configuration (on-demand c7g.2xlarge, 4 workers, 50 files), it ran to completion. The full run took approximately 40 minutes (subject to variation depending on HF CDN speed).

  uploaded fineweb/CC-MAIN-2024-10/000_00049.parquet
Done: 50 files -> s3://<bucket>/fineweb/CC-MAIN-2024-10/
  manifested fineweb/CC-MAIN-2024-10/000_00049.parquet (957982 rows)
Wrote manifest: s3://<bucket>/manifests/CC-MAIN-2024-10-50files-v1.jsonl (50 entries)

• 50 parquet files (~100GB) in s3://<bucket>/fineweb/CC-MAIN-2024-10/
• A manifest (50 entries, approximately 950,000–970,000 rows per file) in s3://<bucket>/manifests/CC-MAIN-2024-10-50files-v1.jsonl
• The instance automatically terminates 5 minutes after the job finishes

The file count of 50 was back-calculated from the earlier single-file measurement of ~2.0GB per file. 50 files gives a scale of roughly 100GB. Note that in actual LLM training, data size is rarely fixed at an exact number of GB, so the 100GB here is just a rough ballpark for demonstration purposes.

Here is a rough cost estimate.

The transfer from HuggingFace (egress) is covered by HF, so it's $0. However, EC2 and S3 storage are billed normally. This is beyond the scope of the AWS free tier, so if you're trying this personally, be aware that leaving it running will incur ongoing storage costs (~$2.3/month for 100GB). When you're done, you can delete the bucket entirely with terraform destroy.

When readers want to verify the same slice, they just need to cross-reference against the manifest. If the hash (ETag) doesn't match, that benchmark result is not comparable — that's the intent of this design.

head = s3.head_object(Bucket=bucket, Key=entry["path"])
assert head["ETag"].strip('"') == entry["etag"]
assert head["ContentLength"] == entry["size_bytes"]

Closing

Before starting, I thought it would be as simple as "just put FineWeb in S3." In fact, the core of the transfer fits in about 10 lines of Python.

But the real work was getting those 10 lines to run to completion on a real AWS account. None of it was in the transfer code itself — the problems arose in the surrounding authentication, infrastructure, and resource design, parts that don't appear in tutorials that stop at snapshot_download.

If you're also looking to "place HuggingFace data in your own AWS in a reproducible way," I've already stepped on these 5 pitfalls for you, so all that's left is to adjust for your own environment.