
Amazon S3 Vectors' filterable metadata has been expanded to 40 KB, so I tried it out
This page has been translated by machine translation. View original
Introduction
On November 20, 2026, the upper limit of filterable metadata in Amazon S3 Vectors was expanded from 2 KB to 40 KB per vector. For indexes in new vector buckets (those created on or after September 30, 2026), this limit is available by default.
Filterable metadata is metadata that can be used in the filter of QueryVectors. This article introduces the steps to use about 38 KB of filterable metadata in an index in a new bucket. It also introduces the documentation that describes how to use the 40 KB limit with an existing index.
Test Environment
- Region: Tokyo (ap-northeast-1)
- AWS CLI: 2.42.1
- Index: float32, 4 dimensions, cosine
- Metadata configuration:
--metadata-configurationis not specified, so all keys are filterable
Creating the Bucket and Index
Create a bucket and an index with any bucket name, without specifying the index mode.
aws s3vectors create-vector-bucket --vector-bucket-name amzn-s3-demo-vector-bucket
aws s3vectors create-index --vector-bucket-name amzn-s3-demo-vector-bucket --index-name idx-40kb --data-type float32 --dimension 4 --distance-metric cosine
Check the bucket's defaultIndexMode and the index's indexMode. To use up to 40 KB of filterable metadata, the index must be ENHANCED.
aws s3vectors get-vector-bucket --vector-bucket-name amzn-s3-demo-vector-bucket
{
"vectorBucket": {
"vectorBucketName": "amzn-s3-demo-vector-bucket",
"defaultIndexMode": "ENHANCED"
}
}
aws s3vectors get-index --vector-bucket-name amzn-s3-demo-vector-bucket --index-name idx-40kb
{
"index": {
"vectorBucketName": "amzn-s3-demo-vector-bucket",
"indexName": "idx-40kb",
"dataType": "float32",
"dimension": 4,
"distanceMetric": "cosine",
"indexMode": "ENHANCED"
}
}
Test Data
Each string value can be up to 2 KB. Metadata close to 40 KB is split across multiple keys.
I prepared 19 string keys (pad_01 to pad_19) with values of 2,000 bytes each, and added the short keys category and tenant for filtering. In total there are 21 keys, about 38 KB. The byte count is the UTF-8 length when the metadata is serialized as JSON without whitespace.
There are three vectors:
- big-manual-a: about 38 KB of metadata (category is manual, tenant is a)
- big-faq-a: about 38 KB of metadata (category is faq, tenant is a)
- small-manual-b: 34 bytes of metadata (category is manual, tenant is b)
The following gen_vectors.py creates vectors.json in the current directory.
gen_vectors.py
#!/usr/bin/env python3
"""filterable metadata が約 38 KB のベクトルを含む vectors.json を生成する。
1 値あたり 2,000 バイトの文字列キー pad_01..pad_19 を重ねて約 38 KB にする。
末尾に絞り込み用の短いキー(category, tenant)を置く。
"""
import json
PAD_KEYS = 19
PAD_BYTES = 2000 # Within the 2 KB limit per value
def pad(n: int, doc: str) -> str:
head = f"{doc}-pad{n:02d}-"
return head + "x" * (PAD_BYTES - len(head))
def big(key: str, data, category: str, tenant: str) -> dict:
md = {f"pad_{i:02d}": pad(i, key) for i in range(1, PAD_KEYS + 1)}
md["category"] = category
md["tenant"] = tenant
return {"key": key, "data": {"float32": data}, "metadata": md}
vectors = [
big("big-manual-a", [0.1, 0.2, 0.3, 0.4], "manual", "a"),
big("big-faq-a", [0.2, 0.1, 0.4, 0.3], "faq", "a"),
{
"key": "small-manual-b",
"data": {"float32": [0.4, 0.3, 0.2, 0.1]},
"metadata": {"category": "manual", "tenant": "b"},
},
]
with open("vectors.json", "w") as f:
json.dump(vectors, f)
for v in vectors:
size = len(json.dumps(v["metadata"], separators=(",", ":")).encode())
print(f'{v["key"]}: metadata {size} bytes ({size/1024:.1f} KiB)')
python3 gen_vectors.py
Verifying Writes and Storage
Pass vectors.json to put-vectors without any additional parameters.
aws s3vectors put-vectors --vector-bucket-name amzn-s3-demo-vector-bucket --index-name idx-40kb --vectors file://vectors.json
Nothing was output, and the command finished successfully.
Next, retrieve the stored content with get-vectors, then use check_stored.py to count the metadata bytes and number of keys and verify that the content matches vectors.json.
check_stored.py
import json
import sys
got = {v["key"]: v["metadata"] for v in json.load(open(sys.argv[1]))["vectors"]}
src = {v["key"]: v["metadata"] for v in json.load(open(sys.argv[2]))}
for k in src:
size = len(json.dumps(got[k], separators=(",", ":")).encode())
print(f"{k}: stored_metadata_bytes={size} keys={len(got[k])} identical_to_put={got[k] == src[k]}")
aws s3vectors get-vectors --vector-bucket-name amzn-s3-demo-vector-bucket --index-name idx-40kb --keys big-manual-a big-faq-a small-manual-b --return-metadata --no-return-data --output json > get-vectors.json
python3 check_stored.py get-vectors.json vectors.json
big-manual-a: stored_metadata_bytes=38262 keys=21 identical_to_put=True
big-faq-a: stored_metadata_bytes=38259 keys=21 identical_to_put=True
small-manual-b: stored_metadata_bytes=34 keys=2 identical_to_put=True
Filtered Search
Run query-vectors without a filter and with three types of filters, and compare the keys of the returned vectors. The query vector is saved in query-vector.json.
{"float32":[0.1,0.2,0.3,0.4]}
The contents of filter-category.json:
{"category": {"$eq": "manual"}}
The contents of filter-and.json:
{"$and": [{"category": {"$eq": "manual"}}, {"tenant": {"$eq": "a"}}]}
filter-pad19.json specifies the value of pad_19 of big-faq-a (2,000 bytes) in $eq. It can be created with the following command.
python3 -c 'import json; print(json.dumps({"pad_19": {"$eq": "big-faq-a-pad19-".ljust(2000, "x")}}), end="")' > filter-pad19.json
Replace <filter-file> with the file name of each filter and run it. "None" in the table is the result of running without --filter.
aws s3vectors query-vectors --vector-bucket-name amzn-s3-demo-vector-bucket --index-name idx-40kb --top-k 10 --query-vector file://query-vector.json --return-distance --no-return-metadata --filter file://<filter-file>
| Filter | Keys of returned vectors | Count |
|---|---|---|
| None | big-manual-a, big-faq-a, small-manual-b | 3 |
| filter-category.json | big-manual-a, small-manual-b | 2 |
| filter-and.json | big-manual-a | 1 |
| filter-pad19.json | big-faq-a | 1 |
Vectors with about 38 KB of metadata could be filtered both by the short-valued category and tenant, and by pad_19, which has a 2,000-byte value.
Using It with an Existing Index
How to change a CLASSIC index in a bucket created before September 30, 2026 to ENHANCED is described in the following documentation.
Summary
I confirmed that about 38 KB of filterable metadata can be written to S3 Vectors with an ENHANCED index.
S3 Vectors hybrid search (vector + full text) has also become available.
If you previously passed on adopting S3 Vectors because of reasons such as the 2 KB filterable metadata limit, we recommend evaluating it again, including this update.
