The story about metadata being stored in an unexpected format with hierarchical chunking in Bedrock Knowledge Bases

Introduction

When trying to display custom metadata such as document creation dates in search results for a RAG system using Bedrock Knowledge Bases, I got stuck because the format of the metadata stored in OpenSearch was completely different from what I expected.

Since I had put metadata in the sidecar .metadata.json on S3, I thought it would be as simple as "just retrieving it from OpenSearch," but when using Hierarchical chunking, things are different.

This article introduces the actual structure of AMAZON_BEDROCK_METADATA stored in OpenSearch Serverless when using hierarchical chunking, and how to correctly retrieve custom metadata.

Prerequisites & Environment

• Amazon Bedrock Knowledge Bases
• Chunking strategy: Hierarchical chunking
• Vector store: Amazon OpenSearch Serverless
• Data source: Documents on S3 + sidecar .metadata.json
• Backend: Python

Metadata Structure on the S3 Side

In Bedrock KB, you can attach custom metadata to documents on S3 by placing a sidecar file (.metadata.json). For example, the format looks like this:

{
  "metadataAttributes": {
    "category": "FAQ",
    "priority": "high",
    "created_on": "2024-04-01T00:49:42Z",
    "author": "yamada"
  }
}

Expected Metadata Structure

I thought the AMAZON_BEDROCK_METADATA field stored in OpenSearch would contain the sidecar's metadataAttributes as-is.

# Expected code
meta = chunk.get("metadata")  # AMAZON_BEDROCK_METADATA
attrs = meta.get("metadataAttributes")
created_on = attrs.get("created_on")  # Should retrieve "2024-04-01T00:49:42Z"

After actually deploying it, I ended up in a situation where metadata could be retrieved for some search results but not for others.

Actual Metadata Structure

After adding logging to check, the actual content of AMAZON_BEDROCK_METADATA was as follows:

{
    "source": "s3://my-bucket/document-001.md",
    "parentText": "# Document Title\n\n## Basic Information\n- Created on: 2024-04-01T00:49:42Z\n- Category: FAQ\n..."
}

metadataAttributes is not included. Instead, only two fields are stored: source (S3 URI) and parentText (the text of the corresponding parent chunk).

This is a structure specific to hierarchical chunking. With hierarchical chunking, child chunks are vectorized and searched, and when a search hits, the corresponding parent chunk is passed to the LLM. In OpenSearch documents, the child chunk's content and the parent chunk's text are stored as parentText in AMAZON_BEDROCK_METADATA.

Note that when using Bedrock KB's Retrieve API, the response's metadata field does include the sidecar's metadataAttributes. However, since it is not included in the AMAZON_BEDROCK_METADATA field of documents stored directly in OpenSearch, caution is needed when querying OpenSearch directly.

Why Only Some Could Be Retrieved

In the original code, as a fallback when retrieval from metadataAttributes failed, I was searching for Created on: in the child chunk's content using a regular expression.

# Fallback: extract from child chunk's content using regex
cm = re.search(r"Created on[：:]\s*(\S+)", chunk.get("content", ""))

There is a section containing Created on: near the beginning of the document. For some child chunks, this section happened to be included in content, so the fallback succeeded. However, when a child chunk was split at a location that didn't include that section, the fallback also failed.

Fix: Also Search parentText

Since parentText contains the full text of the parent chunk, it always includes the relevant section. By adding parentText as a search target for the fallback, it became possible to reliably retrieve created_on for all documents.

_CREATED_ON_PATTERN = re.compile(r"Created on[：:]\s*(\S+)")

def _extract_created_on(chunks: list[dict]) -> dict[str, str]:
    """Extract the creation date for each document from a list of chunks"""
    result: dict[str, str] = {}
    for chunk in chunks:
        source = chunk.get("source_uri", "")
        if source in result:
            continue

        # 1. Retrieve directly from metadataAttributes
        meta = chunk.get("metadata")
        if isinstance(meta, dict):
            attrs = meta.get("metadataAttributes", meta)
            raw = attrs.get("created_on", "")
        else:
            raw = ""

        if not raw:
            # 2. Extract using regex in order: child chunk's content → parentText
            parent_text = meta.get("parentText", "") if isinstance(meta, dict) else ""
            for text in (chunk.get("content", ""), parent_text):
                cm = _CREATED_ON_PATTERN.search(text)
                if cm:
                    raw = cm.group(1)
                    break

        if raw:
            result[source] = raw
    return result

The key is a three-tiered extraction strategy:

1. Retrieve metadataAttributes.created_on directly (when metadata is properly stored by a custom indexer)
2. Extract from the child chunk's content using regex (when the child chunk includes the relevant section)
3. Extract from parentText using regex (reliable since the parent chunk always contains the full text)

What I Learned

Structure of AMAZON_BEDROCK_METADATA in Bedrock KB's hierarchical chunking:

The metadataAttributes from the sidecar .metadata.json are not included in OpenSearch's AMAZON_BEDROCK_METADATA. Since they are included in the Retrieve API response, be careful not to confuse the two.

Summary

• With hierarchical chunking in Bedrock KB, the sidecar's metadataAttributes are not directly stored in OpenSearch's AMAZON_BEDROCK_METADATA
• Instead, source (S3 URI) and parentText (full text of the parent chunk) are stored
• When you want to retrieve custom metadata, it is reliable to prepare a fallback that extracts from parentText text using regex or similar methods
• When the actual data structure differs from expectations, the fastest route is to add logging and verify the actual data rather than guessing your way through