The story about metadata being stored in an unexpected format with hierarchical chunking in Bedrock Knowledge Bases

Introduction

While trying to display custom metadata such as document creation dates in a RAG system using Bedrock Knowledge Bases, I got stuck because the format of the metadata stored in OpenSearch was completely different from what I expected.

Since I had placed metadata in the sidecar .metadata.json files on S3, I assumed it would be a simple matter of "just retrieving it from OpenSearch" — but when using Hierarchical chunking, the situation is different.

This article introduces the actual structure of AMAZON_BEDROCK_METADATA stored in OpenSearch Serverless when using hierarchical chunking, and how to correctly retrieve custom metadata.

Prerequisites & Environment

• Amazon Bedrock Knowledge Bases
• Chunking strategy: Hierarchical chunking
• Vector store: Amazon OpenSearch Serverless
• Data source: Documents on S3 + sidecar .metadata.json
• Backend: Python

Metadata Structure on the S3 Side

In Bedrock KB, you can attach custom metadata to documents on S3 by placing sidecar files (.metadata.json). For example, the format looks like this:

{
  "metadataAttributes": {
    "category": "FAQ",
    "priority": "high",
    "created_on": "2024-04-01T00:49:42Z",
    "author": "yamada"
  }
}

Expected Metadata Structure

I assumed that the AMAZON_BEDROCK_METADATA field stored in OpenSearch would contain the sidecar's metadataAttributes as-is.

# Expected code
meta = chunk.get("metadata")  # AMAZON_BEDROCK_METADATA
attrs = meta.get("metadataAttributes")
created_on = attrs.get("created_on")  # Should return "2024-04-01T00:49:42Z"

After actually deploying it, I found myself in a situation where metadata could be retrieved from some search results but not from others.

Actual Metadata Structure

After adding logging to investigate, the actual contents of AMAZON_BEDROCK_METADATA looked like this:

{
    "source": "s3://my-bucket/document-001.md",
    "parentText": "# Document Title\n\n## Basic Information\n- Created: 2024-04-01T00:49:42Z\n- Category: FAQ\n..."
}

metadataAttributes is not included. Instead, only two fields are stored: source (the S3 URI) and parentText (the text of the corresponding parent chunk).

This is a structure specific to hierarchical chunking. With hierarchical chunking, child chunks are vectorized and searched, and when a search hits, the corresponding parent chunk is passed to the LLM. In the OpenSearch document, the child chunk's content and the parent chunk's text are stored in AMAZON_BEDROCK_METADATA as parentText.

Note that when using Bedrock KB's Retrieve API, the response's metadata field does include the sidecar's metadataAttributes. However, since it is not included in the AMAZON_BEDROCK_METADATA field of documents stored directly in OpenSearch, caution is required when querying OpenSearch directly.

Why Only Some Results Could Be Retrieved

In the original code, when retrieval from metadataAttributes failed, I had a fallback that searched for 作成日: (creation date) in the child chunk's content using a regular expression.

# Fallback: extract using regex from child chunk's content
cm = re.search(r"作成日[：:]\s*(\S+)", chunk.get("content", ""))

There is a section near the beginning of the document that contains 作成日:. For some child chunks, this section happened to be included in content, so the fallback succeeded. However, when a child chunk was split at a point that did not include that section, the fallback also failed.

Fix: Also Search parentText

Since parentText contains the full text of the parent chunk, it always includes the relevant section. By adding parentText as a search target for the fallback, it became possible to reliably retrieve created_on for all documents.

_CREATED_ON_PATTERN = re.compile(r"作成日[：:]\s*(\S+)")

def _extract_created_on(chunks: list[dict]) -> dict[str, str]:
    """Extract the creation date for each document from a list of chunks"""
    result: dict[str, str] = {}
    for chunk in chunks:
        source = chunk.get("source_uri", "")
        if source in result:
            continue

        # 1. Retrieve directly from metadataAttributes
        meta = chunk.get("metadata")
        if isinstance(meta, dict):
            attrs = meta.get("metadataAttributes", meta)
            raw = attrs.get("created_on", "")
        else:
            raw = ""

        if not raw:
            # 2. Extract using regex from child chunk's content → parentText, in that order
            parent_text = meta.get("parentText", "") if isinstance(meta, dict) else ""
            for text in (chunk.get("content", ""), parent_text):
                cm = _CREATED_ON_PATTERN.search(text)
                if cm:
                    raw = cm.group(1)
                    break

        if raw:
            result[source] = raw
    return result

The key is a three-stage extraction strategy:

1. Retrieve metadataAttributes.created_on directly (when metadata is correctly stored by a custom indexer)
2. Extract using regex from the child chunk's content (when the child chunk contains the relevant section)
3. Extract using regex from parentText (reliable, since the parent chunk always contains the full text)

What I Learned

Structure of AMAZON_BEDROCK_METADATA in Bedrock KB hierarchical chunking:

The metadataAttributes from the sidecar .metadata.json are not included in OpenSearch's AMAZON_BEDROCK_METADATA. Since they are included in the Retrieve API response, be careful not to confuse the two.

Summary

• With Bedrock KB hierarchical chunking, the sidecar's metadataAttributes are not stored directly in OpenSearch's AMAZON_BEDROCK_METADATA
• Instead, source (S3 URI) and parentText (the full text of the parent chunk) are stored
• When you need to retrieve custom metadata, it is safest to prepare a fallback that extracts it from parentText using regex or similar methods
• When the expected and actual data structures differ, the fastest path forward is to add logging and verify the actual data rather than guessing