Chunk Anchoring — A Simple Technique to Prevent Ticket ID Hallucination in RAG

Introduction

Have you ever experienced an LLM fabricating non-existent ticket IDs when you want your RAG system to return answers with citations?

When building a RAG system for an internal service desk, I loaded over 1,000 tickets into a Bedrock Knowledge Base and created a mechanism where an LLM generates answers based on search results from chunked content. The search accuracy was good, but I noticed a problem: ticket IDs included in answers sometimes didn't match those from the actual chunks retrieved from the KB.

This article introduces a technique called "Chunk Anchoring" and its complementary "source_uri validation" that I adopted to solve this problem. Both are simple but applicable techniques for any RAG system where citation accuracy is required.

Prerequisites & Environment

• Amazon Bedrock Knowledge Base (Titan Embed V2, Hierarchical Chunking)
• Claude 3.5 Sonnet (via Bedrock Converse API)
• Python 3.13 + FastAPI
• Knowledge source: Approximately 1,300 Redmine tickets converted to Markdown and stored in S3

The Problem: Ticket IDs Disappearing During Chunking

How Hierarchical Chunking Works

When using Bedrock KB's hierarchical chunking (parent chunk 1,500 tokens / child chunk 512 tokens), a single Markdown document is split into multiple chunks.

The original ticket document has the following structure:

# Ticket #12345: VPN Connection Becomes Unstable

## Basic Information
- Status: Resolved
- Assignee: Taro Tanaka
...

## Description
An issue occurred where the internal VPN connection disconnects intermittently...

## Response History
### [2026-03-15] Jiro Sato
- Status: In Progress → Resolved
Resolved by updating the router firmware...

What Happens

When split by chunking, the "Description" and "Response History" chunks may not contain the ticket ID:

Chunk 1: "# Ticket #12345: VPN Connection Becomes Unstable\n## Basic Information\n..."  ← Has ID
Chunk 2: "## Description\nAn issue occurred where the internal VPN connection..."         ← No ID!
Chunk 3: "## Response History\n### [2026-03-15] Jiro Sato\nResolved by updating..."      ← No ID!

When the LLM generates an answer from the content of chunks 2 and 3, it tries to fill in the ticket ID and fabricates a non-existent number. For example, it generates something like "A similar issue was reported in Ticket #12300" — a plausible but non-existent ID.

When service desk operators try to look up the original ticket, presenting them with a non-existent ID breaks their workflow and destroys trust in the RAG system.

Solution 1: Chunk Anchoring

The Idea

Make every chunk carry its own source information — that is the concept behind Chunk Anchoring.

Specifically, immediately after each ## section heading in the Markdown document, we insert the ticket ID and subject in block quote format:

# Ticket #12345: VPN Connection Becomes Unstable

## Basic Information
- Status: Resolved
...

## Description
> Ticket #12345: VPN Connection Becomes Unstable
An issue occurred where the internal VPN connection disconnects intermittently...

## Response History
> Ticket #12345: VPN Connection Becomes Unstable
### [2026-03-15] Jiro Sato
> Ticket #12345: VPN Connection Becomes Unstable
Resolved by updating the router firmware...

This ensures that no matter which chunk is returned as a search result, the ticket number and subject are always included.

Implementation

In the conversion process (transform.py), anchors are embedded when generating Markdown from ticket data:

def ticket_to_document(raw_ticket: dict, lookups=None) -> TicketDocument:
    issue = raw_ticket["issue"]
    issue_id = issue["id"]
    subject = issue.get("subject", "(No Title)")

    # Chunk anchor: repeatedly inserted into each section
    _anchor = f"> Ticket #{issue_id}: {subject}"

    sections = [
        f"# Ticket #{issue_id}: {subject}",
        "",
        "## Basic Information",
        # ... Basic information fields (no anchor needed since # title immediately precedes)
    ]

    # Description section
    if description:
        sections.extend(["", "## Description", _anchor, clean_desc])

    # Response history section
    if meaningful_journals:
        sections.extend(["", "## Response History", _anchor])
        for journal in meaningful_journals:
            sections.append(f"\n### [{created}] {user}\n{_anchor}")
            # ... Journal content

Key points:

• No anchor in ## Basic Information: Since # Ticket #ID immediately precedes it, they are always included in the same chunk
• Insert anchors into each journal entry in response history: Response history tends to get long and often spans multiple chunks
• Use block quote format (>): Makes it easier for the LLM to recognize the anchor as meta-information rather than answer text
• Idempotent processing: The same anchor is generated every time, so re-converting won't create duplicates

Why It's Effective

This technique is effective because it does not depend on chunk granularity. Even if you change chunking strategies (fixed-length, semantic, hierarchical, etc.), ticket IDs will always be included in chunks as long as anchors are embedded in each section.

The LLM no longer needs to "guess" the source — it can directly reference the anchor text within the chunk.

Solution 2: source_uri Validation (Defense in Depth)

Chunk Anchoring significantly reduced ticket ID hallucinations, but it's not a 100% guarantee. The possibility remains that the LLM could mix up information from multiple chunks and return a different ticket ID.

Therefore, I added a filter on the backend side that post-processes LLM output and only passes through IDs that exist in the retrieved chunks.

Turn 1: Extraction from source_uri

Each chunk retrieved from the KB has a source_uri (file path on S3) attached to it. Since ticket IDs are included in file names, they can be extracted with a regular expression:

valid_ticket_ids: set[int] = set()
for c in chunks:
    m = re.search(r"/(\d+)\.md", c["source_uri"])
    if m:
        valid_ticket_ids.add(int(m.group(1)))

Like s3://bucket/tickets/12345.md → ID 12345, this is the most reliable method, requiring no text analysis.

Turn 2 and Beyond: Extraction from Anchors

In multi-turn conversations, Turn 1 builds an augmented message containing KB search results, and subsequent turns inherit that message. In this case, source_uri is not directly available, but we can extract anchor strings embedded by Chunk Anchoring using a regular expression:

valid_ticket_ids = {
    int(m) for m in re.findall(r"Ticket #(\d+)", messages[0].content)
}

Chunk Anchoring proves doubly useful here: it's used not only for ID identification during search but also for ground truth extraction during validation.

Filtering

Ticket information returned by the LLM via streaming is validated against valid_ticket_ids:

async for item in generate_chat_answer(bedrock_messages, source_chunks_count):
    if isinstance(item, TicketResult):
        # If valid_ticket_ids is empty, fall back (pass all through)
        if not valid_ticket_ids or item.issue_id in valid_ticket_ids:
            yield item
    elif isinstance(item, ChatResponse):
        if valid_ticket_ids:
            item.results = [
                r for r in item.results if r.issue_id in valid_ticket_ids
            ]
        yield item

Fallback design: If valid_ticket_ids is empty (e.g., URI format differs from expectations), validation is skipped and all items pass through. This prevents silent failures where "results return zero items" due to filtering failures.

Overview of the Two-Layer Architecture

Layer 1 (Chunk Anchoring) is a preventive measure, and Layer 2 (source_uri validation) is a detective measure. This two-layer structure can almost completely prevent ticket ID hallucinations.

Scope of Application: Can It Be Used Beyond Tickets?

Chunk Anchoring can be applied to any RAG system where source identifiers are important:

The common thread is cases where source tracking is needed even after chunks have been split.

Alternative: Metadata JSON Sidecar

Bedrock KB supports metadata attachment via .metadata.json sidecar files. It's natural to ask: "Shouldn't ticket IDs be separated into a sidecar rather than repeatedly embedded in the document body?"

Limitations of Sidecars

Bedrock KB metadata sidecars function as filtering attributes. They are not automatically inserted into chunk text:

{"metadataAttributes": {"ticket_id": 12345, "subject": "VPN Connection Unstable"}}

This metadata can be used for narrowing down KB searches, but it is not included in the chunk text the LLM receives. In other words, sidecars alone do not solve the problem of IDs being missing from chunks.

Sidecar + Injection at the Orchestration Layer

If you're building a custom orchestration layer, an approach of attaching sidecar metadata to chunks after retrieval is possible:

for chunk in retrieved_chunks:
    meta = get_sidecar_metadata(chunk["source_uri"])
    chunk["text"] = f"> Ticket #{meta['ticket_id']}: {meta['subject']}\n{chunk['text']}"

Comparison

Why This Article Chose Chunk Anchoring

1. Infrastructure-independent: Works even when changing the search backend or switching to managed RAG
2. Synergy with Layer 2 validation: Anchor text can be used for ground truth extraction in multi-turn conversations
3. Implementation simplicity: Completed with a few-line change to the conversion script, with no metadata lookup required at query time

If you're working with custom orchestration as a prerequisite and expect frequent metadata updates, the sidecar + injection approach is also a viable option.

Implementation Considerations

Token Increase Trade-off

Repeating anchors in each section increases the total token count of the document. Measured across approximately 1,300 tickets, the token increase rate was about 3-5%. While there is a slight increase in embedding costs and storage, this is a sufficiently acceptable trade-off compared to the improvement in citation accuracy.

Compatibility with Chunking Strategies

With very fine-grained sentence-level chunking, anchor overhead becomes noticeable, but in practice, splitting at that granularity is rarely necessary.

Fallback Design Is Important

When introducing source_uri validation, explicitly design the behavior on validation failure. Making validation too strict can cause silent failures where "zero answers are returned" when URI format changes or metadata is missing. The fallback of "pass all through if valid_ticket_ids is empty" is an effective design choice that errs on the side of safety.

Summary

• Chunk Anchoring: Embed source identifiers (> Ticket #ID: Subject) in each chunk section to prevent ID loss from chunking
• source_uri validation: Cross-reference LLM output against KB chunk ground truth to exclude hallucinated IDs
• A two-layer structure (prevention + detection) provides near-complete assurance of citation accuracy
• Simple implementation, applicable to any RAG system where citation accuracy is required

If you're experiencing challenges with "citation accuracy" in your RAG system, start by trying Chunk Anchoring. With just a few lines of code changes, citation quality should improve dramatically.