The Story of Implementing Knowledge Base Search by Connecting Vertex AI RAG Engine to Google Chat Bot

Introduction

In Part 1, I built a Google Chat Bot with Cloud Functions + Python + uv, and in Part 2, I implemented a progressive update UX using cardsV2.

This time, we finally get to the main topic — integrating a knowledge base search (RAG) into the bot. I'll walk through the entire process of organizing approximately 300 internal knowledge data entries, loading them into Vertex AI RAG Engine, and enabling the bot to automatically answer user questions.

To cut to the chase, the RAG Engine setup itself was straightforward, but there was a Japanese-specific pitfall in selecting the embedding model, and the process of identifying the root cause was the most valuable learning experience.

Architecture

Data Organization: QA Data → KB Entries

Evaluating the Quality of Source Data

In this case, the source data consisted of approximately 600 QA data entries (question and answer pairs) accumulated internally. The source data was in the format of staff working notes, not user-facing procedure documents.

Quality Filtering

First, I excluded entries that were clearly unsuitable as knowledge base content.

As a result, approximately half of the "question + meaningful answer" pairs remained.

Policy of Keeping Individual Entries

I adopted a policy of keeping QA pairs as individual KB entries. Initially, I considered an approach of grouping by solution pattern and creating consolidated articles, but I switched to individual retention for the following reasons.

1. Preserving variations — Even for the same symptom, solutions differ depending on the situation. Consolidation would lose individual context.
2. Simplifying the pipeline — Rule-based matching and manual article creation become unnecessary, making it easy to add new data.
3. Synthesis by Gemini — When multiple entries of the same pattern appear in search results, Gemini follows the system prompt instructions to summarize common response methods.

Handling Source Data in Memo Format

When source data is in the format of staff working notes (e.g., "Resolved by changing the XX setting"), rather than showing it directly to users, you can instruct Gemini via the system prompt to transform it.

## About the Knowledge Base Format
- The knowledge base contains response records (staff notes).
- For records in the format "Resolved by doing XX", please rephrase them as easy-to-understand steps for the user.
- If multiple similar records are included in the search results, summarize the common response methods in your answer.

Answer quality directly depends on the quality of the source data. To get better answers, the most effective approach is to improve the descriptions in the source data itself.

KB Entry Format

Each entry is saved as a Markdown file.

# Excel macros cannot be executed.

**Category**: office

## Response

Please change the Trust Center settings.

The question is placed in the title (= the match target in vector search), and the answer is placed in the response section. The reason for keeping the question is that without it, lexical matching with search queries becomes weak (e.g., the answer may not contain the word "macro").

In the end, approximately 600 entries were organized into approximately 300 KB entries.

Setting Up Vertex AI RAG Engine

Why I Chose RAG Engine

With over 300 entries, "context stuffing" — cramming all content into the prompt — is not realistic. I adopted Vertex AI RAG Engine (managed RAG) for the following reasons.

• Scales without configuration changes even as the KB grows
• Chunking, embedding, and search are managed with no maintenance required
• Self-contained within the Vertex AI ecosystem, requiring no additional external services

What Is a Corpus?

A corpus in RAG Engine is a container for documents that are the target of search. Uploaded files are chunked, vectorized, and stored in a searchable state within the corpus. It is equivalent to an index in OpenSearch, and you can create multiple corpora for different purposes (e.g., for IT support, HR, technical documentation).

With OpenSearch, you need to build your own embedding pipeline and k-NN settings, but with RAG Engine, you simply specify files and choose an embedding model, and chunking, vectorization, and search index construction are all handled in a managed manner.

Creating a GCS Bucket and Uploading

First, upload the KB articles to GCS.

# Create a GCS bucket
gcloud storage buckets create gs://YOUR_BUCKET_NAME \
  --location=asia-northeast1

# Script to upload KB articles
uv run python scripts/upload_kb.py \
  --project YOUR_PROJECT_ID \
  --bucket YOUR_BUCKET_NAME

upload_kb.py is a script that uploads .md files of KB articles to GCS, creates a RAG corpus, and imports the files.

from google.cloud import aiplatform, storage
from vertexai import rag

LOCATION = "asia-northeast1"

def main():
    # Initialize the SDK only once at the beginning
    aiplatform.init(project=project_id, location=LOCATION)

    # 1. Upload to GCS
    client = storage.Client(project=project_id)
    bucket = client.bucket(bucket_name)
    for md_file in KB_DIR.glob("*.md"):
        blob = bucket.blob(f"kb_articles/{md_file.name}")
        blob.upload_from_filename(str(md_file))

    # 2. Create corpus (first time only)
    corpus = rag.create_corpus(
        display_name="my-it-support-kb",
        description="IT Support Knowledge Base",
    )

    # 3. Import files
    rag.import_files(corpus_name=corpus.name,
                     paths=[f"gs://{bucket_name}/kb_articles/"])

If you omit location in aiplatform.init(), us-central1 is used by default. If the corpus is in asia-northeast1, you will get the following error when calling rag.import_files():

FAILED_PRECONDITION: Request resource location asia-northeast1
does not match service location us-central1.

Since rag.list_corpora() and rag.create_corpus() work fine but only import_files() fails, it took time to identify the cause. Always run aiplatform.init(location=...) before calling rag.*.

Implementing Search

Search from RAG Engine is performed using rag.retrieval_query().

from vertexai import rag

def retrieve_context(query: str) -> list[dict]:
    response = rag.retrieval_query(
        text=query,
        rag_resources=[rag.RagResource(rag_corpus=corpus_name)],
        rag_retrieval_config=rag.RagRetrievalConfig(
            top_k=5,
            filter=rag.Filter(vector_distance_threshold=0.6),
        ),
    )
    results = []
    for context in response.contexts.contexts:
        results.append({
            "text": context.text,
            "score": context.score,
            "source": context.source_uri,
        })
    return results

vector_distance_threshold is the cosine distance, where a smaller value indicates higher relevance. Using 0.6 as the threshold, values above this are judged as "low confidence."

These are easy to confuse, so let me clarify. Cosine similarity ranges from -1 to 1, where values closer to 1 indicate semantic closeness. Cosine distance is the inverse, calculated as 1 - similarity, ranges from 0 to 2, and values closer to 0 indicate semantic closeness. Since the score returned by RAG Engine is cosine distance, a smaller value means a better match.

Metric	Range	Better Direction
Cosine Similarity	-1 to 1	Higher (1 = identical)
Cosine Distance	0 to 2	Lower (0 = identical)

Answer Generation and Fallback Strategy

Two strategies are used depending on the confidence of search results.

def query(question: str) -> str:
    # 1. Search KB
    contexts = retrieve_context(question)

    # 2. Confidence check
    if not contexts or all(c["score"] > 0.6 for c in contexts):
        # Low confidence → honestly return that no information was found in KB
        return "No relevant information was found. Please contact the responsible department."

    # 3. Generate answer using KB information
    return generate_answer(question, contexts)

There is an important design decision here. An architecture that falls back to Google Search grounding (a feature where Gemini answers based on Google search results) for low confidence cases is also technically possible. However, I adopted the policy of not answering with information not in the knowledge base. For an internal bot, "I don't know, please ask the person in charge" is safer than inaccurate general information.

Obstacle: Japanese Search Quality Was Disastrous

The setup went smoothly, but when I actually tested it, some queries returned completely irrelevant search results.

Symptoms

=== "VBA macro cannot be executed" ===
  0.26 | office_macro_blocked.md  ✅ Correct

=== "Cannot log in to the internal system" ===
  0.18 | system_user_lock.md  ✅ Correct

=== "I received a suspicious email" ===
  0.44 | smartphone_google_photos.md  ❌ Completely wrong

Queries containing English or katakana technical terms like VBA and Google Drive worked correctly, but queries in pure Japanese like "suspicious email" failed completely.

Yet the article title was "How to handle a suspicious email" — nearly identical text to the query.

Isolating the Cause: Chunking or Embedding?

Two possible causes were considered.

1. Chunking issue — RAG Engine is splitting files inappropriately, separating the title from the body.
2. Embedding model issue — The default text-embedding-005 is not correctly capturing the semantic similarity of Japanese text.

Verifying Chunking

I ran a search targeting only a specific file and checked the chunks created by RAG Engine.

response = rag.retrieval_query(
    text="I received a suspicious email",
    rag_resources=[rag.RagResource(
        rag_corpus=corpus,
        rag_file_ids=["<suspicious_email_file_id>"],
    )],
    rag_retrieval_config=rag.RagRetrievalConfig(
        top_k=5,
        filter=rag.Filter(vector_distance_threshold=1.0),
    ),
)

The result showed that the chunk contained the entire article text. Since the file was small (approximately 30 lines), it was not split — 1 file = 1 chunk — meaning chunking was not the issue.

Verifying the Embedding Model

Next, I manually embedded the same text and calculated the cosine distance.

from vertexai.language_models import TextEmbeddingModel, TextEmbeddingInput

model = TextEmbeddingModel.from_pretrained("text-embedding-005")

query = "I received a suspicious email"
article = "How to handle a suspicious email..."
unrelated = "Google Photos sync settings on smartphone..."

# Embed without task type (default)
embeddings = model.get_embeddings([query, article, unrelated])
# → query vs article: 0.27 (good!)
# → query vs unrelated: 0.46

With the default task type, it correctly identifies at 0.27. So why does RAG Engine return 0.46?

The True Cause: Asymmetric Task Type Pairing

text-embedding-005 has a concept called task types, where different vectors are generated for the same text depending on the task type.

How Task Types Work

Task types do not switch the model's structure or weights. Internally, a prefix (instruction text) corresponding to the task type is simply prepended to the text. Conceptually, it works like this:

RETRIEVAL_QUERY    + "suspicious email"  →  Vector A
RETRIEVAL_DOCUMENT + "suspicious email"  →  Vector B
(no prefix)        + "suspicious email"  →  Vector C

Same model, same weights, same Transformer — but because the prefix differs, the attention pattern changes, resulting in different vectors being generated. It's the same principle as getting different outputs from an LLM with "please summarize" vs. "please translate."

This asymmetric pairing (using different task types for queries and documents) is designed to absorb the differences in nature between short search queries and long documents. The query-side prefix is trained to "expand the intent to approach the document space," while the document side is trained to "be positioned where relevant queries can easily reach."

However, this training depends on the training data. Since text-embedding-005 is primarily trained on English query-document pairs, it can correctly "stretch" for English, but the core problem here is that the prefix influence distorts the vectors for Japanese rather than helping them.

When Task Types Are Applied

The important thing is that task types are applied both at ingest time and at query time.

• At ingest time (when importing files into the corpus): Each chunk is embedded with RETRIEVAL_DOCUMENT and stored in the index.
• At query time (when calling retrieval_query()): The search query is embedded with RETRIEVAL_QUERY and compared against the stored document vectors.

In other words, the asymmetry is baked into the index. To change the document-side task type, you need to recreate the corpus and re-import the files.

RAG Engine internally uses the following pairing.

• Query: RETRIEVAL_QUERY
• Document: RETRIEVAL_DOCUMENT

When I manually tested with this combination:

q_input = TextEmbeddingInput(text=query, task_type="RETRIEVAL_QUERY")
d_input = TextEmbeddingInput(text=article, task_type="RETRIEVAL_DOCUMENT")
# → Cosine distance: 0.4605  ← Exact match with RAG Engine result!

It became clear that text-embedding-005's RETRIEVAL_QUERY / RETRIEVAL_DOCUMENT pair cannot correctly capture the semantic similarity of pure Japanese text. Since queries containing English or katakana technical terms work fine, this is an easy problem to miss.

Comparing Embedding Models on Vertex AI

Before getting to the solution, let me summarize the embedding models available on Vertex AI.

Both 005 and multilingual-002 support output dimension reduction, allowing cost/speed tradeoffs. For new projects, you'll generally choose between these two.

Solution: Migrating to text-multilingual-embedding-002

I ran the same test with text-multilingual-embedding-002.

model = TextEmbeddingModel.from_pretrained("text-multilingual-embedding-002")

q_input = TextEmbeddingInput(text=query, task_type="RETRIEVAL_QUERY")
d_input = TextEmbeddingInput(text=article, task_type="RETRIEVAL_DOCUMENT")
# → Cosine distance: 0.2090  ← Dramatic improvement!

I recreated the RAG Engine corpus with text-multilingual-embedding-002.

corpus = rag.create_corpus(
    display_name="my-it-support-kb-v2",
    backend_config=rag.RagVectorDbConfig(
        rag_embedding_model_config=rag.RagEmbeddingModelConfig(
            vertex_prediction_endpoint=rag.VertexPredictionEndpoint(
                publisher_model="publishers/google/models/text-multilingual-embedding-002",
            ),
        ),
    ),
)

Search results after recreation:

=== "I received a suspicious email" ===
  0.21 | suspicious_email.md  ✅

=== "VBA macro cannot be executed" ===
  0.19 | office_macro_blocked.md  ✅

=== "My computer is running slowly" ===
  0.18 | pc_slow_troubleshooting.md  ✅

=== "Cannot log in to Salesforce" ===
  0.18 | salesforce_login.md  ✅

The correct article now comes up as top-1 for all queries.

Integrating the RAG Pipeline into worker.py

I incorporated actual RAG processing into the 4-step progressive card built in the previous article.

def process_message(space_name, user_text, sender, ...):
    # Step 1: Analyzing inquiry
    _advance_step(state, "analyze", patcher, message_name)

    # Step 2: Building search query
    _advance_step(state, "build_query", patcher, message_name)

    # Step 3: Searching knowledge base
    contexts = retrieve_context(user_text)

    # Step 4: Generating answer
    if not contexts or all(c["score"] > 0.6 for c in contexts):
        answer = NO_RESULT_MESSAGE
    else:
        answer = generate_answer(user_text, contexts)

    # Add answer to card paragraph by paragraph
    for para in answer.split("\n\n"):
        state.content_paragraphs.append(para.strip())
        patcher.patch(build_progressive_card(state))

Deployment Notes: Out of Memory

512Mi → 1Gi (Initial Deployment)

After the initial deployment, a problem occurred where the card got stuck at "Analyzing inquiry."

Checking the logs:

Memory limit of 512 MiB exceeded with 529 MiB used.

The google-cloud-aiplatform SDK is heavy, and at 512Mi it was being killed by OOM. Increasing to 1Gi resolved it temporarily.

1Gi → 2Gi (After Production Operation)

However, 1Gi was not sufficient either. After operation started, an issue occurred where no answer was returned when queries were sent in rapid succession.

Symptoms and the Difficulty

The symptom was vague: "sometimes no answer is returned when multiple queries are submitted in quick succession." The fact that it was "sometimes" rather than always made it tricky. Error handling on the application side was working correctly, so it didn't appear to be a code bug.

Debugging Steps

First, I checked the error-level logs in Cloud Logging.

gcloud logging read \
  'resource.type="cloud_run_revision"
   AND resource.labels.service_name="YOUR_SERVICE_NAME"
   AND severity>=ERROR' \
  --limit=50 \
  --format='table(timestamp,severity,textPayload)'

Two types of errors were found.

1. OOM (Memory Limit Exceeded)

Memory limit of 1024 MiB exceeded with 1033 MiB used.
Memory limit of 1024 MiB exceeded with 1029 MiB used.

This is the direct cause of the forced instance termination.

2. Connection Errors to Chat API

ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number
http.client.IncompleteRead: IncompleteRead(290 bytes read, 400 more expected)
Failed to create initial card

After an instance is killed by OOM, a new instance starts, but the HTTP connection pool from the previous instance can sometimes be taken over in a corrupted state, causing SSL errors and incomplete read errors in a cascade. In other words, it was a chain of failures: OOM → corrupted connection pool → Chat API call failure → unable to even create the initial card → no response.

3. Checking the Timeline

I checked all logs (including INFO) before and after the errors to understand the flow of the failure chronologically.

12:06:49  ERROR  Memory limit of 1024 MiB exceeded with 1029 MiB used
12:06:50  INFO   Starting new instance (AUTOSCALING)
  ...(normal operation)...
12:26:33  ERROR  Failed to create initial card (SSLError)
12:26:33  ERROR  Failed to create initial card (IncompleteRead)
  ...
12:29:56  ERROR  Memory limit of 1024 MiB exceeded with 1033 MiB used
12:29:57  INFO   Starting new instance (AUTOSCALING)

It was repeating the pattern of OOM → recovery → normal for a while → OOM again. This matched the report that "sometimes no answer is returned."

Visualizing Memory Usage

Since the cause was identified as OOM, the next step was to retrieve memory usage trends from the Cloud Monitoring API to determine the appropriate memory limit.

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v3/projects/PROJECT_ID/timeSeries?filter=..." \
  | python3 -c "..."  # Extract average values from distribution data

Results:

Even in idle state, 70% was already consumed, and just a slight increase during query processing was enough to reach OOM.

Why This Becomes a Silent Failure

This bot processes requests in background threads and immediately returns {} as the HTTP response. When an instance is killed by OOM, the background thread disappears with it, but from Google Chat's perspective, the HTTP request succeeded (200 OK), so no error like "the app is not responding" is displayed. The user receives no notification whatsoever — the answer simply never comes back.

Thinking About Memory Sizing

Google Cloud's best practices recommend keeping peak memory usage at 50–80% of the allocated limit. Below 50% is excessive cost; above 80% risks OOM during spikes.

By increasing to 2Gi, peak usage came in at around 50%, securing sufficient headroom.

Python's (CPython) internal memory allocator (pymalloc) retains memory pools for freed objects and rarely returns them to the OS. This means that even after processing completes and threads terminate, the process's RSS (Resident Set Size) does not decrease. Even calling gc.collect() explicitly will free Python-internal objects, but the memory usage as seen by the OS won't change. For this reason, peak memory usage becomes the resident memory, and the memory limit must be set according to the peak.

Deployment Command

gcloud functions deploy YOUR_FUNCTION_NAME \
  --gen2 --runtime=python312 --region=asia-northeast1 \
  --source=. --entry-point=handle_chat \
  --trigger-http --no-allow-unauthenticated \
  --memory=2Gi --cpu=1

# First time only: Disable CPU throttling (for background threads)
gcloud run services update YOUR_FUNCTION_NAME \
  --region=asia-northeast1 --no-cpu-throttling

The Cloud Run annotation run.googleapis.com/cpu-throttling: 'false' is maintained in subsequent deployments once set. You only need to run gcloud run services update --no-cpu-throttling once during initial setup.

Managing Environment Variables

RAG configuration values are managed via environment variables. Once set with Cloud Functions' --set-env-vars, they carry over to subsequent deployments.

# First time only
gcloud functions deploy YOUR_FUNCTION_NAME \
  ... \
  --set-env-vars="GCP_PROJECT_ID=your-project,GCP_LOCATION=asia-northeast1,RAG_CORPUS_ID=your-corpus-id,GEMINI_MODEL_ID=gemini-2.5-flash"

Summary

I connected the Google Chat Bot to Vertex AI RAG Engine and implemented knowledge base search.

The biggest takeaway is that when building RAG with Japanese content, you should use text-multilingual-embedding-002 rather than text-embedding-005. Since text-embedding-005 works fine for English and katakana technical terms, this is a trap that can go unnoticed depending on your test cases.

Another takeaway is that the quality of KB data determines the upper limit of answer quality. Since the source data is in the format of staff notes, there are limits to how much you can do by instructing Gemini to transform it via the system prompt. To get better answers, the most effective approach is to improve the quality of the source data itself.

References

• Vertex AI RAG Engine overview | Google Cloud
• Choose an embeddings task type | Google Cloud
• text-multilingual-embedding-002 | Google Cloud
• Part 1: Building a Google Chat Bot with Cloud Functions + Python + uv in Minimal Configuration
• Part 2: The Challenges of Implementing a Progressive UX with cardsV2 in a Google Chat Bot