How I Built a System to Automatically Sync KB Articles from Google Drive to a Vertex AI RAG Corpus — Giving Up on Event-Driven and Arriving at Smart Polling
This page has been translated by machine translation. View original
Introduction
We operate a mechanism that embeds RAG (Retrieval-Augmented Generation) into an internal Google Chat bot and generates answers based on KB (knowledge base) articles. Documents are imported into Vertex AI RAG Engine, and the bot searches the corpus to answer user questions.
Initially, KB articles were managed as Markdown files in a Git repository and uploaded to the RAG corpus via GCS using a CLI script. However, adding or editing articles required git operations and running the CLI, which was a high hurdle for non-technical staff.
"When an article is edited in Google Drive, it is automatically reflected in the RAG corpus" — this is a record of the investigation and implementation to achieve that. I first designed it to be event-driven, but ran into the wall of OAuth constraints and ultimately settled on a practical solution: smart polling.
Overall Architecture
The final architecture we adopted is as follows.

The key point is that we adopted Google Docs as the format for KB articles. Instead of placing Markdown files in a Drive folder, using Google Docs lets staff edit articles in a WYSIWYG editor. Even without knowing Markdown syntax, they can intuitively work with formatting such as headings, lists, and bold text.
Vertex AI RAG Engine natively supports Google Workspace files, including Google Docs, so you can import them simply by passing a Drive folder URL to rag.import_files(). However, passing a folder URL to import_files() reprocesses all files every time, which is inefficient for polling at 15-minute intervals. So we implemented smart polling that places GCS in between as an intermediate layer to detect changes and deletions.
Initial Design: Event-Driven with the Workspace Events API
The first thing I designed was a real-time synchronization architecture.

Using the Google Workspace Events API, you can deliver Drive file creation, update, and deletion events to Pub/Sub. The supported event types are as follows.
google.workspace.drive.file.v3.created— File createdgoogle.workspace.drive.file.v3.updated— File updatedgoogle.workspace.drive.file.v3.trashed— File deleted (moved to trash)
With this design, the RAG corpus is updated the moment an edit occurs, so the bot can always answer with the latest information.
Creating a Subscription
A Workspace Events API subscription is created as follows.
EVENT_TYPES = [
"google.workspace.drive.file.v3.created",
"google.workspace.drive.file.v3.updated",
"google.workspace.drive.file.v3.trashed",
]
credentials, _ = google.auth.default(
scopes=["https://www.googleapis.com/auth/drive"]
)
service = build("workspaceevents", "v1", credentials=credentials)
body = {
"targetResource": f"//drive.googleapis.com/files/{folder_id}",
"eventTypes": EVENT_TYPES,
"notificationEndpoint": {
"pubsubTopic": "projects/YOUR_PROJECT_ID/topics/drive-kb-events",
},
"payloadOptions": {
"includeResource": True,
"fieldMask": "mimeType,name,parents",
},
}
result = service.subscriptions().create(body=body).execute()
Important note: The OAuth scope required for Drive events in the Workspace Events API is https://www.googleapis.com/auth/drive. A scope called apps.events.subscriptions does not exist. The official documentation also states that you should use the existing scope corresponding to the target resource (the drive scope for Drive).
Subscription Expiration
Workspace Events API subscriptions have a maximum TTL of 7 days and need to be renewed periodically. You can handle this by renewing every 6 days with Cloud Scheduler, or by handling the expirationReminder event delivered by the subscription itself.
The Wall of OAuth and Organizational Security
As I advanced the event-driven design, I ran into several walls.
Wall 1: Service accounts cannot create subscriptions
Creating a Workspace Events API subscription requires user OAuth authentication. It cannot be created with a service account. Also, Domain-Wide Delegation (DWD) is not supported by the Workspace Events API.
In other words, creating the initial subscription always requires a human to go through the OAuth flow.
Wall 2: The gcloud CLI's OAuth client is blocked
When I tried OAuth authentication using gcloud auth application-default login, an error saying "This app is blocked" occurred in the Google Workspace admin console.
This is caused by the app access control set by the Google Workspace administrator (Admin Console → Security → API controls → App access control), and there are two independent settings:
| Setting | Description |
|---|---|
| OAuth scope allowlist | Whether to allow specific scopes (e.g., drive) |
| App trust level | Classifies individual OAuth clients as "Trusted", "Limited", or "Blocked" |
Even if the scope is allowed, authentication is impossible if the app itself is blocked. The gcloud CLI's OAuth client ID is 764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com, and an administrator needs to set it to "Trusted".
Wall 3: Organizational policy changes are required
In the end, using the Workspace Events API required all of the following conditions to be met:
- The
drivescope is allowed in the organization - The gcloud CLI's OAuth client is set to trusted
- An administrator approves the app in App access control
Depending on the organization's security policy, a process to request and approve these changes may be necessary.
Decision: Switching to Polling
Weighing the coordination cost of having the organization change its security settings for the event-driven approach against the fact that polling provides a sufficient update frequency, we adopted smart polling.
| Aspect | Event-driven | Smart polling |
|---|---|---|
| Reflection speed | Several seconds to tens of seconds | Up to 15 minutes |
| Organization setting changes | Required (OAuth allowance, app trust) | Not required |
| Infrastructure | Pub/Sub + Workspace Events subscription | Cloud Scheduler only |
| Maintainability | Subscription renewal (7-day TTL) | Not required |
| Reliability | Risk of subscription expiration | Scheduler runs stably |
Considering how often KB articles are updated, a 15-minute sync interval poses no practical problems. The Pub/Sub topic and event handler code have been left in place so that event-driven operation can be added in the future once the OAuth settings are in order.
Implementing Smart Differential Sync via GCS
Even with "polling", reimporting all files every time is inefficient. We implemented smart polling, which syncs only the files that have changed.
Change detection: Storing modifiedTime in GCS metadata
When uploading files exported from Drive to GCS, we record Drive's modifiedTime in the GCS blob metadata.
def _export_doc_to_gcs(file_id: str, file_name: str, modified_time: str) -> str:
drive = _get_drive_service()
content = drive.files().export(
fileId=file_id,
mimeType="text/plain",
).execute()
client = _get_gcs_client()
bucket = client.bucket(GCS_BUCKET)
blob = bucket.blob(_blob_name(file_name))
blob.metadata = {"drive_modified_time": modified_time, "drive_file_id": file_id}
blob.upload_from_string(content, content_type="text/plain")
return f"gs://{GCS_BUCKET}/{blob.name}"
Sync logic: Detecting both changes and deletions
full_sync() works as follows:
- Get the list of Google Docs in the target folder via the Drive API
- Get the metadata (
drive_modified_time) of existing blobs on GCS - Compare the two and re-export only the files that have changed
- Detect files that exist in GCS but not in Drive (= deleted files) and delete them from GCS and the RAG corpus
- Call
rag.import_files()only when there are changes
def full_sync() -> dict:
drive_docs = _list_drive_docs()
gcs_blobs = _get_gcs_blobs()
# GCSメタデータからmodifiedTimeを取得
gcs_modified = {}
for blob in gcs_blobs.values():
blob.reload()
meta = blob.metadata or {}
gcs_modified[blob.name] = meta.get("drive_modified_time", "")
exported = 0
unchanged = 0
drive_blob_names = set()
for doc in drive_docs:
bn = _blob_name(doc["name"])
drive_blob_names.add(bn)
gcs_mtime = gcs_modified.get(bn, "")
if gcs_mtime == doc["modifiedTime"]:
unchanged += 1
continue
_export_doc_to_gcs(doc["id"], doc["name"], doc["modifiedTime"])
exported += 1
# 削除検知: GCSにあるがDriveにないファイル
deleted = 0
for bn, blob in gcs_blobs.items():
if bn not in drive_blob_names:
file_name = bn.removeprefix(GCS_PREFIX).removesuffix(".txt")
blob.delete()
_delete_rag_file(file_name)
deleted += 1
# 変更があった場合、またはコーパスに欠落がある場合にインポート
imported = 0
needs_import = exported > 0
if not needs_import:
corpus_names = _list_corpus_display_names()
missing = drive_blob_names - {_blob_name(n.removesuffix(".txt")) for n in corpus_names}
if missing:
needs_import = True
if needs_import:
response = rag.import_files(
corpus_name=_corpus_name(),
paths=[f"gs://{GCS_BUCKET}/{GCS_PREFIX}"],
)
imported = response.imported_rag_files_count
return {"exported": exported, "unchanged": unchanged, "deleted": deleted, "imported": imported}
Note: If you control the import with only exported > 0, then in the case where the export succeeded but the import failed, a "divergence" where files exist only in GCS and are not imported into the corpus becomes permanent. In fact, because of the SDK's bucket ownership check (described later), the import raised a ValueError, and 15 files were not imported into the corpus for several weeks. The code above includes self-healing logic that compares the corpus file list with GCS and reruns the import if anything is missing.
rag.import_files() itself also has a mechanism that internally detects file changes and skips them, but the differential check on the GCS side lets us skip unnecessary export processing, which makes the whole sync faster.
Why put GCS in between as an intermediate layer?
You can import Google Docs by passing a Drive folder URL directly to rag.import_files(), but this method reprocesses all files in the folder every time. Reimporting more than 170 files every time with 15-minute polling is inefficient.
By using GCS as an intermediate layer, the application side can perform differential detection and process only the files that have changed:
- Change detection: By storing Drive's
modifiedTimein the GCS blob metadata, we can accurately determine whether there has been a change. If there are no changes, both export and import are skipped - Deletion detection: By diffing the list of GCS blobs against the list of Drive files, we can identify deleted files.
rag.import_files()does not detect deletions, so this mechanism is essential - Ease of debugging: You can directly inspect files on GCS, which makes troubleshooting the sync state easy
Cloud Function entry point
The Cloud Function (2nd gen / Cloud Run-based) handles both periodic invocations from Cloud Scheduler and Pub/Sub events (for future event-driven use).
@functions_framework.http
def handle_sync(request):
body = request.get_json(silent=True) or {}
try:
# Pub/Sub push message (for future event-driven use)
if "message" in body:
return _handle_pubsub_event(body["message"])
action = body.get("action", "")
if action == "full_sync":
summary = full_sync()
return (summary, 200)
return ("Unknown request", 400)
except Exception:
log("ERROR", "sync_error", error=traceback.format_exc())
# Return 200 to prevent infinite Pub/Sub retries
return ("error logged", 200)
Prerequisite: Enabling the Cloud Resource Manager API
The import_files in agentplatform (google-cloud-aiplatform v1.160.0 and later) verifies GCS bucket ownership on the client side. This verification requires the Cloud Resource Manager API. If it is not enabled, ValueError: Bucket ... does not belong to project ... occurs even if the bucket is in the correct project.
gcloud services enable cloudresourcemanager.googleapis.com --project=YOUR_PROJECT_ID
For details, see Pitfall 4 in the migration article.
Deployment and scheduler configuration
# Deploy the Cloud Function (sync)
gcloud functions deploy kb-drive-sync \
--gen2 --runtime=python314 --region=asia-northeast1 \
--source=sync/ --entry-point=handle_sync \
--trigger-http --no-allow-unauthenticated \
--memory=512Mi --cpu=0.5 \
--set-env-vars="GCP_PROJECT_ID=YOUR_PROJECT_ID,GCP_LOCATION=asia-northeast1,RAG_CORPUS_ID=YOUR_CORPUS_ID,DRIVE_FOLDER_ID=YOUR_FOLDER_ID,GCS_BUCKET=YOUR_BUCKET"
# Run periodically at 15-minute intervals with Cloud Scheduler
gcloud scheduler jobs create http kb-drive-sync-scheduler \
--location=asia-northeast1 \
--schedule="*/15 * * * *" \
--uri="https://kb-drive-sync-HASH-an.a.run.app" \
--http-method=POST \
--headers="Content-Type=application/json" \
--message-body='{"action":"full_sync"}' \
--oidc-service-account-email="YOUR_SA@developer.gserviceaccount.com"
Migrating KB Articles
We needed to migrate several hundred Markdown files to Google Docs.
Challenge: Cannot call the Drive API from local
Due to the OAuth constraints described above, we cannot upload files using the Drive API from a local environment. So we took the approach of performing the migration via a Cloud Function.
- Upload the Markdown files from local to a temporary prefix in GCS (
kb_migrate/) - Call the Cloud Function's
migrateaction - Inside the Cloud Function, read the files from GCS and create them as Google Docs with the Drive API
def migrate_from_gcs() -> dict:
client = _get_gcs_client()
bucket = client.bucket(GCS_BUCKET)
drive = _get_drive_service()
blobs = list(bucket.list_blobs(prefix=GCS_MIGRATE_PREFIX))
md_blobs = [b for b in blobs if b.name.endswith(".md")]
# Skip duplicates (safety on re-execution)
existing_docs = {doc["name"] for doc in _list_drive_docs()}
for blob in md_blobs:
file_name = blob.name.removeprefix(GCS_MIGRATE_PREFIX).removesuffix(".md")
if file_name in existing_docs:
continue
content = blob.download_as_bytes()
media = MediaInMemoryUpload(content, mimetype="text/markdown")
metadata = {
"name": file_name,
"parents": [DRIVE_FOLDER_ID],
"mimeType": "application/vnd.google-apps.document",
}
drive.files().create(
body=metadata,
media_body=media,
fields="id",
supportsAllDrives=True,
).execute()
When you specify application/vnd.google-apps.document as the mimeType in the Drive API's files().create(), the uploaded file is automatically converted to Google Docs. Markdown files sent as text/markdown have their headings, lists, code blocks, and so on appropriately converted into Google Docs formatting.
Handling timeouts
The default timeout for Cloud Functions (2nd gen) is 540 seconds (9 minutes). It wasn't enough time to upload all several hundred files, and the first run timed out at about half of them.
However, because the existing_docs check skips duplicates, simply rerunning processes the remaining files. By designing for idempotency, a timeout is not a fatal problem.
Summary
What we learned:
- The Workspace Events API requires OAuth: It cannot be used with service accounts or DWD. Coordination with the organization's security policy is required
- The OAuth scope trap: There is no dedicated scope for the Workspace Events API (like
apps.events.subscriptions). Use the existing scope corresponding to the target resource (such asdrive) - The role of the GCS intermediate layer: RAG Engine can import Google Docs directly, but it reprocesses all files every time. Performing change and deletion detection with GCS blob metadata greatly improves polling efficiency
- Idempotency matters: Designing the migration script to be re-runnable lets you handle timeouts and temporary failures
Future outlook:
Since the event-driven code (Pub/Sub handler) has been left in place, once the OAuth settings are in order in the future, smart polling and event-driven operation can be used together. Keeping polling as the baseline for reliability while adding real-time capability through event-driven operation — we believe this hybrid "polling + event-driven" configuration is the most robust choice.