The story of fixing httplib2 thread safety issues — A trap in google-api-python-client

Introduction

This is a series of articles about Google Chat Bots. In Part 1 we implemented the minimal configuration with Cloud Functions + Python + uv, in Part 2 we covered cardsV2 progressive UX, in Part 3 we added rich text support, and in Part 4 we implemented knowledge base search using Vertex AI RAG Engine.

This article is about a bug fix. Right after fixing an OOM error, another tricky issue surfaced — httplib2's shared TCP/SSL connections breaking under multithreading — and I'm sharing the process of investigating and fixing it.

Symptoms: Random Errors When Sending Messages in Quick Succession

Right after increasing the bot's memory from 1Gi to 2Gi to resolve the OOM issue, a new problem appeared. When users sent multiple messages in a short period of time, some messages were returned with an "An error has occurred" card.

Checking Cloud Logging, the following errors were recorded intermittently:

05:11:31  ERROR  Failed to create initial card (SSLError)
05:12:21  ERROR  Failed during pipeline execution (SSLError)
05:13:21  ERROR  Failed during pipeline execution (SSLError)
05:15:01  ERROR  Failed during pipeline execution (SSLError)
05:16:01  ERROR  Failed to send error card (TimeoutError)

There were three specific types of errors:

Key point: This is not OOM. Memory is sufficient. All errors occur at the HTTP communication layer.

Root Cause: httplib2.Http Is Not Thread-Safe

Architecture Review

First, let's outline the request processing flow for this bot:

The moment main.py receives a request, it returns {}, and the actual processing runs in a background thread (Cloud Functions gen2 with --no-cpu-throttling allows CPU usage even after the HTTP response is returned).

The Problematic Code

ChatApiClient is the class responsible for communicating with the Google Chat API, and it uses a service object from googleapiclient internally. Since parsing the discovery document for this service object is costly, it was being generated only once using a singleton pattern:

# bot/chat_api.py (before fix)
_default_service = None
_lock = threading.Lock()

def _get_default_service():
    global _default_service
    if _default_service is None:
        with _lock:
            if _default_service is None:
                credentials, _ = google.auth.default(scopes=SCOPES)
                doc = json.loads(_DISCOVERY_DOC_PATH.read_text())
                _default_service = build_from_document(doc, credentials=credentials)
    return _default_service

The initialization itself is thread-safe thanks to double-checked locking. The problem lies beyond that — the httplib2.Http instance generated internally by build_from_document() ends up being shared by all threads.

"Thread-safe" means that multiple threads can access the same resource simultaneously and still behave correctly.

When a program runs multiple operations concurrently, each operation runs as a unit called a "thread." The problem occurs when multiple threads simultaneously read from or write to the same data (variables, connections, files, etc.).

A relatable analogy: imagine two people writing in the same notebook at the same time:

1. Person A writes "Today's weather is"
2. Person B simultaneously writes "Meeting agenda is" on the same line
3. Result: "Today'sMeeting weatheragenda is is" — a meaningless string

The same thing happens inside programs. Thread-safe code prevents this problem by mechanisms such as "only one person writes at a time while others wait" (locking) or "each person uses a separate notebook" (instance isolation).

In this case, the TCP/SSL connection (= notebook) held by httplib2.Http was being used simultaneously by multiple threads (= people), causing communication data to become corrupted.

httplib2 is a Python HTTP client library, in the same category as requests and httpx. It appeared in 2006 and at the time offered advanced features not found in the standard library's urllib, such as cache control and automatic authentication handling. google-api-python-client has depended on this library from its early days, and it is still used as the default HTTP transport.

In modern Python development, requests and httpx are the mainstream HTTP clients, but a key difference from httplib2 is thread safety:

requests.Session is also not thread-safe, but requests is typically used via module-level functions like requests.get(), so it rarely becomes a problem. httplib2, on the other hand, is designed to be used by explicitly creating an Http() instance, and since google-api-python-client hides that instance inside the service object, it's easy to miss that it's being shared.

Why It Breaks

httplib2.Http holds TCP/SSL connections internally, but has no mechanism to safely share them between threads. When two threads call .execute() at the same time:

1. Thread A starts a TLS handshake
2. Thread B sends a different request on the same socket
3. The TLS state machine reaches an inconsistency → WRONG_VERSION_NUMBER

Or alternatively:

1. Thread A sends a request and waits for a response
2. Thread B sends a request on the same socket
3. Thread A ends up reading Thread B's response → IncompleteRead

At 1–2 requests/second it may work by chance, but sending 10 simultaneous requests will break it with high probability.

Checking the Official Google Documentation

The official thread safety documentation for google-api-python-client states clearly:

The httplib2.Http() objects are not thread-safe. If you are running as a multi-threaded application, each thread that you are making requests from must have its own instance of httplib2.Http().

The official documentation recommends two solutions:

Approach 1: Override requestBuilder

def build_request(http, *args, **kwargs):
    new_http = google_auth_httplib2.AuthorizedHttp(credentials, http=httplib2.Http())
    return googleapiclient.http.HttpRequest(new_http, *args, **kwargs)

service = discovery.build('api_name', 'api_version',
                          requestBuilder=build_request, http=authorized_http)

This approach embeds behavior into the service itself to create a new Http object for each API call.

Approach 2: Pass http to execute()

http = google_auth_httplib2.AuthorizedHttp(credentials, http=httplib2.Http())
service.spaces().messages().create(...).execute(http=http)

This approach explicitly passes an Http instance when calling .execute().

Comparing Five Approaches

In addition to the official documentation, I researched DoIt's production operations article and GitHub Issues, and compared five approaches.

Why Approach 1 (Per-Instance AuthorizedHttp) Was Chosen

In this bot, a new ChatApiClient() is created every time process_message() in worker.py is called. Since each process_message() runs in an independent background thread, giving each ChatApiClient instance its own HTTP connection naturally achieves isolation between threads.

During a single pipeline execution (1 create call + several patch calls), the same AuthorizedHttp is reused, so connection reuse is also possible.

The connection pool (Approach 3) was appealing as introduced in DoIt's blog, but the maximum number of concurrent threads in this bot is around 10. I determined the scale didn't warrant adding pool management code.

Implementation

The change is limited to a single file, bot/chat_api.py. No new dependency packages are needed (google_auth_httplib2 and httplib2 are already installed as transitive dependencies of google-api-python-client).

# bot/chat_api.py (after fix)
import google_auth_httplib2
import httplib2

_default_service = None
_credentials = None  # Added: hold credentials at module level
_lock = threading.Lock()

def _get_default_service():
    global _default_service, _credentials
    if _default_service is None:
        with _lock:
            if _default_service is None:
                _credentials, _ = google.auth.default(scopes=SCOPES)
                doc = json.loads(_DISCOVERY_DOC_PATH.read_text())
                _default_service = build_from_document(doc, credentials=_credentials)
    return _default_service

def _build_http():
    """Generate an independent HTTP connection per instance"""
    return google_auth_httplib2.AuthorizedHttp(_credentials, http=httplib2.Http())

class ChatApiClient:
    def __init__(self, service=None):
        if service is None:
            self._service = _get_default_service()
            self._http = _build_http()  # Each instance has its own HTTP connection
        else:
            self._service = service
            self._http = None  # For testing: use mock service as-is

    def create_message(self, space_name, body, thread_name=None):
        # ... (omitted)
        response = (
            self._service.spaces()
            .messages()
            .create(**kwargs)
            .execute(http=self._http)  # ← This is the key point
        )
        return response["name"]

    def patch_message(self, message_name, body, update_mask):
        return (
            self._service.spaces()
            .messages()
            .patch(name=message_name, updateMask=update_mask, body=body)
            .execute(http=self._http)  # ← Same here
        )

Impact on Tests

Existing tests inject mocks like ChatApiClient(service=mock_service). In this case self._http = None, and .execute(http=None) falls back to googleapiclient's default behavior (using the service's built-in HTTP). MagicMock ignores http=None and simply returns the return_value, so all 98 tests passed without any changes.

Verification

Local Tests

$ uv run pytest -v
============================= 98 passed in 1.05s ==============================

Deploy and Spam Test

$ gcloud functions deploy google-chat-bot \
    --gen2 --runtime=python314 --region=asia-northeast1 \
    --source=. --entry-point=handle_chat --trigger-http \
    --no-allow-unauthenticated --memory=2Gi --cpu=1

After deploying, 10 messages were sent in quick succession within a few seconds in Google Chat. Results:

• Before fix: 3–5 out of 10 messages returned as error cards
• After fix: All 10 messages returned with correct answers

Log Check

$ gcloud functions logs read google-chat-bot \
    --region=asia-northeast1 --gen2 --limit=50 \
    --start-time="2026-06-15T06:31:00Z" \
    | grep -iE "error|failed|SSLError|TimeoutError"

# No output — zero errors

Summary

When calling APIs in a multithreaded manner using google-api-python-client, sharing httplib2.Http will reliably break things. This is clearly stated in the official documentation, but when creating a service with the singleton pattern, sharing happens naturally, making it an easy trap to fall into.

The fix is simply to pass an independent AuthorizedHttp to .execute(http=...). In this case, I adopted the approach of isolating HTTP connections per ChatApiClient instance, but depending on the scale, a connection pool is also worth considering.

I hope this is helpful for anyone who has run into the same problem.

References

• Thread Safety | google-api-python-client Official Documentation
• Using Google API Python Client in Production-Grade Apps | DoIt
• Thread safe of httplib2 and credentials · Issue #808