The story of fixing httplib2 thread safety issues — A trap in google-api-python-client

Introduction

This is a series of articles about Google Chat Bots. In Part 1 we implemented the minimal configuration with Cloud Functions + Python + uv, in Part 2 cardsV2 progressive UX, in Part 3 rich text support, and in Part 4 knowledge base search using Vertex AI RAG Engine.

Today's topic is a "bug fix". I'll share the process of investigating and fixing another tricky problem that surfaced right after resolving an OOM error — httplib2's shared TCP/SSL connections breaking under multithreading.

Symptoms: Random Errors When Sending Messages in Rapid Succession

Just after resolving the OOM issue by increasing the bot's memory from 1Gi to 2Gi, a new problem emerged. When users sent multiple messages in a short period of time, some messages came back with an "An error occurred" card.

Checking Cloud Logging, the following errors were sporadically recorded:

05:11:31  ERROR  Failed to create initial card (SSLError)
05:12:21  ERROR  Failed during pipeline execution (SSLError)
05:13:21  ERROR  Failed during pipeline execution (SSLError)
05:15:01  ERROR  Failed during pipeline execution (SSLError)
05:16:01  ERROR  Failed to send error card (TimeoutError)

There were three specific types of errors:

Key point: It's not OOM. Memory is sufficient. All errors are occurring at the HTTP communication layer.

Root Cause: httplib2.Http Is Not Thread-Safe

Architecture Review

First, let's organize the request processing flow for this bot:

The moment main.py receives a request, it returns {}, and the actual processing runs in a background thread (Cloud Functions gen2 can use CPU after the HTTP response with --no-cpu-throttling).

The Problem Area

ChatApiClient is the class responsible for communicating with the Google Chat API, and internally uses a service object from googleapiclient. Since parsing the discovery document for this service object is expensive, it was being generated only once using a singleton pattern:

# bot/chat_api.py (before fix)
_default_service = None
_lock = threading.Lock()

def _get_default_service():
    global _default_service
    if _default_service is None:
        with _lock:
            if _default_service is None:
                credentials, _ = google.auth.default(scopes=SCOPES)
                doc = json.loads(_DISCOVERY_DOC_PATH.read_text())
                _default_service = build_from_document(doc, credentials=credentials)
    return _default_service

The initialization itself is thread-safe with double-checked locking. The problem lies further down — the httplib2.Http instance that build_from_document() creates internally ends up being shared by all threads.

"Thread-safe" means operating correctly even when accessed by multiple threads simultaneously.

When a program executes multiple processes in parallel, each process runs as a unit called a "thread". The problem occurs when multiple threads simultaneously read and write the same data (variables, connections, files, etc.).

To explain with a familiar example, it's like two people writing in the same notebook at the same time:

1. Person A writes "Today's weather is"
2. Person B simultaneously writes "The meeting agenda is" on the same line
3. Result: "Today'sThe meeting weather agenda isis" — a meaningless string

The same thing happens inside programs. Thread-safe code prevents this problem through mechanisms like "others wait while one person is writing" (locking) or "use separate notebooks in the first place" (instance isolation).

In this case, the TCP/SSL connection (= notebook) held by httplib2.Http was being used simultaneously by multiple threads (= people), corrupting the communication data.

httplib2 is a Python HTTP client library in the same space as requests and httpx. It appeared in 2006 and at the time had advanced features like cache control and automatic authentication handling that weren't available in the standard library's urllib. google-api-python-client has depended on this library from the beginning, and it is still used as the default HTTP transport today.

In modern Python development, requests and httpx are the mainstream HTTP clients, but the major difference from these is thread safety:

requests.Session is also not thread-safe, but since requests is typically used with module-level functions like requests.get(), it's less likely to be a problem. On the other hand, httplib2 is designed to be used by explicitly creating an Http() instance, and since google-api-python-client hides that instance inside the service, it's easy to miss that it's being shared.

Why It Breaks

httplib2.Http holds TCP/SSL connections internally, but has no mechanism for safely sharing them between threads. When two threads call .execute() simultaneously:

1. Thread A begins a TLS handshake
2. Thread B sends a different request through the same socket
3. The TLS state machine becomes inconsistent → WRONG_VERSION_NUMBER

Or:

1. Thread A sends a request and waits for a response
2. Thread B sends a request on the same socket
3. Thread A reads Thread B's response → IncompleteRead

At 1–2 requests/second it may work by chance, but sending 10 requests simultaneously will break it with high probability.

Checking Google's Official Documentation

The official thread safety documentation for google-api-python-client states clearly:

The httplib2.Http() objects are not thread-safe. If you are running as a multi-threaded application, each thread that you are making requests from must have its own instance of httplib2.Http().

The official documentation recommends two solutions:

Approach 1: Override requestBuilder

def build_request(http, *args, **kwargs):
    new_http = google_auth_httplib2.AuthorizedHttp(credentials, http=httplib2.Http())
    return googleapiclient.http.HttpRequest(new_http, *args, **kwargs)

service = discovery.build('api_name', 'api_version',
                          requestBuilder=build_request, http=authorized_http)

A method that builds the behavior of "create a new Http for each API call" directly into the service.

Approach 2: Pass http to execute()

http = google_auth_httplib2.AuthorizedHttp(credentials, http=httplib2.Http())
service.spaces().messages().create(...).execute(http=http)

A method that explicitly passes an Http instance at the time of the .execute() call.

Comparing Five Approaches

In addition to the official documentation, I researched DoIt's production operation article and GitHub Issues, and compared five approaches.

Why Approach 1 (AuthorizedHttp per Instance) Was Chosen

In this bot, a new ChatApiClient() is created each time process_message() in worker.py is called. Since each process_message() runs in an independent background thread, giving each ChatApiClient instance its own HTTP connection naturally achieves isolation between threads.

During a single pipeline execution (1 create + several patches), the same AuthorizedHttp is reused, so connection reuse is also achieved.

The connection pool (Approach 3) was introduced in DoIt's blog and was attractive, but the maximum number of concurrent threads in this bot is about 10. I decided it wasn't a large enough scale to justify adding pool management code.

Implementation

The change is to a single file, bot/chat_api.py. No new dependency packages are added (google_auth_httplib2 and httplib2 are already installed as transitive dependencies of google-api-python-client).

# bot/chat_api.py (after fix)
import google_auth_httplib2
import httplib2

_default_service = None
_credentials = None  # Added: hold credentials at module level
_lock = threading.Lock()

def _get_default_service():
    global _default_service, _credentials
    if _default_service is None:
        with _lock:
            if _default_service is None:
                _credentials, _ = google.auth.default(scopes=SCOPES)
                doc = json.loads(_DISCOVERY_DOC_PATH.read_text())
                _default_service = build_from_document(doc, credentials=_credentials)
    return _default_service

def _build_http():
    """Generate an independent HTTP connection per instance"""
    return google_auth_httplib2.AuthorizedHttp(_credentials, http=httplib2.Http())

class ChatApiClient:
    def __init__(self, service=None):
        if service is None:
            self._service = _get_default_service()
            self._http = _build_http()  # Each instance has its own HTTP connection
        else:
            self._service = service
            self._http = None  # For testing: use mock service as-is

    def create_message(self, space_name, body, thread_name=None):
        # ... (omitted)
        response = (
            self._service.spaces()
            .messages()
            .create(**kwargs)
            .execute(http=self._http)  # ← This is the key point
        )
        return response["name"]

    def patch_message(self, message_name, body, update_mask):
        return (
            self._service.spaces()
            .messages()
            .patch(name=message_name, updateMask=update_mask, body=body)
            .execute(http=self._http)  # ← Here too
        )

Impact on Tests

Existing tests inject mocks like ChatApiClient(service=mock_service). In this case, self._http = None, and .execute(http=None) falls back to googleapiclient's default behavior (using the service's built-in HTTP). Since MagicMock ignores http=None and returns return_value as-is, all 98 tests passed without any changes.

Verification

Local Testing

$ uv run pytest -v
============================= 98 passed in 1.05s ==============================

Deploy and Spam Test

$ gcloud functions deploy google-chat-bot \
    --gen2 --runtime=python314 --region=asia-northeast1 \
    --source=. --entry-point=handle_chat --trigger-http \
    --no-allow-unauthenticated --memory=2Gi --cpu=1

After deploying, sent 10 messages in rapid succession within a few seconds in Google Chat. Results:

• Before fix: 3–5 out of 10 messages returned error cards
• After fix: All 10 messages returned answers normally

Log Check

$ gcloud functions logs read google-chat-bot \
    --region=asia-northeast1 --gen2 --limit=50 \
    --start-time="2026-06-15T06:31:00Z" \
    | grep -iE "error|failed|SSLError|TimeoutError"

# No output — zero errors

Summary

When calling APIs from multiple threads using google-api-python-client, sharing httplib2.Http will definitely break. While it's explicitly stated in the official documentation, creating a service with a singleton pattern naturally leads to sharing, making it an easy trap to fall into.

The fix is simply passing an independent AuthorizedHttp to .execute(http=...). This time I adopted the approach of isolating HTTP connections per ChatApiClient instance, but depending on scale, a connection pool may also be worth considering.

I hope this is helpful for anyone who has fallen into the same problem.

References

• Thread Safety | google-api-python-client Official Documentation
• Using Google API Python Client in Production-Grade Apps | DoIt
• Thread safe of httplib2 and credentials · Issue #808