I tried finding disappeared threads in a frozen Python process using py-spy

Introduction

A process involving a resident process froze with CPU at 0%. The process was alive but nothing was progressing, and no obvious errors appeared in the logs.

This article introduces my experience of peering into this Python process using py-spy. To state the conclusion upfront, the cause was "one thread that was supposed to read standard output had terminated, leaving nobody to read the output." I will walk through the steps of noticing that absence and tracing it back to the reason for termination from a single line in the startup log.

What is py-spy

py-spy is a sampling profiler that can extract the currently executing stack from a running Python process externally without stopping it. Using the dump subcommand, you can list the stacks of all threads at that moment.

Target Audience

• Those who are stuck investigating a freeze in a process involving Python
• Those who want to know what is happening inside a daemon or resident service without stopping the process
• Those who have heard of py-spy by name but have never used it for freeze investigation

References

• py-spy (GitHub)
• Python codecs error handlers (strict and replace)
• Python subprocess (encoding and errors)
• Anonymous pipe fullness and blocking

Problem: Cannot See Inside a Process That Is Alive but Idle

The frozen Python process was stuck with CPU at 0%. The process itself was alive, and no glaring errors appeared in the logs. It was alive but nothing was progressing.

To find the cause, I wanted to look at the threads inside, but this turns out to be surprisingly difficult. Restarting the resident process causes the state at that moment to disappear. Attaching a normal debugger to set breakpoints stops the target, and the act of stopping it can change the symptoms. I needed a way to observe the state of threads while keeping the process alive and running.

Investigation: Finding the Missing Thread with py-spy

So I used py-spy. I extracted the stacks of all threads without stopping the target and examined them.

Extracting All Threads with py-spy

Run dump as follows. Adding --native lets you see not only Python frames but also native frames from C extensions and the like. This is effective when a read operation is blocked on the C side.

py-spy dump --pid <pid> --native

The output lists all living threads, each with their current stack.

A Thread That Should Exist Is Not There

While looking through the dump, I noticed that a thread that should have been there was missing. This daemon was supposed to have a thread that continuously reads the standard output of a subprocess, but it was nowhere to be found. Meanwhile, the thread reading standard error was alive and paused waiting for input.

Thread (idle): "MainThread"
    ...
Thread (idle): "stderr reader"
    readline (...)
# The "stdout reader" thread is absent from the list

The absence of the output reading thread means nobody is reading the subprocess's standard output. The value of py-spy lies in its ability to notice this kind of "something that should be there is not."

Confirming the Reason for Termination in the Startup Log

The reason the thread had disappeared was preserved in the daemon's startup log.

Exception in thread stdout reader:
Traceback (most recent call last):
  ...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc7 in position 2: invalid continuation byte

The cause was that the subprocess's standard output was being read as strict UTF-8. When a single unexpected byte (here, 0xc7) arrived, a UnicodeDecodeError was raised, and that exception brought down the reading loop thread along with it.

Investigation Results and Response

It became clear that the cause was a thread reading standard output terminating due to an exception, after which nobody read the output anymore.

When the reader stops, the subprocess's standard output pipe (a few tens of KB by default) eventually becomes full. Once full, the subprocess blocks on writing, which in turn causes processing in upstream processes to stop returning. The freeze that appeared on the surface was the result of this chain reaction.

A loop that reads the output of an external process as text is safer with lenient decoding. Replacing invalid bytes with a replacement character (errors="replace"), or receiving the data as bytes and decoding explicitly, prevents the accident of an entire thread crashing due to a single unexpected byte.

After making the decoding lenient so the thread would no longer terminate, the pipe was continuously drained and the CPU 0% freeze stopped reproducing.

Summary

Using py-spy, you can extract the stacks of all threads from a frozen Python process without stopping it. The key to solving this case was noticing the absence of "a thread that should have been there." Making the decoding lenient in loops that read the output of external processes prevents the accident of an entire thread crashing due to a single unexpected byte. I hope this article serves as a helpful reference when you find yourself stuck investigating a freeze.