I tried finding disappeared threads in a frozen Python process using py-spy

Introduction

A process involving a resident process got stuck at 0% CPU. The process was alive but nothing was progressing, and no obvious errors appeared in the logs.

This article shares my experience of peering into this Python process with py-spy. To state the conclusion upfront, the cause was "one thread that was supposed to read standard output had terminated, and nobody was reading the output anymore." I'll walk through how I noticed that absence and traced it back to a single line in the startup log to find the reason for the termination.

What is py-spy

py-spy is a sampling profiler that can extract the currently executing stack from a running Python process without stopping it. Using the dump subcommand, you can list the stacks of all threads at that moment.

Target audience

• People who are stuck investigating a frozen process involving Python
• People who want to know what's happening inside a daemon or resident service without stopping the process
• People who have heard of py-spy by name but have never used it for freeze investigation

References

• py-spy (GitHub)
• Python codecs error handlers (strict and replace)
• Python subprocess (encoding and errors)
• Anonymous pipe capacity and blocking

Problem: Can't see inside a process that's alive but idle

The frozen Python process was stuck at 0% CPU. The process itself was alive, and no dramatic errors appeared in the logs. It was alive, but nothing was progressing.

To find the cause, I needed to look at the threads inside, but this turned out to be surprisingly difficult. Restarting the resident process destroys the state at that moment. Attaching a normal debugger with a breakpoint stops the target, and the act of stopping it can change the symptoms. I needed a way to observe the threads while they were still alive, without stopping the process.

Investigation: Finding the missing thread with py-spy

So I used py-spy. I extracted the stacks of all threads without stopping the target and examined them.

Extracting all threads with py-spy

Run dump as follows. Adding --native lets you see not only Python frames but also native frames such as those from C extensions. This is effective when a read is blocked on the C side.

py-spy dump --pid <pid> --native

The output lists all living threads, each alongside its current stack.

A thread that should have been there was missing

What I noticed while looking through the dump was that a thread that should have existed was gone. This daemon is supposed to have a thread that continuously reads the subprocess's standard output, but it was nowhere to be found. Meanwhile, the thread reading standard error was alive and stopped, waiting for a read.

Thread (idle): "MainThread"
    ...
Thread (idle): "stderr reader"
    readline (...)
# The "stdout reader" thread is absent from the list

If the output reader thread is missing, it means nobody is reading the subprocess's standard output. The value of py-spy is precisely that it lets you notice this kind of absence — something that should be there but isn't.

Checking the startup log for the reason behind the termination

The reason the thread had disappeared was left in the daemon's startup log.

Exception in thread stdout reader:
Traceback (most recent call last):
  ...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc7 in position 2: invalid continuation byte

The cause was that the subprocess's standard output was being read as strict UTF-8. The moment one unexpected byte (in this case 0xc7) arrived, a UnicodeDecodeError was thrown, and that exception brought down the entire reader loop thread.

Investigation results and remediation

It became clear that the cause was the stdout reader thread terminating with an exception, after which nobody was reading the output anymore.

When the reader stops, the subprocess's standard output pipe (a few tens of KB by default) eventually fills up. Once full, the subprocess blocks on its write call, and as a result, the processing of even further upstream processes stops returning. The freeze that appeared on the surface was the end result of this chain.

A loop that reads output from an external process as text is safer with lenient decoding. Either replacing invalid bytes with a replacement character (errors="replace") or receiving the data as raw bytes and decoding explicitly will prevent one unexpected byte from taking down the entire thread.

After making the decoding lenient so the thread no longer terminated, the pipe was continuously drained and the 0% CPU freeze stopped reproducing.

Summary

Using py-spy, you can extract the stacks of all threads from a frozen Python process without stopping it. The decisive clue this time was noticing the absence of a thread that should have been there. Making the decoding lenient in loops that read output from external processes helps you avoid accidents where a single unexpected byte brings down the entire thread. I hope this article proves useful when you find yourself stuck investigating a frozen process.