I tried tracking down the cause of a process frozen at 0% CPU using cdb and handle64

Introduction

A long-running process stops dead at a certain point. No errors appear in the logs. The Task Manager shows CPU usage at 0%, and while the process is alive, it's doing nothing — this kind of silent freeze is difficult to trace because it leaves behind no exceptions or stack traces, making investigations drag on.

This article describes an experience of actually tracking down such a freeze. To state the conclusion upfront: the cause was "a write to a named pipe with no reader never returning." I'll walk through the journey from drawing a blank with Process Monitor to finally identifying the culprit using cdb and handle64.

What is Process Monitor?

Process Monitor (hereafter procmon) is a Sysinternals tool that records file, registry, and network operations issued by processes in real time. Since it lets you trace what was being accessed in chronological order right before a freeze, it's useful as a first step for narrowing things down.

What is cdb?

cdb is a command-line debugger included in the Debugging Tools for Windows SDK. By performing a noninvasive attach to a frozen process, you can read the stack and registers at that moment without stopping the target.

Target Audience

• People who are stuck investigating a process that freezes without any errors
• People who want to isolate a frozen native Windows process within the user-mode scope
• People who use Process Monitor but haven't yet tried cdb

References

• Getting Debugging Tools for Windows (cdb)
• FlushFileBuffers (Win32 API)
• Sysinternals: Process Monitor
• Sysinternals: Handle (handle64)
• Named Pipes

The Problem: The Frozen Operation Doesn't Show Up in procmon

A freeze at 0% CPU means the process is in a waiting state. A busy loop would consume CPU, so 0% suggests it's sleeping inside the kernel, waiting for an I/O or synchronization object to complete.

In other words, the three things to investigate are: "which thread," "in which system call," and "what it's waiting for." I first tried procmon as a tool to check this from the outside.

However, I ran into procmon's limitations. What procmon records are completed operations. A frozen process is most often in a state of waiting for an already-issued I/O to complete, so the very operation it's waiting on does not appear in the records. Even if you start capturing after the freeze occurs, you can't catch the critical operation.

In practice, even after scrutinizing a capture of over 2 million lines that should have included the frozen interval, not a single flush operation — which should have been the cause — was recorded. An operation stuck waiting never generates a completion event, so by its very nature it won't appear.

Note that there is an option to run procmon before the freeze occurs. By running procmon as a resident process, you can start recording before processing begins.

Procmon64.exe /AcceptEula /Quiet /Minimized /BackingFile C:\temp\capture.pml

Even so, the waiting operation itself won't be captured. I decided to treat procmon as a tool for narrowing down the cause, and leave the task of confirming what is being waited on to cdb.

Investigation: Identifying the Handle with cdb and handle64

From here, I'll look directly inside the process using cdb and track down exactly what is being waited on, right down to its name.

Checking the Stack with cdb

After performing a noninvasive attach to the actually frozen process and displaying the stack with kb (a cdb command that shows the call stack and the first 3 arguments of each frame), the frozen thread showed the following sequence:

ntdll!NtFlushBuffersFile
KERNELBASE!FlushFileBuffers
...

FlushFileBuffers is a function that ensures written content is flushed out to disk or similar for a given handle. Since NtFlushBuffersFile appears below it, this can be read as "this thread is waiting inside the kernel for a flush on a certain handle to complete." So I set out to identify the value of that handle.

Pinning Down the Handle Value

The first argument of FlushFileBuffers is the handle. However, which register carries that argument depends on the calling convention and the compiled output. In x64, the first argument is normally in rcx, but there's no guarantee the register hasn't been overwritten inside the function. Therefore, the reliable approach is to disassemble FlushFileBuffers, visually confirm that the argument register is preserved without being overwritten until it's passed to the system call, and then read that value.

Converting the Handle to a Name with handle64

The handle value itself is just a number, so it needs to be converted to a name. The tool I used for this was Sysinternals' handle64. It provides a list of handles held by a process and the name of the object each one points to (file paths or named pipe names).

The access mask of the handle also serves as a clue. The target handle was opened as write-only (no read, mask 0x120196). This is a characteristic that fits the write end of a one-directional named pipe rather than a local log file. Checking the name confirmed it was in the form \Device\NamedPipe\<pipe-name>, establishing that the other party was the write end of a named pipe.

It would normally be unusual for a flush to a local file to never return. On the other hand, a named pipe has the property that a write-side flush will block indefinitely as long as the reader doesn't drain it. The symptoms in this case are consistent with the latter.

In other words, the reason Process A's flush never returned was that the intended reader on the other end of the pipe (Process B) was not reading the contents. Before agonizing over why a local file flush wasn't returning, the shortcut was to first identify the true nature of the handle by name.

Investigation Results and Response

Using cdb and handle64, I was able to identify by name that the target the frozen thread was waiting on was a named pipe whose reader was not draining it. The cause lay not in Process A, but on the Process B side.

Shifting the investigation to Process B, one thread that was supposed to be reading the pipe had silently terminated, which had created the condition where there was no reader. Once the issue of that thread's termination was resolved, the 0% CPU freeze stopped occurring.

Summary

By combining cdb and handle64, I was able to identify by name that what a process frozen at 0% CPU was waiting on was a named pipe with no reader. While procmon is useful for narrowing down the cause, incomplete I/O during a freeze is not recorded, so pinning it down required a noninvasive cdb attach, disassembly rather than guesswork, and handle name resolution via handle64. I hope this article proves useful when you find yourself stuck in a freeze investigation.