Just wanted to share something that’s happened a couple of times while downloading very large files (>10gb) via qBitTorrent. The problem manifests as the entire system or at least every application appearing to hang (showing not responding). The clue to the problem is very high kernel time in Window’s task manager (View->Show Kernel Times) and drilling into Resource Monitor’s ‘Disk’ tab – if you sort by ascending order of ‘Response Time’ a handful of the files being downloaded will show enormous response times (thousands of milliseconds). If you see this then congratulations: these files have become cursed.

The fix is simple, restart the computer, then ensure that “pre-allocate space for all files” is checked in qBitTorrent config, and any pre-existing cursed files should be moved to a different drive and back again, which un-curses them. Keep resource monitor open in the background for a few days in case you missed one of the cursed files.

The root cause of the problem is faults in the design of NTFS sparse files. Being sparse means that when a file is created, rather than pre-allocating empty space for the file according to the file’s size, the file system only allocates space for parts that have been written to. For smaller files and large files that grow from start to end linearly, this works nicely. For very large files being written in small chunks in random order, this doesn’t work so well. Eventually as the large, randomly written file grows, NTFS needs to do housekeeping, this house keeping grows rapidly in complexity until each operation takes minutes of CPU time.

In a well designed operating system, the fact that one file being written by one process is taking a lot of house keeping time shouldn’t impact other processes, however eventually in Windows this leads to some sort of priority inversion and blocks the network stack, leading to hanging browsers, and eventually ends up blocking all file-system reads and writes.

Using Contig -a will tell you that the files have 100k+ fragments. However, Contig itself doesn’t seem to be able to unfragment the files. The fix is to move them to another drive and back again. On later versions of Windows (8.1 and up) – when creating drives it’s possible to use the Format /L option, which will make this less likely to be a problem. But I don’t believe it’s supported in Windows 7.

Remember to leave resource monitor on the disk tab running in the background once you think you’ve fixed the problem, so that if you hit another hang, you can identify which files you need to fix.

Good luck!