Windows Hangs: Cursed NTFS Files

Just wanted to share something that’s happened a couple of times while downloading very large files (>10gb) via qBitTorrent. The problem manifests as the entire system or at least every application appearing to hang (showing not responding). The clue to the problem is very high kernel time in Window’s task manager (View->Show Kernel Times) and drilling into Resource Monitor’s ‘Disk’ tab – if you sort by ascending order of ‘Response Time’ a handful of the files being downloaded will show enormous response times (thousands of milliseconds). If you see this then congratulations: these files have become cursed.

The fix is simple, restart the computer, then ensure that “pre-allocate space for all files” is checked in qBitTorrent config, and any pre-existing cursed files should be moved to a different drive and back again, which un-curses them. Keep resource monitor open in the background for a few days in case you missed one of the cursed files.

The root cause of the problem is faults in the design of NTFS sparse files. Being sparse means that when a file is created, rather than pre-allocating empty space for the file according to the file’s size, the file system only allocates space for parts that have been written to. For smaller files and large files that grow from start to end linearly, this works nicely. For very large files being written in small chunks in random order, this doesn’t work so well. Eventually as the large, randomly written file grows, NTFS needs to do housekeeping, this house keeping grows rapidly in complexity until each operation takes minutes of CPU time.

In a well designed operating system, the fact that one file being written by one process is taking a lot of house keeping time shouldn’t impact other processes, however eventually in Windows this leads to some sort of priority inversion and blocks the network stack, leading to hanging browsers, and eventually ends up blocking all file-system reads and writes.

Using Contig -a will tell you that the files have 100k+ fragments. However, Contig itself doesn’t seem to be able to unfragment the files. The fix is to move them to another drive and back again. On later versions of Windows (8.1 and up) – when creating drives it’s possible to use the Format /L option, which will make this less likely to be a problem. But I don’t believe it’s supported in Windows 7.

Remember to leave resource monitor on the disk tab running in the background once you think you’ve fixed the problem, so that if you hit another hang, you can identify which files you need to fix.

Good luck!

Sony PMW-EX1 Maximum SD Card Size: 78.9GB

I dusted off my Sony PMW-EX1 Camera this month to shoot a bit of work-related video, I’d last used it a few years ago, and I was curious to see how well it aged. Well the first casualty was the Sony brand battery which couldn’t hold a charge at all. Everything else however seemed to be working fine. A replacement battery from SWIT seems to have sorted things out.

The next problem was getting some fresh storage for it. When I originally worked with the camera I’d used a combination of 16gb and 8gb SxS cards and few awkward SD Card adaptors. However I decided to try out a Sonnet SD card adaptor which allowed the data compartment door to be closed.

The next experiment was to try a variety of different SD card sizes to see what the largest size my latest firmware EX-1 could handle. The good news was that even cards up to 512GB work, however only the first 78.9GB seems to be seen by the camera.

It’s certainly an odd one.

In summary: there isn’t a huge advantage to buying SD Cards bigger than 64GB for the Sony PMW-EX1, but there isn’t any harm either. 78.9GB is enough for nearly five hours of highest quality footage – certainly more than enough for my needs.

I’m hoping the camera will continue to be useful for a few more years, however, the lack of 1080p60 is starting to feel a bit limiting in this youtube era.

FTC Disclosure: I am an Amazon affiliate advertiser – clicking on one of the links above and buying something may result in me being paid referral fees.