Cluster size in Windows

[Karim Ghantous]

My main drive (on my Windows machine) was formatted with 64KB file

allocation units (yeah, I like extremes, so sue me). I was recommended

that this could be a performance advantage (or was it something

else?). Anyway I don't mind the 'wasted' space that results (because a

file on the disk has a lot of overhead associated with it). But I

haven't noticed any real-world advantages.

 

Some weeks ago I tested a drive and formatted it with 4KB units and

then with 64KB units. In each case I did a file transfer test. The

file was just a large video file or something. When the drive was

formatted with 64KB units the transfer speed advantage was negligible.

 

Is it worth it to have such large FAUs? If the larger units help to

protect the data (e.g. photographs) then I'm all for them. I'd like to

see what you all think.

[beauh44]

Hi Karim, I believe large block sizes (or FAU's) work best with large files, and vice-versa. If your average image file-size is considerably less than 64KB, then you probably won't derive much of a benefit. This is usually a little more of a concern in a file-server environment where the server is routinely shoveling around quite a few bits and the modest performance gains can add up much more quickly, so it's not too surprising that you didn't see much of a real-world benefit with your test. Perhaps taking an average of your image file-sizes and adjusting the FAU's accordingly might help. Having too large of a block-size can just waste disk space if your typical image size is considerably smaller than what you've allocated. Best wishes . . .

[jbq]

The "cluster size" has nothing to do with protection. When a sector dies, it does, and it doesn't matter whether it was part of a 4KB or 64KB cluster. It's a space vs. speed tradeoff that only becomes visible in some edge cases.

 

A big cluster size is an advantage if you have many files which all get resized by small increments (it reduces file fragmentation), but the cost is a big overhead if you store many files (64KB vs 4KB wastes 30KB per file on average). I wouldn't expect any significant difference in performance as long as fragmentation isn't involved. Put in some fragmentation and things will be different.

 

Some filesystems even have strategies that allow to get the best of both worlds, by using a small cluster size to reduce overhead but by growing files in big chunks to avoid cases of extreme fragmentation. The filesystem used by BeOS does exactly that, and it works wonders.

 

If you deal with few files (say, fewer than 10000), you can use big clusters. With many files (say, one million), you want smaller clusters. If you expect that you'll just write files and not resize them afterward, go for small clusters. If you expect to often resize files, go for big clusters. If you expect that you'll often delete files, go for big clusters. If you expect that you'll delete few files, go for small clusters. If you have a defragmenter, go for small clusters.

 

Honestly, the consequences are so small in general that I wouldn't worry about it. It's like asking whether you'll get more sharpness with a 50/1.4 at f/6.7 or f/8.

[oskar_ojala]

I'll add to Jean-Baptiste's good answer: cluster size does reduce performance penalties of a fragmented drive. Studies done indicate that the average file sizes in Unix environments are about 1-4 KB, so cluster sizes should be set accordingly. However, if you have on disk/partition solely for photo use, then setting a large cluster size is good, since it improves performance when fragmented and block management becomes simpler, which will, depending on your file system, affect performance on a few critical sizes (assuming I don't remember the workings of NTFS completely wrong.)

 

However, cluster size has a negligible impact since there are so many factors at play: the filesystem used, the fs driver, the OS, the disk, the disk controller etc., so in practice cluster size is nothing worth losing sleep over.

[gordonr]

Cluster size is directly relevant to the total size of the FAT (File Allocation Table), and inversely proportional to the amount of overhead processing involved in file I/O (fragmented files). However, with today's CPUs and RAM available, cluster size is unlikely to have any performance effects.

 

If you make the cluster size too large and have many hundreds of thousands of files, you may run out of clusters before the disk is full, because of the wasted space.

 

Cluster size has no effect on the underlying hardware, and the biggest effects of disk performance is latency (the time taken for one rotation of the disk, i.e. RPM), and track-to-track seek-times.

[beauh44]

This is from the "horse's mouth", otherwise known as Microsoft: "If the files are typically smaller than the default cluster size (for example, 4 KB) and do not increase, use the default cluster size to reduce wasted disk space. However, smaller clusters can increase fragmentation, especially when files grow to fill more than one cluster. Therefore, adjust the cluster size accordingly when you format the volume. If the files you store tend to be large or increase in size, use 16- or 32-KB clusters instead of the default 4-KB cluster size."

 

The link is one of those easy-to-remember ones:

http://www.microsoft.com/resources/documentation/Windows/XP/all/reskit/en-us/prkc_fil_punq.asp

 

I would think that if your image files are very large - for example scanned medium-format slides or negs - that would certainly qualify you for the first criteria for considering a large block-size: files that "tend to be large". The second criteria - files that "increase in size" certainly would seem to go hand in hand with multi-layered work in Photoshop, not to mention a lot of resizing for web and print.

 

I don't think the advice that "the largest cluster size is best for everyone" is very sound. However, it seems that Microsoft is recommending that you evaluate your own situation and I'd tend to agree because image files are typically large. You might see a modest increase in performance by bumping it up from the default of 4K and keeping your disk defragged. Here's another link with a few tips for XP-ers out there: http://www.tweakxp.com/. Best wishes.

[steven_clark]

Large clusters would only prevent fragmentation on files that will increse in size little enough that they will not need to allocate additional clusters when they increase in size. For photo quality image work this should be effectively nill.

[scott_eaton]

Cluster size has more to do with the drive controller optimization than anything else. RAID is where it tends to make a difference. Not single drives.

[Karim Ghantous]

Thanks, guys.