32 vs 64 bit, file sizes etc

<p>In a recent post, Shun observed of 32 bit Windows "... limited to 3G of RAM. That is not going to do so well with those huge D800 NEF files."<br>

Can someone enlighten on the arithmetic of memory space required for file processing of single images (ie not batch jobs) ?<br>

A D800 NEF file is I understand about 50MB. I know that during processing significantly more memory space than that is used up, but 3GB is about 60 times that much.<br>

This post was prompted by me recently doubling my RAM to 8GB (Win7 64 bit) and finding that it made virtually no difference to any photo processing except possibly big batch jobs.</p>

<p>First of all, Shun was right in confirming that any 32-bit operating system cannot access more than 3GB of RAM, thereby severely limiting the memory available for any application running under that operating system and any files opened and processed through that application. As for the mathematics, here are some basic concepts.</p>

<p>Out of the 3GB, approximately 1GB is pretty much automatically taken by the operating system and the various background tasks it performs, even when you're not really using your computer. This leaves approximately 2Gb. Even if you open one application (say Photoshop), that will take (assuming you're using CS4) another 700MB-1GB, leaving you with 1GB. Out of that, PS will also grab an additional 250MB (approximately) to use as buffer, instruction and preview scratch etc. This means you're left with, well, around 700MB. A 50MB D800 NEF file will almost certainly translate to a 150MB TIFF or PSD file when opened in Photoshop. Now, if that was the case, all might have been okay, but PS does not really work like that.</p>

<p>It will require, in order to avoid having to write and then subsequently read image data to and from your scratch disk (which is usually set to be your hard drive), free RAM approximately to the tune of file size * channels, meaning that for an RGB file, PS will need approximately 450MB of free RAM. And that is BEFORE you add a layer or any type of post processing.</p>

<p>So, with 3GB, opening a single file will almost certainly cause performance problems...and that is without having a single other application open.</p>

<p>I cannot tell why you find no difference in performance - back in the days when I updated my system from 4 to 8GB I found a significant improvement in performance...</p>

<p>Chris, I think it very much depends on the image-processing software used. I've always found that Photoshop's memory management is very poor, and even after images have been closed, PS hangs onto the memory previously used like a limpet. The only way to get it to release space is to close and restart the entire program.</p>

<p>The panic and stress over D800 file and image size is greatly exaggerated IMHO. I used to (and occasionally still do) scan 5x4 negatives at 2400 ppi and 16bit depth. Now THAT's what I call a huge file size at over 600 Megabytes. OK it slows things down a bit, but I never had any trouble using a 32 bit OS to process the image. Same with scanning 645 negs and slides at 4800 ppi/16bits. You end up (eventually - the scanning itself ain't fast) with a file size of around 500 Megabytes. So, a piffling 200 or so Megabyte D800 RAW file shouldn't give any decent computer indigestion, nor its owner a minor heart attack.</p>

<p>Search the MF forum. You don't read people constantly bleating about how their OS can't handle their scans. But then MF users generally don't machine-gun their cameras at 6fps and end up with 6,000 shots per session to open and edit.</p>

<p>WRT to memory used Chris; although the NEF file off the camera is only 50 megabytes, the uncompressed image data expands to fill around 206 Megabytes, and as a minimum I would expect an image processor to keep at least one mirror of the file in RAM plus working space equal to another copy of the image. Say 3 or 4 times the expanded image size in total, which if my maths is correct is around 828 Megabytes. That's still a long way short of 3GB, not counting virtual memory, which should be set and fixed to around 3 times the RAM size. Then there's the "scratch disk" memory of Photoshop available as well. A modern SATA disk system should cope without a sweat, especially if setup as a striped RAID array.</p>

<p>The issue isn't how much RAM a single image file is using, but how much RAM in total is being used by everything in the computer. If there are applications, background, tasks, browser tabs, widgets, whatnots, etc. running, then loading one image may pass the threshold of available physical RAM and things have to be written out to virtual (hard drive) RAM. Different operating systems/versions of Windows manage memory differently, and the 32/64 bits is the length of the address for the RAM. 32 bit is 32^2 addresses and 64 bit is 64^2 addresses: 64 bit gives a much bigger address space, so more RAM can be used.</p>

<p>Shun's observation is a truism, but it's truth is dependent on each individual's computer usage.</p>

<blockquote>

<p>A modern SATA disk system should cope without a sweat, especially if setup as a striped RAID array.</p>

</blockquote>

<p>RAM is Random Access Memory and rotating storage media is Sequential Access Memory. Once access times, interrupts, and actual, transfer rates are factored in, it becomes obvious to the casual observer why "virtual" memory wants to be avoided: it's about 1/10 the speed of real, physical RAM.</p>

<p>Actually, it is 2^32, not 32^2. 2^32=4GB. I'm not sure where the 3 GB figure comes from, but perhaps it is because less than the full 4 GB is usable by apps. There are computer designs that allow a 32-bit processor to access more than 4 GB, but microsoft lists the limit of Windows 7 32-bit as 4 GB.<br>

Have you tried changing the application's settings, like the cache size?</p>

 

<p>Bruce explanation is quite a bit better than Marios Forsos;</p>

<p>There are actually two different parameters; 64 bit kernel (the machine can access 2^64 addresses) and 64 bit applications (which can only run on 64 bit kernels). Most applications are in fact 32 bit.</p>

<p>The D800 file sizes are no where close to 1GB much less 4GB; but the application still might use quite a bit of memory dependent on many factors.</p>

<p>Upgrading to a 64 bit kernel will not increase the address space for 32 bit applications but it will allow for more memory that can be used by other applications (web browser and similar). If you primarily use your machine for a single application; there is still some benefit of a 64 bit kernel over 32 bit kernel (mostly because windows reserves quite a bit of address space for internal use) but those benefits are less significant.</p>

<p>One thing to keep in mind is that windows paging (and linux for that matter) tends to be very inefficient with regards to iops; so if you are forced to page (do not have enough real memory) things will be incredibly slow. This is somewhat tangent to your question; but most people who observe a 'slow' system do so due to lack of memory (this is one reasons laptops tend to be slow; because laptop disks are significantly slower than desktop disks; though this is changing as ssd become more common).<br /> BTW; you actrually answered your own question. You found that D800 files were not that large (relative to 3GB); and therefore you saw little performance difference when you upgraded memory. Even better if you run task manager you can watch the amount of memory being used by both photoshop (if that is your chosen software for photo processing) as well as the system as a whole. If you are not using more than 3GB in most cases then it makes sense that upgrade would not boost performance. Btw the performance gain (if you needed more memory) would be non trivial as swapping is very very slow relative to direct ram access.<br /> Last but least when you have a 32bit system; and 4GB of real memory; you simply cannot swap (this is a slight lie; but lets ignore that detail)*. I.e, what you would expect is that some applications are either forced to shutdown or refused to run if you had memory issues; not the system running slower.</p>

<p>*(some software supports Physical Address extension (PAE); which does increase the amoutn of memory the kernel can use - 64GB is common but there are other values; I do not believe windows xp has PAE support turned on by default; though it can be easily enabled)</p>

<p>I do not understand your question. Are you asking about a file or about operating system ?<br /> A file is a collection of bytes, and regardless of operating system, 32 bit or 64 bit, it occupies certain space on hard disc.</p>

<p>How the file is used in 32 or 64 bit system is different, and it was already explained a bit by others.</p>

<p>So, your picture file is independent of 32 or 64 bits, and can be processed on both systems, if system limitations are not exceeded, and the D800 still picture files will not exceed 32 bit system limitation.</p>

<p>For practical reasons, some longer HD video clips from HD camcorders that do not impose limitation on length of single video clip, will exceed 32 bit systems capability, and the video file canot be played or edited, and a 64 bit system is needed.</p>

<p>For D800 produced picture file, is far away from 32 bit system limitation, and any system will process your D800 output.</p>

<p>I think some of you have gone a bit off track !<br>

The question summarised was why does processing a photo file, such as a 50MB NEF, need so much memory that it might benefit from more than 4GB RAM.<br>

My recent experience suggests that it does not, HOWEVER it seems other information is pertinent:<br>

1. I have several different user logons on my PC - the one I use for photo handling is dedicated to that task and consequently has the minimum of other tasks running at the same time.<br>

2. I use Capture NX2, not any Adobe product, and maybe that has a better grip on the memory use.</p>

<p> </p>

<p>OK. I just opened the same ~10,000 x 8,000 pixel scan from an MF colour neg onto two systems. The 16 bit uncompressed TIFF file is reported as 600MB on disk. That's around 3 times the expanded size of a single D800 file. <br>

The file was opened in CS3 on a system running Win XP 32 SP3, and in CS5 (X64) on a Win7 64 bit system. The 32 bit system was maxed out at 3.12GB of RAM, while the Win7 OS has 8GB RAM. Also the Win XP system has a slower dual-core 2.7 GHz processor and the Win7 system has a 4-core 3+GHz processor. Oh, yes the Win7 system also has a faster graphics card with more video memory and is set up to use the GPU for some image processing tasks. So it should be no contest, right? Well, er, not quite!</p>

<p>I did the same sequence of Invert, Auto Levels, Curves adjustment and USM filter on the two systems.<br>

The "table" below shows the timings I got:<br>

Process ________X64 time (secs)_____32 bit time (secs)<br>

Invert _____________<2______________2 <br>

Auto levels __________2_______________5<br>

Curves _____________2_______________4<br>

USM _______________5______________7<br>

Not a lot of difference really, considering the extra CPU power and memory available to the 64 bit system. However, the real shocker was the time to open the file; 12 seconds for the 32 bit system and 25 seconds for the 64 bit - and no, I haven't got that the wrong way round. That extra loading time completely annihilates any advantage the 64 bit system had! Not sure why, since the same USB2 external drive was used with both systems. I'll need to investigate further.<br>

Also at no time did the 32 bit system appear to use a scratch disk or virtual memory.</p>

<p>Of course YMMV.</p>

<p>Well you should test how long it takes to the read the file on both system (for linux I woudl use dd if=file of=/dev/null; but not sure of a good method with windows :) ). The persumption is that it is an issue with usb configuration or how the software is accessing the disk (for example small reads would kill performance).</p>

<p>If the user (not relevant to your synthetic test) had 4 or 5 photo sessions open or a number of browsers open or whatever such that they ended up using 6GB of ram for user applications; then I'm sure the test would show a big win for the 64 bit system :)<br>

-</p>

<p>It's a given that if you are not using a page file and you have ample memory for all your application that using 64bit address space provides no benefit. Heck using more than 1GB has no benefit (if you are not using a page file and you are not measuring the benefit of buffered data). Naturally with newer os and applications i might be difficult to fit everythign in 1GB - esp if you are editing 600MB tiff files...</p>

 

<p>Every 32-bits process on Windows is limited to a maximum of 2GB. This limit is regardless of the amount of memory installed. This even affects 32-bits software running on 64-bits Windows. Photoshop, for example, is a process (plugins it spawn run within that process space normally). So, the limit on 32-bits windows for any application is by definition 2GB.<br>

All the "calculations" sound nice, but Windows shuffles quite some memory around. It can write quite a lot to the swap-file and use the RAM instead to cache the access to the swap-file for faster access. It's not as static as the calculations here seem to imply.</p>

<p>In the other thread, I already shared a very simple experiment. I downloaded a D800 NEF from dpreview, loaded it in CaptureNX2 on my oldish Vista 32-bits system with 4GB of RAM and it ran fine. Did normal actions such as sharpening, zooming in and out, B&W conversion, selective colourpoints etc. No problem, and in fact not much slower than my D300 files. I did not run out of memory at all. Yes, the D800 won't mind a PC with 16GB of memory and 64-bits applications. No, it's not demanding it.</p>

<cite>I'm not sure where the 3 GB figure comes from, but perhaps it is because less than the full 4 GB is usable by apps. There are computer designs that allow a 32-bit processor to access more than 4 GB, but microsoft lists the limit of Windows 7 32-bit as 4 GB.</cite>

 

<p>The full story is quite complex, so here's a <em>very</em> simplified version. First, let's look at the software limits imposed by Windows.</p>

 

<p>By default, a 32-bit Windows app gets a maximum of 2 GB of memory for the app itself, on any version and edition of Windows from the very first version of NT in 1993 right up to the present. Some apps are "large address aware", and that changes things a bit: on 32-bit Windows versions in the last decade or so, if you specify the /3GB switch in your boot configuration (Google will find you the details, including warnings), such an app can see 3 GB. On a 64-bit version of Windows, such an app can see 4 GB with no OS configuration required. Meanwhile, a 64-bit app (running on 64-bit Windows, since 32-bit Windows can't run 64-bit apps) can see vastly in excess of this.</p>

 

<p>On the hardware side, most PC-compatible CPUs since the mid-to-late 1990s have had the ability to access more than 4 GB of RAM, but many motherboards (particularly workstation motherboards) limit this. For instance, the computer I'm using right now has a CPU that can handle more memory, but the chipset in the motherboard cannot, so it's limited to 4 GB. But some peripherals (most notably graphics cards) have to map some of their memory into that 4 GB address space, which leaves less than 4 GB for Windows and programs. This is the mostly commonly-cited "3 GB" limit, although that's not strictly true. Windows can access about 3¼ GB on my computer, which clearly is more than 3 (but less than the 4 I actually have installed). The actual limit will vary depending on your computer hardware, but typically it's in the 3-3.5 GB range.</p>

 

<p>Like I said, that's highly simplified, but hopefully it answers the "where does this 3 GB limit come from" question.</p>

<p>Too late to edit my previous post - minor correction, it is possible to make a 32-bits program use more than 2GB, but the program needs to be developed that way, and there can be speed concerns, AFAIK.</p>

<p>So if a know-nothing might interject a question related, I'm thinking of upgrading a d200 to a d7000. Obviously those are a lot bigger files and greater bit depth if I understand correctly is one of the advantages. But is my measly 3 GB ram 5 year old vista computer running CS5 going to choke on those big files?</p>

<p>Harry, no. My 4,5-year old 4GB Vista 32-bits PC is not choking on D800 files, so the relatively modest increase from 10 to 16 MP should really be no issue.</p>

<p>That's it guys, completely ignore a practical, real life example test and concentrate on the theory. Way to go!</p>

<p>Alan, that WAS the read time I was talking about. Reading/loading - what's the difference? And even disallowing the read time, the 32 bit system didn't choke on that 600 MB file, and the processing time difference wasn't that startling. So IMHO all you new or wannabe D800 users are making a bit fuss over nothing.</p>

<p>"If the user (not relevant to your synthetic test) had 4 or 5 photo sessions open or a number of browsers open or whatever such that they ended up using 6GB of ram for user applications...." Then they'd be an idiot that deserved all they got!</p>

<blockquote>

<p>That's it guys, completely ignore a practical, real life example test and concentrate on the theory. Way to go!</p>

 

</blockquote>

<p>LOL. I appreciated the info, and your summation. That seems to have put the original question to rest very nicely. The remainder was just everyone else piling in to help explain what memory is, isn't, or doesn't. :)</p>

<p>I thought at first I would do some more of the same, but realize now that this isn't the time, place, or audience. It suffices to say here that there are several concepts all bunched up and tangled together when we start talking about "memory" space. In the end, the suggestion made of keeping the background processes to a minimum about sums up the important points.</p>

<p>I would only add, Joe, that external USB2 drives are quite a bit slower than internal SATA3 drives; a theoretical maximum 6 Gbps transfer rate versus 480 Mbps for high speed USB2. I expect the difference to be much smaller if you first copied the file to an internal drive. You might also experiment with plugging the drive into a different port. My PC, for example, has a small hub feeding the front panel, but the high speed ports are on the back panel. It also makes a difference what else is attached and active on that particular hub. External audio or a webcam on that hub will compete for bandwidth. That 480 Mbps seemed huge when USB2 first came out. The latest standard is 5 Gbps, which is pretty close to the limit possible for copper with current technology.</p>

 

<p>The USB 3.0/eSata external ports are much faster than the USB 2.0.<br /> When purchasing a new computer, make sure it has USB 3.0 ports.<br>

<br /> Up till now Express 34 / Express 54 external computer ports were needed for faster transfers on higher grade laptops.<br /> It seems that in newer laptop designs the USB 3.0 is replacing the Express 54/34 port. <br /> Perhaps because one can have multiplicity of USB 3.0 ports, while Express cards were mostly for one external device?</p>

<p>An FYI: 5 Gbps is the speed of the bus; not the speed of the disk. 1 TB disks can stream data at close to 750 Mb/s if you are lucky and reading the inner tracks. Significantly slower if reading the outer tracks or the data is (significantly) fragmented. Naturally if you are using raid you can up your transfer speed but the typical user likely has a single drive; and if a smaller size or a few years old they might be lucky to see 240 Mb/s.<br>

<br />If your external usb2 disk is a single 1 tb drive then yes usb2 might cause a bottlneck; but if your computer is a few years old and you have sata2 don't sweat it.</p>