How to manage those massive files?

[guido_h]

<p>I shoot and scan 6x6 color negatives in a hybrid workflow. Scanning those negatives on my Hasselblad Flextight scanner at max. resolution (3200dpi/16 bits) results in 6800px x 6800px TIFF files which are some 300MB large on average. Add some adjustment layers in Photoshop, and I'm left with image files half a gigabyte in size EACH. Yikes!!</p>

<p>I'd like to have something more storage efficient here - like my digital camera snaps, which I manage with Lightroom and which occupy around 38MB each as DNG files. Unfortunately, LR does not provide the full range of features I need for film processing (like dust spotting, dodge/burn with layers etc.), which is why I use Photoshop for my scanned film.</p>

<p>But still - is there anything I can do to bring this digital deluge down to more manageable proportions? A 16-bit file format other than TIFF? A different workflow? I already thought of discarding the actual image data (the background layer) in the Photoshop files and keeping the adjustment layers only since I can always rescan the original negative if needed. However, then I'd have to do pixel operations (like dust spotting) all over again each time. Or should I just resign to the fact that in order to keep playing in this league, you have to have a tower of terabyte drives sitting on your desk?</p>

<p>I'd appreciate suggestions from digital and hybrid MF shooters out there, because eventually I'd like to afford myself a digital back at some point. But I fear that if I already have trouble juggling gargantuan files now, it's not going to get any better later on.</p>

[mattlaudergallery]

<p>I also have my own Imacon and scan my 6x6 slide, 617 slide and 624 slide and I then shoot with a 5D MkII and I have all my files on a Drobo system with 4 TB of space. Works a treat and saves bogging down my Mac's operating hard drive..</p>

[lolver]

<p>[thought of discarding the actual image data]<br /> Hmm, interesting concept, have you tried to do it. Adjustment layers contain process information on the adjustment, no image data. So you would be left with a blank image/PSD file. Actually I don't think Photoshop allows you to remove the layer anyway (if it's associated with adjustment layers).<br /> I think you're stuck with some form of large image management. You could delete the TIFs once you had a saved PSD which would save space, but that feels wrong to me.</p>

[david_henderson]

<p>Here's some thoughts , though I'm not guaranteeing that you'll like them.</p>

 

<ul>

<li>Do you actually need a 6800 x 6800 pixel file. What do you do with these things in volume? Do you actually make ( as against theoretically could make) prints of maybe 30" sq. from lots of originals? If you don't you could scan smaller and use a smaller masterfile. </li>

<li>If you do make really big prints, I'm kind of assuming that you don't do so from every transparency you scan. Try selecting those images that you are going to print large earlier in your workflow. There's no point scanning that size and spending all the time optimising a huge file if you're going to end up with screen-based applications or small prints or stock agency files. Personally I don't scan anything till I know I have a use for the scan, and most of my scans are relatively small. Its only if I know I want to make a print for my wall or someone else's, that I get a big Imacon scan made. Even then, some applications (eg stock agency acceptances) can be far smaller than the max your Imacon will deliver. </li>

<li>I must have a different version of Lightroom to you but mine has dust spotting and dodge/burn capability. You don't need layers with non-destructive editing. Even if you do (as I do) tend to prefer some of the PS tools to those in Lightroom, I find I can get far enough through the process in LR to mean that I don't need to bother with layers in Photoshop, because I can always revert to the LR version which has maybe 80% of the work done.</li>

<li>I can understand the benefits of editing in 16 bit, but is there any need to keep a completed masterfile as 16 bit? Certainly the printers I use don't demand 16 bit files. Again if you learn to do the bulk of the editing in LR you can always get back to the point of final export from LR easily and recreate a 16 bit Tiff if you really have to make significant amendments after. </li>

</ul>

[guido_h]

<p>@Laurence: I think the concept would work. I managed to delete the background layer (which was possible), and save the file (which consisted of adjustment layers and masks only). This yielded a file size reduction of approx. 30%, but I'm not really delighted about the prospect of manually keeping track of filenames and associated sleeved negatives.</p>

<p>@Matt: Your files must be incredibly large. What kind of Mac configuration do you use? I found that my configuration with CS4, 2.4GHz dual-core and 6GB RAM probably is close to the limit at which one can still comfortably edit 6x6 scans.</p>

[guido_h]

<p>@David: I find that scanning at the highest optical resolution provides the best results, even when downsizing for web publishing or printing later. Comparing downsampled high-res scans to lower-res scans of the same size, I feel that the former tend to have a more appealing "look" and micro-contrast than the latter. Difficult to proof scientifically; just my personal impression.</p>

<p>Nevertheless, the legitimate question remains: how much resolution is needed in the final storage version? When I started shooting digital in the early days, I shot everything in JPG small size because memory cards were expensive, I didn't know a thing about RAW, and my PC then was too lame to handle anything above average anyway. Needless to say, there were a couple of good shots which from today's point of view I wish I had shot in full quality back then, because now I have the computing power and software which I lacked back then to realize the full creative potential of the image. So my rational for high-res scanning and storage stems from a similar "you never know" gut feeling: maybe a couple of years from now on, with my Flextight long dead or sold, I might browse through the old archives and feel that I could do something with a couple of yet unprocessed shots, regretting that I didn't scan and store them at full resolution when I was able to. So that's basically the thinking behind my current approach.</p>

<p>Regarding Lightroom, I'm using LR3, but I must admit that I didn't pay attention to the dust/spotting/dodge/burn features too closely. Last time I looked, it seemed like I had to place individual spot healing circles on every single speck, which would be fine for the occasional digital sensor spot, but didn't appear to me as being intended for elimination of several dozen specks and hairlines. I should probably take another closer look here.</p>

[leicaglow]

<p>TIFF is one of the largest file formats you could choose. Could you save them in PSD instead, converting them after the scan to a PSD? In my workflow, I use a 16bit file and make all the corrections, including resizing, then save as an 8bit file to go to the printer. I also save the layered TIFF until I get the print back, and make sure I haven't missed a spot or something I didn't see on the screen. Then I delete the 16bit version.</p>

[mendel_leisk]

<p>I scanned 35mm at 5400 DPI. This resolved grain nicely, and while debatably overkill, was my preference: it ensured I had all the detail the (slide) film had to offer. All my master files are 16 bit per channel with red, green and blue channels. Once cropped, they are around 200 Megabytes in size.</p>

<p>Going up to medium format, 3200 DPI is a not unreasonable compromise. The potential detail and grain size in your case is likely similar, but 3200 DPI is a good medium, considering the increased area of your film. And file sizes of 300 Megabytes are managable.</p>

<p>My strategy was to use 2 internal drives exclusively for storage of such files, one primary, the other a routinely updated mirror of the first. In my slide scanning project, whenever the drives approached 17 percent full (the minimum freespace requirement for defragmenting on a Windows XP system), I would burn 2 DVD's of the current content, then (gulp) delete those files from the hard drives, defrag the drives and carry on scanning.</p>

[guido_h]

<p>@Michael: I already load the scanner's TIFF file and save the retouched version as PSD in Photoshop, but it doesn't seem to make any difference as far as size is concerned. I just pulled some arbitrary image of the web and saved it as a TIFF file to make sure: 3.3MB. Loaded that file into CS4 and saved it as PSD. The PSD file too is 3.3MB in size. By the way, TIFF is what comes out of the scanner's Flexcolor software, that's why I'm dealing with that format at all. Am I missing some compression setting in CS4?</p>

[lolver]

<p>Guido<br>

sorry to go back to the adjustment layer bit - I really don't think it's possible to save only the adjustment layers, I've tried (with CS4) and can't do it. As I said before an adjustment layer only contains a process (curves, levels,...) and the settings, that have been applied to image data in a layer below it. There has to be at least one layer with image data, Photoshop won't let you remove them all.<br>

Are you sure you're not creating a new layer, copy of background for instance, and doing image adjustments on that layer. If you are you're incrementing the file size by the same amount of the original layer, your file will grow huge and quickly. A true adjustment layer adds nothing to the size (well maybe the odd byte or so).</p>

[guido_h]

<p>@Lawrence: I stand corrected - you're right. In my earlier test, I played with a PSD and missed one image data layer which was still in the stack. As soon as there are only adjustment layers left, the background layer indeed can't be deleted anymore. So that approach definitely isn't feasible.</p>

[leo_papandreou1]

<blockquote>

<p>I'd like to have something more storage efficient here</p>

</blockquote>

<p>Film!</p>

<p>Film is its own archival medium, why archive twice? I scan on average maybe 5 frames per roll (120) and frequently delete as my enthusiasm for the photographs diminishes with time. After two years I've managed to keep maybe 100 scans on disk. Some of them are even good. I look at everything else on a light table.</p>

[john_a5]

<p>I also use a Flextight and certainly there is a difference in most cases between the size of a Tiff and a PSD and it doesn't have anything to do with setting compression--there are none. It might have a lot to do with whether you set your PSD's for maximum compatibility. That doesn't do anything for you unless you use all of the other Adobe products, like Flash, In-Design etc--and I am not sure it does much for you there either!</p>

<p>I personally keep all of my files. I scan in FFF, which is the scanner's version of a RAW file. If you aren't doing that, you aren't getting the most out of the scanner. These are huge files, bigger than your 16bit Tiff, but can be discarded once you have the outputs you want in a PSD file (Yes, the output is Tiff, but I throw the Tiff away as soon as I save the PSD file) You could save the adjustment layers, but then if you don't get the exact same scan, they are worthless--most of my time is in the layer masks which would never work with this method. The same exact scan, or output from an FFF file, is not that easy unless you just scan and don't adjust from one of the standard profiles. You can save specific profiles for each scan and as long as the software isn't upgraded, you should get a reasonably similar scan and it might work.</p>

<p>The solution suggested above, to flatten the image and convert to 8 bit is probably the best solution to be sure you have that original look in the smallest size possible (saved as a PSD--or some even save as a #12JPEG if you are really worried about space). I personally save my 16 bit layered files and many are in excess of 1.5gb as well as a flattened and saved 8 or 16 bit final. I tend to tweak things over time as my tastes change. I do make large prints from much of my work, but if I didn't--and I didn't in years past--you can get the quality you discuss in smaller scans--I like those around 50mb (8bit equivalent) and they will look as good as a full res scan. Really small scans don't hold up to much post work--or look very good, so I understand your comments in that regard, but 50-60mb (100-120mb 16bit)files will give you results on a par with full res scans.</p>

[rob_piontek]

<p>I think the only reason you might want a 16 bit file is if you have any major processing you need to do, like lots of dark shadow areas that need to be dodged. If the file is close to what you want you don't need 16 bits. I would flatten and convert to 8 bit as soon as possible in PS and archive the 8 bit psd along with the original 16bit tif. Plus 8 bit files are much more fun to work with in PS.</p>

<p>I find spotting to be much quicker in LR than PS.</p>

[Ed_Ingold]

<p>When I scan using an LS-8000, the files are over 440MB. I save the originals in tact, do what editing I need in layers and save that as the TIFF master. In short, it takes 3-4 DVDs to hold one roll of film. I work on an internal drive, which fills up pretty quickly. I archive the results on optical discs and move the files to an external drive. Disk space is cheap, but desk space is not. I keep a library of hard drives and only have one or two installed at any one time. DVDs and (occasionally) BDs take time, but manageable if you keep up to date with your work. I store them in archival file pages (8/page) and notebooks for easy retrieval. I usually keep JPEGS sized 10x10" at 300 ppi on disc for ready reference.</p>

[kelly_flanigan1]

<p>With my 1996 35 megapixel scan back; files are about 100 megs. The recomended CPU then was a pentium 133Mhz. We mostly used NT4 with a PPro at 200Mhz with 512 megs of ram.</p>

<p>Our 50 megapixel scan back from 1998 makes about a 145 meg file.</p>

<p>Our current 36" wide RGB scanner for mapping is a 600 dpi device normally it is used in the 200 to 400 dpi region.</p>

<p>With big maps or big photos; and with a high dpi ; it is rather easy to get files in the 300 meg to 1.3 gig region.<br>

We normally use a pentium III @1 ghz with 1 gig of ram with the smaller sub 1/2 gig stuff; and a more modern computer with the larger files. Scanners like this have been around for about 14 years now; thus it is actually easier as ram is so cheap.</p>

<p>Some of us were messing with 400 megs files well over a decade ago.</p>

<p>Files that size are really nothing new</p>

<p>At one time the max practical size folk scanned at was 100 megs; dictated by the ZIP100 disk.</p>

<p>Then we got out 600 buck 2x CD burners; and we buried out Jazz drives when they broke!</p>

<p>I really never much got into the ZIP250's or ZIP750's</p>

<p>l but got into the 2x CD burners</p>

<p> </p>

[kelly_flanigan1]

<p>You need to see where your bottlenecks are when you *Process* larger files. Some folks have poorer settups; not enough ram.</p>

 

[leighb]

<blockquote>

<p>You need to see where your bottlenecks are when you *Process* larger files.</p>

</blockquote>

<p>This is very true.</p>

<p>Locating the choke point in a digital process stream can be quite a challenge, even for professionals who have access to profiling tools.</p>

<p>The single biggest gain per $$ is usually achieved by increasing the amount of RAM in the system, up to the maximum it will support. If you can avoid "paging", which means copying data from RAM to the hard drive and retrieving it later, you can greatly increase the efficiency of the system.</p>

<p>- Leigh</p>

 

[kelly_flanigan1]

<p>Typically the more modern the OS; the more bloat it has. Thus if I run NT4, Win2000, XPPro, Vista, Win7 all on one box that is capped at 2 gigs of ram; NT4 and win2000 has the most photoshop ram. There is less bullcrap running; hogging up ram. </p>

[leighb]

<blockquote>

<p>NT4 and win2000 has the most photoshop ram.</p>

</blockquote>

<p>Of course, NT is OS/2, which is a much more efficient o/s than any of the u$oft offerings.</p>

<p>You can't really judge the o/s by the amount of RAM consumed when it's running with all the associated apps and support programs. Those can chew up huge chunks of RAM for no reason. If you really want efficiency you need to unload all the extraneous govno and get down to basics.</p>

<p>- Leigh</p>

 

[thomas_janik]

<p>Hi Guido:</p>

<p>I have been scanning 67 film with a Nikon 9000 for several years. The files are over 500 MB when 16 bit and 4000ppi. I am not sure why you're concerned about storage space. A 1 TB external HD is around $100US - enough to store almost 2000 scans. The real problem is computer speed working with such big files and that means RAM in this case. When I was using a PC limited to around 4 GB of RAM, I tried and used a number of the workarounds suggested to handle those large files. I couldn't use smart sharpen at all. Even with reduced file sizes it might take 20 minutes. Last year I bought an i7 with 12GB RAM and 64 bit operating system for around $1100 ( Dell refurb). Smart sharpen on a 500 MB file takes about 15 seconds. Heck, I can have a duplicate of the background without noticeable effects.</p>