Raw and DNG: what is your strategy?

<p>You can batch convert cr2 or nef to DNG with Lightroom overnite as you sleep. On my computer Lightroom converts 600 21mp cr2 to DNG per hour, so that's 5,000 per night. 3 or 4 nights and the average portfolio is pretty well taken care of.</p>

<p>It doesn't have to be done at the time of import. You can bring in your cr2 or nef, adjust them, assign metadata, etc. and it will all carry through seamlessly into the dng. In this sense if you convert overnite it takes zero extra time.</p>

>>> I think the raw files don't matter for archival purposes. It's the jpegs that you produce today that you

want to be around in the future. Do you really ever think you might want to go back to a raw file 10 years

down the road and make some minor adjustments?

 

Not for me. It's only the RAW files and edits that I want to archive. And be able to re-interpret in the

future. Saving jpegs is pointless. If I need a (another) jpeg, I'll just export one off the (re) edited RAW at the

resolution and sharpening I need at the time, for whatever use (web, print, magazine, book, whatever) at

the time.

 

Why would I want to archive single-purpose lossy jpegs with baked in processing edits?

<p><em>'I think the raw files don't matter for archival purposes. It's the jpegs that you produce today that you want to be around in the future. Do you really ever think you might want to go back to a raw file 10 years down the road and make some minor adjustments? Even if you do my guess is there are plenty of other people that will want to do the same and your raw files will be readable.'</em><br>

I shoot jpeg and raw together, and with my D300 or a D700 when available, I edit extensively in the camera in slower periods, so my thoughts are fully formed by the time I download in most cases -- e.g., I convert for testing in camera to monochrome, do some minor cropping (as allowed minimally by the camera software), etc., and even try some color balance from time to time.<br>

I archive extensively with duplicate sets of downloads on two (or three) continents, safely out of reach of a region-wide disaster. Fires don't just stop at the house's sides or even a city limits but can engulf whole regions, even much of a county or a good part of a state; same with a major earthquake or a tornado.<br>

I've heard of too many people losing their entire archives to a fire, a spouse or lover who threw their negatives out a window on the grass, a member here whose movers lost his precious negatives during a move while he was away, and so forth.<br>

An older friend, a former editor of Ladies Home Journal and Harper's Bazaar (a friend of Richard Avedon) lost his entire archive when the publisher's warehouse burned to the ground in Florida, I have heard. Nothing was backed up; all was film. Photos of Jackie O' (burned) Sophia Loren (burned) and so forth. He has almost nothing to show for his life's work except old copies of the magazines that have survived with their faded dies.<br>

Anything that can go wrong will go wrong, and redundancy is the main answer.<br>

Even in those places where I have a duplicate file set, I also keep more than one identical archive hard drive with each set of numbered download photos. In certain cases DVDs too, but those can suddenly become unreadable, whereas with a hard drive, so long as the drive spins and its read arm is working there is software available which will scan the disk almost an infinite number of times --- days or weeks if need be -- until it reads all or nearly all the data. (source -- Leo La Porte)<br>

Dropped or damaged drives in other senses are another matter, which is why if I have one drive nearby copying, I almost invariably have from two to four more copies spinning almost immediately and often simultaneously. You can even duplicate download from Adobe download software, but I don't recommend it; it's cumbersome if you make an error.<br>

At ten minutes a downloaded 8 gig CF card, and multiple copies to various hard drives from one computer possible at one time, it's a small distraction. <br>

Then physically isolate the drives once again.<br>

I have had every brand, make and size of drive I've bought fail on me at least once, most form internal difficulties - a couple from problems of my own making (I tipped over two upright drives identically, days apart with the toe of my shoe, and vowed nevermore to use an upright drive stand.)<br>

JPEGS are OK for printing, though frankly the print house I consulted prints from files at 700 dpi or above on their light jets, so if you're starting with JPEGs (8-bit) then they're having to upconvert from jpegs to a higher dpi that they can use, you're not going to get the same quality large or very large print from a jpeg file that you can get if you send them a 16-bit .tiff of equal size (they don't accept 'raw' files such as NEF without charging for Photoshop time.)<br>

Just the fact that a world quality print house won't accept proprietary formula files, such as NEF should tell you that NEF is fine so long as you believe Nikon will support them or support will exist through third parties such as Adobe or others, but bankruptcy can happen to anybody or any company for a variety of reasons. (who foresaw that BP might be floating bankruptcy trial balloons, eight months ago - it was a 'widows and orphans, dividend paying stock' and now lately has suspended its dividend and has had to sell off assets and even is considering changing its name to AMOCO - a name it discarded when it bought out AMOCO'.<br>

Things change, companies go bankrupt, undocumented file formats can go unsupported.<br>

If that troubles you, pick up a copy of Corel's photo editing software, and look at the 'save as' choices and gaze on the tens of possible formats that one can save a photo as. It makes Adobe's 'save as' choices look positively anemic.<br>

There currently are two universally accepted file formats: jpeg and .tiff that I would rely on.<br>

JPEGs by definition are 8-bit so one cannot get the greatest quality from them, though one can get high quality prints and files from a jpeg. Editing choices are very limited, though. .jpegs are lossy, .tiffs are not.<br>

.TIFF files that are saved from Photoshop (.PSD) files, saving all the 'layers' and not 'flattened' provide for me the greatest flexibility, and since hard drive space is dirt cheap these days and headed cheaper, I've decided (and counseled my photoshopper(s) to save every change, even if they do 'flatten' a photo for use, also to keep a 'layered' version for tweaking later, and it's the layered version that's the 'archive' version.<br>

But unless and until Nikon goes bust and Adobe which makes conversion software goes out of that business, I'm also keeping for archive NEFs, which have all the data, not just the data about, say, white balance, that Adobe has (1) been able to unlock and (2) been able to fit into its common photo editing 'raw' software.<br>

I'll be damned if I will buy Nikon's lousy software for doing any sort of substantial processing, though for just a print and if someone wants to sponsor me by paying the fee, and the photoshopping time, then I might consider using it, but would have to have substantial justification (since it is so slow and cumbersome) in trying learn how to manipulate it.<br>

Proprietary software is Nikon's Achilles heel, in my opinion.<br>

Nevertheless I have tens of terabytes of hard drives filled with (from first in my career) digital jpegs, then NEFS and JPEGS shot simultaneously, and will continue to have both for the foreseeable future.<br>

For processed photos, I process them in to .psd (Photoshop) files, keeping the initial download (it's a small move to save that version) from the raw, then the layered .TIFF version, and any 'flattened' .tiff version.<br>

From the .tiff versions, I make at least one size of jpeg file, as almost all applications require a genuine jpeg file for practical use.<br>

It seems like a lot of work, but if one structures one's workflow just right, and saves in batches, it's not a lot more work to process AND save NEFs, .PSDs, layered .TIFFS and then the resulting jpegs . . . .<br>

Consider the following:<br>

In transferring files from a camera, I invariably review my transfer before erasing the chip (usually just before reusing it in case of an interim hard drive or computer failure and a need to redownload), and on more than one occasion, I've found a file that needed to be recovered because of a voltage drop, a file error (a misplaced + or - in the file, that can't or I don't want handled by file recovery software).<br>

The result will be half a file or photo in my download; or the same can happen during a copy job from one hard drive to another and it might not be discovered immediately among the thousands of photos copied. The error might be perpetuated through copying and recopying until discovered.<br>

If one shoots raw plus jpegs, one always has ONE file to go back on, as it's almost unheard of for two adjacent files to be affected by a voltage spike, a cosmic ray or some such.<br>

If this saves from total loss 'THE GREAT FILE on which a reputation is made', it may be worth the extra trouble.<br>

I've used these methods to rescue a substantial number of damaged files which if I had not used such redundancy would be lost forever . . . . and some are among my best (and most viewed) photos.<br>

It takes a lot of work to get our photos; they're in effect the photographer's children.<br>

It doesn't take much to orphan them, and not much more to keep them for several lifetimes, if one uses some extra care . . . . .<br>

So:<br>

I opt initially for jpegs and NEFs shot together, then for archive, jpegs, .PSDs, and layered .tiffs for worked up photos. In a real pinch, I can alway grab a jpeg from any source and use it rather than bringing out photo editing software, too, so I always shoot jpegs at 'fine' setting, for highest quality - it's saved my bacon a few times for otherwise lost captures that a bad chip , a voltage spike, or a cosmic raw partially demolished.<br>

All duplicated files are kept in different countries on more than one hard drive for both archives of initial captures and worked up captures.<br>

[About 'cloud' backups]<br>

A member here highly touted 'Digital Railroad' to me as 'The Way to Go' for my captures.<br>

I told him 'no, I didn't think form a legal standpoint or a business choice that was wise.'<br>

A year of so later, I read that a bankruptcy judge gave the trustee who was running the bankrupt company permission to erase the hard drives so many photographers' photos were stored on. Many had rescued their photos; I'm sure a few didn't get notice.<br>

In another case of on-line photo storage, without anyone's permission, some bonus-hungry executive, not conversant with the finer points of photo quality, unilaterally decided to reduce those big floppy customer photo files to little files without telling anyone or asking anyone's permission . . . .<br>

One day there was a huge computer full of photos from all over the USA and other countries and the next there was just a bunch of thumbnails or just larger.<br>

The only person you can really trust with your photos is you. Even the most trusted company can fail.<br>

Then it's a bankruptcy judge (or some hotshot executive with a bonus in mind) who may have control over your precious captures, and you may never get a notice of their intentions.<br>

[Don't ever trust a company to tell you 'it's financially sound' either. I once was a business reporter, and the mantra of the executives of every failing company was 'we're basically financially sound' just before they pulled the bankruptcy plug . . . . and they knew and people in their industry often knew they were in trouble. They just don't want their customers to know, so they playact and keep secrets]<br>

Best to trust yourself and be super redundant.<br>

Even now trusted girlfriends get future jealous boyfriends . . . . . if you get my drift.<br>

The more copies the better.<br>

One stored photo in a less desirable format of a great capture is better than a 100% lost capture in the most desirable format.<br>

john<br>

John (Crosley)</p>

<p> </p>

<p>When importing in Lightroom I convert to DNG and save a raw in external drive. They both open in raw and don't see any major difference.</p>

<p>When importing in Lightroom I convert to DNG and save a raw in external drive. They both open in raw and don't see any major difference.</p>

<p>Never saw the reasoning behind DNG. My co-worker swears by it. For me it's an extra and un-needed step.</p>

<p>DNG is an archival, open standard that eliminates the need for sidecar files and creates essentially no extra work (see my previous post). It's a much nicer way to work IMO.</p>

<p>Also, you could lose your Lightroom catalog and all of your sidecar files, and, provided you have written the metadata to the files, years of work would be maintained if the files are dng. Just open Lightroom, import, and you're done, all of your metadata, all of your develop adjustments. With cr2 or nef you would be screwed in this scenario.</p>

>>> Also, you could lose your Lightroom catalog and all of your sidecar files,... With cr2 or nef you would

be screwed in this scenario.

 

Well, if you allow for the possibility of "losing" a catalog, or sidecars (haven't lost one yet), then you can

just as easily lose DNGs. Then what? Screwed as well. That's what archiving is about.

 

As with others, no real advantage going to DNG for me... When mainstream cameras start recording in DNG, the decision

would make a lot more sense. But that's not going to happen.

<p>I use Capture One 5 for raw conversion, since Adobe announced that the new .3-version of Lightroom would bring a better conversion than the .2-version. The by Adobe so promoted DNG-format may have a better archival value, but the basic quality of DNG made by LR has a certain level, I concluded.<br />I am happy with the qualtity of the conversion by CO5. IMHO it beats LR. I also hate sidecar files. It is an odd and confusing way of working. After conversion I switch over to LR to catalogue and archiving the files. My basic archive of the original files contains NEF, my master archive TIFF. I seldom use DNG as step between anymore.</p>

<p>...Just another question: can I extract a .xmp file from a developed DNG file? And can I re-convert a DNG file to the original raw (e.g. CR2) file?<br>

Final: are sidecar .xmp file generated by all raw converters?</p>

<p>My vote goes to staying with Raw files. Editing software will keep improving and you will be able to extract more information out of your Raw files that otherwise could be lost by converting them to DNG. Why convert unless there is a benefit and I can't see any just the negative that i have already mentioned.

<p>Just my thought.

 

</p>

</p>

<p>John McCosh wrote: ... <em>"Editing software will keep improving and you will be able to extract more information out of your Raw files that otherwise could be lost by converting them to DNG"</em> ...<br>

But the DNG file is Raw file by definition. Original raw data is not going anywhere. Only the package is different. The only thing you lose is the possibility to edit with camera manufacturer's software.<br>

I personally prefere DNG package because it is better documented and all my edits and jpg preview can be saved within the file.<br>

Regards, Marko</p>

<p>Personally I shoot and archive both NEF and JPEG as well as PSD. For me I make some adjustment in LR, then go to PS, and then back to LR with sometimes a few minor tweaks. Then output JPEG with LR. In the end I archive everything including the PSD, but to me the JPEGs are most valuable as they are the finished product. The fact that they are lossy is a total non issue for me as they will never be edited again. You don't have to have any special software to read JPEGs, which is something to consider. When I'm dead my kids will have no trouble with the JPEGs, but they probably won't know what to do with NEFs or DNG.</p>

<blockquote>

<p>Thank you for your suggestions! I was thinking about DNG mainly for storage and maybe for future compatibility.</p>

</blockquote>

<p>That’s good reason to consider DNG among others (see http://www.ppmag.com/reviews/200709_adobedng.pdf)</p>

<p>Thanks for that link Andrew, I needed a good and current overview.</p>

<p>I convert to DNG at import and then move the NEFs to an independent external drive, like<a href="../photodb/user?user_id=2138430"></a> Iftikhar Ahmad mentioned earlier... t</p>

<p>'But the DNG file is Raw file by definition. Original raw data is not going anywhere. Only the package is different. The only thing you lose is the possibility to edit with camera manufacturer's software'</p>

<p>It's 'raw', but it's not the original file. Throwing away the NEF can have unfortunate consequences, e.g.:<br>

http://www.photo.net/casual-conversations-forum/00WQjJ<br>

http://www.photo.net/digital-darkroom-forum/00Usgn<br>

You also permanently lose access to Nikon's camera profiles that specify how colour values are converted between the camera's internal colour space to a standard RGB space - these are only available when using Nikon's own software. Adobe, of course, has its own proprietary camera profiles, but these aren't identical to Nikon's. Apart from this issue, there's no reason to fear that NEFs (or DNGs) will become unreadable in the future, provided your files have been successfully backed up! Although Nikon does not publish the technical details, all the key features of NEF (including the silly encryption used for white balance etc.) have been reverse engineered, and Open Source code is available to process them (e.g. http://www.cybercom.net/~dcoffin/dcraw/ ).</p>

<blockquote>

<p>You also permanently lose access to Nikon's camera profiles that specify how colour values are converted between the camera's internal colour space to a standard RGB space - these are only available when using Nikon's own software. </p>

</blockquote>

<p>Unless there is some spectral sensitivity data there (and I don’t know if its supplied or not, its proprietary), its simply an assumption like any other raw processing engine. One could say, there is no color space of raw data, or one could say there is an assumed color space which each converter can decide upon. The only other and really useful way to define this is having the spectral sensitivities of the sensor and ideally the illuminant of each capture (otherwise assume some more, daylight, D50 etc). One could argue the same algorithms and data are used to convert camera JPEGs and ultimately raws with host software and I’d say for those who (for whatever reason) use raw+JPEG workflows, that be useful. Newer tools like the DNG profile editor and X-Rite Passport make it possible to get a close match using Adobe raw processing.</p>

<p>I’m not saying you don’t lose some proprietary data, you do. Is it useful? Outside of a Nikon proprietary accessible converter, its not useful at all. </p>

<p>You can have your cake and eat it if you are OK with increased storage (which cost little compared to images), keep both. Lightroom can spin off the proprietary raws to a drive, convert to DNG and import at the same time. </p>

<p>You can argue, of course, that Nikon's camera profiles are no more or less valid than (e.g.) Adobe's. Subjectively, I find the default NX output more pleasing than ACR's, even with Adobe's 'camera matching' profiles applied (haven't tried ACR6, though). With NEF you have the option to use either (unless you're Marko!), but with DNG you're locked out of the Nikon workflow. Saving the NEF as well seems like a very sensible precaution even for hardcore Adobe users, and the potential need to show the original file to competition judges (or perhaps news organisations, etc.) should be a further consideration for any photographers who might find themselves in this situation.</p>

<p>I convert all raw to DNG. In Adobe Lightroom that is the efficient way to work on your files. I don't think that Adobe will get rid of that file format for a while, even if so I am sure there will be a way to convert to the new file format in the future if the need arises. Adobe is a pretty smart company, I am not affraid to use their DNG format. Is it better then any other raw format. Probably not, but I got used to converting all the files a while back and there is no turning back for me. I used to keep the original raw files in the past, but not any more.<br>

Tom</p>

<blockquote>

<p>I convert all raw to DNG. In Adobe Lightroom that is the efficient way to work on your files.</p>

</blockquote>

<p>In what way is this "efficient"? It's more efficient to leave them alone; then imports don't take as much time because there's no conversion process. Other than that, there's no performance difference.</p>

<p>And all the discussion of sidecar files misses a key point -- sidecars only matter if you need them. If your workflow doesn't have any other programs which read the XMP data (e.g. Camera Raw), you don't need 'em. And if you don't write out the XMP data at all, then you don't have to back it up, which saves backup time and space as well.</p>

<p>My workflow doesn't use external XMP data at all. Which means that the folder hierarchy containing my image files is write-once. The raw files get renamed and backed up at Import time, then never need to be backed up again... That's the efficient way to work with Lightroom.</p>

<p>If you do need external XMP data, then arguably raw files + sidecars are more efficient than DNGs, because the XMP files are what changes, and they're much smaller -- which means they take less time to back up. If your XMP data is written into the DNG, then the entire DNG must be backed up again.</p>

<p>Mark</p>

<blockquote>

<p>then imports don't take as much time because there's no conversion process</p>

</blockquote>

<p>DNG conversion can and should be done overnight as you sleep. There's never any need to do them at the time of import. Any changes you make to cr2 or nef carry over. Doesn't matter when you do then conversion.</p>

<blockquote>

<p>because the XMP files are what changes, and they're much smaller -- which means they take less time to back up</p>

</blockquote>

<p>You're very right about this. It's the only major drawback to using dng. However, as long as you are backing up your LR catalog, covering yourself as far as metadata and develop changes, it's very easy to tell your backup software to skip existing dng files, so it's really not a big deal. Personally I back up the dng once a month overnight, so no big deal. LR catalog is two or three times per week.</p>

<blockquote>

<p>In what way is this "efficient"? It's more efficient to leave them alone; then imports don't take as much time because there's no conversion process. Other than that, there's no performance difference.</p>

</blockquote>

<p>Data verification is done when converting to DNG. Its real useful to know you have good data before you format that card. Just one of the advantages to doing DNG conversion and backup from cards as you import. </p>

<blockquote>

<p>Data verification is done when converting to DNG. Its real useful to know you have good data before you format that card. Just one of the advantages to doing DNG conversion and backup from cards as you import.</p>

</blockquote>

<p>Good point. I was harping on the use of the term "efficient", but data verification is indeed an advantage of DNG and of converting early in the workflow. (And those reading should know that this is a relatively new function -- if you're reading old articles about the pros and cons of DNG, they may not include this.)</p>

<p>The point made by Richard Williams about saving original captures for competition judges and news organization inquires should never be denigrated.<br>

Remember the member here who posted an urgent question? He had converted all his captures to DNG but wanted to enter a competition, and if he won he would be required to show his original files. He was in a quandary and no one really could help him other than to advise him to ask leniency from the judging if he happened to win . . . . and it was a major competition. (no one could assess his chances at winning, but someday it's going to happen for someone who converts captures to DNG and discards their 'originals' . . . they'll 'win' a competition maybe to 'lose' it later on the issue of 'not having the original file' to show authenticity.<br>

Until the issue is resolved, I'd advise, like Richard Williams, keeping all raw files if there's any thought of news use or competition entry.<br>

john<br>

John (Crosley)</p>

<p>Well. I guess I should have said "In Adobe Lightroom that is the efficient way to work on <strong>my</strong> files" I don't really know why everyone thinks their way is better or more efficient then others. What works for you may not work for someone else. Point of interest. I convert my files to DNG at import. I input all the information that I need and let it rip. My catalog right now has about 32,000 photographs and I do not see much performance difference when it had only few thousand. That said, any computer (hard drive) that fills up with data slows down, that is just the nature of the system. The conversion process and photo processing can be done pretty much at the same time. Once I import first 20 images from a particular CF card, I do my initial walk through the files, rating, star system and so on. I can work pretty fast, but to this day I have not out ran the computer yet (unless most of my photos are _crap_001 and that is a different case. Do I see a did a slow down on the system performance. Not too much. Not enough to bother me. And that is what<strong> "efficient" </strong>means to me.<br>

TOm</p>