Raw image files; what to archive?

OK, after more than 60 years of photography exclusively with 35mm film I have

finally and firmly put one leg into the digital stream. The learning curve is

not so steep as I anticipated, but there remains much to be digested. Today's

question concerns file management and archiving. I am shooting with a Nikon

D200.

 

After trying a few different raw "converters/editors" (Nikon Capture NX,

Bibble, etc.) I have settled on plain vanilla ACR via Bridge. I don't see a

substantial quality difference between this and the others, and the workflow is

much more straightforward. The question is: what to save/archive after the raw

conversion has been accomplished. The following are my assumptions based on

what I have read. I would be very glad to have them substantiated or corrected

as the case may be. Are there other advantages or disadvantages to these

options? Are there alternative workflows that you would recommend?

 

If I understand correctly, after making the adjustments to the raw (NEF) file

in ACR there are a few options.

 

(1) Hit "done". No new file is created. The adjustments are saved as

metadata, either in the Camera Raw Database, or as a sidecar (.xmp) file.

Although the adjustments are applied to the raw image when it is opened in

Photoshop, the data in the raw file remain intact and can always be recovered

with no degradation. Thus the raw (NEF) file itself, along with the metadata,

are all that one needs to archive (the digital negative).

 

(2) Save the ACR processed image in one or more formats (JPEG, TIFF, PSD or

DNG). The option that seems attractive to me is the DNG format simply because

the metadata and camera raw settings are embedded within the file and one need

not worry about moving this data along with the raw file when archiving. Again

I assume that the intact (unadjusted) raw file data can be recovered from the

DNG file whenever desired. I would then archive the DNG file and could discard

the NEF.

 

Following this my workflow would remain essentially unchanged: Open

the "converted" (NEF or DNG) file in Photoshop. Make my edits and save them as

PSD. Convert the edited files to jpegs as needed.

 

Thanks in advance for your comments/suggestions.

 

Mike.

Here's what I do.

 

I keep all the raw files in their original unchanged state. I think of them as a digital negative. I

save the file out as a psd to work on (just the images I'm editing, not every single one) and

use adjustment layers as much as possible so as to make changes later... I save that file, too.

 

I use a lot of blank CDs obviously...

.

 

Technical point: RAW, (a latent image file) is not equal to a film negative (a real image), rather it is more equal to exposed but undeveloped film (a latent image), which RAW file can be redeveloped over and over, over the years, as our skills and tools evolve - not so with a film "negative" that is developed once and forever.

 

Archive wise, I'd save RAW chronological in directories by day onto CDs - cheap, and a forever reference in dark storage. Nothing is foolproof (since fools are such geniuses), but large hard drives are cheap too, and may be easily read if you also keep working gear that can read them (IDE, SATA, SCSI, FAT, FAT32, NTFS, and so on). I still have 5 1/4" and 3.5" diskette drives - why not keep 'em?

 

Good luck, Mike. You'll probably change your archiving every 6 months until you get tired. Let us know how it goes, and if you find any PERFECT solution to an accessible, crash proof archive (that is NOT in the basement of the twin towers - ouch!).

 

-- Click! Peter Blaise, Minolta Rokkor Alpha DiMage Photographer

The answer depends on how much external hard-drive space and how many DVDs you are willing to devote to archiving. I shoot RAW but decided not to save everything. Instead, I first weed out non-keepers, then process all keeper RAWs to maximum-quality JPEGs. I then discard all RAW files except for my "hero" shots. I convert the hero RAWs to DNG, keeping both the original RAW and the DNG. Then, I archive (on DVD and an external hard drive) my JPEGs for all keepers, and the JPEGs, original RAW files, and DNG files for the hero shots. My thinking is a maximum-quality JPEG processed from RAW is sufficient for my everyday shots, while I may want access to RAW data later for my hero shots. I don't save TIFFs because they are huge (my camera is a 1Ds2) and, in my opinion, offer no advantage over RAW files.

.

 

Geesh I love it when someone asks for archive advice and someone inevitable asnwers to throw stuff away! =8^o

 

I guess it depends on the stuff, and the person, and I know some commercial photographers who tend to discard a lot to lighten the load of each project, then toss the entire project archives when it has aged even a bit. It has no value to them, professionally, so why save it? The customer's got their copy.

 

Personally, I treasure shots from 40 years ago that I probably would have tossed if I were "commerce" minded. Heck, they're just files now, so I save everything, and a 500gb drive is the same physical size as a 250gb drive, so it's not like it takes up more space to save everything forever. Free http://picasa.google.com/ and other cheap and expensive programs scan and display images very fast, so finding tons of personal archive stuff quickly is getting easier and easier.

 

So, Mike, I see some broad and varied advice here, including to reconsider why you're saving your files in the first place. Great question, and for me, the answer is, "because it's easy, and I have no idea why I'll want to see these 40 years from now, or why anyone else will want to, but I can't see 'em then if I don't save 'em now, so ..."

 

-- Click! Peter Blaise, Photographer, archivist, accidental historian ...

I completely agree with opinion that raw files should be saved. My ideal combination includes Raw Shooter premium 2006, because it's quick, rendering is good, raw files are not actually changed (modification parameters are saved in a sub-folder)... But, you need a lot of disk space for that. I suggest you western digital external HDD with BOTH Wi-Fi and USB connections (I'll buy one of these new, with ethernet link) and, of course, at least one dual core with 2GB RAM, 20" monitor... that's most of it.

What & How to Archive?

 

Preworkflow set up info:

 

All of my files are on external hard drives. I set up the hard drive by year, the a folder for each month, a folder for each gig, a folder for each cf card from that gig.

 

Workflow:

 

After a gig, I bring the CF cards to my studio. I have my camera on continuous numbering. I upload each card into a file folder for that card with the setup on the hd as mentioned above.

 

Once the files are on the hd I take the cf cards and place them in a drawer just in case I need them again. Later on I will format them, but that's later on.

 

I take the RAW files that are in each folder for each card and copy them onto DVD's.

 

Then I get the images ready so as I can use them in CS2.

They are categorized with a folder for each CF card. So I have a folder that says something like #1RAW and another that says #1 TIFF or psd or jpeg whichever format I want to save the RAW files.

 

Then I burn the images onto CD's where they are put on the server of my internet fulfillment company.

 

At last I can erase/format the CF cards and use them again. I have enough cards to do at least 3 gigs without worrying about things.

 

As far as changing the RAW files, as I understand it what is changing is the metadata not the RAW file and you can always go back to the original settings. Someone else can confirm this.

 

In conclusion, I archive all of my RAW files in three ways, on an external hard drive, on DVDs and on the server of the internet provider.

 

Hope this helps you.

 

Best in 2007!

Remember guys..........count on CD's and hard drives as 5-8 years life span.

 

I just had this conversation with a professor at the Rochester Institiute of Technology.

Film archives much longer under proper conditions, so don't be lax on digital re archiving.

 

Cheers.

Hard drives are getting so inexpensive that this is hardly a concern any more. Nowadays I can get a 500G drive for well below $200. Using just one command you can copy the entire content from one drive to another. The process may take a couple of hours, but you could be sleeping while the computer does the work.

 

I actually have a hard drive with my important images in the safe deposit box. My wife thinks that is way an overkill, but I swap it maybe twice a year and also keep a 3rd copy in the office.

 

There are archival DVDs that are supposed to last 100 years.

 

The biggest problem is, as usual, that I don't shoot enough great images to fill those media. Storage itself is hardly a problem.

Many thaks to all who have answered so far. Your suggestions have all been helpful.

 

Perhaps my first post did not sufficiently emphasize the thrust of my question. I am not so much concerned with <how> to archive (i.e. what media, what hardware, number of backups, organization and cataloging, etc.). I am more concerned with <what> to archive; not in the sense of which images, but in the sense of what file type should serve as my digital negative (pre or post development). For the past several years of my film work I have been "digital". All film is scanned and the resulting TIF files comprise the base of my archive. Now the switch to all digital has prompted my question.

 

Here and elsewhere it is suggested that one archive the unprocessed NEF (raw) files just as they come from the camera. The first step in my workflow is to process the NEF file in ACR (while at the same time culling out the obvious duds). This is quite efficient since one can often apply the basic adjustments as a batch process to multiple images. Thereafter the result can be saved in either a NEF file plus metadata in an .xmp or a DNG file with the metadata incorporated. If my assumption is correct the original (unaltered) raw file can be retrieved from either. If so, it seems unnecessarily redundant also to maintain an archive of the NEF files as they come from the camera. As I mentioned in my first post I am leaning toward the use of the DNG format since this eliminates the need for the additional .xmp files in the archive. Again the NEF files and their accompanying metadata could be discarded without risking any loss of information.

 

As for what happens next; any images that I feel are worth further editing are brought into Photoshop, and the results are always saved as PSD files. These are added to the archive.

One point I've noticed. In my previous employment I spent a LOT of time and effort using CDs

and DVDs for archiving files. The reliability of DVDs, medium and long term, is SO horrible, I

would not even DREAM of using them as a primary archive medium. CDs, however, have been

great for me (except for in the early 90s when the technology was brand new and things were

a little tough).

 

I HIGHLY recommend archiving to CDs, backup hard drives, but NOT to DVDs. You WILL lose

data at some point. We always did on the really big archive jobs.

If you shoot RAW, I don't see how you can archive to CDs; JPEG may be a different story. Each CD is about 760M. Each RAW file from the D2X is about 20M, not much smaller from the D200 or D80. If you process that in PhotoShop and add a few layers, you can easily have a 100M file per image. Do you really want to store merely a few images on each CD? You'll be talking about stacks and stacks of CDs.

 

For optical type of discs, my problem is that even DVDs do not have sufficient capacity. Hopefully Blue Ray or HD DVD will solve that problem in the near future.

Mike.

 

Think of your RAW files as a film negative.

I am sure You have kept most of, if not all, Your developed films through the years. Do the same whit the RAW files.

Now You do not have to keep bad shots!

If You discard the RAW file after processing it in PS and only have a PSD file left there is no way back.

I now use Capture NX and PS, even the versí¯® ability in NX,(bigger files) and save all my files to DVD, both RAW and all the relevant formats.

 

My only problem is catalogue.

 

Jens

Let's face the truth - we have a serious problem in keeping our files safe. DVD and follow-up technologies are insecure for obvious reasons of unknown shelf life (or even worth because of known instability problems in long term storage). There is no safe storage that is affordable and adequate for safe storage of a few hundred GByte/year. Storing data on external magnetic disks is a "sort of save" and is good to sleep well at night. But the truth is none of these methods are nearly as good as the good old tape. Tape backup is still available but not within financial reach of an amateur with a modern camera with many megapixels.

Why is the storage of digital images such a big concern all of a sudden? It isn't like old-fashioned film was all that safe in the first place, and you can't make more identical copies of your film after the fact. If anything, just the fact that you can make multiple, identical lossless copies of your digital files and store them in different places makes it much safer overall.

 

Some 15 years or so ago, famed wedding photographer Monte Zucker had a electric equipment malfunction that caused a fire which burned down his entire studio. The fire destroyed netatives from some 20 weddings that were never printed. He said he felt fortunate that none of those couples sued him.

 

Before I format any memory cards, I always make sure that I have the images on at least two hard drives, and if it is important images, maybe 3, 4 hard drives and also have a DVD copy.

 

See also:

http://www.photo.net/bboard/q-and-a-fetch-msg?msg_id=00J9H5

> Why is the storage of digital images such a big concern all of a sudden? ... If anything, just the fact that you can make multiple, identical lossless copies of your digital files and store them in different places makes it much safer overall.

 

Same answer as always: Digital files need constant active maintenancee to keep. Film just needs passive care. As long as the maintenance is provided, files can be kept safer than film (redundant copies, as you mentioned), once the maintenance stops the gambling begins. The files will be lost somewhere between a few months and several years ...

 

> Before I format any memory cards, I always make sure that I have the images on at least two hard drives, and if it is important images, maybe 3, 4 hard drives and also have a DVD copy.

 

I hope you also verify the integrity of original files as well as copies from time to time? Just a few months ago we had a thread from someone who found out that his elaborate backup system had dutifully spread the corruption which had destroyed his original files to all his backup copies ...

 

Andreas

"I hope you also verify the integrity of original files as well as copies from time to time? Just a few months ago we had a thread from someone who found out that his elaborate backup system had dutifully spread the corruption which had destroyed his original files to all his backup copies ..."

 

This is what makes a tape system so popular among administrators. One can store a "history" of data on relatively cheap media. This is not possible on magnetic disks. It was possible with "moderate cost" of a tape system near the price of a pro camera body until 2-3 years ago when the capacity of such systems was about 80GByte per tape and hard disks were typically about 20-40GByte.

 

External disk drives - even if these are normally without power and in running state occasionally - can be corrupted just when these are needed: in case of data loss without notice e.g. due to system instability or malicious software attack. At least I do not know consumer hard disk drives where you can deactivate the write head in a case where you want to restore data onto a computer that suffered data loss due to an undetected virus. Just one example of what can go wrong.

It seems to me that the responses to my original post (while informative and interesting) have wandered away from the question I posed. While searching the web I have found some information that is pertinent to the original question. For those who may be interested the link is

 

http://www.barry.pearson.name/articles/dng/safety.htm

 

Cheers/Mike

I keep several copies of my keepers on different hard disks with at least one copy offsite. My

main concern now is that raised by Walter, that a file gets corrupted which is then propagated

to the other backups. My solution is to just keep a history of backups where the hard drive is

written once and never over-written. In my opinion, this is feasible even with hard-drives as

the cost is so low (say once a year write to a hard drive and don't overwrite).

What I currently do:

 

1 - Copy CF to hard drive (CF usually not reused until I have backups made)

 

2 - Rename images names to a series of consecutive five digit numbers. Some people use dates, but I'd like to keep the file names short. I use folders named by date, so it seems redundant to have it in the file name.

 

3 - NEFs converted to DNGs, though I'm not sure right now if this is worth the extra work. Too bad the camera doesn't write DNGs natively (Right now I've got my D200 Quality set to RAW/JPEG Basic option and also saving those jpegs) One dated folder will have separate separate subfolders for the DNG and JPEG files.

 

4 - Every 3 to 5 gig, I archive to three separate DVDs. Each DVD is from a different manufacturer.

 

5 - I keep ISO disk images of each DVD on an external hard drive. I actually make the ISO disk image first and write that to the backup DVDs.

 

The weakest point in all of this is that I have all the media in a single room.

Tom--

 

In reference to item #3, you might wish to look at the article by Jeff Schewe at

 

http://photoshopnews.com/2005/05/23/dng-workflow-part-i/

 

He describes a very simple workflow whereby one takes the images directly from the memory card, renames files and converts to DNG in one step. What you do about duplicating and archiving thereafter is up to you.

 

Cheers/Mike