Long Term Digital File Storage

Hi folks,

 

Recently bought an a7riii and I'm very quickly learning just how much space 80+ MB photos can take up.

I have been using a separate internal 1TB hdd in my desktop, backing up with all the rest of my files (3 other various internal hdd/ssds) to an external 4TB backup drive.

 

When I was using my old Rebel, 1TB was by far and away enough space for 10s of thousands of photos. With the addition of some 4k video and huge raw files, that's no longer the case. I'm looking into other long term storage options. So far I'm aware of:

 

Simple external backup drives, housings that can have drives added and expanded upon, using either of these in RAID, or buying larger hdds for my pc. There's also the cloud, but that feels very expensive over time.

 

Does anyone have any particularly good (or honestly bad, good to know what to avoid) experiences with data storage? What are your preferred methods?

The issue is both storage media and file format. Even if you have intact digital media in 10/20/50 years, will you have software that can read the files on that media? Here are some thoughts on each of these:

1. There are three file type you should consider for long term storage of still images, based on the likelihood (not surety) that they will be readable in the long term. These are Adobe digital negatives (DNG), TIFF, and JPEG. Native camera raw files are very inconsistent and changeable. DNG is touted as a universal, cross-platform format that should be viable for as long as Adobe is active. I translate all of my raw files into DNG when importing into Lightroom.

2. Storage media is and will always be a question. Archival grade "gold" CD's seem to be the best standard right now, along with multiple site storage using the various drive types and/or cloud. Having your data on various media types in multiple locations is the best approach, insofar as I am aware. Our firm continually wrestles with how to maintain our archives using digital media, and this is the approach we have taken.

 

I'm sure there are others with more knowledge and experience. I'm anxious to hear what they have to say.

I wasn't thinking at all about filetypes so thanks for pointing that out! I usually import raw into lightroom, work with the file, and export in whatever form the situation calls for. I've never used dng conversion although I always wonder about it when I import - maybe a question for an entirely different thread but do you lose anything by going to dng?

Another media choice for long term storage would be M discs, available as DVD and BluRay blanks. As you probably know every storage medium comes with advantages and disadvantages--hard drives are cheap and fast to retrieve files, but have a limited life span, optical media are more expensive per megabyte and can also be easily damaged and are dependent on playback devices being available, while the cloud an be expensive and isn't particularly practical if you don't have decent upload speeds. Also, cloud companies come and go, which could put your files at risk if the repo man comes for the servers. I use a mix of multiple hard drives and M disc storage, with the M discs at my home and the hard drives at my studio. Is it absolutely fool proof? No, but so far I have been able to retrieve files when I need them. A last note--ruthless editing of out takes and marginal small variations can help reduce the amount of storage you need.

ruthless editing of out takes and marginal small variations can help reduce the amount of storage you need.

+1 to this advice. With the D810 now I'm having to be much more "ruthless" in culling all but absolute keepers. Also, I've quit storing RAW/DNG for snapshots and informal stuff. I keep those in JPEG format, even if they started out in RAW, once I've done my desired edits.

 

do you lose anything by going to dng?

No. At least that's what the advertising says. I have not experienced any loss of core data insofar as I can tell. By contrast, I have had trouble updating all of my software to stay current with the latest NEF raw file formats from Nikon. As a result, I'm ever more reliant on DNG to assure interoperability between platforms. There are even free raw-to-DNG converters available that can be used outside of PS/LR, etc.

I've been doing digital imaging of one kind or another back to 1978.

A lot of "universal" formats have been proposed. Some are no longer easily read.

 

(DNG), TIFF, and JPEG

 

are the best current bets, as he says; but not only do you need to choose one that has some chance of being readable in the future, you need to regularly check the files and update formats as that becomes necessary.

 

Culling is one answer, but it, too, is a "lossy" answer. Memory is cheap and getting cheaper all the time.

Also, new post-processing methods may in future recover things that seem lost now such as now pulling out detail in what seemed to be totally black Kodachrome shadows....

I concur with JDM: storage is cheap! But I also partially agree with David: once you've edited, corrected and exported your photos, maybe there's no need for the RAW file. In my case it's not yet an issue, so I just keep the RAW files by default.

 

An alternative to RAW is log JPEGs. Some Sony cameras have that feature, but not the ones that I own.

 

Perhaps a combination of at least two storage media is good insurance. Also, off-site backups are worth considering. I had a friend who lost all her data because both of her hard drives were in her car when it got stolen.

 

FWIW, cloud storage is for syncing, not for backing up. It's a very important distinction. More detail:

 

The Story Behind That Picture 152 - How to backup your photo archive

I’m glad I’ve been keeping all my RAW files, even ones that may not have seemed like “keepers” when I first saw them. I’ve gone back to many unprocessed files I passed by at the time only to appreciate and fulfill their potential years later. I’ve also gone back to files I was “happy” with the way I’d processed only to realize years later how much more or differently I could do now. Since my vision is always growing and changing, I want to maintain the original RAW files for future ways of seeing and processing.

 

As to storage, I have 3 backups where I store photos, 2 at home and one at a remote location. I periodically replace the drives with new ones. That’s worked out well for me so far.

 

Good luck!

A bit off topic, but I have to mention it. I was talking one day with the curator of the Clark County, Nevada, museum. You might have seen him on Pawn Stars--Mark Hall Patton. I asked him what the museum does for archiving photos. He said the first thing they do when they receive a photo is to make an archival quality print on acid free paper and then store it in a folder they keep in their safe. He said that way, no matter what happens to file formats, they still have the original print they can make a scan from.

High quality, large format film and prints remain the standard for the National Register of Historic Places. There's no suggestion this will change any time soon.

In regards culling: When I attended the Nikon School some 40 years ago the presenter made a direct point that bad photos don't get better by sitting in a shoebox on the shelf in the closet. I don't believe the change to digital has defied this essential truth. True, it might be possible to salvage currently unavailable data. Still, that's not a reason, for me, to store, sort, and wade thru thousands of less-than-satisfactory images. I am a hobbyist, after all, and my living is not dependent on my photo archive. I do make copies of my family photos and distribute them to my kids. I can see how this issue is critical for pro's, just as it is critical for my firm. It is essential that we preserve the record of our work. Still, we are very, very selective of those items we make the effort to preserve. It is not only storage capacity, but also the time and effort required to identify and extract the desired item when it is hiding amidst thousand or tens-of-thousands of like material.

He said that way, no matter what happens to file formats, they still have the original print they can make a scan from.

Unless there’s a fire, flood, earthquake, or war!

 

I think we have to face the fact that we live in an imperfect world and no means of backup is perfect, whether file or print. It does seem, however, that making prints does cover some vulnerabilities of files just as saved files would cover some vulnerabilities of prints.

 

The best antidote to possible file corruption or print loss is to keep making new pictures and pay enough attention to preservation as seems reasonable while not succumbing to obsession.

Good point on the RAWs ditching discussion by The Shadow above. To summarize: If you are fortunate to have a calibrated screen and almost fully grown editing / PP skills, letting (at least some) RAWs go, seems an option.

 

I am planning to bet on multiple computers, here at home.

• I don't want to keep a huge RAID of disks spinning to do something that really doesn't require those at all (due to concerns about surplus wear, tear and skyrocketing utilities bill).
  
• Saving a card to two independent machines before formatting and reusing seems safe / feels good.
  
• I heard Lightroom performance suffers from infinite catalog size?
  

I want to limit the capable editing machine to 3 drives; maybe one SSD for "to do RAWs" and one disk for storage of keepers or such, to get viewed, which means I'll want JPEGs and if that drive gets cluttered with those; I might even downsize them to 4K stills. - I am talking viewing and handling convenience here. To enjoy one's photographic results flipping through screen sized JPEGs seems more fun than staring at an hourglass symbol while another RAW gets opened. - Storing RAWs (somewhere else) in case you 'll learn editing later or find another reason to need them is a different issue.

 

Surviving RAWs will have to get moved on a less spectacular storage machine, on which serious backup efforts will get focused.

I can imagine deleting older files on the other initial download machine, to make room for new ones. This seems a simple way to have my files on two computers and no need to weed them out on both.

 

Not sure about RAIDs. - They sound cheaper than multiple stand alone disks. - OTOH: What are you going to do, to retrieve data from surviving disks, after your weathered RAID controller failed? Why buy multiple drives when you haven't yet shot enough, to fill one of them?

Buying a NAS: Maybe worth it, for some. - See RAIDs.

My wealth is limited. I can get (old! & unspectacular) computers for free, cheap etc. I had the tiny PSUs of ready made external drives fail on me. KVM switches became dirt cheap. Energy cost isn't my big main issue, if I can go for an entire month or even two without needing my data storage machinery running. - Whoever needs archive access multiple times per week or every day should probably think differently. I like the idea of storage computers being of a hard to misplace size.

 

Backups & eternity: Let's be realistic: Nobody cares enough about our (<-99.7% of the population's) photography, to wait for us hitting our coffins to finally get hold of all our RAWs and spend months, years, whatever it takes to edit and retouch the heck out of them.

If we are lucky(!) people will like some finished pictures enough, to keep them. For that reason I think off site storage of presentable JPEGs might be sufficient, for me, professionals' MMV.

Backing up on tiny external disks tends to get messy. - Better come up with a system to keep track what is where and backed up to what. Also buy yourself some permanent felt marker to note installation dates on HDDs, to figure out which lasted how long, to conclude what you might have to replace soon(ish).

There's also the cloud, but that feels very expensive over time.

Over time is right.

If you have ADSL broadband you're lucky to get 1 MB/sec upload speed - mostly it's a lot less. But even if it is 1MB/s, that's still 1 million seconds to upload a Terabyte - 16.7 thousand minutes - 277.8 hours - or 11.6 days solid. Worth the wait?

 

SATA drives are about £30 per TB at the moment, and you can get a simple USB cradle for them for about the same price. And it's hot-swappable. No need for an entire machine just to run one or two hard drives, which can be packed about 4 to a sandwich-box sized container. Add a roll of sticky labels and you're sorted.

Edited April 7, 2019 by rodeo_joe|1

Thank you for all the replies and discussion! I think I'm probably going to do a version of what I already do: recent imports (6 months or so, until space is gone) kept on both my desktop and laptop. I'm going to grab a 1 drive usb 3.1 hdd bay, find a reliable archive drive, and backup there. Keep them in an archival box with some silica beads, labelling each drive with dates of included content. I'll keep a smaller jpeg library of keepers on my computers for use on socials and sending to friends, etc.

 

Maybe one day if google fiber or something comes to my city and I get a chance at proper high speed internet I'll look into the cloud again, but until then it just doesn't make sense with my 20 Mb down / 5 Mb up.

 

I'm going to look into print services too, I've been thinking about old family albums for a while and miss the tactile experience. Paring down my 30000 photos into a couple hundred for a book or two will be the real challenge here, definitely a lesson on staying on top of triaging photos on first import :rolleyes:

 

> edited for spelling

The ability to read the file is always going to be an issue.

I recall having to dig through old floppy discs to find a software that could read one of my files.

Will DNG software 10, 15 or 20 years from now still be able to read todays DNG files? I do not know.

Will Nikon's editor 10, 15 or 20 years from now be able to read todays NEF files?

 

I do physical backup and rotate through 3 large external drives (a,b,c) to backup what I have on the computer.

That way I have 3 generations of backup to fall back on.

And they are NOT kept in the house, they are sent off-site. And if you are paranoid, send them to two different sites.

The more critical the data is your business, the more attention you have to pay to them.

 

Hard drives have been getting bigger and bigger, so I expect the drive size increase will keep up with my ability to fill them, as they have. Although I don't shoot video.

 

 

However, at a certain point, I will have to start archiving off my OLD files and purging intermediate work files, and have to think of a robust workflow for that.

Maybe a separate archive drive(s) that I move old files to, then backup that drive every 6 months.

 

As for RAID.

I determined that for home/small business use, RAID is just not worth the hassle.

I am a techie and thought RAID was neat, so I built RAID into a couple of my computers. But as was mentioned, if the RAID controller crashes, can you get the data off the drives, maybe not. And you can't even buy some of the older controllers. Or the system architecture changed, and your replacement computer uses a different card interface than your old computer, so you can't even use your old RAID controller. If you are using striping or volume joining, if the RAID controller crashes, and you can't replace the controller, you lost everything. If you use simple mirror, you "might" have a chance at recovering one of the drives.

 

KISS. So now, with the benefit of hind-sight, rather than RAID, I now use the drives individually, so each drive is recoverable.

I backup quarterly + after any important gig. The more you shoot, and the more you change data, the more often you have to backup. For a commercial business, I would backup 'at least' monthly, if not weekly + after any important gig.

 

As for online backup, I use them for short term backup. I do NOT rely on them for long term backup. A few too many online sites have gone under, or shut down parts of their business. If you use online backup, you better use at least TWO of them, or one + your own physical backups.

 

I also do NOT like the idea of using the cloud as working storage. Think about how slow it takes to upload a file, then multiply that by a few hundred files that you may shoot at a gig. What was tolerable for a 16MP image may not be for a 45MP image from a D850. Then think about editing, where your software is reading and writing to the cloud. Your RAW to JPG conversion that may take 5 seconds to your local HD might take 30 seconds or more to the cloud, then multiply this by a few hundred. The problem is the size of the file and how fast your upload pipe to your ISP is. Not everyone has uncapped fiber optic speed, I don't.

 

So what do I do:

I do daily online backup + quarterly physical backup to an external drive that is rotated.

I think spinning HDDs and slow transfer speeds will soon be a thing of the past. The large 2.5" packaged SSD also. It's already mainly a box of air.

 

RAID as well, is a legacy of the days when HDDs would routinely wear out, with an average lifespan of about 3 years.

 

M2 and whatever follows is obviously the future, but that's almost totally irrelevant because the stored data remains the same regardless of the physical medium.

 

Reading 'old' image formats is simply a matter of software. For example; the free Irfanview reads almost every media format ever invented - most of which are now rarely used.

 

Likewise with RAW formats. Adobe and the like would have us believe that every new digital camera needs a complete software re-write, but that's total BS. In most cases only the file header is changed, and the de-mosaicing remains bog standard.

 

If obsolete software worries you that much, then simply backup the reading software alongside the files.

 

I mean, who has a slide projector or enlarger and darkroom readily to hand these days?

Edited April 9, 2019 by rodeo_joe|1

I mean, who has a slide projector or enlarger and darkroom readily to hand these days?

 

me

me

Me too, but the blackout blinds in the darkroom/computer/printer room haven't been pulled fully down in years, and the slide projector is a little portable curio that folds up to a 6" x 4" x 3" metal box!

16-TB HD is only about $500. Start with 4 or 8-TB, to make sure you're "serious." Subscribe to Backblaze to back up your entire archive (on your HD) for $50 annually. They keep things synced automatically. Start while your archive is still small, so it'll only take a few days to upload to BackBlaze. Don't confuse the issue with a bunch of discs. On my HD, I can search 20,000 images and, usually, find an old image in less than five-minutes, often seconds. (Tag your images copiously. It'll make searching much quicker and easier down the road).

 

DON'T keep images that you're not going to process. Don't even let them onto your HD. I use FastRawViewer to select those that I'll process. I do wildlife and often shoot at 20-fps. I'll take 500 and process 20, sometimes less.

 

DON'T keep images that you're not going to process. Don't even let them onto your HD. I use FastRawViewer to select those that I'll process. I do wildlife and often shoot at 20-fps. I'll take 500 and process 20, sometimes less.

Another possible solution is to export the masters once you've edited in the RAW converter. DxO doesn't seem to have this, but Aperture did. You can export as DNG in DxO, but that's not what I actually want.

Another possible solution is to export the masters once you've edited in the RAW converter. DxO doesn't seem to have this, but Aperture did. You can export as DNG in DxO, but that's not what I actually want.

 

I Export from DxO to LR, then add my logo when I Export from LR to disc. I don't allow LR to the "Backup" that it wants to do when I close it. (If DxO would finally make good on their promise to include logo affixing in RAW conversion, I would probably almost never use LR or PS).

I have a 4x2TB RAID 5 array SAN that I use for my home storage, not just pictures, but

also WAV files for audio, and anything else I do on my home computers.

 

One that hasn't been mentioned, though I haven't been keeping up quite fast enough,

is Ultrium tape. Ultrium 3 is 400GB per tape cartridge, which is much more than

CD-R, DVD-R, or even BD-R.

 

Someone above mentioned gold CDs, but for 80MB files, that is about 8 per CD.

I have some files from my D70s on CD-R, 200 JPGs/CD-R but even at hobbyist

production rate from a D70s, that is a lot of CD-Rs.

 

But yes, much more complicated than some negative strips in Print-File pages.

 

I have negatives that I shot 50 years ago, and some from my grandfather older than that.

I have a 4x2TB RAID 5 array SAN that I use for my home storage, not just pictures, but

also WAV files for audio, and anything else I do on my home computers.

 

One that hasn't been mentioned, though I haven't been keeping up quite fast enough,

is Ultrium tape. Ultrium 3 is 400GB per tape cartridge, which is much more than

CD-R, DVD-R, or even BD-R.

 

Someone above mentioned gold CDs, but for 80MB files, that is about 8 per CD.

I have some files from my D70s on CD-R, 200 JPGs/CD-R but even at hobbyist

production rate from a D70s, that is a lot of CD-Rs.

 

But yes, much more complicated than some negative strips in Print-File pages.

 

I have negatives that I shot 50 years ago, and some from my grandfather older than that.

 

Glenn, I'm not dissing what you do, but "not keeping up" is what moved me from trying to maintain my own off-site backup, to getting a 16-TB HD and backing up continuously with BackBlaze.

 

Please, tell me, why you persist with piecemeal storage. Now that we can buy many TB at relatively low cost and use cloud back-up for $50 annually, I don't see why anyone we not put all archives on one device. Retrieving various files is much easier when they're all on one device. What advantages do you see in your strategy?

Glenn, I'm not dissing what you do, but "not keeping up" is what moved me from trying to maintain my own off-site backup, to getting a 16-TB HD and backing up continuously with BackBlaze.

 

Please, tell me, why you persist with piecemeal storage. Now that we can buy many TB at relatively low cost and use cloud back-up for $50 annually, I don't see why anyone we not put all archives on one device. Retrieving various files is much easier when they're all on one device. What advantages do you see in your strategy?

 

I suspect mostly because I am lazy, and haven't tried so hard.

 

The reason for not putting all on one device is redundancy.

 

How you do the redundancy is the question.

 

RAID gives you some built-in redundancy, but there is still the possibility of

more than one drive dying at the same time. (Well, then there is RAID 6.)

 

In addition to the 4x2TB RAID array SAN, I also have a 3GB SAN, which I use more for ordinary

file storage, NFS mounted on different house computers.

 

Keeping files on many different computers seems to be the easy way to lose them, after you

forget which are on which computer.

 

Yes I haven't tried cloud back-up yet. That means finding one that will be around for many years,

which doesn't always happen. What guarantee to cloud providers give you?

 

Or maybe it is that I like playing with computers.