How to archive 100,000 images?

Hello:

I am hoping some of you will share what you do.

I have a burgeoning image catalogue of >120 000 images, which I need to backup

in a way that will make them both safe and accessible. At the moment they are

spread across 3.05 Tb of HDD space. As a working pro, I am constantly adding to

that~!

i want ot be able to archive them safely, without using DVD ( which aren't that

safe anyway, and I don't own a warehouse), using HDD's where possible. Currently

I back them up to externals, which gives me a 2nd layer of protection, but I

want to add a 3rd.

What do the rest of you do, and what suggestions do you have? NAS? Xternal? network?

Incidentally, before anyone asks, I use a PC.

Kind Regards

Tony

<p>Would the <a href="http://www.drobo.com/products_demo.aspx">Drobo</a>, or two of them, be useful? I don't have 3 TB of data to worry about so really have no idea how to manage that.</p>

We recently got a NAS (Dlink NAS-323) for our home network. A NAS is pretty neat because it's very easy store/retrieve/share files between computers that even run different OS's. Even if you only use one computer, having a NAS lets you put it anywhere you have a network connection. All the two disk NAS's can be configured as: each disk is an individual volume, JOBD - all the disks are combined to look like a single large volume, RAID 0, RAID 1. A NAS with 4, or more, disks can also do Raid 5.

 

The biggest drawback to a NAS is low transfer speed relative to direct attached storage (DAS). If you have a gigabit LAN and you get one of the fast NAS's (500mhz processor with 128 meg of memory) it's about the same transfer speed as a fast USB cardreader with a quick memory card. You wouldn't want to make a habit of working on large number of files stored on on the NAS.

 

Depending on the NAS the storage cost can run from about the same as an external HD to about twice as much for a "small business" NAS. (Whe n I got ours, I only considered NAS's that came without drives so I could tailor the drive selection to our storage needs.) Our Dlink cost about $180. A pair of 1 Tb drives would add ~ $500 (If you don't do a RAID configuration you get 2Tb of storage). A NAS that takes 4 disks costs at least $500 (w/o drives). You can use multiple NAS's, but if you want something that looks like one really big drive (4Tb), you need a 4 disk NAS.

 

Decent looking brands of NAS's for home/small business are Dlink, Synology and Qnap. Small Net Builder has lots of good info and reviews of these devices (http://www.smallnetbuilder.com/content/view/30324/75/).

 

I'm sure that other folks will suggest just building your own file server, which is a perfectly appropriate technical solution, but it's not a "plug and play" device.

Try these hard drives for on shoot archiving they are very tough:

http://digitalfusion.net/process/hardfilm.php

 

And this offsite backup service:

 

https://mozy.com/home

Putting the files on another hard drive (or hard drive array) is called "backing up" not archiving. Sooner or later all hard drives will fail.

 

Another form of backup would be on tape. Tapes don't "burn out" like drives but they do deteriorate due to crosstalk and bleed through in 5 years or less. You would need to refresh the tapes periodically. This is the medium of choice in the business world, where 5 years might as well be eternity.

 

Optical media, including CDs and DVDs will last much longer than HD or tape copies. Furthermore, they can't be accidently erased or changed. The greatest danger is the evolution of technology so that readers become obsolete. Most of my images (about 3 TB/year) go on DVDs. Larger files, such as video AVI, I'm now putting on Blu-Ray discs. My oldest CDs are from about 1994, and are as good as new.

 

If you want images to survive technology (and you), I suggest you print them and store them in a controlled envirionment. Eyes aren't going out of fashion any time soon :-)

Ignore for the moment the arguments about types of media to use. What is really important is what library and keywording system / software are you using to keep track of where the iamges are and what the content of the images is?

 

Peter Krogh's <A HREF = http://www.thedambook.com> The DAM Book</A> is a useful resource.

Tony, I am in exactly the same situation, with the same volume of photos, and have been

looking around myself. (Only difference, I'm a Mac user.)

 

I find that offsite storage like Photoshelter is prohibitively expensive--about $1000 per TB

per year. Plus, even with a fast DSL connection, when I'm shooting a lot, there's not

enough time in a week to upload all my RAW files.

 

I consider DROBO, but don't understand it that well. I think I'd prefer a system where I

know exactly what data is on any given drive, and I can swap it in and out as needed.

 

Here's what I consider my best idea right now: Purchase an eSATA multi-bay enclosure

like one of these:

http://www.macgurus.com/productpages/sata/BurlyPortMultiEncl.php

 

With something like the 8-bay tower, I can fill it with smaller drives that I've since replaced

with 1 TB internal drives (of which I have three). So I can put in 2 500 GB drives and 1 750,

and can add more larger drives as needed. I can use some to keep data online, and can

swap drives in and out of the remaining bays, backing up the other drives, then labeling

them and shipping them off to an out-of-state friend for off-site backup.

 

I just need to find out whether my Mac Pro can accommodate two eSATA cards or modules

or whatever they're called.

When you need to store that much data, it will be both expensive and time consuming. Whether you need to store that much data only you can answer.

 

As Edward points out, hard disks can and do fail. CDs and DVDs also fail. One answer to this problem is multiple backups and diverse media. You may want to keep a copy on an internal disk drive, another on an external disk drive, and yet another on DVD or CD.

 

When you choose media, you should consider the impact of the loss of any one piece of media. You will lose 8 time the data if a DVD fails than if a CD fails. You will lose over 1,000 time the data if a hard drives fails than if a CD fails (assuming a 500 Gigabyte drive).

 

You will also need a good catalog system so you can find a given image on your backup.

 

The external disk drives should not normally be attached to the computer. The drives should be attached only for backup or, less frequently, for restore. This is to guard against corruption of the back up files due to a computer virus or, more likely, deleting files due to "operator error" i.e. mistake.

 

Yes, you do need a warehouse or at least some form of off site storage (a Safe Deposit Box) to guard against physical destruction of the data due to fire, flood, theft, etc.

 

A good, but not absolutely foolproof backup system would look like this:

 

1) A mirror RAID array in your computer.

2) An external mirror RAID array attached to your computer, normally turned off, turned on only for backup or restore.

3) Periodic backups taken to CD or DVD and sent off site. These backups can be incremental i.e. only changed files are backed up, but the entire backup set should be refreshed periodically - monthly, quarterly, or yearly - and more than one generation retained.

 

There are online backup services where you upload your data to their servers and they take care of backing up the servers. They are not cheap and there are security and reliability considerations.

 

Now, do you really need to archive 100,000 images, or would a mirror RAID array do for most and only a subset need to be archived and protected? Only you can answer that question?

I guess that you could spend a lot of money setting up your own NAS or Server array. But a relatively easy and safe way to manage them would be to get two Drobo's with the Drobo-share and put 4TB in each one. $500 + $200 for each unit+share, + $200/drive = $1,500 per unit.

 

Connect everything via Gigabit ethernet and you'll have a fast solution that would require minimal administration. You'd have full access across your network to everything and full redundancy to all the data on each Drobo (assuming two drives didn't fail at exactly the same time). One thing to watch out for is that the Drobo software will split anything above 2TB on a single Drobo into multiple logical volumes.

 

I know that systems that use the ZFS file system would offer huge pooled storage soutions, but that's way out of my league to setup. The Drobo is SO easy, it's not even funny.

 

Skip

[[i find that offsite storage like Photoshelter is prohibitively expensive--about $1000 per TB per year. Plus, even with a fast DSL connection, when I'm shooting a lot, there's not enough time in a week to upload all my RAW files.]]

 

Why do you need to upload /all/ of your photos to such a service? Why not upload keepers and images you've printed or sold?

my friend as a Xserve and tape backup with a serious DAM software to keep track of everything. 2 location.

Xserve looks awesome but is awesomely expensive too.

 

Rob, there are two reasons I keep and backup all my files.

 

1. I look at what people order and realize I can't predict what will sell.

 

2. I don't have a fast and efficient way of selecting which ones I wouldn't keep. Sure, I can

select my four and five star images and save just those, but I've made more money on

sales of images that I never even rated at all, and don't necessarily stand out as anything

amazing. So I'm back to point 1 again.

Agree with the expensive part..but whats more expensive between lossing 100 000 images, or buying something a ultra secure way of backuping it?

 

For a normal consumer, hard drive or other should be enough..but for someone that need a professional way of keeping is image, because he is a pro himself..money shouldtn be a limit in a certain way.

 

at least thats my way of seing it..just my oppinion : )

Multiple HDs, multiple locations...

"What do the rest of you do, and what suggestions do you have?"

 

Archive them to hard drives. Change your mindset: these things are are disposable.

 

Buy bare drives, transfer the images over, send them out for remote storage. Space rental in environmentally controlled, guarded data vaults can be had for about $50/month. Google Iron Mountain, then go down the list. The price differentials can be large; do your own due diligence.

 

Oh, don't bother with tapes, for many reasons. The biggest issue is that it's a niche technology with a handful of incompatible formats. An archive isn't any good if you can't find a drive to read your particular media in a few years.

Guys:

My aoplogies for the delay in getting back to you. I have beeen out of town on jobs.

So many suggestions that I can't thank each of you personally. As i think of it there are 2 main issues:

1 Back storage of the images already in my machine. I think this may be the nub of the problem. I have begun using MyBooks and duplicating when I import, using Lightroom, which I am going off, since it doesn't give the file quality of say Bibble or CS3, and the catalogues get flaky above 30k images. So I need to have a way of backing up all that data so I can retrieve them when needed.

2. Forward storage. I am looking for a good downloader that will enable me to save in multiple locations, ie to NAS/server. Wouldn't it be great if you could burn at the same time. Any suggestions for good backup software?

I really do appreciate you comments and help

Kind Regards

Tony

O . and I have an offsite fireproof safe..which I do use...