External Storage solutions?

[fxt1]

<p>I need a bit of advice on external storage options - I currently have 5 1TB HDs in my PC, all dedicated to photos only, and all almost full up.. my current PC has run out of drive bays to fit more in.. Was thinking about building a new PC in a case that can house up to 8 HDs, but that's still only a temporary fix, as I have to archive so many photos that space goes pretty fast. <br>

I need a system that will allow me to access all 6+ HDs instantly, and be fast enough to work off and save to. Preferably something very reliable and expandable that will last for a very long time. Any pointers?</p>

[mendel_leisk]

<p>I'm just curious: do you have things set up so your files are always on two drives, ie: there's a pair of drives, one mirroring the content of the other?</p>

[fxt1]

<p>No.. I wasn't that clever when I built the system (that's RAID - right?) They're just NTFS Hard Drives - All Western Digital 1TB Caviar Black drives.. Because there is SO much data, I'm open to building a RAID system if it's the most sensible solution, but I'm guessing it'd probably take about a week to copy 5+TB to a new system? </p>

[fxt1]

<p>Oh.. and also - I'm upgrading computer soon anyway.. but may be adding a mac system.. so I need something that could be accessed from both PC and mac.. in my head, it's like an external box that I can just put all my hard drives in, and just keep adding more drives, and they'll all work happily there forever, and I can just connect E-sata cables (or something faster?) to both PC and MAC to use it as a giant storage box... but I don't think it's that easy?</p>

[starvy]

<p>Why are you thinking about MAC? With so much data at your disposal I think one platform offers a better solution. I did a quick search on the auction site and came across a 10 bay case. It wasn't that expensive. I wonder if you would consider getting a bigger case to your existing setup and add more drives?</p>

[fxt1]

<p>Considering getting a mac as it's better for video editing, which I do a little as well.. and I'm told that the workflow is slicker than that of windows.. I'm very pro-PC, so if I can build a substantially faster machine at a lesser cost then I'll stick to it. Yes, I've considered getting a bigger case.. but again, it'll only solve the for a few more years.. I get through about 32GB a week.. Would be nice if I could find something expandable to make it that bit more future proof.. but if there is nothing, then that's what I shall do.</p>

[MichaelChang]

<p>It seems to me that <a href="https://www.google.ca/search?q=network%20attached%20storage&hl=en&biw=1440&bih=813&ie=UTF-8&sa=N&tab=iw&ei=wAo1UfznH-e_0gGk4YHgDA">Network Attached Storage (NAS)</a> and <a href="https://www.google.ca/search?q=Storage+Area+Networks+(SAN)&aq=f&oq=Storage+Area+Networks+(SAN)&aqs=chrome.0.57&sourceid=chrome&ie=UTF-8">Storage Area Networks (SAN)</a> will likely be the most suitable solution.</p>

<p>There are many tutorials and details in the links for hardware/software requirements and system capability and expandability. </p>

 

[walter_degroot]

<p>since both platforms often use similar programs to edit video files and the gap<br>

has been steadily decreasing. . It may offer only a small advantage to changing or<br>

adding a different system,. mac's today use intel cpu's. so what is the great difference .<br>

you will need different spoftware to run under the different operating system.<br>

I think you really have a HUGE about of data. how much is duplication?<br>

Since the data is all in one pc case. what happens in case of a disaster,<br>

a virus a flood a power surge or a fire or theft.<br>

all or part would be lost.. It is time to think of off site storage.<br>

Not a upload and save service like carbonite.<br>

But external drives stored off site.<br>

Raid systems are used in industry. but some that mirror your data on two drives<br>

are useless if one of the two fails.<br>

other raid solutions are more protective of your data.<br>

You have chosen good drives. but if you buy several at once and the same lot.<br>

it is possible all or several could fail at almost the same time.<br>

a big plus to you for buying drives by brand and not an iuknown drive in a box by a " box maker"</p>

[fxt1]

<p>So... what about this solution: <br>

1. Initially upgrade to a new PC with more bays for HDDs - which should last me another few years.. and I'll see what technology is available then.<br>

2. Start backing up to external RAID based NAS/SAN system (not the kind that mirrors data) and keep it somewhere else.<br>

That's probably the most sensible course of action, I guess?</p>

[goldbergbarry]

<p>Ran across a similar thread on another board. The consensus was that Synology had a good line of devices worth looking at (http://www.synology.com/products/product.php?product_name=DS413j&lang=us)</p>

<p> </p>

[michael_young3]

<p>Good consumer grade, 3-TB SATA III drives are under $150 each. It wouldn't be difficult to scheme up a way to shuffle the contents of 3 current drives onto a single new drive.</p>

<p>Striping alone for faster access might not be such a great idea. A single drive failure loses up to 6 TB at a time, more than your entire current library. RAID 1 mirroring gives some safety by storing two copies of everything.<br>

RAID 10, mirror and block level striping, is a reasonable plan. 4x 3-TB drives makes a total of 6 TB striped+mirrored storage. 5 total drive bays is just enough to clone your system drive, and then copy one drive at a time to its new home.</p>

<p>(And then you get to do it all over again in a few years. How long does it take to generate 1 TB of keepers? Longer than it takes drive manufacturers to meaningfully increase cheap drive space?)</p>

[David_Cavan]

<p>I'm just wondering what you are using now for external storage and archiving options? Could you build on that idea? Do you have a need for all 5TB on-line now?</p>

[fxt1]

<p>When you say 'stripping alone'.. do you mean, just sticking a bunch of drives in a workstation and working off them as they are, without any clever mirroring? Sorry.. a little clueless with this terminology..<br>

Right now I'm not using anything for external storage - I have 5x 1TB HDs in my current computer (and a few more in a very temperamental old machine, and most of my edited finals are hosted on my web server.. but that's only a very small percentage).. I need to upgrade to a 64x system anyway, so that's happening - I shall ask you guys for spec advice shortly.. (hah)..<br>

I'll leave the system disk in this computer, and just use this machine for admin - and start a clean install on the new machine.. which will also have lots of drive bays.. I'd thought about using higher capacity drives, for some reason I didn't think they were as stable as the 1TB drives, but perhaps that's old information? I do need to be able to access them pretty quickly - and as each shoot (at least from the past few years) is around 3-5GB, having them anywhere online or some sort of slower access storage wouldn't really work... </p>

[mendel_leisk]

<p>Not necessarily RAID. Look into Windows RoboCopy. It can ensure 2 drives, or various folders in 2 drives, have the same files.</p>

[gduffy]

<p>“I currently have 5 1TB HDs in my PC, all dedicated to photos only, and all almost full up"<br>

Frank, that’s a lot of photos! Something like 200,000 25mb files. Even if they are PS files that’s still a lot. With that big of a collection I would suggest that you come up with a solid back-up strategy soon that includes remote storage. It would be catastrophic to lose so much.</p>

 

[hjoseph7]

<p>"<em>1. Initially upgrade to a new PC with more bays for HDDs - which should last me another few years.. and I'll see what technology is available then."</em></p>

<p>Have you considered deleting some stuff ?</p>

[francisco_salaquanda]

<p>If one looks back over the last 20 years the computer industry has had five media format changes not including mid range and mainframe systems. In another five years it will have changed again, so one needs to be ready for another upgrade or format change. Even the granite disk format, though offering bomb proof durability, will have to upgrade or change as well.<br>

There is one intrepid New York company who won a tender to archive their digital copies of precious documents in their collection. What was the solution? LF film! Yes folks, what goes around, comes around.</p>

 

[stevenseelig]

<p>This question makes me ask a couple of questions and a few comments</p>

<p>1. What is your backup and archive strategy. RAID systems can NOT be considered a backup system. It is more useful to think of them as more stable non-backedup systems. Bottom line...you need to have separate backup drives even for your RAID drives. I would also encourage you to have an archive for each of backup drives.</p>

<p>2. I have found that my storage requirements continue to expand. I currently have 28 TB of pictures/video attached to my computer. 16 TB are two different 8 TB DROBO RAID 5 drives, and 12 TB of RAID 0 Western Digital Thunderbolt drives (2 x 6 TB). One of the of the 6TB drives holds my images from 2011, 2012 and hopefully will be adequate for 2013. Each of the DROBO drives and the RAID 0 Western Digital drive has a backup and archive drives. RAID 0 drives are fast, but do not replicate itself so they are sensitive to any error.<br>

3. For portability, I also place working projects on Western Digital Passport USB 2.0, 3.0 or Firewire and work off of those drives with several different computers. I also backup those portable drives on a nightly basis.<br>

I buy drives based on price/GB and performance. I get as big as I can get.</p>

[cudacar2000]

<p>Pay attention to what Steven said. RAID IS NOT A BACKUP PLAN! Even aside from the potential issues recovering form a failure (disk failures are pretty easy to recover from, but controller failures may not be so easy), RAID controllers are designed to calculate and distribute the redundant data quickly and with as little latency as possible. That means if you overwrite a file, the redundant copy (or parity) is overwritten within milliseconds. A program, virus or memory problem that causes disk corruption can destroy data just as fast on a RAID system as it can on a single drive (sometimes faster). User error (probably the largest cause of lost data) is not protected against - delete a folder of file and it's just as hard to retrieve (or harder) as it would be on a single disk.<br>

RAID is not a backup solution, it is an availability enhancement. Corporations that spend many hundreds of thousands on redundant data storage systems still use backup systems for a reason. With or without RAID, some other system for backup (hopefully off site) is the ONLY way to be remotely sure that your data is safe. In my opinion, some read only media in addition to hard dives is a minimum level of protection. I will not rely on only DVDs (or Blu Rays) but they will always be part of my backup strategy.</p>

[walter_degroot]

<p>a persom I knew worked for a medical billing company.<br>

they kept data on large portabel drives. this were different then.<br>

there were at least three copies<br>

on premises , in this mans car trunk and a third copy at his home<br>

I suspect there were others. Thos was when viruses were not a srious problem and they were not running windows</p>

<p>another considerationb is backing up your operating drive.<br>

keep that drive separate. it does not have to be huge.,<br>

a second drive kept locked up nearby will save hours or days of hard work.<br>

clonezilla is a useful too to make a copy or the operating systm.<br>

it is free.<br>

put date on data drive and programs an OS on a smaller drive.<br>

You are wise to be concered and seeking practical solutions and safeguards</p>

[ellis_vener_photography]

"When you say 'stripping alone'.. do you mean, just sticking a bunch of drives in a workstation and working off them as

they are, without any clever mirroring? Sorry.. a little clueless with this terminology.."

 

there are three basic storage techniques used in RAID arrays:

 

Striping writes data across the drives in the array using the fastest available sections of the drives. This is good for speed

and total capacity. -the total capacity of all of the drives is seen as one giant volume. Striping alone is lousy for storage

because lose one drive and you lose all of the data across all of the drives. RAID 0 is just striping.

 

Mirroring requires a minimum of two drives and after that an even number of drives. But the total capacity is half of the

total capacity of all of the drives.in mirroring you can lose 1 drive and your data remains intact and when you replace the

faulty drive what was on the bad drive one is replaced by the remaining copy.

 

RAID 10 , also known as RAID 1+0 , uses striping plus mirroring for redundancy and speed. It requires a minimum of four

drives.

 

Distributed block level parity used in RAID 5 & 6 is where RAID gets really interesting for storage. It requires a minimum

of three drives (four or more is better though). RAID 5 & RAID 6 use striping for speed but also create a set of parity

check numbers. This allows you to lose one drive in RAID 5 and the check parity numbers are used to re-create the data

that was striped on the drive that died. RAID 6 works the same way as RAID 5 does except that that there are two sets of

block parity check numbers.this lets you lose up to two drives simultaneously. But the cost for this second set of parity

values is less total space than RAID 5 but more than RAID 1+0.

 

 

That's as simply as I can describe in layman's terms what the words striping, mirroring and distributed parity mean in the

context of RAID and how different levels of RAID work.

 

I use RAID 5 as it hits the sweet point for me in balancing capacity, redundancy and speed.

 

I also use Drobo's BeyondRAID systems but that is a different discussion.

 

 

A caution: redundancy in a RAID is not the same thing as having independent backups.

 

As for hard disk drives the best value on the market these days for RAID or NAS storage are the 2 and 3 TB Western

Digital Red series.

[David_Cavan]

<p>My opinion is that its very important that as part of this re-architecting process you come up with a back-up strategy that has multiple copies of your photos, separated in different locations. I personally couldn't sleep if I had 200,000 photos on a machine that is subject to fire, flood, viruses and user-error. I expect that will end up being the most valuable part of the plan - as someone pointed out earlier you can likely upgrade your current system with a larger cabinet capable of holding more drives for now. </p>

[MichaelChang]

<p>MaximumPC has an article on a DIY server based on FreeNAS and some pretty fancy hardware for a total of 12TB of storage:<br>

<a href="http://www.maximumpc.com/print/article/features/roll_your_own_home_server2013">http://www.maximumpc.com/print/article/features/roll_your_own_home_server2013</a></p>

<p>On a smaller scale and also based on FreeNAS is a system based on a less expensive ASUS/AMD mobo which will accommodate 16GB of RAM and 6 SATA drives, and in this DIY example the author configured an 8GB system:<br>

<a href="http://blog.brianmoses.net/2013/01/diy-nas-2013-edition.html">http://blog.brianmoses.net/2013/01/diy-nas-2013-edition.html</a></p>

<p>Here are some links to FreeNAS and there are also many YouTube videos describing its application. It can even run in virtual mode within your primary OS if desired:<br>

<a href="http://www.freenas.org/">http://www.freenas.org/</a><br>

FreeNAS videos: <a href="http://www.freenas.org/features/videos">http://www.freenas.org/features/videos</a><br>

<a href="http://doc.freenas.org/index.php/Main_Page">http://doc.freenas.org/index.php/Main_Page</a> - WIKI page<br>

and [<a href="http://www.youtube.com/results?search_query=freenas&oq=freenas&gs_l=youtube.3..0l10.1799.3275.0.3551.7.6.0.1.1.0.98.444.6.6.0...0.0...1ac.1.sgQ99dXS5dk">YouTube videos</a>]</p>

[ellis_vener_photography]

Going back to the original question

 

You say you have 5 1TB hard disk drives installed in your current PC (I'm not going to get into a discussion of Windows

vs Mac OS X vs Linux, because it really is a matter of personal preference and this is a pointless political "my way is

better than your way" non-discussion)

 

Unless your system sees all of those drives as a single volume, you don't have a RAID, you have what is affectionately

known as JBOD or Just a Bunch Of Drives.

 

I can make a couple of suggestions that will help performance

 

1) keep the drive that has your OS and applications as open as possible and keep your photos and music on other drives.

Ideally switch this main drive to an SSD (Solid State Drive) from an HDD (Hard Disk Drive).

2) Install as much RAM as your machine will hold.

3) make one or two of your internal drives your primary scratch disks for your applications that use scratch disks (

Photoshop and other Adobe Creative Suite applications).

4) if you use Lightroom make one of those drives your Lightroom cache drive. With a image library the size of yours make

the cache around 50-100GB.

5) use your remaining two internal drives as a RAID 0 array for hot active projects -especially for video - but definitely not

for storage.

 

6) External storage: A RAID 5, RAID 6 or Drobo 5N (set up for either single or dual redundancy) array as a NAS

(Network Access Storage). If you have a wireless hub with multiple gigabit Ethernet ports, connect your primary editing

computer via Ethernet cable. Your laptop can use wireless access.

 

I really like the new Drobo 5 systems for three reasons: RAID 6-like dual distributed redundancy, a built in power backup

supply that provides just enough power for the system to shut down safely in case of power loss; this keeps your data file

structure healthy. Premature shut down causes a lot of the file structure corruption problems that both RAID and DROBO

users see. And finally in the bottom panel of the DROBO 5 cases there is a slot for adding an SSD (64 or 128GB is fine)

that will serve as a cache (repository) for frequently accessed bits of data like Lightroom previews. That can speed hints

up.

 

The Synology, OWC, QNAP, and Western Digital Sentinel 4000 RAID arrays are all terrific.

Keep in mind that when you set up a RAID array you should be looking down the road a couple of years to project what

your storage needs are likely going to be when deciding what make, type and size HDD'S to populate it with.

DROBO'sWhen you say 'stripping alone'.. do you mean, just sticking a bunch of drives in a workstation and working off

them as they are, without any clever mirroring? Sorry.. a little clueless with this terminology.. BeyondRAID technology lets

scale up capacity as you need it but as with RAID the fuller the system is everytime you swap a drive it takes a long, long

time to write the data to the new drive.

 

This has been a long post and I apologize for that, but I've recently been wrestling with the same problems and a similar

number of images.

[ellis_vener_photography]

And as everyone has been saying neither RAID or Drobo, despite the built-in redundancy is backup.