File corruption

<p>Hi Forum<br>

Over the last five years I have seen this problem on a number of my stored images. I work on a mac and now store to an iomega drive. Although before that, they were on other drives and I moved them over to the iomega about four years ago. Has anyone else experienced this form of file corruption?</p>

<p>Thanks<br>

Karl</p>

<p>Digital data ages just like an old VCR tape. Eventually some bits won't be reliably readable. Or your copy from one media to another over time will eventually miss a bit and fail write it correctly, especially if you don't do a read of the data to make sure it's good. I'm on the PC and use FastSum which creates an MD5 hash of my files in an open source format (md5summer can also read the same files - its free, but the interface is not as good as FastSum and its not as fast). I end up with a single file in the folder where I keep all my finished photos that I want to keep for many years. This file contains a checksum for every photo. If the photo changes, the checksum won't match.</p>

<p>I'm not totally paranoid though and only run the checker every several months or so on my photo archive to check the old stuff and add the new stuff - FastSum simply shows a list of whats changed, whats missing and whats new. A photo from the 80's should not have changed. A photo from last week should have. So its easy to spot problems and its fairly fast if you let it rip while you are not doing anything else.</p>

<p>I've found occasional old stuff that was corrupt and I simply restored those files from backup (you need backups) and I know my old photos are still good. It also lets you test a backup if you copy the entire thing to a new drive. The errors are rare, but over the years I would have lost work. Now I know it's good.</p>

 

<blockquote>

<p>Digital data ages just like an old VCR tape.</p>

</blockquote>

<p><br /><br>

That's not true at all. Digital data never ages. <em>Media</em> may develop problems, but it's not digital data aging.</p>

 

<blockquote>

<p>Or your copy from one media to another over time will eventually miss a bit and fail write it correctly</p>

</blockquote>

<p><br /><br>

Copying almost always includes verification. If it doesn't, it can be turned on. This is never a problem for anyone that take a minute to understand how their system works.</p>

 

<p>As Jeff said, digital data--the zeros and ones--don't degrade over time. The media they're stored on are, mostly, mechanical devices and thus subject to wear and tear, which is why you should have redundant backup system in place. I use three backup sets--a network area storage at home online, which is the main backup. Then a USB drive, which is not online but backed up to on a regular interval (once a week or so, depending on how much shooting I do). Then an identical USB drive that's stored at my work, which is rotated with the home USB drive on a monthly basis. I feel having a regularly rotated set of backups offsite (i.e., not at home) is crucial.</p>

<p> </p>

<p>Digital files usually hase some sort of Cyclic Reduncay Check (CRC), or CRC32, or CRC64, etc..<br>

This assures that perfect copy is obtained. If the CRC comparizon fails, the entire file is most likely considered corrupted. If this happens, and is not recoverable, you need to get the copy from a backup media, or a second copy of it. So, it is crucial to have backup, or even second backup, or offsite storage.</p>

 

<p>I do MD5 sums similar to what Stephen does. I have about 3TB of data and in 5 years I have seen 2 files go bad. Both files were large video files and not in a critical part. The files still played fine without any noticeable artifact or damage. The hard drives report no bad sectors. I restored the files from backup anyway because my 2 backups matched while the main drive in the computer did not.</p>

<p>There has been research like this that show cosmic rays can and do damage computer chips. Hard drives and tapes use magnetic fields which could be damaged from other magnets.<br>

http://www.newscientist.com/blog/technology/2008/03/do-we-need-cosmic-ray-alerts-for.html</p>

<p>Just a quick addition.</p>

<p>The Mac comes with an MD5 checksum generator -- via the Unix command terminal. There are several books and online sources for information on how to access the terminal on the Mac, I won't expand on it here. The command is simply MD5and it returns the checksums, which you can put into a file or pipe to diff to compare against an existing set of checksums.</p>

<p>To be safe, I have three sets of backup drives and I occasionally erase, reformat and reload a backup drive just to be sure all my files have been recently recorded. It's not a lot of work, since the computer does it all, but it takes a long time to finish so I do it overnight. (The backup drives are 1TB in size)<br>

Gerald</p>

 

<p>I stand by my post. Magnetic media (ie hard drives) do actually degrade over time just like a video tape. The difference is that the bits are either 1 or 0 so its much easier to discern the value since its either there or not there. This is opposed to an analog recording on a video or cassette tape. Hard drives have use extensive error correction (http://en.wikipedia.org/wiki/Hard_disk_drive#Error_handling) to keep them looking like they have zero problems. Also all hard drives fail just like car engines. Its not if, its when.</p>

<p>All that aside, the Md5 and other CRC applications give you data integrity. Just because you write the data and get a valid CRC today doesn't mean that 3 years from now when you go to read it that it will still be in the same exact shape and you will still be able to read it.</p>

<p>Like Walt I've only had a few errors over time, but it really depends on what your intent is. If like me you want your files to last the rest of your life and be passed down generations, even a few errors every few years add up over the long run. And you don't get to choose which files have the errors. This is an easy way to make sure what you copied over is what you intended. No OS errors, no data errors. Of course it all depends on you fault tolerance. Race photos don't need to last the same as family photos for me at least.</p>

<p>Sorry if that sounds like a defensive rant. Good backups first, error checking is optional if you want.</p>

 

<p>And rereading I agree that 1's and 0's don't degrade, but the media does.</p>

<p>Thanx guys. Some good ideas here. I do have two sets of back ups, I will look into error checking.</p>

<p>Hi Karl, I have never experienced corrupted files on any medium (exept on some .UDF format files when writing to an external CD drive many years ago). I always move the files by copy and paste, not by the move command (I don't have a Mac). I was adviced to do so once by someone who knows more about data than I do. It has worked fine for me. I have been using Western Digital external drives for some years now.</p>