Total Digital Archiving Solution

[eddiem]

Greetings,

 

<p>I'm working with an uncle who's now 70 to create a searchable digital archive/database of his 20,000 plus family slides, created over the past 50 years. (He's 70 now). </p>

<p>We can't afford to have someone scan everything for us, and I haven't been able to find any software that really meets our requirements at a decent price. (see link to our requirements, below)</p>

<p>So, I'm probably going to get some friends to write a database for us, and get a Nikon SuperCoolscan and a hefty PC. I've posted the <a target="_blank" href="http://www.wematthews.com/photodb.html">initial project specifications</a> and invite you to take a look. </p>

<p>What are photo.net readers doing to archive their digital images, and how are you planning for a lifetime collection of them?</p>

<p>Brian M., have the Photo.net coders thought about selling a packaged stripped down photo.net archiving software product?</p>

<p>Note that I am aware of Portfolio Extensis, and also have found a very nice inexpensive indexer called BR Software. Neither is OS agnostic at this point, and at least BR isn't likely ever to be (written with a MS coding tool for Windows)</p>

Thanks<br/>

Ed Matthews, g8orade@Yahoo.com

[carl smith]

If it's 35mm you're scanning, don't get the Nikon 8000. IN fact I wouldn't get it if I were scanning medium format because of the banding problem (see <a href="http://www.rit.edu/~cgs2794/comparison.htm">here</a>).<br><br>

 

What scanner should you get? If you're scanning mounted trannies (slides), you probably won't wont the Nikon 4000 unless you want to spend another $500 on the bulk slide loader. It will hold 500 slides and allow you to scan them all relatively easily. As far as how easily it jams, well that depends on whether your slide mounts are flat.

 

Otherwise you might have to look at other scanner options.

 

Also, if you intend for these files to keep and be most versatile and compatible, a low resolution tiff would be better than the same resolution jpg. Just a thought.

[gwilburn]

<p>Ed,</p>

 

<p>I'm doing something similar with our family photos. Scanning the entire lot. I've created system in Perl that works for my needs and have spent some time developing a rational file-naming convention.</p>

 

<p>Check this <a href="http://www.northernjourney.com/photo/articles/">link</a> for full details. I use a tab-delimited database for cataloging and indexing and creating an index.html "contact sheet" with info, for every CD I burn. Also a subject-index module (which is still in beta but works).</p>

 

<p>Gene</p>

[kelly_flanigan1]

20,000 images at 5 minutes per image is 100,000 minutes = 1667 hours =208 days @8hours/day.<BR><BR>The 5 minutes per image is for prescans; scanning image; naming file; adding data; and burning a CD. <BR><BR>Scanning can be a sink hole for time; in scanning slides for customers; I adjust colors and contrast etc to revive tired old faded slides. many times this requires calling up the customer to ask them what color their old car was etc.

[b2 -]

Ed,

 

Sounds like a great project. My father-in-law has a simular blackhole of slides that would be great to do. In looking over your requirements, you missed something that Kelly started to do, backup.

 

I am not sure of the best way to do this, but I know I would not use tape. CDs are not big enough, DVDs? You need to be able to back up your system, and the data. Part of me says get an inexpensive PC with a lot of dasd (disk space) and back up your primary system to it. Long term,you need to look at burning DVDs.

 

Another thought is to look into using XML to export your data into a more universal format. I do not think that trying to keep everything (images and data) in one large database will work well, but I do not have any first hand experience with the software you are looking at. You might look at UDB (DB2 version) from IBM. I think that they have a single user version for Linux for free. That may have changed though.

 

You are building a wonderful thing that would be great to share with family members on CD or DVD if you can find/build the right software to export images and data in a way that you can view it from a DVD player.

 

Best of luck and let us know how you make out. I think a lot of folks out there have one or two of these in our families (I think I might be one and not know it!?!!?).

 

B2 (;->

[tony_terlecki]

Ed,

 

When you say you want a piece of software that is OS agnostic what exactly do you mean? Given the swift movement of technology where OSs and software can be obsolete after a few years I would think that only the database containing all the information would be of importance - as long as that can be exported via an open format then you should be in good shape.

 

I think IMatch is a very good program for the cost and the author's committment to open database standards bodes well for ensuring access to your data in the future whatever way technology goes.

[eddiem]

<p>Wow, Thanks to everyone who has responded so quickly. I'll have to incorporate more details on the project spec page in the coming week, as my friend Pat and I talk through it some more.

In the mean time, here are responses to some of your posts:</p>

<p>To Carl Smith, we are scanning all mounted 35mm Kodachrome and Ektachrome. We do want the bulk loader, so I guess we're looking at the 4000? An option we ruled out was to have someone put these down on a high end professional flatbed scanner and do a bunch at once like that--too expensive for the labor, given the numbers.<br> Can you tell me more about the low res tiffs? We want to get the file format right the first time... :-)</p>

<p>To Gene Wilburn and Bill Bingham, I'm glad to see others are home brewing their own solutions. Regarding name, we are not planning on a human readable filename at all, rather storing all of the information, including the path, in the database. The data will be text in a format that can be exported and imported as plain text. Bill, we aren't planning on storing the file itself in the database, just the metadata, with a pointer to the files. The files will remain in a filesystem. Hopefully, Linux will stay in a state that lets us move the files and data to an upgrade as needed. We are considering how to export the metadata into XML, the XML schema shouldn't be too complicated--but I'm still learning XML. As far as backup, we'll probably put the whole thing onto a mirrored RAID drive system of some kind, and for backup, maybe just copy the whole thing off to an external hard drive like you said. We plan on putting it up on the web for access, and maybe giving copies to family members who want them also on said external HardDrive(s). How many MB/GB does a DVD hold? We'd have to have an index that pointed to the CD or DVD number, something we haven't considered yet. Hmmm.</p>

<p>To Kelly Flanigan! Yes, we have run similar numbers, but thanks for pointing out for everyone what a huge effort it is. That's why we need to get started asap. My uncle is mostly retired from a pathology practice he helped found and barring any calamities plans to work on this maybe for the rest of his life. All of the slides are at his house. Obviously we need to automate as much as possible. The DB will contain a pointer back to the physical slide location based on box/tray number, so until the slides go bad, there's that, too. </p>

<p>To Tony Terlecki--We want something that can run today on Windows, Macintosh, and X Windows in Linux/Unix. Thus, we either have to write the front end to run in a browser that runs on all three (Mozilla), or we have to use something that has its own libraries that will run on all three. We don't know java, but we do know tcl, and could probably pick up PHP to run in a browser. So, it's not just where and how the data are stored, it's what OS OR cross OS application are they going to use to look at it. <br>Also there's eliminating all software license fees for the audience, which this solution does. So far, there are NO SOFTWARE COSTS (building nor running it) involved in this project.</p>

<p>Again, THANKS for the swift and helpful responses; please keep them coming. I'll try to add more details to the "official plan" as posted on my website and post an update message here when they're there. We hope to have the pilot running by the end of November.</p>

[carl smith]

FYI: even with the slide loader, batch scanning with the Nikon 4000 of 50 slides at even 2000 dpi will take quite a while.

 

You mention you don't need a whole lot of information, and that you don't need big files. Unfortunately, the TIFF format has been around a while and hasn't changed. It's maintained better compatability than jpg has because who's to say that in 10 years when you want to go back to these, you can still read today's jpegs? The TIFFs have a better chance of still being readable.

 

The bad thing is your files will be a good bit bigger. If you want to print good 5x7s at 300 dpi, you can save your tiffs as 8bit files and they come to roughly 9.01222 megabytes. You can probably average them out to being roughly 9 megabytes because the file size ultimately depends on image detail and all that jazz. Certainly none will go over 10mb. In contrast, if you were to use jpgs, each file would likely be around a megabyte if saved as a relatively high quality jpg. Can you afford the 9 to 10 times increase in storage? Hard drives aren't too pricey, and for your archiving, if you're not accessing frequently you probably don't need fast drives, but also consider that you'll want to back this up.

 

Even still, if I were in your shoes I'd be using the tiff (.tif) format files.

[eddiem]

Hello all. Sorry I've been away from this for a little while. We're getting ready to buy the scanner.

Has anyone used the Super Coolscan 4000 from Nikon with the slide loader?

Can you load it up with 50 slides and walk away and it will scan and auto name/number each resulting image in the directory of your choice and with other default settings?

We want to get the scanning and storage moving and come back and import once we have the software.

We may end up using something that's already built, not building our own, but we're still researching a few products. I haven't updated the web site, but hope to this weekend.

I read in one review that slides in cardboard mounts may require some attention to keep the loader from grabbing two. Anyone with experience using the scanner and bulk feeder?

 

Thanks,

Ed

[mikko_issakainen]

Hello Ed!

 

I am also interested in developing/finding a digital archiving software & database solution. In my case "solution" sounds quite grand. :) I've thought about similiar archiving system for some time but until now I haven't written down any requirements. Here are my thoughts on XML.

 

Database products usually provide a way to export the data but I'm not totally convinced that it is what I want. I've worked with XML for a couple of years now and in my opinion it seems to be quite flexible. It requires more work than a simple flat file but it also is "self-documenting" (if done right) and more structured. The default values provided by DTDs and schemas are also an advantage. XML is (IMO) readable so chances are that it is usable for a very long time. XML is relatively easy to modify using XSLT transformations.

 

At the moment I only have perl scripts that use ImageMagick for transforming images for www use. I plan to develop a some sort of simple XML-file 'database' that can hold information on the photos. You have a very interesting project so I'm eager to see what will come out of it.

 

Regards,

Mikko Issakainen

[frank_bomarius]

Ed,

 

I am using The Nikon Coolscan 4000ED with batch slide scanner since more than a year. I have done several thousands of slide scans with it.

 

I used the ICE features to automatically restore color and eliminate fungus from slides which were shot in the sixtees (from my father). ICE worked very well - some of the slides were almost only orange and the scans came out 'like new'. BTW: did the same with color negs from the fifties.

However, ICE does not work well on frames with dirty glass. I take the slides outand put them in new frames without glass before scanning.

 

Regarding the mechanics: you CANNOT go away. Sometimes it does 50 slides' batches fine, sometimes every 2nd frame stops the scanner...

I now use very thin frames (no glass, CS-System). Old frames were often cartridge (Kodak) or thick glass frames. Had mecanical problems with all kinds of frames. I often just sit next to the scanner and have it do batch scanning while I read. When I hear that the frame sticks I know where to press a bit and it slips in before the scanner software bails out. That's my workaround :-)

 

Bottom line: the batch scanner could be better (mechanically) but I would have never done that job without it.

 

-- Frank

[paul_hirst]

Just in case anyone is still monitoring this thread -

 

I'm about to start something similar myself - I'd be interested ho hear what people here ended up with... I have in mind something much more capable than a filename convention - I'm using a postgresql database backend, with some perl scripts to populate the database. I'm currently at (I think) the final design stage of my database layout, when it occured to me that this must be a wheel being re-invented, so I thought I enquire what everyone else out there is using, or is everyone inventing there own?

[eddiem]

Paul,

 

Well, it's quite a while later and my project at least has not progressed.

I did notice the spec link wasn't working, and I fixed it, if you want to see what I was considering. I have yet to see something like a 'myphoto.net' or 'myphotosig' or 'myshutterfly' in an installable form as polished as BR's product. (photo.net is running on Oracle? I think, which immediately puts it out of the common man's hands...) The photoalbum approach being offered in several of the windows editor programs like photoshop elements, paintshop pro etc. isn't really it.

My uncle did finally get a new pc this spring and he may be coming around to buying a Supercoolscan.

I did get a suggestion of putting the keywords in the files themselves (custom exif tags?) and then building the indexes off of them that way. Any comments about that? It would keep the file and its metadata together.

Postgres can handle blobs and Pat and I looked at storing image files inside the postgres db in a very cursory fashion this spring, but that is all.

Note that the pgaccess project is up and running with a fairly active group of contributors now. I've heard pgadmin (windows only) is also worth a look, but haven't used it...

If you'd like some help critiquing functional or workflow requirements I'd be glad to take a look. I've been pondering this for a while, but the rest of my life (making money, raising family, uncle not really ready to do it) have put this on hold.

[matthew_george1]

I heard about your scanning project you have under taken with your uncle. I saw some posts online. If we can be of any assistance let us know. We are a scanning business. We are used to scanning large quantities for a lot less than other companies. If you would be interested in talking about our services maybe we can come up with a price that would satisfy your needs. Our regular scanning prices are online but we could come with some competitive pricing for larger orders. I hope your project goes well for you and your uncle. www.ExtremeHiRes.com

 

 

Matthew George

 

AFG Digital Film Scanning

[brainbubba_motornapkins]

If your time is worth anything at all, surely the most (and perhaps only) cost-effective way out of this is PhotoCD. DIY on a project of this magnitude will get real expensive, and tedious, real fast.

[sanaka_thompson]

If you don't need your scans to be exceptionally high res., you might think about a really good flatbed like the Epson 3200. On the DigitalFineart Yahoo forum I read that it can be quite adequate for up to 8 x 10 prints. Cool thing is you scan 12 neg. frames at a time. You could set up cropping in VueScan to auto name your files etc. Could be quite fast. $.02 :-)

 

Peace,

Sanaka

[chip l.]

Just came across this thread. As to software for cataloging, Extensis Portfolio seems to fit the bill. An advantage is that it is PC and MAC based.