Organizing and archiving thousands of photos - which is best?

[jacques c pelletier]

<p>Hi to all!<br /> I have not posted here for a VERY long time but I can assure you that I visit the forum on a daily basis.</p>

<p>I have read through a few threads here and elsewhere regarding a system which:<br /> will help me organize and archive litterally tens of thousands of photos taken over the past three years and stored on external hard disk drives.<br /> I am now totally confused as there are dozens of good-great-excellent software suggestions, one always claiming to be better than "the others".<br /> Some are free (ex.: Picassa for instance), most are for sale from a few dollars to a few hundreds of dollars!<br>

My very inefficient storage and "archiving" set up at the present time is:<br /> 2 external HDD's of 1 TB each in which I have hundreds of "named" folders containing "named" photos. There are no keywords or "ratings" attached to the image.<br>

I have RAW (DNG) and JPEG files, so it would be nice to be able to visualize the RAW images as thumbnails as well as the JPEG's.<br>

The problem is when I want to search for a particular image, it takes forever, of course. Especially when I am not sure on which external HDD the said photo is supposed to be.</p>

<p>I am looking for a solution before this really gets out of hand ...<br /> Any suggestion would be much, much appreciated.</p>

<p>Thanking you all in advance,</p>

<p>JP</p>

<p> </p>

[Mary Doo]

<p>I think nothing beats Adobe Lightroom for organization and workflow.</p>

<p>Nat Coalson's book, <em><a href="http://www.amazon.com/Lightroom-Streamlining-Digital-Photography-Process/dp/047060705X/ref=sr_1_2?ie=UTF8&qid=1316662432&sr=8-2">Lightroom 3: Streamlining Your Digital Photography Process</a></em>, is terrific.</p>

<p>Good luck!</p>

 

[Charles_Webster]

<p>Check out "The DAM Book - Digital Asset Management for Photographers" by Peter Krough. It will help you understand the consequences of any organization system you choose.</p>

<p><Chas><br /><br /></p>

[will_daniel1]

You have already seen all of the software suggestions, so it won't help you much to say that I use ThumbsPlus from www.cerious.com. But I will offer one piece of hardware advice: Figure out what size hard drive you need in order to put all of your images on ONE drive only. Then buy two of them, one for backup (or three of them). I had your same problem, and placing them all on one drive solved it. Also, in any given folder I keep raw, JPEG and sometimes TIFF formats of the same image in subfolders. I generally use the JPEG folder for viewing purposes, and can quickly get to the raw folder when necessary. Hope this helps.

[nicolas_bl]

<p>I also use Lightroom 3 this is the best not only for archiving but as well for post production treatment...<br>

My favorite future is to be able to edit an image without out touching at all the original one, also to be able to work with very little few CPU memory (it use a low definition image then make a render at the end on the exportation but not on the treatment, it's a very smart way of working images...)<br>

good luck bye.</p>

[patricklavoie]

<p>lightroom 3</p>

[jim_service]

<p>I'm in agreement with Will. Thumbs+ is great. Also, you can get a 3TB USB 2/3 external drive at Best Buy for less than $140. How much is that per Gig?! Get two and you're set.</p>

[Tim_Lookingbill]

<p>I'm in your same situation, Jacques. Been that way since I began my 1000 or so Raw/Jpegs collection when I started shooting with my Pentax K100D DSLR in 2007.</p>

<p>I'm using Adobe Bridge CS3 and don't use keywords and am still culling through all my images tossing out ones that just don't hold up aesthetically.</p>

<p>The way I remember and find my images is by assigning a unique name to each folder of images taken during one shoot session that fit on a 1gig SD card according to place, date and subject . I don't bother naming through keywording every Raw and jpeg shot. Too much work.</p>

<p>I now just click on my collection of uniquely named folders in my Picture folder in Bridge set to Filmstrip View Mode which displays small thumbnails across the bottom with the main Preview on top and start editing or tossing.</p>

[patricklavoie]

<p><em>I don't bother naming through keywording every Raw and jpeg shot. Too much work.</em></p>

<p>in Lightroom, it could be done in 2sec for 1 or 100000000000 images... simply select them all (or the one you want to rename) hit F2.. and apply the name you want with increasing number, or (too) many option that you need.</p>

<p>I dont use keyword either, but all my images have a unique name, same as the folder they are in... easy to find them even if i dont have acces to photoshop / bridge / lightroom by simply using the search feature in my Mac (spotlight).</p>

[Tim_Lookingbill]

<p>Patrick, what's the point of giving each individual image a keyword that describes it so it's easy to search for among a thousand others that are different which is the reason why you'ld want to do such a thing? That would require more than 2 seconds. Naming one folder containing images that were shot during the same session is easier.</p>

<p>It's a logistic not an execution issue which will be different according to what and how the photographer shoots. Giving a unique name to over 1000 images sounds counter productive and a PITA.</p>

<p>If all a photographer shoots is models in a studio for media reproduction, then some other naming and archiving convention would be implemented because they're all generally the same image subject, location but different dates. The name of the client and for what publication the model is being shot for (and maybe the clothes being modeled) would be a much more appropriate way to remember where that photo is located among a thousand or so.</p>

[patricklavoie]

<p>I meant i name the images itself, not the keyword sorry.</p>

<p>My folder is call let say PLAVOIE/CLIENTX_SEPT212011<br>

so each image inside this folder are call:<br>

plavoie_clientx_sept212011_0001<br>

plavoie_clientx_sept212011_0002<br>

plavoie_clientx_sept212011_0003 etc....</p>

<p><em>It's a logistic not an execution issue which will be different according to what and how the photographer shoots. Giving a unique name to over 1000 images sounds counter productive and a PITA.</em></p>

<p>not really, if you keep thing simple, a generic name can be use for 1 or 10000 images.. for example;</p>

<p>Im back from a trip to Mexico, i can simply call all my image MEXICO_SEPT2011_0001, 0002, 0003 etc... of course, if you start calling your image mexico_real_de_catorce_june_2011_a_nice_tree _over_a_edge_0001 its another story, and as you say insane and counter productive (in my point of view).</p>

<p>If i go back to mexico next month, just call all my file MEXICO_OCT2011_0001, 0002, 0003 etc...it is easy to find within Bridge or Lightroom.. but also with any search engine part of your system.</p>

<p>Keyword then can be use (if you have time) to mark some important element in the image (name of the town, exact day, people name etc...) Then again, if you have a lot of image let say from a certain town in Mexico, you simply select one, adjust the keyword, then select all the image that need that or those keyword and press the SYNC button.. again, all this is done (the SYNC) in sec.</p>

<p>Digital asset management shouldtn be hard to do.. i think it look hard because people want it hard.. i have a client that call is file by photographer name + camera + day+month+year+location+client name+shot01+product+file number.... thats is kind of seriously anal.. since every frame have not the same product in it ; )</p>

<p> </p>

[tom_mann1]

<p>Hi Tim - My system is similar to yours in that I have dated and named folders (eg, c:/20110921-staff_headshots), but I also embed the same information in the metadata fields. The reason is that should a copy of the image be moved from that folder, I still can easily find it, figure out where it came from, etc.. With PhotoMechanic, LR, and probably many other programs, it's just a few mouse clicks to embed the path, filename, etc. in metadata fields.</p>

<p>Cheers,</p>

<p>Tom M</p>

 

[patricklavoie]

<p><em>The reason is that should a copy of the image be moved from that folder, I still can easily find it, figure out where it came from, etc..</em></p>

<p>this is why i name the file itself.. Tom put this info in the metadata field.. (that mean that your image are call with the generic camera name like _MG_61090987.nef ?)</p>

[Tim_Lookingbill]

<p>This was the archiving system that was in use during my graphic art/prepress production years before digital imaging working for mom and pop ad agencies and ad specialty screenprinting companies in the '80's and late '90's.</p>

<p>Throughout the years these businesses accumulated tons of custom artwork occupying banks of file cabinets and homemade vertical flat files for artwork larger than standard letter size. Client "ABC" would come in wanting to reorder an old job he doesn't have the invoice for and now has to resort to visually describing the artwork. Our filing and artwork archiving system was organized by invoice numbers so we don't have a clue what client "ABC" is talking about. However, we could enter the client's name into our database which listed all invoice numbers associated with it.</p>

<p>Now that did actually only take 2 SECONDS to find the original artwork with that setup. Without it, we would've had to re-create the artwork from scratch.</p>

<p>You have to organize large tombs of individual objects/images according to how its going to be searched for years later. You sort of have to establish a method by thinking ahead to how future searches and cross referencing schemes-(invoice number to client name) will be conducted in case memory fails both the photographer and the client during a search among tons of images. </p>

[tom_mann1]

<p>That's right, Patrick. </p>

<p>First, I make sure that all my cameras assign unique file names. For example, my old d200 named files d2aXXXX.nef, d2bXXXX.nef, etc. My 1st d700 names files d7AaXXXX.nef, d7AbXXXX.nef, etc. My 2nd d700 names are d7BaXXXX, d7BbXXXX, etc. (where XXXX are all numbers). The actual prefixes I use have changed a bit over the years, but you get the idea.</p>

<p>After the in-camera file labeling is set, I don't do anything to unique identifier (ie, file name) assigned by the camera. If, in post processing, I make several variants of an image, I'll keep the file name as a root and append a description of what I did, e.g., d7Ba1234-700px_wide_for_web.jpg</p>

<p>That way, if I need to find a different variant of a particular image, the search couldn't be easier.</p>

<p>My technique is just an implementation of the old maxim to always use a unique identifier for each entry in catalogs / databases / libraries / etc. </p>

<p>Like you, if I don't have any image search and retrieve software running, I can just use the OS to find the right folder and search manually within that folder. If I have a simple search and retrieve software running (eg, Picasa), I easily can find (for example) all staff headshots no matter if that text string appears in the path name, the caption, as a keyword, etc. If I have LR or Portfolio running, I can do much more complex searches, if needed. The truth be told, even with well over 100k images in my system, I rarely have to use complicated searches to find what I'm looking for.</p>

<p>Tom M</p>

[Tim_Lookingbill]

<p>So ten years from now you'll be able to remember to search for "d7AaXXXX.nef" or "plavoie_clientx_sept212011_0001" when you or a client wants that particular image among 1000's.</p>

<p>Good luck.</p>

[patricklavoie]

yes. since all u have to do is search for MEXICO, all the mexico image will pop up.. if i type the year, well i narrow the

search... then visually i cant find what i need in particular.

 

if i had enter keyword, the search will be even faster.

 

not sure i still follow what whe are talking about Tim? are u saying that it is faster to search in all your archive visually

for what u need? because it seem whe use the same merhod to start with ( naming a folder) with a unique name... the

only thing is that i add this info also as the file name to all the included images... witch seem to be even more foolproof

dont u think?

[Tim_Lookingbill]

<p>My point is one has to consider the ability and methods used in remembering what search term to use on any given image ten years into the future and anticipate or know how it's going to be asked for. To get slightly metaphysical here, does an image really exist if no one asks for it or even remembers it ten years later and if not should it be kept? Would you know if you lost it due to corruption? Try searching for that and see how much time you spend/waste.</p>

<p>Keyword is really an appropriate term in this regard in that there must be a "key" (reminder string around the finger) associated with the person searching for an image forgotten ten years later. There must be an incentive to remember an image that's buried in 1000's of images on a photographer's hard drive. The "ABC" client mentioned above sort of illustrates this in that the client was associated with an invoice number "the key" at the point of knowing what to ask for. The next person may have a different key or memory description but no invoice number associated with them.</p>

<p>The key, Patrick, in your example above was MEXICO, but will you be able to rely on your memory of that key ten years later if someone asks for that one image associated with it but doesn't remember MEXICO either. Should you rely on associating an image with a description or should you associate it with its usage such as with a client's name>the name associated with an invoice number>invoice number associated with a job>associated with a check/credit card payment? What's your incentive for remembering? You're going to remember someone paying you money for an image, I'm sure.</p>

<p>IMO a photographer must anticipate how a future client or anybody asks for a needle in a haystack. Will you be compensated for the time it takes searching for it ten years down the road among a million of other images?</p>

<p>Ten year old images most likely will have to be archived to a backup system stored on a hard drive in another building, maybe in another town. Who knows. By then the photographer might have accumulated over a million images. That photographer's going to need a staff just to do searches on obscurely described long forgotten images.</p>

<p>You might try this test. Have someone, a friend or co-worker, randomly pick out and view an old image deep in your archive and do a cropped screen grab showing no number, name or anything to tell you what and where it is on the hard drive. See how fast it takes to find it.</p>

<p>Now have the same friend pick out an old image and just describe it to you verbally and see how fast it takes to find that.</p>

<p>Now think of doing this same routine ten years later.</p>

[Tim_Lookingbill]

<p>Well I went ahead and gave it try and attempted to provide descriptive keywords in the xmp files of some of my Raw images in Bridge. All keywords are different requiring individual descriptions for each. Did a test search using Find (Command-F) in Mac OS and the xmp file associated with image shows up immediately at the top of the list using the corresponding keyword search words.</p>

<p>I'm assuming the keywords are subsequently embedded into a final jpeg or tiff converting from Raw as EXIF data if I happen to later save the Raw in those formats.</p>

<p>It took far more than 2 seconds to enter the keywords for each image because it required coming up with appropriate descriptive keywords that might be used if no other data is available in describing the image like EXIF or file name or number.</p>

<p>Guess I should've done this while first viewing and editing each Raw. It wouldn't have saved time but at least it would've kept me awake during the process.</p>

<p>Photography can't be all fun, I guess.</p>

[dan_south]

Before you use any software package make sure that your files are organized with respect to how they are stored on

your drives. Hopefully, they're not all in one huge folder. Maybe you'll want to group them by year, place, or subject.

If you have done shoots for clients, you might want to have separate folders for each client and the date. E.g.

20110923-Simpson-Wedding. Or in subfolders such as Weddings\2011\Simpson.

 

 

IMO that's the place to start.

[patricklavoie]

<p><em>It took far more than 2 seconds to enter the keywords for each image because it required coming up with appropriate descriptive keywords that might be used if no other data is available in describing the image like EXIF or file name or number.</em></p>

<p>Tim, i dont know why you still buggy about 2sec+keyword... i never said that... i said 2sec to change THE FILE NAME.</p>

<p>Whats the point of having a Folder call (whatever method you use) for example PLAVOIE/MEXICO_SEPT2011...if the file inside are call _MG_3456789.cr2? So i said, that in Lightroom, you can select all those <strong>IMAGES</strong> and by pressing <strong>F2</strong>, change the <strong>NAME of the IMAGE</strong> itself... and it will take <strong>2SEC</strong> even if you have 1 or 100000000 IMAGES in your folder.</p>

<p>Im not talking about <strong>KEYWORD</strong> here, just the <strong>NAME of the IMAGE</strong>.</p>

<p>You can do the exact same thing in Bridge by going in the top menu under tool / Batch rename.. and quickly change the name to whatever you want to call them. You obviously dont need to change the name of each file one by one like a monk!</p>

<p>Of course, keywording take time... its a obvious thing. And of course, you can speed up this process also by using generic keywrod while importing or after import by using set of keyword related to your image ie: MEXICO, 2011, SEPTEMBER, TRAVEL, FAMILY, etc.... and then, you can go specific on each image or group of images ie: HOTEL X, LOCATION X, BEACH, CHURCH etc...</p>

<p>Since you can also use the SYNC button to apply them in batch, you can also save some time there.. it is a time consuming task at that stage, but later can save yourself a lot of trouble.</p>

<p>___</p>

<p>And yes, if a client call me 10 years later asking to find a image from Mexico i will find it.. how and why? Because i will using a search engine or lightroom (or else) will start by searching for is client name ie: tim_lookingbill, then with the result i have i will add MEXICO, from there it will be already pretty much obvious to find the correct image by sending you the link of the image whe work together, so you can choose the one you like.</p>

<p>But what about if whe work together often? and always in Mexico? Well i would have simply use your reference number or the month, or the year to get a even smaller number of image result.</p>

<p>What append if a client send me just a capture screen of a file without the name and anything? i will politely ask him to grab another one and make sure he get the name of the file on top of that image ; )</p>

[tom_mann1]

<p>Patrick - Your methods are so close to mine that we need to look into the possibility that we were twins separated at birth ;-) , but below is one minor difference in the way we work. </p>

<p>With respect to (w.r.t.) keywording vs file re-naming, Photo Mechanic provides for user-defined templates (...they call them "stationary pads") that allow one to store a complete set of variable substitution commands, boilerplate text, info on whether to replace or append the new info to each field, etc..</p>

<p>After I have given a descriptive name to a folder of images, I then open the contents of that folder in Photo Mechanic, hit Cntrl-A to select all the images, select the correct "stationary pad", and click "Apply to all Selected Images". With these three keystrokes (not including typing in the text that makes the folder name descriptive), I can have the folder name, and/or other parts of the path copied into the keyword field and/or any other IPTC field(s) that I want.</p>

<p>After this initial IPTC editing step, I then usually select subsets of the images in a folder and keyword each of these subsets as a group. For example, if the folder is for a wedding, I might add keywords like "preparation", "ceremony", "formals", "reception" to the corresponding groups of images in that folder. Even if I have more than one camera in use, I can sort by capture time instead of by file name, and select all the "ceremony" shots by little more than an initial mouse click followed by a shift-click on the image at the end of this group. It's an extraordinarily fast process, especially considering the advantages it provides when you need to pull some images.</p>

<p>BTW, I don't make the folder name part of the file name because I tend to append other descriptors to the file name and don't want the file name to get too large.</p>

<p>I'm sure one can do the same thing in other programs, but Photo Mechanic is the program I've used for the longest time and hence am most familiar with it for this type of task. I also know it's fast (both in # of keystrokes required and in processing time).</p>

<p>Tom M</p>

<p>PS - In case anyone is interested, info on Photo Mechanic's basic IPTC editing capabilities can be found here: http://www.camerabits.com/site/PhotoMechanic.php#Annotate</p>

<p>A full description on how to perform variable replacement in Photo Mechanic is here:<br>

a) http://camerabits.fileburst.com/Photo_Mechanic_4.5.3_Manual.pdf (page 46 and following pages)</p>

<p>A reasonably complete list of available variables begins on p. 48 and continues through p. 54. It should be pretty obvious that one can most anything you can imagine using their variable replacement system to enter/modify IPTC info in any set of files.</p>

<p>b) More useful info on this topic is here:<br>

http://www.camerabits.com/site/PMHelp/Help_Mac/pages/Captioning.html#code</p>

<p>... and no, I have absolutely nothing to do with Photo Mechanic except that I own and like their program.</p>

[Tim_Lookingbill]

<p>I have all my images in separate subfolders located in the one Mac OS Picture folder. That's not a good idea? If not, do I store each folder on the root level of a User Account? I've been putting all my image folders in the Picture folder going on 4 years now and have been backing them up to other hard drives since then without one lost or corrupt image. I even "drag and drop" copied the entire Picture folder once but now use Carbon Copy Cloner to clone the entire bootable hard drive to another. Takes 45 minutes in Mac OS 10.6.8 on my 2010 Mac Mini. Nice way of defragmenting the entire HD at the same time as well.</p>

<p>Sorry Patrick for misreading your "2 second" comment. There's quite a bit of ambiguous referencing going on in your translated posts...</p>

<blockquote>

<p>it could be done in 2sec for 1 or 100000000000 images...</p>

</blockquote>

<p>What's ambiguous here is how one could select that many at one time and name them in 2 seconds or are you referring to that it only takes 2 seconds to select and hit f2 which I did in a folder of images and it deselected all the selections down to one highlighted name entry to where I had to reselect individually. Bridge may be different than Lightroom in this respect.</p>

<p>Secondly I'm not sure how you could select that many images and give them <strong>different</strong> names that quick or are you talking about just naming them all the same name which I have to wonder if they would all default to a truncated number included in the new "same name" since you can't have all files have the same exact name in one folder. That wouldn't work anyway in my case from a practical standpoint since all my images are different.</p>

<p>At least this discussion is hashing out a lot of these image management strategies in a practical real world scenario which is going to get folks reading this a bit to think about. There should be different strategies and approaches between the hobbyist and the professional having many clients.</p>

<p>I know I couldn't keep nor dissect this much info in my head trying to read a 200-300 page DAM book on the subject.</p>

<p>Thanks for your input and clarifications, Patrick.</p>

<p> </p>

[tom_mann1]

<p>Tim L: <em> </em></p>

<p><em>"...My point is one has to consider the ability and methods used in remembering what search term to use on any given image ten years into the future and anticipate or know how it's going to be asked for..." </em>and<em> </em></p>

<p><em>"...Have someone, a friend or co-worker, randomly pick out and view an old image deep in your archive and do a cropped screen grab showing no number, name or anything to tell you what and where it is on the hard drive. See how fast it takes to find it. ..."</em></p>

<p>Your post raises some very good points, and I believe we had a bit of discussion on this in the past. My POV is that for the scenario you outlined, ie, millions of images acquired over decades, you essentially have no choice but to regard searching your archive no differently than one would search the archive of one of the stock agencies. Specifically, you probably won't remember if a specific image is in your archive, just like you certainly don't know ahead of time if the archive of a stock agency has an image of the type you are looking for. </p>

<p>So, what do you do about this? Well, you do exactly what the good stock agencies do: make sure each image (or set of images) not only has detailed keywords, but a set of good general keywords as well. </p>

<p>This way, if your hypothetical friend pulls and crops one of your images from 20 years ago, and all you can see is that it's a backlit B&W shot of a cardinal on a bird feeder in snow, and you don't remember anything else about that image, if you had entered the following general keywords: BW, bird, feeder, snow, winter, backlit, and then searched for some of these terms, your search would almost certainly yield a very small set of images that you could then quickly flip through to see which one you like. Even if you had also entered the exact date and time of the film capture, the camera, lens, and keywords like: female, cardinal, Cornell, laboratory, ornithology, christmas, "field trip", Ithaca, you would have no need to recall such specific info to pull up a reasonably small set of images.</p>

<p>The problem, at least for me, is that entering such incredibly general terms is boring, and hardly seems necessary at the time of entering the image into your catalog -- "Of course it's a bird. Any idiot knows that, so I'm going to enter "brown", "female", "molting", and a few other highly specific terms, that you may or may not recall years later.</p>

<p>Cheers,</p>

<p>Tom M</p>

[Tim_Lookingbill]

<p>Just checked out the Bridge's "Batch Rename" and I see what you're talking about, Patrick. I'm going to have to spend a long while sorting out what all the menus do. It's not a very intuitive interface to indicate what's going to happen with 90 selected Raw files once I hit "Rename".</p>