Camera review standards for photo.net -- opinions solicited

Discussion in 'Photo.net Site Help' started by philg, Mar 3, 2002.

  1. I'm trying to make sure that a whole new crop of people can be involved in writing camera reviews for photo.net. At the same time we want a reasonable degree of consistency among these reviews. To that end, I've drafted http://www.photo.net/editorial/camera-review-standards What do you guys think? Please respond in this thread. Thanks.
     
  2. Good idea, a bit like the comments on photographyreview.com but better implemented. Will make it easy to redirect questions of the type "Eos-5" or "Eos-30" (sorry, I'm in Europe). It may be of use to know which function of a camera is not only available but also easily accessed. I think it's not only important to know which functions in a camera are available, but also how many hands/fingers/brain cells you need to use them. Seems like a good idea to share knowledge between photo.net users. Together we know, well maybe not everything but at least a lot.
     
  3. Philip - this looks very sensible. If I wanted to do a review, would I 'reserve' that with you, then deliver a couple of weeks later (rather than just start writing and risk someone else duplicating this work)? Do you require writers to supply their own photographs, or do you prefer straight text?
     
  4. I agree, this is a good idea. And perhaps a more matured equipment opinions forum would be a possibility for a location for this sort of thing. As it is now, the equipment opinions will collect some possibly accurate reponses to a generally effective product (duds aside), but will eventually switch over to why this body sucks and the next brands is better. I noticed this on the Elan 7e, where it becomes an argument over whether the N80 is better.

    I am curious as to how exactly this system will work. Will people offer to produce a review and you will tell them "go ahead?" Or perhaps, its first come first serve, and new reviews get added to it, as an addendum? It would take a lot of work, but a few moderators could take existing and new reviews and take all the parts and fit them together, attributing them to the authors. I doubt that's really a likely or logical solution, but I am curious as to how you propose to choose which review goes up? Do we send them in, and based on your criteria, the existing one may get replaced?

    Well, I think this is a great idea, a good start for getting a good facility going and allowing real user opinions on possibly a wider array of equipment.
     
  5. This is just a thought: this is a popular enough web site that manufacturers might be willing to lend new releases to photo.net for review purposes. Of course, there would be some work involved in investigating this, but it might help broaden the review base (and maybe get reviews of new products up early).

    PS: Re SLR model naming conventions: "North American names" should be "USA names"... here in Canada, and I imagine Mexico too, we have the ever-dreaded "gray market" cameras
     
  6. How about daily unachieved announcements for a couple weeks before the archived thread on a product.

    I don't know how complex it is but you could connect a ratings system to the posts.
     
  7. Andrew: yes we do want the reviewers to submit photos (large high quality scans, PhotoCD or JPEG) along with the reviews. Otherwise it will be hard for a reader to judge the authority of the reviewer.

    Carl: we won't be replacing old reviews with new, putatively better ones. That's what the comment server is for! Besides, there are plenty of new cameras out. We prefer to focus our limited energy on becoming more comprehensive rather than the ultimate review of the Nikon F2 or whatever.

    Patrick: yes, it is true that photo.net is popular enough that manufacturers are willing to loan us equipment (most of our new reviews are done with manufacturer loaners). But first we need to ask! Instead of having all of the cameras come through Cambridge we want to delegate to brand editors and finally writers the authority to request a camera and have it shipped directly from, say, Nikon USA to their house and then back.
     
  8. In an admittedly quick read, I didn't see anything about lenses. Will they be reviewed, or only bodies ;-)
     
  9. Duh, that should read "daily un-archived"
     
  10. Good good. I think the older reviews are still valid and useful, and above all, neutral. I was just curious how this will be implemented, whether it would be in addition or aside from the already existing reviews.
     
  11. Mr. Greenspun: I like to keep two steps behind... it gives my ideas a nice 'retro' feel to those who are two steps ahead :)

    The lenses comment is a good point. There are a few Nikon lens reviews, but not many (I crossed over to the dark side and saw that there were a good number of Canon lens reviews). That might be another category that could use some description. As for actual reviewing: I'm ALWAYS looking for something to come between me and my thesis (or, for that matter, the freshman claculus classes I always end up teaching). I don't have a huge background to draw on, but I'd be more than happy to try to give something back to the website which has given me so me free education. The new 80-400 VR from Nikon is missing on the Nikon lens list.....
     
  12. "... so MUCH free eduction."
     
  13. I like the standards as drafted i.e. as broad standards.

    1. I might have been a bit more explicit of the requirements for reporting handling ("what are the ergonomics of setting exposure? Of focusing? Of getting the shot?") and convenience of carrying around these cameras ("Is it possible to carry this camera all day comfortably in your hand? Around your shoulder? In a bag?") .

    But these are perhaps too explicit, revealing my bias toward 35mm street, people and travel photography.

    2. Without necessarily regarding cameras as gadgets, it might be useful to require reviewers to illustrate any ergonomic points with little jpegs. E.g. an adverse comment on the placement of an exposure compensation dial should have a little jpeg showing where it is on the camera...
     
  14. Sounds good to me. If you're looking for submissions, I'd be happy to write reviews for three of the cameras that I've been using lately...Nikon Fm3a, Voigtlander Bessa-L, and the Hasselblad XPAN.
    <p>
    Not exactly the mainstream bodies of the new millenium, I know, but very satisfying to use.
    <p>
    I assume this is how you want people to volunteer to write these things. You said to respond in this thread.
     
  15. Ooh-ooh... how is the XPAN? I like the idea... but aren't the lenses all a bit slow?
     
  16. I would add one point. It is far more useful to read a review that has been written by someone who has actually used the camera fairly extensively, rather than just "taking it for a spin" for a few hours. I have noticed that a few of the reviews that have already been posted here have been based on only a very brief period of use of the camera. These reviews are not particularly helpful. I can read about the features of any camera on the manufacturer's website. What I look for on photo.net is someone's opinion about what it is like to actually use the equipment on a daily basis. Which features seem great at the beginning but turn out to be worthless or annoying in the long run? Which features are winners and might be a reason to buy this particular camera over another? I think that Kirk Tuck's review of the Leica M6 is a good example of the kind of review that I find useful.
     
  17. I can see where Steve is coming from, but I'm not sure how realistic that is for new equipment. And I think it's possible to, in a trial run, figure out most of the bugs and issues of a given piece of equipment. Don't forget, PG pointed out in the editorial plan somewhere that long-time users can (, will, and do) comment on the reviews over time with these sorts of observations.
     
  18. I agree with Steve, but I do see a problem. I bought an Olympus E10 and before doing so I looked at all the reviews. The one by PG was least informative and least accurate, but I then looked at the others and bought the camera, which is excellent for my own purposes. Here's the problem - I have daily experience and detailed knowledge of 3 cameras (the E10, Mamiya RZ & Arca Monorail) but I don't have enough knowledge of the features of competetive cameras to make subjective comments, for example I know that the rotating back on the RZ is generally excellent (although it can't be used on my tripod unless it has a motordrive fitted) but I can't compare that to the Bronica which has no rotating back but which has an optional gizmo that rotates the camera. Also, my opinions are biased towards the type of work I do, which in my case is commercial/fashion - I said earlier that the E10 is excellent, and so it is as far as my own needs are concerned, but it's useless for any kind of action shots.

    I think that most people would have the same limitations as me.
     
  19. I would prefer that the reviews focus on the relative merits of different cameras. You sort of mention some issues in your bullets but i'm worried that people will take your list as explicit and not suggestive. Aka, while it might be harder to use the dof on slr a slr a might retain accurate focus ability with 1/4 the light of camera b at 1/2 the cost even though camera b has a easy to use dof button. Alas, low light focus ability isn't one of your bullet.

    Hum - I guess what I am also trying to say (in a convoluted way) is that there are clear classes of cameras based on feature/price point (aka: olympus 3030, nikon 995, canon G?, ...) - since prices and resolution are similar - it would be esp useful to comment on specific trade off between cameras (nikon has better macro, olympus has faster lens with less chromatic aberration)...

    Hum -- I guess I am mixing two issues and being a poor and unorganized writer they are hard to sep.

    a) Reviewers might take your bullets too literal (even if you meant them to be literal) and omit other important details.

    b) It is often important to directly compare items in the same class to give a better indication of the trade off (harder since the reviewer needs 'expert' knowledge of multiple items).
     
  20. > think that Kirk Tuck's review of the Leica M6 is a good example of the kind of review that I find useful.

    exactly. Kirk's review sets the standard in my view. one requirement is ample objectivity to overcome the natural reluctance to fault equipment they own and have paid good money for. most reviews are clamourings of justification for their purchase. I can read the Calumet catalogue and find the fluff. I want the dirty details.
     
  21. hub

    hub

    Philip: Something that have not been mentionned (or I missed it): what about review for old or discontinued cameras ? Given how large is the used camera market and how useful it is for people that don't really have a lot of money but want to do serious photography, I think it would be a nice idea to have that sort of review. Do you think it is worth ?
     
  22. Perhaps it reflects my shooting preferences, but I would like to see examples of low-light performance in reviews of digital bodies. By low-light I mean exposures ranging from few seconds to several minutes. That's the area they are presumably weak in (noise, hot pixels), but I saw very little discussion of this.
    <p>
    Remote release? Bulb mode? Maximum exposure time? Star trails with a digital camera? Moonlit landscape? Night cityscape (how long an exposure, exactly)? Dim indoors with large depth-of-field? Multiple exposure on bulb? Correction for fluorescent lights (filter/built-in)? Am I better off with film, finally?
     
  23. I think it would be very helpful if there was also a standard template that could be filled in. This way the formatting would be consistent (H3..? level 3 heading??), and section headings/field titles would prompt people to supply appropriate comments.
     
  24. Philip, in addition to the guidelines you've laid down, I think it would be valuable, in the digital camera section, for users to give their opinion of the colour accuracy of the camera, and also to indicate the length of time that they've been using it. It's very easy to give a rave review in the first flush of enthusiasm for a product, but after a few weeks its drawbacks begin to be more obvious.<br>For example: Some digital cameras appear to be geared toward giving saturated colour above everything else. Their CCD filters are so narrow cut that it makes it almost impossible for them to give accurate colour (are your ears burning yet, Fuji?).<br>The colour might look impressive initially, but when you've been unable to get the camera to distinguish between shades of red, or even between red and magenta, then your enthusiam might fade, even if the colour rendering doesn't.<br>Just a suggestion.
     
  25. fpa

    fpa

    I would add a cautious second to the idea of some reviews of 'classic' cameras, but would suggest restricting it to those you might reasonably find and get repaired in the outside world today.
    For instance, a group roundup of the Nikon FM/FE/FG/FM2, and compatibilty with modern lenses, or someone who owns an F3 comparing it against the F2a that Phil reviewed, and versus the newer F4/F5 would be useful and interesting, since a used manual or Semimanual Nikon from the 80's may be a better choice for a student or amateur on a budget than a more automated and plasticized current consumer model. (sorry for the Fx references; I'm not a Canonista, but a follower of the other camp). Case in point, the Mamiya C330 review, while not a current production model but easy to find, afford, and repair, was helpful when I went MF shopping two years ago. Partly based on that review, which included what it was like to handle in the field, I settled on a C220. Monaghan's page at SMU provided a complementary resource of the competing older models (Bronica S2, Kowa 66), but also made clear the difficulty of getting them repaired, and hence potentially lower interest to this community. Someone willing to face down a few of the low-priced MF options (Used Fuji Rangefinder vs. Seagull vs. used YashicaMat, for instance), may appeal to the budget-conscious MF users reading this forum.

    Personally, I'd admit to being curious about field reports of people who use the Mamiya RB/RZ system, or the Bronica GS-1, since I met a hiker in Arches last year (a middle-aged man about 2/3 my size, and I'm not large) with an RB plus a couple lenses, and wondered about what that weighed and how it operated outdoors versus my field-camera + lenses. That's been the joy of photo.net reviews; opinionated, and most of the opinions based on trying to actually get a picture with the camera, not the glossy brochures the manufacturer packed along.
     
  26. I agree that the review should follow a format. However, it shouldn't just recite the camera specs. The review should state the specs, possibly even in tabular form. I also think it makes sense to have commentary restricted to that paticular family of camera - as a Maxxum user, I really don't care about what things a F5 or a 1V do better or not as well as a 9. I have far too much money into lenses to care whether I should have held off and saved and bought Leica. When it coes to handling, we do need a prose report of how the camera functions at its user interface - the downloaded photos will show the the camera's ability to produce images (and possibly flaws or shortcomings too) but the reviewer's report has to tell us whether it's easy to swich to spot and lock AE in a backlight or other uneven lighting situation, how easy it is to call up the onboard flash to get fill at -1.5 FC. Also, and especially because the reviews will be here for decades, we need to know something about how the camera fits into the brand's evolution, which would (along with all the other info & specs) help people decide on used bodies. There has to be flexibility in the reporting format; how else is a reviewer going to give a worthwhile assessment of the panoramic option in the XPan, the film door screen in the Maxxum 7, or the AF bracketing of the N1? But I welcome this development, and although the comments from the community at large are great they should supplement, not supplant, a thoroughgoung review done to a rubric.
     
  27. It's a pretty lame request, I know, but I'd like to see a picture of the camera along with the review. Perhaps it is not important for modern 35mm bodies, which all look fairly similar, but if the camera is something much different from that (such as some spiffy new Happy Magic Family C5 medium format camera which no one has ever heard of), then it's nice to see how the controls are laid out on the body.
     
  28. Philip, I think this is great. As someone who has only recently gotten back into photography (and therefore had to buy in), I've read a lot of reviews in the last year or two, and I agree that standards would be immensely helpful. Your proposed standards reflect the general approach of photo.net well.

    I would suggest that this camera-review-standards document could be rewritten for a different target audience -- the reader of the reviews -- and a link to it provided at the top of every review page. That way people won't wonder why this article isn't comparing a Nikon body to a Canon body.

    Keep up the good work!
     
  29. Sidney has a good point.<p>
    Pictures of the camera in the review. With 35mm, not so important but with MF & LF it
    would help the reader in understanding the review better. Also most of us live in an area
    that limits us to seeing these items in person. Where I live a LF camera is a rare beast, so
    it's Ebay or nothing.
     
  30. The tool focus means that we're interested in how a camera performs when asked to do a task.
    I suppose only a technical writer like myself would get excited when someone writes a good set of writer's guidelines, but I say Bravo Phil! Your directions for writers will make for good and useful reviews -- the sort I'd expect to find on photo.net.
    I'm looking forward to strong opinions backed up by intelligent observations. That's what I find when I go searching through the photo.net forums, but it will be fun to see them focused upon a given piece of equipment. (I mean tool.)
    Thanks, Phil. And thanks in advance to the writers who will take the time to do the work!
     
  31. Just one point to add: I second the suggestion of submitting a pic of the camera being reviewed but I would go further and ask that the photo be taken with the camera in someone hands. This gives an idea of size, which is difficult to otherwise determine. Depending on where some readers live they may not have easy access to a store with that model in stock for real world comparison.

    Or, If possible a photo of the camera alongside another camera that's better known or more commonplace.

    Such a photo comparison would easily answer, "Is the Oly E-10 the size of an F5 or an MX?"
     
  32. Sidney, Daniel and Robert have an important suggestion -- including pictures of the item under review, and shown with a familiar object for scale. Indeed, many of us live in the hinterlands and don't have access to stores with a wide variety of bodies, lenses, etc. (Thank goodness we have access to the Internet!)

    I remember the first time I saw a display of digital cameras. Are you kidding? I thought. In magazine articles, their photos seem to show them as about the size of a standard SLR, but that's deceiving. In "real life," they seemed ridiculously small.
     
  33. camera-review-standards looks good. How about some guidelines for lenses and other accessories? Do we also need some guidelines for 'Member opinions' in ezShop? I wrote a couple comments there, and I would be happy to write some more. I'll say right away there are certainly many people much more qualified than I to review some particular 'tool', just perhaps I had more time at the moment to write something.

    Maybe I missed it, but is there a planned submission or posting process?
     
  34. One more thought: the anti-spec people have me on their side, but it might be nice to also compile some spec charts comparing the features of all of the major slrs out there today. I know that when I went shopping for my first slr a while ago (needing something to replace the older slr borrowed from my mother), I spent a lot of time flipping between spec sheets to see which bodies had which features. Not everyone finds this important, but it might be useful to have a basic "features at a glace" chart.
     
  35. I find it easy to locate information about the features and performance of cameras. I would like to see reviews that concentrate on the feel and usability. Feel: is it satisfying/solid/cheesy/finicky/whatever. Usability: is it logical/confusing at first, and once you get used to it; are things that should be easy actually easy; what kinds of photographic tasks are natural or awkward?
     
  36. Please request the "warts" feature you mention, and the reverse, of every review. Learning what someone likes best and least is often very revealing, both about the product and the reviewer's viewpoint. It also helps the reviewer to get off the fence and take a position. Also, please ban use of the word "cool."
     
  37. I think I rating system will help as well. The best ones I have found are the consumer reports ones - eventhought I disagree with their SLR camera reviews (last one was in 1993). Please let me explain by what I meant about the ratings. <p> We could use ratings on: 1) handling - how easy is it to hold the camera steady? How does a big lens affect the usage? How does it feel light despite having a fairly heavy weight? 2) easy of use - how are the controls laid out? 3) Features 4) exposure consistency - how accurate the metering system is? 5) Weight. Note that I have omitted a final score. I think this is very personal, so one has to judge by oneself. <p> The downside of a rating system is that it implies consistancy in judgment. Someone needs to have a number of cameras and use them enough.
     
  38. What a great idea! Reviews from real users. Most of the time you can find reviews of guys which don´t own the camera but own just the leaflet. Endless lists of features and technical data together with comments like well built, very good image quality, ...
    Only if you really know something you are able to write a useful review. I plan something like that on my own website ( www.joachimgerstl.com ) but think I will concentrate on things like tripods, ballheads and other useful accessories.
    Reviews like this and www.photo.net together with www.dpreview.com ( for all technical data and pictures of the product ) will be the perfect sources to preselect equipment.
     
  39. Personally I wouldn't trust a review from anyone who hadn't lived with a piece of equipment (and used it!) for at least a month, preferably 3 months. Initial delight with a new camera/lens can turn into dislike/frustration when you really get to know its bad points as well as its good points.

    Of course this means you don't get the first reviews out there, but what you do get are worth a lot more in the long run.

    "Instant reviews" such as those you find in Petersen's Photographic magazine are a prime example of what not to do. They reprint the spec sheet, show a few images and rave over how great it is. I'm sure manufacturers love them and I'll bet they NEVER have problems getting equipment on loan. The trouble is the reviews are useless other than as marketing tools. They could probably be written (uncluding doing all the "testing") within an hour of opening the box, in fact I'm not sure some of them aren't.
     
  40. Mr. Atkins, do you think it would be possible to borrow something for that long from Nikon, Canon, or any of the big producers (I'm actually asking, not rhetorizing). The problem is that, if it isn't, you're only going to get reviews by owners of the equipment. And my psyc. major girlfriend tells me that it is well established that people will give favourable feedback on products that they have spent a lot of money on (to avoid feeling like chumps who threw money away). Just look at the Nikon/Canon debate here on pdn. Is one really so much better than the other in ANY area that someone would get so worked up about it? I think most of the flame wars stem from the fact that people have invested a great deal in these systems.
     
  41. Well, it's possible, though perhaps not as easy as a 1 week loaner. Or photo.net could do what the better consumer magazines do and purchase the equipment, test it and then sell it (by auction). That way you also get around problems of manufacturers sending out "hand picked" samples for review (not that they'd even think of doing that of course). It costs more of course, but if photo.net is running as a business there will be operating costs. Then again, if you buy at a really good discount price and sell on Ebay you might even make a profit. People do!

    As for psychological factors, they do exist of course. Since Canon users should test Canon equipment, Nikon user test Nikon etc. there's already some built in bias. Personally, if I have a piece of equipment for over three months, I like it. Otherwise I would have sold it. Either way I can give an unbiased opinion.
     
  42. If someone could write a page even based on a three month+ use of a camera/equipment and that person is a Community member of some standing I would likely buy on that basis. I did with the Hexar two years ago. (Your writing guidelines make good sense and as guidelines I see no holes)I think the point maybe is-'Go and Start Writing, Others will Help to Edit as Needed.' Or that could be the message.There are volunteers who can do that sort of thing,I trust, even I could collaborate on broad editorial review,looking for things that need fleshing out and getting ready for publication part, if that would help,even at this distance. Note: I got an emailed comment I solicited from a person I queried about the Minolta Scan Elite II, a hot new item in digital land,for film scanning(that is my next buy). This sort of networking goes on and is a real asset to the community.I think the identification of interesting items for review and soliciting of writeups is a worthy step. Good project. I hope to be able to participate. I use old stuff and my new stuff like the little Leica Mini is out of production. Need a writeup on the Stereo Realist,no sweat.
     
  43. Phil,

    Thanks, it looks good so far to me.

    photo.net attracts the attention of so many photographers and wannabes that it will be impossible to serve them all equally well.

    It will be a sufficient service if photo.net injects a degree of rationality into the camera market. The general corporate trend to keep consumers on a price escallator cannot be totally overcome by the easy availability of rational information, but it will help. Most manufacturers change models and features continually....so it becomes difficult to evalute the ratio of "manure to mollasses" on any given model. And I don't want to see photo.net reviews simply becoming recounts of slack-jawed amazement over every new transistor or photocell added to a camera, and enforcing that seems to be the trickiest part. Maintain the focus on "usable" and "practical", of course. And do it without getting way too serious or formal. That's the ticket.

    Cheers,

    Charlie
     
  44. In my opinion, reviews need to be written by people who are not in love with cameras. Basically photographers, not collectors or equipment junkies. The trouble is that it's the equipment junkies who want to get their hands on the latest stuff and write reviews. Photographers are probably too busy with photography to bother about such things!
    <p>
    Photographers go through 3 phases:
    <ul>
    <li> Phase 1 - Equipment is everything!
    <li> Phase 2 - Equipment is nothing!
    <li> Phase 3 - Equipment is equipment
    <p>
    Some photographers get stuck in Phase 1 or Phase 2. Reviewers should be in phase 3.
     
  45. I agree with Bob A. I have lusted, purchased, used, and then discarded several systems and formats over the decades. I rarely get excited about any gear unless I've laid hands on it and put it through my paces. I would trust some reviews from some writers, especially if I got to know them to a degree and learned how they arrived at their conclusions. I still get excited when I buy a new toy, but usually I've already shopped a solution prior to finding the good reviews. How many of you have used reviews to justify your purchases or choices after the fact?
     
  46. I'm surprised by how many people in this thread accept the idea that someone who owns something will be biased in favor of it. Maybe that is true of a 13-year-old kid who is proud of his new toy. But how many adults do you know who say "My house is the greatest house in the world" versus who say "My house is falling down and the electrician won't return my calls and I wish the kitchen and dining room weren't separated by such a long hallway..." And a house is a lot more expensive than a camera. In the world of aviation, which is renowned for attracting opinionated blowhards, you don't find Cessna 172 owners saying "My plane is so much better than that overrated Beech Bonanza"; they all say "Gee, if I had $600,000 to spend, I'd be in that Bonanza right now."

    And really why would any bias matter? The reviews are supposed to be authored by someone carrying out a project with a camera or lens. If the reviewer is diligent about noting what went well and not so well in the project and the reader is critical in looking at the same images, people get the information that they need. If the author says the Holga is the world's best camera and her pictures are 100X better than mine, I really can't argue with her results, can I? If her pictures suck, why would I want her unbiased opinions since she obviously doesn't have any practical photographic ability.
     
  47. Philip - We all know that people cannot be trusted to be unbiased, no matter how scientific the test and how much they try not to be.
    For example, look at medical research. The only true tests are double blind where neither the doctor nor the patient knows if a particular drug was taken. In science in general, if a result is expected, it is often obtained. If it isn't, the experiment is repeated until it is!
    A photo.net equivalent might be for one person to do the testing (take the pictures) while a second person - who does not know anything about the equipment involved - evaluates the results.
    About bias you say " Maybe that is true of a 13-year-old kid who is proud of his new toy". Well, have you read through some of the postings lately? It's amazing how much passion people feel about their equipment. It almost amounts to a religion with some people. They will defend it to the death. Photographers (of almost any age) are generally pround of their new toys. Bias is alive and well and living right here. I wouldn't expect anything else.
     
  48. I don't think bias should be an issue in camera reviews.

    At some level, camera reviews are sort of meaningless and
    boring anyway. No one can really tell you if you will like a camera.

    All they can do is tell you what a camera is like to use and then
    tell you if they like the camera. The first part of this can be
    relatively objective (i.e. the camera has no auto modes and a
    100% viewfinder vs. the camera has only an auto mode and an
    optical viewfinder with the rangefinder in it).

    The second part is bound to be somewhat subjective (the
    camera is too heavy, the camera is too small, etc).

    I think most people reading reviews want information of the first
    kind. I think when they get information of the second kind the
    flame wars start.
     
  49. Now, I don't want to prattle on about bias, because I don't think it's a MAJOR problem in reviewing (though something to think about, always), but I must respectfully disagree with Philip. Yes, people complain about their houses and cars... and I am the first to admit that, if I had the money, I would likely buy a Leica system and leave my Nikon stuff at home (yes, equipment is equipment... but the simplicity and quality are so appealing). That said, I still think you'd get a more objective review out of me by dropping some kit in my lap and asking me to wander around shooting with it for a couple of weeks than you would by asking me to review my own stuff. For starters... I picked my stuff over the other stuff out there (of course, this could be a useful bias in a review, since readers might be able to determine if they have the same shooting interests as you). And I stick to my story that many people are hesitant to admit to themselves that they may have wasted money... especially on stuff purchased recently (it's easier to say "I was so dumb" than "I am so dumb"). And Bob A. has a point: some people on pdn are, with respect to equipment, exactly like the 13 year old boy (and some are stuck in phase 2).

    Is bias a critical factor in a review? I guess if people write reviews well it isn't. The guidelines posted are certainly a leap in the right direction. So, Mr. Greenspun... how might I get involved?
     
  50. Unless you chose reviewers carefully, all you'll end up with is a clone of "epinions.com". Random reviews from random users. If you want to know where all the controls are, what the specs are etc. it's usually all on the manufacturer's web site. Might as well just copy that. No reviewer is going to weigh the camera, calculate viewfinder coverage, verify exposure range etc. And who is going to risk testing weather sealing on a camera they don't (or do!) own?
    <p>
    There's not a chance that any photo.net reviewer without access to an electronics and optics lab is going to come anywhere near a Popular Photography SLR or lens review for example. You might not BELIEVE the Pop Photog tests, but they have the equipment and the time to do it right (at least in principle).
    <p>
    The question then is what can photo.net do better than the rest? I'd say carefully considered long term reviews of cameras. Perhaps something like the long term tests that automobile magazines do. I know of no photography magazines that do anything similar, nor web sites for that matter.
    <p>
    The only problem is that equipment may become obsolete (in the sense that it's replaced by a new version) before the long term test is up unless you start the test right after an item is released!
    <p>
    There's also certainly a place for "epinions" too - and photo.net really already has that.
     
  51. Let me give specific instances of typical camera body limitations which are seldom, if ever, mentioned in reviews:

    1) The built in flash only works with (physically) short lenses because longer ones block part of the flash output.

    2) The built in flash will cause red eye, only a little less severe than a point and shoot camera.

    3) The built in flash requires several seconds to recover, but if you shoot in continuous mode, the camera doesn't wait for the flash to recover, thereby wasting film.

    4) When autofocus assist illumination comes from an external flash, taking the flash off camera can direct the pattern away from where the autofocus sensor is pointed, i.e. no autofocus.

    5) If the SLR sync speed is 1/125, almost any point and shoot camera can do flash sync faster than it.

    6) SLR lenses have to let in enough light for human vision, and must be designed to clear the mirror. Accordingly, inexpensive SLR zoom lenses are not optically much better than point and shoot lenses.

    7) The focusing screen in some SLR models is a poorly designed for fine manual focusing.

    8) The goals of a 100 percent view, a short shutter release lag time, and low mirror vibration cannot simultaneously be provided in a lightweight inexpensive SLR body.

    9) One of the stronget advantages of an SLR over a point and shoot camera is depth of field preview. However, several SLR model don't provide a DOF function.

    10) The number of people who think that a microchip in a camera body is capable of reading their mind is amazing. That function isn't available yet.

    11) LCD panels that are not backlighted are useless in the dark.

    12) The mere listing of a feature in a camera brochure in no way gaurantees that feature has been implemented in a practical way. As Phil points out several places on photo.net, having to remember and perform an obtuse programming sequence using "leetle buttons" is a bummer.

    13) Many SLR cameras provide an exposure lock function which uses an older analog electronic circuit known as "track and hold". This is accurate for only a few seconds before the charge on a capacitor bleeds down. These are the SLR models that have a tiny, difficult to push, impossible to hold, spring loaded buttons for exposure lock.

    14) If you like taking pictures of people, a quiet camera is a big advantage. Children in particular are distracted by any noises made while zooming or focusing just before the shot. This rules out several SLR bodies as well as most point and shoot comeras.

    15) With any camera that uses removable lenses, there are many extra opportunities for dust to enter and accumulate inside the lens and inside the camera.

    Well, I've got other thing on my plate now, but I'm certain the above is only a short list of the deficiencies in typical camera reviews. Anyone else care to add?
     
  52. Great discussion, but instead of arguing, I'd just like to say that photo.net's Minolta section is badly out of date.
     
  53. I have a few moments now, so I will continue my ode to under-reported camera inadequacies....

    16) In almost all SLR bodies, metering doesn't function while the mirror is locked up, so you have to use manual settings and hope the light doesn't change. And in almost all SLR bodies, the camera will not automatically bracket exposures in the manual mode. So the photographer has to change the exposure settings manually, and with most cameras this risks bumping the camera out of position during macro shots.

    17) If you use an eyepiece shade on an SLR, you have to take it off to get the film back open.

    18) If the camera only has a central spot meter, and the camera is on a tripod, the meter will probably be reading off something in the scene which gives unreliable results.

    19) In most SLR models, results from the evaluative metering mode will sometimes defy any analysis.


    Come on guys. Help me with this.
     
  54. Wow Charles! None of my cameras have even half the features you mention. Makes me wonder how I manage to take any pictures at all...
     
  55. Andrew, I use an old and simple Nikon N2000 as well as other cameras. I guess one point I was trying to make about SLRs is that in the rush to incorporate gads of "features", some models have been rendered less than desirable. I could name a couple of such models, as could most folks at photo.net. I would simply be more impressed with camera reviews if some of these short comings were revealed more thoroughly and clearly.
     
  56. Charles - most of your points apply to every SLR. I suppose you could add:
    <p>
    (20) If you drop it it may break<br>
    (21) If the batteries go dead it won't work<br>
    (22) You can't get sharp images with a hand-held 300mm lens at 1/15s<br>
    (23) It won't take a 3/8-16 tripod threaded screw<br>
    (24) It won't do TTL metering with a $10 Vivitar flash<br>
    <p>
    And so on. My guess is that the list is almost endless.
     
  57. One minor correction to Charles' list:

    13) The circuit is called "sample and hold", not "track and hold". If it "tracked", it wouldn't be much of a exposure lock, would it?
     
  58. "My guess is that the list is almost endless"

    That sounds like it may become the basis of an excuse not to consider the short-comings problem at all. Boo, hiss! Why not just put the top thirty possible short comings in a standard check list to be considered by the reviewer?
     
  59. Phil is looking for input, right? I think the short one is best one, so here is mine;
    <p>
    1. Review must be written by a person who used reviewed gear extensively.<br>
    2. Review mustn't be a mere reproduction of manufacturer's specs<br>
    3. Review must be accompanied by images taken with it. Ultimately, every major statement should have a "Figure" or "Figures". Sorry, that's where my scientific background kicks in.<br>
    4. Review must be thorough, cover all issues related to actual picture taking (AF/AE accuracy), ergonomics and built quality.<br>
    5. Review mustn't focus on direct comparison with competing product.<br>
    6. Review must be well organized, divided in sections and have "Conclusions" part.

    <p>Other than that, suggested guidelines cover all essentials.
     
  60. I guess there is one more thing;
    <p>
    Submitted draft must be reviewed by a Photo.net editor prior to posting.
     
  61. Mr. Miller... I think Mr. Atkins has rather a different point. Some of these shortcomings seem not to be problems with specific camera models, but rather problems with the universe as we know it (i.e., 6, 8, 15, 18 to name the most obvious). In place of 18 one might as well simply say that if the camera is pointed randomly, it's probably not looking at something you want to take a picture of. Some of the other comments are more about cameras, but I still wouldn't want every review to mention that the camera in question cannot meter with the mirror locked up, just as I don't want to be reading that a certain SLR body won't iron my shirts for me. It would be a nice feature, but so far no bodies have it so it's not much of a review issue (more of a "send a letter to Nikon/Canon R&D" issue).

    I think it's great if reviews mention a few obvious oversights on the part of the designers, but those should probably be restricted to features often available on other units (i.e., mirror lockup).
     
  62. I'll comment on Large Format only:

    What's important to me is:

    Type of camera (bench, field?)
    Format (4x5, 5x7, 8x10 ?)
    Range of swing/tilt
    Lens board type ('standard'? easy to find?)
    Ease of use
    Construction


    What I mean is - why do some people use Gandolfi, if not for the beauty of the construction :)????
     
  63. The issue with displaying images taken with the camera is that they are essentially useless. They make nice decorations but there's no way you can get any sense of image qulity from a small web image. If you post a 20MB tif file you might get some idea, but even then the quality probably depends more on the scanner than the lens. You could post photomigrographs of resolution test chart images and they would, in fact, be useful, but you'd need to specify a standard film, standard processing and a standard chart - and you'd need the ability to take photographs through a high quality microscope. The number of reviewers able to do this properly would be very small.
    0021z0-7446684.jpg
     
  64. Bob, if you are talking about sharpness/microcontrast, then yes, online scan tells little about this. However for things like vignetting, flare/ghosting and distortions web image is a valid way of demonstration.
     
  65. True to some extent, but things like flare and ghosting are very difficult to quantify. I can take pictures with a Canon "L" lens which show flare and ghosting and I can take pictures with Samyang 28-300 zoom that don't. Distortion is easier to show. Vignetting is tricky since it depends on what contrast (gamma) level you use for the scanned image. You can reduce or increase the visibility of vignetting quite easily depending on your exposure/scanning parameters - and the scene you shoot.

    Attempting to do "scientific" testing without a standard is a route to confusion.

    I'm not saying images hurt, just that their actual (as opposed to perceived) utility is small.
     
  66. Well, you brought that up, Bob. Next step would be a development of lens testing guidelines. Assuming that only a few of us have opportunity to use optical bench, the set methods should be easy to perform and available to anyone (camera and lens is still required).
    <p>
    1. Test in contra-light conditions to test flare/ghosting (sun in a viewfinder)<br>
    2. Distortion test (brick wall, architecture shot, anything with straight lines within peripheral areas of image where distortions are most prominent).<br>
    3. Vignetting test (open sky shot, wide open and stopped down)
    4. Rendering of out-of-focus areas at maximum apertures (background blur, bokeh, whichever term is preferred)

    <p>These are criteria and techniques that I've used to test my lenses for a few online reviews.

    Yuriy.
     
  67. Patrick said, "

    I think it's great if reviews mention a few obvious oversights on the part of the designers, but those should probably be restricted to features often available on other units (i.e., mirror lockup). "

    Yes, thanks for your comment. Any idea must used with discretion, perhaps more so than I indicated with my lists. But the advertising and promotional campaigns for many cameras are deceptive, and I simply would like photo.net reviews to provide a tactful, yet EFFECTIVE, counterpoint whenever it is deserved.
     
  68. Can you cite a specific example of a deceptive ad, i.e. one that makes you think a camera has a feature it doesn't really have or can do something it can't do?
     
  69. OK Bob, here goes,

    Nearly all camera brochures for entry-level cameras show pictures taken of a professional model at an exotic location, not mentioning the use of thousands of dollars of lighting or light modifiers, the use of several professional quality lenses, or that the whole shoot was aided by an assistant or two who used hand-held meters. No starter SLR with a cheap zoom lens will do that. No big SLR with the best lens and an attached flash will do that either.

    Quotes like "We take the best pictures". This is in the same class of silliness as a feature once listed on the box of a cheap vacuum sweeper, i.e. "Dirt Seeking Headlamp".

    Specifications like "multiple exposures are possible". "Possible" if you know in advance how many exposures you want. "Possible" if you can remember how to program the camera. So every time you see the word "possible" in a camera specification, subtitute the phrase, "possible but difficult".

    Pictures on a Contax TVS carton I saw were taken using MF, but the brochure inside the box showed a smaller image taken by a TVS, implying the whole image was taken by a TVS. Tsk, tsk.

    The N6006 I had only took sharp pictures when it was bolted down to a large tripod because the mirror vibration was ferocious. It was said to be popular with photojournalists because of its relatively low shutter lag. Was I supposed to guess from that "information" that the N6006 mirror-up goes KA-WHOP?

    The N80 has all sorts of complicated autofoucusing features to track moving subjects, but only a 1/125 flash sync, which is an obvious mismatch of capabilities. How long until the average N80 buyer will realize he really needs something more for action photography?

    I never saw even the slightest notice from Canon that a camera with a pelicle mirror has special problems. Any light entering through the eyepiece not only throws the exposure off, but it reflects onto the film off the pelicle and affects the final image too. And any dirt on a pelicle mirror degrades the final image, which is a problem not experienced with a regular return mirror. Come on Canon, puleeeze give us some sort of little hint about things like that.

    Every manufacturer produces a super sharp and inexpensive 50mm lens, and the photo rags really reach their stride saying a $100 lens is as good as a $500 lens. And of course, bokeh rhimes with hokey-pokey, right?

    Beginners are encouraged to get an autofocus prime lens or two right away, but many autofocus prime lenses are formulated for extra sharpness in the center at the expense of edge sharpness. The overall quality of autofocus lenses may be about the same, or even a little better, than manual focus lenses. But if someone is going to do landscapes, high uniform sharpnes is more important than fantastic center sharpness. It is amazing, but some very recent cameras don't even work with manual focus lenses. (expletive deleted).

    A friend was complaining about inconsistent focus and exposure on her Rebel 2000, and when I tried out a Rebel 2000, I found that it often would not refocus when pointed at something closer. But the Rebel G I tried worked flawlessly. Many of the same features of the Rebel 2000 are in the EOS-7, where they are said to work well. So think of the Rebel 2000 as more of a "training bra" than a real camera, for instance.

    Bob, we are in the Fraud Neighborhood. Can YOU say "APS"? Try it now. And how many of those amazingly superior APS SLR outfits have you bought so far? Exactly how many...hold up your fingers.

    I still hope that photo.net will bring useful objectivitiy to camera evalutions, but if photo.net is going to have a Unique Selling Position (USP), it can't play dumb about camera shortcomings forever. Over a dozen commercial rags already do that very well. On the other hand, nothing succeeds like success, so maybe photo.net should just emulate them. It seems that you are debating it, and I don't want photo.net to fail from a debilitating case of honesty either. Good luck following the "Luminous Path".
     
  70. I'm not sure that I agree with everything Mr. Miller says, but there is a very good point about camera ads, and in particular camera ads that we don't know are camera ads. No-one expects camera ads to point out obvious flaws. It would be nice if they did, just like it would be nice if instead of "We love to see you smile" McDonald's ads said "We spit in your burgers but don't worry, they're made out of rat ass anyways". But that's not gonna happen since the small increase in sales from people thinking "Honesty? Wow!" would be overwhelmed by the large decline in sales from people thinking "They can't even make their product SEEM good." My question is: Are reviews in magazines really more objective than ads when the magazines are largely funded by advertising dollars from the companies that make the products that the magazine reviews? I'm not saying there's some grand conspiracy, but Pop Photog writing a negative review of the F80 seems a lot like biting the hand that feeds it. That is the advantage that PDN has, and that is why the reviews here are not redundant. Of course, if we're asking for loaners we might be in a similar situation, but there are ways around that.
     
  71. Charles

    All that can be summed up by "Advertsing exagerates", pretty much a self evident observation that everyone is aware of.

    Most of your points equate to "if you stick your fingers through the shutter it won't work", or "A $250 body doesn't focus or expose as well as a $1200 body". Yes, correct, what else do you expect. If reviews point out the obvious they're going to be very dull and boring.

    I suppose if you're writing reviews for the terminally clueless, you'd have to include all that stuff.

    Obviously we disagree, so if you'd like to take a camera body and review it, submit the review to photo.net and see what sort of feedback you get.
     
  72. Bob,

    Well, I'm better at hyperbole than I am at eloquence, so please accept my sincere apology for any unwarranted directness. I simply wanted to express my view very clearly, but of course it is indeed only one view.

    Best regards,

    Charlie
     
  73. Directness is fine. Different users want different things from reviews. Personally I don't find camera body reviews very useful. I can read the manufacturer's spec sheet and brochure and learn pretty much everything I want to know. Lens reviews are more useful since the spec sheets don't tell you about performance, but the problem there is objectivity. What one users may regard as good performance, a more critical user might not. Getting objective data is tricky and comparing "objective" data from a random set of reviewers is even trickier.

    Some users may want to know what I think is obvious. Clearly you can't meter when the mirror is locked up because the metering sensors are in the viewfinder (actually on the prism) and that light path is blocked by a locked up mirror (and the sub mirror used for spot metering in some cameras is retracted). It's not a design defect and all cameras suffer from the same "problem". You can't autofocus with the mirror locked up either, but again that's obvious, all cameras suffer from the "defect" and I really don't need to be told that in every review.

    If I had no idea of how a camera worked, I suppose I might need to be told that, but by the same token I'd need to be told just about everything. A reviewer has to assume some level of basic knowledge on the part of the reader or he/she would have to explain everything from the meaning of "f-stop" to the ISO rating of film.
     
  74. C'mon, Bob... camera manufacturers could just put a second metering system on a small retractable arm behind the mirror... then when the mirror locks up you can still meter, and when you want to shoot the second metering system would swing out of the way and... oh. :)
     
  75. Thanks Patrick and Bob.

    I think part of the problem I have with most consumer gear, including cameras, is that some great features have been offered only in the past. With some Minolta models, you could plug a hand-held spot meter directly into the camera metering system. With some Olympus models, the OTF sensor was also accurate for normal exposures with the mirror up. Also with some Olympus models, several spot meter readings could be averaged.

    With one of the Contax models (or is it a Leica model), you can trigger studio strobes with the camera, and the camera will set the lens aperture correctly for the subsequent shot. The Nikon F5 allows exposure bracketing in the manual mode, which is a super convenience for macrophotography on slide film. Canon models compute DOF, which is really nice.

    So why can't a top Nikon, Canon, or Minolta body have several optional backs, each designed for a specific working requirement? Presently, top models usually have only one optional back, but it enables a motley assortment of features. Most buyers only use one or two of these features, and often only the features that are built into other camera bodies, often from the same manufacturer.

    In the PC world, if I want to change from Word Perfect to MS Word, I don't have to swap computers. In the audio realm, if I want to upgrade to a twenty-bit player, I don't have to sell off the speakers and amplifier. Ho hum.

    Sorry guys, but I'm still not impressed with the design decisions which have been made for most SLR bodies. I want the camera manufacturers to make enough money off casual shooters that they can continue to design and sell the really neat, limited quantity, pro stuff at reasonable prices. The alternative is that an F5 would be priced about the same as an equivalent Hassy or Rollei. But few consumers are attracted to SLRs, and a useless or inappropriate feature set simply compounds the problem, in my opinion.
     
  76. Being about to buy a camera or a lens it is all the time nice to read several reviews on photographic equipment. Imo opinion a well established review site will lead into a lot of technical disputes which is fine as long it is clearly separated from other aspects of photography. The risc that photo.net will be more commercialized due to the high interest of manufactures who might be willing to support as long as they agree with the reviews should be avoided.
     
  77. Having standards is a good thing, but attacking one brand (Nikon) in the standards document is not good.
    Standards and reviews should be unbiassed, and your document sets a tone of "if it's Nikon point out the weaknesses, if it's Canon point out the strengths".
    Only times I've seen that before was in reviews sponsored by one manufacturer wanting to give a bad name to another...

    Why should digital camera and LF camera reviews mention price but not 35mm reviews? In all fields, price determines what to expect of a camera (I too can write a comparison between an F60 and an F5 and determine the F60 is crap in comparison, but that's no surprise when the F5 costs 10 times as much or more. When compared with cameras in the same pricerange from other manufacturers it stands up nicely).

    Maybe also mention the intended audience for the equipment. For some people, manufacturers produce a camera for a certain type of photographer (no dealer will advise an F65 to a professional wildlife photographer, so why should a review of that camera be written from the point of view of one?).

    etc. etc.
     

Share This Page