What Do PopPhoto Lens Test "Subjective Performance" Tables Really Show?

[r_rubinstein]

Can someone explain the PopPhoto "subjective" lens performance table?

What does "subjective" mean? Are lenses graded "on a curve" (i.e.

against products of similar cost and quality) or on a straight scale?

The reason I ask is that the tables for a $119 dollar sigma 24-70

3.5 - 5.6 zoom and the canon 17-40l were pretty similar; honestly, I

find it hard to believe that performance levels would be so close.

[frank_skomial]

I think Popular Photography's "subjective" means that the tests may not necessarily be completely technically correct, complete or repeatable. This way Popular Photography stays away from troubles. If your tests are contrary, then ... you see ... they were subjective, or subject to interpretaion of the testing person. Saying it otherwise, those tests can be tinted or biased towards vendors who improve the bottom line of the publisher.

[neal_shields]

It means that:

 

They are too cheap to buy test equipment,

 

to lazy to use it,

 

or that they think that you are too dumb to understand objective data.

[larry_underwood]

I think they don't want to say to much bad about any lens, because they all advertise in their magazine. To them, they all ride a white horse.

[kelly_flanigan1]

It is a dumbed down table; to reflect the lowering of folks knowledge. Pop Photo has been around before WW2. Older how to articles carried alot more meat. Folks dont think as much today; and demand a simple answer; to a complex problem.

[stephen hazelton]

What it's intended to show is the overall quality of a photo taken and viewed at the size shown. Contrary to the remarks above, they DO show many lenses as not being so hot. You'll notice, though, that they don't often test the cheapest brands (Phoenix, etc). The image quality doesn't tell you anything about build quality, closest focus, feel of the lens, etc., for which you have to read the article. They are not graded on a curve. You'll notice too, that the two lenses you're comparing are not anywhere near the same focal length, so it is no surprise that a "standard" zoom range lens by Sigma turns in comparable performance to a "superwide" zoom range lens by Canon. Comparing a 24-70 from each or a 17-40 from each would mean a lot more. And, as I recall, the results are based on actual testing- reading the articles will show that.

 

Regardless of their testing, you will find some people that like either lens, some people that wouldn't own one or the other, etc. One of my favorite lenses is one which another poster complained was "not critically sharp".

[stephen hazelton]

I just saw a post farther down about a Sigma 24-70 f/2.8 lens (IE, not the same exact lens, a faster pro lens) and thought the responses were enlightening:

 

"The Sigma lens you ask about is one of their best professional quality lenses"

"I have the Sigma 24-70, and love it... Very sharp, very fast AF, very solid build. I consider it $ well spent."

"I have it to and it is well worth its $$ ! So, yes it is better than yours (but exactly comparable focal length). And No, there is no better Nikon at the same price..."

 

Now, keep in mind, that with those testimonials above, there are other people on photo.net who would consider this lens to be junk and would never buy it.

[stephen hazelton]

Somewhere down there was another post complaining about the Canon lens- specially, indifferent performance compared to an older Canon zoom of similar focal length. I quote:

 

"Camilla, nothing is wrong, the 17-40 is not a brilliant example of canons lense quality (although it is very good)"

 

"Keep in mind that the extra $$ for the 17-40 pays for more than just image quality....a good friend of mine...longs for the USM speed, constant aperture, manual focus ring, distance scale, and build quality of the "L" lens".

[bob_rannells]

Kelly, perfect answer.

[mike_elek]

I've been disappointed in Pop Photo for some time. The writing is too folksy with terrible grammer, although I notice that they've made an effort to improve that in recent months.

 

Equipment reviews are still too soft. They need to just say a dog is a dog sometimes.

 

One of the better magazines is the UK publication Amateur Photographer. They pull no punches in their equipment reviews. If a camera is disappointing, they clearly state that in their conclusion and throughout the review.

 

I think Pop Photo needs to use that as a model. Their velvet-glove approach is a disservice to the readers and to the advertisers. Much better to be straightforward.

[chris haake]

Stephen H is absolutely dead-on. I'm guessing that many who have railed against PopPhoto's SQF tests haven't bothered to actually read the articles. I have. While PopPhoto may be diplomatic about a lense's deficiencies, they list them nonetheless. I recall (from an issue I no longer have) a lense--I think it was the Sigma 24-135--which they specifically said had "significant" distortion of this type and "very noticeable" distortion of another type, and their SQF table corroberrated what they said. Their concluding paragraph, of course, said something like the lens was "remarkably well corrected for a design of this zoom range" (or something like that), which was about the nicest thing they had to say about it. No, they didn't trash it, but if you just look at their data and what they DON'T say, you get it. I've seen tests where they do the same to Nikon, Canon, Pentax, and Minolta lenses. Remember, PopPhoto gets at least as much ad money from Sigma as any of those other companies...just look at the four-page ads they run. So, is PopPhoto as ruthless (or, rude) as some of the British mags? No. Diplomatic while still presenting their findings? You bet.

[bruce_rubenstein]

The SQF numbers/table, is actually based on tested MTF values, presented in a somewhat simplified, comprehensible for the lay person, format. The subjective part is what the "average" person can see, as opposed to just a set of numbers. For instance, 60 is better than 50, but can you see the difference in a 4x6 picture? The table tries to convey that information. The biggest issue I have with Pops testing is that they may be using hand picked samples. They have stated that if they get something in that doesn't do well, they'll send it back for a different one. They also won't publish test results of bad products.

 

If they ever figure out that Herb Keppler stopped breathing in 1986, and get someone else in there to run the mag they may have something worth reading.

[dan_fromm2]

SQF is an unfortunate name for a very useful measure of lens performance. It was invented by Kodak, and it reflects the results of a large survey of people's ratings of carefully made prints. One of the interesting results of the survey is that subjective quality ratings are very closely tied to system -- taking lens + film + enlarging lens + paper -- MTF.

 

I believe that if the survey were redone, PP would award higher SQF ratings than they now do. This because, thanks to peoples' habituation to TV and computer monitors and the spread of digital cameras and the crappy printers often used to print images taken with digicams, generally accepted standards of minimum acceptable print quality have fallen.

 

The news from PP's SQF charts is interesting and, for 35 mm still photography, not particularly good. The question SQF answers is "If I shoot at f/x and enlarge to a x b, will most people think the print looks good?" The news from PP's charts is that getting an ok print larger than 8x10 from a 35 mm negative requires using a very good taking lens at a 'good' aperture and perfection elsewhere.

 

As for why lenses with different prices produce similar SQF charts, well, the purchase price of a lens buys many things, e.g., build quality, tighter quality control, marketing budget, that have little to do with how well the lens performs. Also remember that PP tests just one example of a lens. There can be considerable variation between examples of the same make & model, so the tests are at best a suggestion. If you luck into an example of a lens that's at least as good as the one PP tested, well, then, ... Otherwise, ...

 

Cheers,

 

Dan

[douglas_greenberg]

In response to the direct question asked about the two lenses in question, it's actually quite common for very cheap lenses to be close to a premium lens in terms of image quality under most circumstances. My guess would be that at the widest apertures there probably are differences, there might well be significant differences in terms of linear distortion correction, and finally, the color rendition of the cheaper lens is very possibly inferior. Oh, and I forgot to mention build quality, which is really important to professionals and very active amateurs, but is not so important to weekend photographers.

 

Some of the previous answers about SQF are better than anything I could write, so I'll pass. However, let me add that some people expect too much from Pop Photo, I think, which if you subscribe is very cheap and usually has enough interesting stuff in it to make getting it worthwhile. The photography community is so varied in terms of interests and skill levels, it would be veritably impossible to create a single magazine that pleases everyone.

 

Yes, Pop Photo is going to be reluctant to publish test results on really bad equipment. No surprise there. Yet what they do is actually puzzling at times. They will publish SQF data that show a lens to be fairly mediocre, and then the written commentary will say that "all slides were sharp and snappy except in the corners at the widest aperture." ???

 

Actually, MOST lens "tests" published in magazines are pretty vague and useless. Very few use objective criteria; despite the flaws in their appraoch, Pop Photo is actually more rigorous than what you get from most photo mags.

 

I used to think that Pop Photo's basic approach was to avoid saying much of anything negative about a major manufacturer's lens, but their review of the Nikon 80-400mm. VR zoom, which I own, dissuaded me of this (and continues to puzzle me). Based on the Pop Photo review, no one in his/her right mind would buy this piece of junk, which tested "F" in terms of SQF at 400mm. at the wider apertures. Yes I use this lens almost every day to take bird photos and am completely satisfied with its quality at these "F" graded settings.

 

All I can think is that indeed, there are variations in lens samples, and once you realize this, then you have to take ANY test of a single lens with a grain of salt.

[r_rubinstein]

Thanks, everyone!

[e._m.]

"Their velvet-glove approach is a disservice to the readers and to the advertisers."

 

No, actually the velvet glove approach does a great service to their advertisers, who want P.P. to not give their crap products a bad review.

[douglas_greenberg]

I guess now I'm curious: is there a widely available photo magazine/publication that does NOT give manufacturers a "velvet glove" treatment? That is, is there one that will consistently, unflinchingly, tell the truth about all products that come their way? And does real testing?

[steve_dunn2]

<p><a href="http://www.bobatkins.com/photography/technical/mtf/mtf1.html">MTF and SQF explained by Bob Atkins</a></p>

[stephen hazelton]

There are some websites that tend to be quite critical of gear, but then again, they just have DIFFERENT biases.

 

And you still have the fundamental problem, that some people like a product while others do not.