Lens Tests (MTFs) for 300+ lenses

For those of us who are interested, the following site has over 300+ 35mm and MF lenses tested with detailed MTF curves and scores.

<p>

<a href="http://www.photodo.com/prod/"> http://www.photodo.com/prod/</a>

<P>I will point out that the data in the Contax SLR body section of

photodo.com is laughably inaccurate. This is a real shame since Contax

provides Dead Trees marketing collateral as well as a corporate web

site that has specifications. It is clear that whoever authored the

Contax SLR section of this web site A.) was very careless about data

entry, and B.) has no familiarity whatsoever with the Contax SLR

product line.

<P>That said, I question the accuracy of these 300+ lens tests.

There are certain Zeiss primes especially the wide angle side have

inferior MTF curves than the third party equivalences in the test.

There are also some Zeiss lens with Macro capability are labelled as

not. I personally won`t trust anything from this web site.

I was interested in both Sean's and Eric's remarks.

<p>

First, the MTF graphs were clearly generated by a different person

than the individual(s) who transcribed the information Sean questions.

It isn't necessary to distrust the MTF data simply because of errors

that have nothing to do with the data.

<p>

Second, the data were generated by a single identified individual

working at one machine in one site for all the measurements. That this

is not the same individual, the same machine, or the same site that

were involved in generating the MTF data that Eric cites is not at all

surprising. For this and other reasons it is also not surprising that

the data are not in complete agreement with the data presented by the

manufacturer.

<p>

I have no idea how the manufacturer generates the MTF data. I suppose

that there are several possibilities, including: (1) test a number of

lenses, and present the data for the best individual lens (2) test a

number of lenses, and then present the data for a lens that is

truly representative of the sample (3) test a number of lenses, and

use the data to generate average curves for the sample. I strongly

suspect, though, that the manufacturer does NOT do what the photodo

site workers apparently do: generate the MTF curve from a single,

random sample of a lens purchased at retail.

<p>

What you trust or mistrust may say more about you than about the

data. (I might be more critical of the information available at

photodo.com if the MTF data presented for my favorite Mamiya 7 lenses

didn't compare so favorably to, say, some of the Zeiss lenses...)

>>> generate the MTF curve from a single, random sample of a lens

purchased at retail.

<p>

i can't tell if you think is this a good thing or a bad thing. i will

tell you think this it's a bad thing. a horrible thing in fact.

<p>

back in my college statistics classes we learned how big a sample was

needed to produce reasonably accurate results. i don't remember

the exact formulas, (they were pretty involved) but i do remember that

ONE was not a good sample size for anything.

<p>

i recently bought a brand new EOS-5 body. the shutter failed on the

very first frame i fired it on. (so i sent it back and got a new one)

if i worked at photodo.com i guess i'd say that the average shutter

lifespan for EOS-5 cameras was 1 firing. nonsense!

<p>

any lens test that reports data based on a sample size of ONE is

beyond worthless.

Most magazines test a sample of one, for economic reasons. I once

talked with someone involved in testing who said that if the

results looked "odd" (very good for a very cheap lens, or very bad

for an expensive lens), they'd test another one. Hardly scientific

I agree.

<p>

The excuse behind testing one sample is thet modern QC techniques

are good enough that there is very little spread in performance.

Of course that's not much good if they happen to test one of

the outliers, and by testing just one they don't know when that

happens. You can probably say that most tests are fine, the

trouble then being that you don't know which one's aren't!

<p>

You could write a test of any modern consumer zoom without even

testing it.<em> "Slightly soft at maximum and minumum apertures, but

sharp and contrasty at intermediate apertures. Excellent

performance at the short end with a slight drop when zoomed out

fully. Light fall off gone by 2 stops down. Slight to moderate

barrel distortion at the short end changing to slight to moderate

pincushion distortion at the long end. Flare well controlled at

the short end but showing some ghosting in strong backlight at

the long end."</em>

Once again Bob Atkins has hit the nail on the head. What he wrote sounds like almost every Popular Photography test I have ever read. It means almost nothing in the real world, but sounds scientific enough to fool most of the people most of the time. Lens tests are good for getting information on specifications, construction, and handling of a lens, but really not much else.

<p>

The Canon Rebel G ads are right. Image IS everything. The image you put on film that is. Even though it tests badly the Canon 75-300 'IS' lens has allowed me to get a few shots that I wouldn't have gotten with any other lens that was available when I bought it. That is a much more important test of any lens.

Sean quoted me as suggesting that the testers at photodo might

"...generate the MTF curve from a single, random sample of a lens

purchased at retail."

<p>

Then Sean said: "i can't tell if you think is this a good thing or a

bad thing. i will tell you think this it's a bad thing. a horrible

thing in fact..."

<p>

I believe the folks at photodo merely provide information that can be

thrown into the mix and weighted as you wish. Testing a necessarily

limited number of lenses, even a single lens, is similar to Consumers

Report technicians testing one--and not hundreds--of Chevy Suburbans

before presenting data on, say, how long it takes the beast to lumber

from 0-60 mph; if CR says that the single vehicle tested required 0.2

seconds longer than the figure published in Chevrolet's product

literature, is that a horrible thing?

<p>

I don't think so. Instead, while I'd probably ascribe the difference

to the (very) limited sample size, I'd also be glad that apparently

honest work allowed me to consider an unbiased independent assessment,

no matter it's limitations.

<p>

Of course, when I recently purchase a pricey lens, I did exactly what

I think the photodo workers do: I used a sample size of one. Is that a

horrible thing? As Bob suggested, modern quality control techniques

should protect me from purchasing an outlier, but every 43mm lens

Mamiya made cannot be precisely like another. Would the MTF data

generated by the folks at photodo for a single sample of that lens

exactly duplicate the curves from the manufacturer? I doubt it, but

I'll never know: unlike Carl Zeiss, or even Chevrolet, the

manufacture of my new baby doesn't publish test data, so there's

nothing to get excited about.

i'd like to see them test 2 (or maybe 3) lenses. hundreds would not

be necessary. if the results of the 2 sample were similar that

would mean 100 times more (even though there were only 2 tested) then

the results of only one. if the results were wildly different then

the data would obviously be more suspect. they could either test a

third (probably similar to one of the first two) or just leave it with

"take your chances"

<p>

as it stands i have to read several different lens tests to gain any

useful data. if i look at 4 places and 3 say "great lens" and one

says "horrible lens" i go with "great lens". and thank myself i

didn't only read the "horrible lens" test and draw the wrong

conclusion. photodo does provide a useful data point. but i don't

place all my trust in them.

<p>

as for your own sample size of one. when i buy a lens, i do a limited

test of it. if it performs worse then what i expected, i exchange it

for another. i have only done this once, but it was worth it. my 2nd

one was much better then the first.