Roger Cicala does one for the fanboys (sorta)

Unless there is evidence of sampling bias, LensRentals results are statistically valid. However a smaller sample size results in wider confidence limits regarding the results. I sympathize with Roger Cicala's "bias" against testing zoom lenses, which derives from the number of tests across the focal length range required to characterize the lens, times the number of lenses tested. As an intrinsically "lazy" scientist (hence innovative), I would look for a way to automate the process.

Sony 70-200/2.8 lenses seem to compare weil with the competitors. The difference is variability between lenses. Sony had better step up to the plate on this one.

Why would they do that? Are some lens that bad?

Some lenses have more sample-to-sample variation than others. Wider variation is an indication of poor quality control, and possibly design issues.

MTF curves have significantly more falloff at the corners in Roger Cicala's results than results posted by Zeiss* or Sony. Home team bias aside, lenses designed specifically for Sony FF cameras account for the cover glass on the sensor, approximately 2.0 mm in thickness. Cicala uses Zeiss MTF test gear, and inserts a 1.5 mm glass plate for test purposes.

* Zeiss cites MTF data from actual lenses. Sony MTF data is calculated, ignoring any manufacturing variations, including curves and centering.

It's still not usually a statistically valid sample

For those who care about this:

First, validity and reliability are different things. Reliability refers to the size of the error band around the estimate--to oversimplify, the random variation in quality from lens to lens. Validity also refers to whether the estimates are systematically wrong.

There is no cut-off where a result switches from statistically unreliable to statistically reliable. The more lenses, the more reliable. Specifically, the standard error of the mean of a statistic (say, a point on one of the MTF curves) is the standard deviation of the statistic divided by the square root of the number of observations. It's impossible to gauge the reliability of a test of a single lens, as one has no indication of the variability of the statistic.

The validity question Ed raised is whether the validity of manufacturer's charts are biased upward (or Cicala's are biased downward). I wonder whether the results of manufacturer's tests could be more positive because they are testing lenses that haven't been shipped and handled. I have no reason to expect this other than Ed's comment.

The validity question Ed raised is whether the validity of manufacturer's charts are biased upward (or Cicala's are biased downward).

That's not what I said. Please re-read.

If the test does not duplicate the thickness and placement of the sensor cover glass, resolution at the edges of the field will be degraded. This would constitute a testing bias.

Confidence intervals are wider for small sample sizes. For normal distributions, Confidence intervals are inversely proportional to the square root of the sample size. I refer you to "Student's T Statistics."

If a sample falls outside the small sample confidence interval, it is said to be an outlier, rather than within expected variations. Cicala's article illustrates this principle.

Ed,

This is what you wrote:

MTF curves have significantly more falloff at the corners in Roger Cicala's results than results posted by Zeiss* or Sony.

Doesn't that mean that the values in the corners are wrong in one set of charts or the other set, given that they disagree with each other? If not, I apologize for not understanding what you meant.

I'm not sure why my misunderstanding your sentence above, if I did, led you to conclude that I don't understand first-semester statistics. I taught this stuff for years at the graduate level. You wrote:

Confidence intervals are wider for small sample sizes. For normal distributions, Confidence intervals are inversely proportional to the square root of the sample size. I refer you to "Student's T Statistics."

What I wrote about the standard error of the mean is simply another way of saying the same thing--or to be more precise, it explains your statement about confidence intervals. In theory, a confidence interval can be of any size, although it has long been convention to set their boundaries at +/- 1 or 2 standard errors. So, my explaining that the standard error is a function of the square root of the number of observations is equivalent to saying that the width of a confidence interval is inversely proportional to square root of the sample size.

BTW, "outlier" has no uniform definition, but few people I read would use a 1-standard-error confidence interval to define outliers. Two standard errors, perhaps. Some who offered specific definitions have opted for a more extreme cut yet. If you are intrigued by this stuff, search on "John Tukey outlier" without the quotes.

Dan

The cover glass effectively increases the optical distance between the rear node of the lens and the focal plane by about 1/3rd the path length, with the effect of curvature of field. If the test glass is too thick or too thin, corner resolution will be adversely affected. The finest curve in Cicala's test represents 50 lp/mm, whereas Zeiss limits their test to 40 lp/mm, and most others to 30 lp/mm. 50 lp/mm is a very severe test of lens performance in this context - contrast v distance from center. On the other hand, contrast in the center is 50% or more, which is very good for 50 lp/mm.

Image stabilization in the lens and/or body adversely affects overall resolution, to the extent of about 2 pixels in the A7Riii/iv. Unlike early implementations of IS, artifacts are not exaggerated when a tripod is used, so I don't ordinarily turn it off, except to demonstrate ultimate resolution or make use of pixel-shifting.

"Outlier" is a valid term if you define the confidence interval, typically 95%.

Chill, Phil. You are going OT.

I see a tree, with something hanging from it. Someone has cut a name on it. Phil;)

Some folks like the technical side of photography and like to discuss it among themselves.

Photography encompassing all.

Chill, my friend.

F$ck all this bullshit wannabe geekery circle jerk off nonsense. This is not photography. Who the f$ck are you people? You’re not photographers that’s for sure. This is 1000 % hilarious.

So saith the photographer with no posted images and STEM aversion syndrome.

Really.

Just chill.

Presented as a Public Service:

What is a lens MTF chart and how do I read it | NIKKOR lens technology from Nikon

One man's insult is another man's truth. What I indicate is the severe lack of action. This pseudo technical geek talk about lens charts is inconsequential to both the practice and theory of photography.

Says the guy who posts Jordan Peterson trivia that have no relevance in any real world. I don't think he has any useful ideas about either practice or theory.

A PN gallery is, of course, the publishing standard for photography. LOL

How much more impressive to have your own little site, trying to sell photos. LOL

See what you think about Phil's work at PHILIP SWEECK

I’ve been listening to his lectures long before he became famous

We can tell.

Maybe it's time you listened to what he is really saying!

Oh please. GTFOOH with that old Guardian link. Lol. These hyperbolic smear pieces are old hat; they are certainly nothing new to anyone who follows Peterson (and I’ve been listening to his lectures long before he became famous). There are probably hundreds of these silly articles. Not sure what your point is here in linking to one. Everyone who doesn’t agree with the preapproved idiot narrative of the new left is considered a “dangerous right-winger.” Obama is now even supposed to be right-wing according to this new left and that’s moving ever more to the far left to the point of absurdity and self-parody. Instead of listening to what these radical left-wing dimwit activists pretending to be journalists claim about Peterson, maybe take the time to listen to what the man himself is actually saying.

 

And thanks for the advertising!

LOL, left wing, right wing, don't kid yourself, I have seen both sides of Iron Curtain, they pretty much the same. Left-wing Obama was bombing Middle East the same way, right-wing Bush did. And American doctors, who participated and assisted in government sanctioned torture of prisoners , had to pay to settle out the courts.