Lens Resolution and Sharpness

Discussion in 'Canon EOS' started by rixhobbbies, Feb 16, 2007.

  1. I'm looking for a little better understanding of how the resolution figures
    shown in lens testing equates to perceived sharpness in photos.

    I've been able to perform some comparisons between a Nikon 18-200 VR and a
    Canon 17-55 f2.8 IS and a 70-200 f4 IS and although I am not surprised by the
    results, I just don't get the MTF figures. I was under the impression that
    higher MTF figures meant better sharpness.

    For example, I'll compare the results I found on the lenses I tested (above).
    On Photozone, they list the Canon 17-55 IS and the 70-200 f IS and the Nikon
    18-200 VR as having about 2000 MTF on center focus at f5.6. This would tell
    me that I would expect roughly the same sharpness and resolution of fine
    detail on a 10MP camera shooting raw. In this case, a Rebel XTi and a D200
    was used. All RAW images were processed using Adobe Photoshop Elements 5 at
    default setting. No post processing was done beyond this. Pictures were
    taken at 18mm, 35mm, 55mm, 70mm, 135mm and 200mm at f5.6. A tripod was used
    and the pictures were taken within minutes of each other. Shutterspeeds were
    at or over 1/125 second. ISO was at 100. 5 shots were taken of each sample
    and the best shot was selected. A flash was not used for the outdoor shots.

    I used outdoor foliage and brick walls as well as indoor and outdoor closeup
    photos in my comparison. I realize that this is far from a scientific
    approach, but I felt that it would be a good comparison as to how I perceive
    the sharpness to be between the cameras and lenses.

    I summary, in no single situation did the Nikon D200 with the 18-200 VR lens
    out-resolve the Canon with either the 17-55 IS or the 70-200 f4 IS. While I
    was happy to see this considering the differences in lens costs, it raises a
    quation as to why sites like Photozone show similar MTF numbers. One would
    expect that the center sharpness and detail resolutions between the tested
    cameras and lenses would be similar. In my case, they weren't even close.
    The Canon setup produced pictures that were so sharp in fine detail and
    resolution, that I was able to crop to 100% and produce an image that hardly
    looked cropped at all. With the Nikon, a 100% cropping produces an image that
    looked slightly soft and lacked much of the fine detail presented with either
    of the Canon lenses I tested with.

    Now, after reading the reviews, I thought something must be wrong. So I
    obtained a second sample of the 18-200 VR lens and re-tested, but got the same
    results. So, I must conclude that the 18-200 VR isn't as sharp and doesn't
    resolve the fine detail that the Canon lenses I have do, IMHO.

    Now, my unscientific test aside, I figure I must be missing something with the
    MTF numbers. Can someone clue me in?

    Regards,

    Rick
     
  2. You have just discovered the meaning of "lies, damned lies and statistics"

    Juts because a site prints numbers doesn't mean those number are accurate, either for the actual lens they tested or for any other given sample of the lens.

    Numbers make you think that the test is accurate, precise and the testing is scientifically rigorous, but that may be an illusion.

    MTF (see http://www.bobatkins.com/photography/technical/mtf/mtf1.html) can be a reasonable predictor of image sharpness, though each point in an image has it's own MTF curve, so just one number isn't likely to be able to describe the overall qualities of a lens.

    Lenses with the same MTF (or SQF) given as a single number may yield different image qualities. Any aberration (spherical aberration, coma, astigmatism, diffraction, defocus, chromatic aberration) will change (lower) MTF and SQF, but they can combine in different ways to give different image characteristics, though a lower MTF will pretty much always yield an image that looks less sharp.

    I could easily generate number when I do lens tests (I worked with optics and optical testing in a reserch lab for 20 years), but I don't (see for example http://www.bobatkins.com/photography/reviews/tokina_12-24_f4_review_3.html). I think numbers can fool people and are hard to understand. People think that just because someone prints that lens A is a 7.3 and lens B is a 7.2, that makes lens A better. It doesn't. The numbers may be wrong, or the difference may be insignificant or who ever did the tests may have gotten hold of a better than average sample of lens A and a worse than average sample of lens B.

    So your understanding of MTF is probably fine, it's your trust in published numbers that's the problem.

    You are also assuming that both you and the lens tester achieved perfect focus every time on every target. Unless you know that for sure (by testing), it's not always a good assumption to make. All images shot at the same focal length and at the same aperture should have the same depth of field. If the image of your main subject is sharper with one lens than another, but the background (or foreground) is less sharp, you have a focus problem. Autofocus isn't perfect and can differ very slightly from shot to shot. Maybe not enough to make any difference on real world images, but enough to throw you off if you're pixel peeping at resolution test charts.


    BTW when you quote the "2000" numbers you're presumably quoting resolution measurement, which is different from MTF. MTF is a measurment of local contrast at a given spatial frequency. Resolution is a measurement of when MTF drops below a certain value. It's quite possible to have a high resolution image that look less sharp then a lower resolution image if the MTF at lower spatial frequencies is higher in the second image than the first (i.e. if the MTF curves have a different shape). "Resolution" doesn't tell you anything about the shape of the MTF curve. Or to put it another way, resolution doesn't always give a 100% accurate prediction of how sharp an image will LOOK.
     
  3. Resolution has little to do with perceived sharpness. Sharpness is a function of edge contrast, not resolution. A low resolultion image can look very sharp whereas a high resolution image can look fuzzy, low contrast and unsharp. It's a matter of how the lens has been designed optically--sharpness and resolution usually being a direct tradeoff, i.e. you can emphasize one only at the expense of the other. The reason Leica is so famous (and expensive) is that their designers have seemingly found a way to do both, at least better than anyone else.

    Photozone is a fun and useful website in many ways, but he really only gives resolution figures, whereas what is really needed for a true understanding of a lens is contrast at various resolutions like MTF charts do. In that area the Photodo web site is much better as it provides the actual tested MTF curves, though not for all f stops like Photozone does (I use them both). As such, the Photozone data can be misleading to those who don't realize their limitations, as you've found out.
     
  4. You can't read across different camera systems with the Photozone data, they do tell you this if you look through their info page. This is because the cameras may have different numbers of pixels, different format sizes and different anti-alias filters.

    So it is a good guide for the relative sharpness of lenses measured on the same camera but not across different cameras.

    What the sharpness measure is they are reporting is the number of lines that can be fitted across the frame and have a contrast of 50%. Technically not MTF but SRF, howver the difference is probably small in most cases.

    Canon for example quite MTF (contrast) between lines spaced 10 line pairs apart per mm and 30.
     
  5. It's true the photozone data doesn't apply cross systems, but the poster didn't do cross systems. He used Nikon lenses on a Nikon camera and Canon lenses on a Canon camera just like Photozone. If Photozone got the same sharpness with both systems - and if their results are accurate - he should too.

    It really boils down to the fact that numbers can't be trusted and even when they are accurate, they don't tell the whole story.

    If data looks "scientific" (which in many cases means numerical)people tend to trust it. That can be a mistake.

    "98% fat free" is a good example. It doesn't mean that 98% of the fat has been removed from something. The full fat version might be "97% fat free". Numbers don't lie, but they do confuse at times.

    If your tests show that the 18-200 on a D200 isn't as sharp as a 17-55 or 70-200 on a Rebel XTi, that's what's true. That fact that it should really be no surprise that two shorter range "L" series zooms outperform a single wide range zoom is just a bonus!

    Next time you read photozone data, you'll probably do so with a healthy degree of skepticism.
     
  6. Bob perhaps I am misunderstanding you or the OP but he says;

    "I summary, in no single situation did the Nikon D200 with the 18-200 VR lens out-resolve the Canon with either the 17-55 IS or the 70-200 f4 IS."

    So he is comparing one system against the other on the basis of the IMATEST derived 50% SRF values on each system.
    The point is they have different AA filters and the IMATEST code seems (from what I gather very unclearly) to try to set the sharpening to make the system overall lens+AA+sensor+sharpening system is a first order filter (which is why I use my own software).

    So I still don't think say 1800 lines measured on one system equals 1800 lines measured on the other in terms of sharpness necessarily. The OP did not apply any PP, which presumably means no sharpening so difference in the AA filter characteristic will enter in which is a further complication.

    Other systematic issues with the OP test method may also cause issues. I know Photozone use a precision MF method. I presume the OP used AF, but that can have quite a lot of variation from operation to operation. So I don't know if he has done anything to mitigate that uncertainty factor.

    I still think the Photozone data is useful if it is used within the bounds that it is defined for. Of course it is not perfect (I have my own issues with the some aspects) but as a relative measure on the same camera it is useful as long as it is not taken out of context like this.

    Yes, the is part to part variation between lenses but you get that if you test by just shooting newsprint or a brick wall.
     
  7. In short on the photodo site, an lens rated 3.8 (or so) or better is a "good" lens. The advantage of that site is the same people did the testing, ON THE SAME equipment, using the SAME SCALE. (See Norman Koren's site for a detailed description of MTF's and pitfalls of it). You also can look at the luminous landscape, other landscape sites, and Norman Koren's Imagetest site. Note should you do your own testing with imagetest you will need to test one lens multiple times over several days to get your procedure down pat.

    Lastly as Bob says don't believe everything you read.
     
  8. Zeiss uses very fine grain film to test lenses. Some of the lenses you cite probably resolve greater than 100 lp/mm easily.

    If your sensor is a 10 meg sensor it is probably about a 2500 X 4000 pixels.

    If it is 12 mm tall then it has about 200 pixels per millimeter, not anywhere near enough to resolve 100 lp/mm so your sensor is your weakest link. (A line pair is a white space and a black line. Sinse it is unlikely that your image would line up exactly with the detail in your scene, it is generally accepted that you need at least 3 rows of pixels to resolve a line pair.)

    In the lens testing I have done, I do it inside, lock the mirror up, focus with a ground glass and release the shutter with a cable release. I use an airforce test target and still have to use a microscope directly on the negative to tell the difference between good lenses and great ones.
     
  9. "Yes, the is part to part variation between lenses but you get that if you test by just shooting newsprint or a brick wall."

    True, but generally you are testing the lens you have, so the results are a lot more meaningful (to you) than tests somebody else did on another lens.

    I think the resolution numbers are resolution numbers across systems. They take into account differences in sensors, AA filters, software etc.

    What I think photozone mean about cross comparisions is that you can't take the data for a Tamron lens tested on a Canon XTi (or whatever they use) and apply it directly to a similar Tamron lens used on a Nikon D200 (or a Canon EOS 5D for that matter).

    I'll have to look at their site again. I suppose if they're using dumb measurement units like lines per picture height, it could skew things unless you factor in the actual picture height.
     
  10. I wouldn't expect a 10-1 zoom (Nikon) to outperform a 3-1 zoom (Canon).
     
  11. I never have tested cameras or lenses but I was I was Deputy Director of R&D at a large organization and had other responsible jobs having to do with R&D , development and testing on large systems. We sponsored many studies and gave grants for research. What I have never seen in these internet equipment studies is an inventory of study variables influencing the outcomes (like different sensors) and processes for compensating for them. They are mostly empirical, i.e.; relying on study outcomes to prove a hypothesis. Every study begins with assumptions. Assumptions are articles of faith that can't be proven but without which a study cannot be under taken. We always required that such assumptions be listed and analyzed before proceding. If you want a different outcome vary your assumptions. I have seen this done at the highest levels to prove a point to Congress or to the public. In short, I look at most of these empirical internet lens tests with some skepticism as I think a certain amount of scientific depth and rigor may be lacking.
     
  12. A lot of good info here and it will certainly help me put appropriate weight on the different reviews I read. If nothing else, it will help me know what to look for.

    As for my testing, I understand that it was far from what I would call a professionally controlled test. Basically, what I was trying to do was determine why I was getting noticably softer images from the Nikon compared to the Canon when the resolution tests suggest that they should be comparable at the center.

    This is why I tried to make things as even as I could and try several different attempts with multiple shots to see if I could remove any potential influencing factors. I felt that the tests at least showed me that I could obtain far sharper pictures with the Canon setup regardless of how hard I tried to get the Nikon to at least come up even. I would even go so far as to say that I worked harder to get the Nikon sharp and yet it still never approached the Canon's sharpness when viewing image details at 100% crop using a RAW un-modified image.

    I just wanted to know why this was when the resoltion figures stated on the review site suggested that should at least be comparable, which I think Bob responded to.

    As for focus, I tried several focus modes on the Nikon, including manual focus. Since the Canon was consistently sharper, I felt the "problem" was with the Nikon so I spent a lot more time trying to get the Nikon to measure up. I tried all of the focusing settings as well as numerous manual focusing attempts. I even tried stopping it down to f8 in case it was some DoF issue. I tried autofocusing and tweaking the focus manually to counter any front or back focus possibilities. The best shots were with the camera on auto-focus and my manual focusing could match but not exceed those results. I have to assume the D200 was doing it's job and the lens was falling short. This is why I tried the second sample, which gave the same results. I concluded that the Nikon 18-200, which a nice lens certainly, doesn't hold up to the likes of an "L" lens despite the fact that the review site resolution figures suggested it would (at least at center).

    I put so much time into it because of curiosity, I thought I would bounce it off of the folks here in the forums. Honestly, I simply thought that there was something that I was missing in interpreting the review site's test results.

    In the end, the results make sense though. Nearly $2500 worth of Canon "L" glass (in two lenses) should certainly out-perform a $750 Nikon superzoom. I just got curious because of the quoted resolution figures and I wanted to see how this really translated into perceived sharpness.

    It's also interesting the comment made about the AA filter in the cameras. I can see how this could affect sharpness, but, I would like to think that Nikon would put something into their camera that would hobble it in such a way that a D200 owner could not achieve at least comparable results using their best glass (which the 18-200 VR clearly isn't).

    As a side note, I did note some other differences that I found between the camera's themselves (D200 vs XTi) and the lenses I used. In the Canon's favor, focus was noticable faster and more consistently accurate when using center point focusing mode and WB was more accurate. In the Nikon's favor, it produced noticably better exposure, keeping highlights and shadows from getting lost. I also think that Nikon's choice to include dedicated buttons and switches for the various functions works well. I hope Canon has taken some of this to heart when designing the 40D.

    In the end, it was fun to experiment. I even considered purchasing the Nikon or one like it as I liked the ergonomics of the body so much. But, Canon has a much more desireable lens lineup IMHO and that, I think, is really where the image quality is. The test I did just reinforced that thinking.

    Thanks for the feedback, everyone!

    Rick
     
  13. One more thing. Review sites aside, the real source of quality information and recommendations I got was from this forum. The feedback lead to my selecting the 17-55 IS and 70-200 f4 IS lenses and quite obviously, they are proving to be quite outstanding! I did a photoshoot of red-tail hawks today using the 70-200 and I am simply amazed that I can take a shot and crop it 100% and the image looks fantastic! You wouldn't realize it was cropped unless you tried to zoom in some more. Certainly on print you'd never know! Amazing, I'd say!

    So, thanks to the photo.net users who helped me make this choice! Now I just need to save up for that 100-400 IS...

    Regards,

    Rick
     
  14. They are mostly empirical, i.e.; relying on study outcomes to prove a hypothesis.
    This and Bob's "damn lies..." quote remind me of another. I can't recall who said it and I probably won't get it exactly right but it goes something like this:
    "Statistics are used as drunken men use lamp-posts: for support, rather than illumination." ;-)
     
  15. Bob Atkins: "I'll have to look at their site again. I suppose if they're using dumb measurement units like lines per picture height, it could skew things unless you factor in the actual picture height."

    Yes, I think this is why they use such daft units, because if you used lp/mm people would be inclined to read across. Even after you account for resolution and format size differences you still need to account for differences in AA-filter characteristic.

    That is the real problem you don?t know what the AA-filter is let alone the overall sampling aperture of the sensor and filter, then we have different demosaic algorithms on top, so reading across systems is at a minimum very suspect, but within the same system there is no reason why it should not be consistent as long as the focus etc is consistent.

    As you know I did some research on AA-filter characteristics and try to deembed this in my own measurements so I can scale results in lp/mm, but I can have no idea if I have this right. But if I plot the 10 and 30 lp/mm measurements against Canon?s MTF data it is surprisingly good, I only did this for fun originally.


    Neal Shields: "If it is 12 mm tall then it has about 200 pixels per millimeter, not anywhere near enough to resolve 100 lp/mm so your sensor is your weakest link. (A line pair is a white space and a black line. Sinse it is unlikely that your image would line up exactly with the detail in your scene, it is generally accepted that you need at least 3 rows of pixels to resolve a line pair.)"

    Photozone uses IMATEST software which is based on the ISO slanted line method for the resolution tests. This permits over-sampled measurements up to four times the Nyquist rate so reducing these issues. Also 100 lp/mm sounds like an old style resolution measure which typically recorded around 10% MTF, Photozone are quoting 50% MTF (SRF).


    mike earussi: "Resolution has little to do with perceived sharpness. Sharpness is a function of edge contrast, not resolution. A low resolultion image can look very sharp whereas a high resolution image can look fuzzy, low contrast and unsharp."

    This is very true, but the sort of resolution you are probably talking about is the 10% MTF type, not the 50% one which is much more relevant. Fine detail on 35mm is at 30 lp/mm and on APS-C is at 48 lp/mm. Many lenses wide open will have a MTF of close to 50% at those spatial frequencies.

    It is possible to plot the MTF (SRF) at any line frequency as the ISO method provides a complete spectral response for every test point and orientation. I suspect they don?t plot these to keep the volume of data down and make it less confusing. My own experience is that plotting the 10 and 30 lp/mm curves does not provide much more insight over the 50% MTF frequency.

    It clearly is confusing even so as it is considering some of the comments one sees, people do not really understand the measurement
     
  16. From the Photozone FAQ http://www.photozone.de/8Reviews/lensFAQ.htm

    "Q: Are the figures comparable between cameras or different systems ?
    It depends on the similarities between the image sensor system. A sensor SYSTEM contains the image sensor with or without micro-lenses, an IR filter, a low-pass (Moire) filter and the signal processing. As you can imagine the output quality is largely dependent on the whole chain on not just on the amount of megapixels. The different output quality between the Canon EOS 350D and the Olympus E-300 is a good example (despite a 8MP sensor). If the image sensor system is very similar (like e.g between the EOS 350D and the EOS 20D) the findings remain pretty much valid."

    This is somewhat less strong than the original note they had about this.

    Rather amazingly if you go to the IMATEST site http://www.imatest.com/docs/sharpness_comparisons.html they do try to compare completely different camera systems. I find this procedure suspect for the reasons I have noted above.

    OK so lets try and compare, rather naively neglecting differences in AA Filter, demosaic and NR. The Nikon measurement is on a D200 and the Canon measurement on a 350D. The frame sizes are 3,872 x 2,592 and 3456 x 2304 pixels respectively. A linear sampling rate factor of 1.1225 on average.

    Looking at the Photozone centre results for the Canon 17-55 at 17mm f5.6 we have 2093 lines, for the Nikkor AF-S 18-200mm f/3.5-5.6 G IF-ED VR II DX at 18mm f5.6 centre we have 2105.5 lines. Scaling the Nikon results yields 1875 in Canon terms, quite a bit less sharp in pixel peeping terms, but still perfectly usable.

    However, doing this we have made all sorts of assumptions about implicit signal processing, that the person who prepared the data warned us against. What is surprising is despite this; at least at this point it seems to support the OPs results.
     
  17. Good info. I never realized that there was so much to consider. I guess I over-simplified it in my mind, which perhaps comes from my astonomy background where you can judge a piece of equipment's ability to resolve details across the board, regardless of manufacturer. I assumed that it would be reasonable to expect test results with cameras to reflect this, especially since it seems like they are using a test chart for measuring resolution and resolved details. There are still some pieces missing for me, I'm sure, so feel free to continue the discussion. I'm learning a lot!

    Regards,

    Rick
     
  18. The really ironic part is that, despite numerical measurements, confusing units, cameras of different sensor size, AA filters and resolution, you could pretty much have guessed that the 18-200 wouldn't be as good as the 17-55/4 and 70-200/4 without doing any testing at all. You'd have been right too, so you might wonder if the time spent and the complexity of the anaysis is actually worth doing!

    I've found that given two lenses with the manufacturer's name, the lens specs and the price, I can pretty much guess which one will be better and I'm right most of the time. There are a few surprises, but not very many.

    BTW the simple explanation as to why photozone lens test results aren't comparable between nikon and canon is that photozone doesn't do lens tests and don't test lenses. They do system tests and test systems with a couple of specific cameras. Ironically (again) Popular Photography actually do LENS tests but they present their results "dumbed down", and much of the value of doing that is lost.
     
  19. Good thread. The truth is that testing lenses has always been a very complicated business, both for the tester and the user. With digital, it's even harder.

    Eg contrast is usually far more important than resolution, but is much harder to measure without a 'real' MTF test bench (MTF has become an abused 'marketing' term on some websites these days). Very few of these machines exist outside lens manufactures' development departments and those that are available for hire are very expensive. They are huge, many meteres long, and need to sit on tons of heavy concrete.

    I reckon you need at least 142 test traces to get a reasonable idea of lens sharpness and ideally you should test several samples of each lens, running into hundreds of traces and days of work. That's before any anlayis takes place, and there are also many other important aspects to lens performance than sharpness alone (which is mostly all that MTF can reliable measure).

    Then you have the digital factor. All digital sensors are different so even if you could fit a 'better' Nikon lens on a Canon, you may not get the result you were expecting. In the film days, we could all use the same film for lens testing and the camera 'system' was not such a significant factor.

    All of which leads to the conclusion that unless users are prepared to pay for thorough and conclusive lens tests, then they will never appear on the 'free' internet.

    Richard.
     
  20. Bob Atkins : "The really ironic part is that, despite numerical measurements, confusing units, cameras of different sensor size, AA filters and resolution, you could pretty much have guessed that the 18-200 wouldn't be as good as the 17-55/4 and 70-200/4 without doing any testing at all."

    Ha :) yes, going to be fairly obvious in this case. I may be wrong but expect you would still be hard pressed to tell the difference between an A4 print from each though.

    Bob Atkins : "BTW the simple explanation as to why photozone lens test results aren't comparable between nikon and canon is that photozone doesn't do lens tests and don't test lenses. They do system tests and test systems with a couple of specific cameras."

    Exactly, they can be useful as long as they are not taken out of context. It should however be fairly easy to deembed the rest of the system, probably with a simple scaler adjustment. I am sure the frequency response of the system could have the frequency response of a lens measured on an optical bench subtracted to get that.

    However, I doubt that we will get much better than these as they are relatively cheap and easy to do. People seem to forget that the old Photodo tests were system tests also, but here the lens MTF and film MTF were combined.
     

Share This Page