DxOMark's Sensor Overall Score

<p>There is published controversy regarding DxOMark's Sensor Overall Score. I have owned some of the cameras on the list below and would agree with the scores. What is your opinion about the scores? <br /><br /> <strong>Camera Sensor Ratings by DxOMark</strong><br /> <br /> http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Ratings<br /> <br /> <strong>SENSOR OVERALL SCORE</strong><br /> <br /> Canon EOS-1Ds Mark III (80%)<br /> <br /> Canon EOS 5D Mark II (79%)<br /> <br /> Canon EOS-1Ds Mark II (74%)<br /> <br /> Canon EOS-1D Mark IV (74%)<br /> <br /> Canon EOS-1D Mark III (71%)<br /> <br /> Canon EOS 5D (71%)<br /> <br /> Canon EOS-1D Mark II N (66%)<br /> <br /> Canon EOS-1D Mark II (66%)<br /> <br /> Canon EOS 60D (66%)<br /> <br /> Canon EOS 7D (66%)<br /> <br /> Canon EOS 1Ds (63%)<br /> <br /> Canon PowerShot G1X (60%)<br /> <br /> Canon PowerShot G12 (47%)<br /> <br /> Canon PowerShot G11 (47%)<br /> <br /> Canon PowerShot G10 (37%)</p>

So much for medium format :-) Or are the DxO results a bit flawed/biased?

<p>Not sure. But, so much for Hasselblad.</p>

<p>You scoundrel, Peter. Isn't that a Nikon listed as "best ever?" </p>

<p>What? The D800 better than 35mm and medium format? Oh no! I'll reserve one D800 and one D800E for now just in case the Canon EOS 5D Mark III and Canon EOS-1D X fall short of 95% <em><strong>:)</strong></em></p>

<blockquote>

<p><a href="http://www.dxomark.com/index.php/About/What-is-DxOMark2">To design DxOMark Scores, we have made choices about our photographic use cases and their associated image quality requirements (such as resolution, distortion, noise, dynamic range, etc.). It is clear that other photography experts may see things differently. We are very open on this site about the choices we have made so that anyone interested in creating a different scoring system can do so based on their own analysis of our DxOMark Scores and Measurement Database.</a></p>

</blockquote>

<p>The "problem" with the scores is that they apply their arbitrary scoring to a multiple of metrics to end up with one number, if you don't give the same weight to the metrics they do the scores can be moved substantially.</p>

<p>Whilst they are very open about how and what they test, the relevance to any one individuals shooting needs is so rounded up, or down, to achieve the one number that it becomes almost meaningless. For instance, how many people downsize to 8 MP for everything? For electronic display that is too high, for many prints, way too small, but it puts cameras with higher than 8 MP at a disadvantage, or advantage depending on your opinion, unless you agree with the specific weighting <a href="http://www.dxomark.com/index.php/About/Sensor-scores/Viewing-conditions">they go through to even everything down to that 8 MP figure.</a></p>

<p>Another controversial point, they claim to measure RAW files direct off the sensor, but can make no allowance for on sensor processing, similarly, if nobody can actually use unprocessed RAW files, scoring it seems pointless. For instance there is some in camera processing of most RAW files, if those processes improve actual image quality not testing them doesn't really work. It is like saying a 600hp semi will have a higher top speed than a 450hp Ferrari because it has more horsepower. I'd take them both for a test drive!</p>

<p>The weighting of their metrics end up showing serious anomalies, they used to have a page directly addressing the inability to compare different sensor sizes, the scores don't transfer across size differences. That is why, Medium Format cameras end up scoring so badly, too high an importance is put on high iso performance. </p>

<p>Personally, I rarely shoot above 200 iso and my idea of sports shooting is AF, AF and AF, I don't care if a different sensor can perform 5 points better (or 1/3 stop) I care which will keep my subject in focus. Bearing this in mind, for my uses, DxO marks are worse than useless.</p>

<p>Scott: As an example, concrete compressive test results in the construction industry in my neighbourhood <strong>must</strong> be certified accurate by a <strong>licensed professional engineer</strong> as required by <strong>law</strong>. The same regulation should apply to sensor testing. LOL!</p>

For my three current canon DSLRs I think the weightings are fair (5DII, 1DIIN and 7D) but I think that they must put a

heavy weighting on high ISO. My Leica M8 performs badly (like MF) and it really does have great IQ.

Why 30d is 3 points worse then 20d?

Same sensor, better body? Maybe they

think cameras are like wine?

<p><em> how many people downsize to 8 MP for everything?</em></p>

<p>That's not the point. The DXO sensor metrics attempt to address the question of signal quality from the sensor <em>per area</em> in the final print. The metrics do not attempt to give an indication of overall image quality, but only how good the signal is, in relation to the whole sensor area. This is something that is difficult for the photographer to test themselves, but it gives an indication of how good the tonal & colour information is in a small print (small relative to the resolving capabilities of the camera). Most images are used in small sizes (web, magazine print, small inkjet prints etc.) and this is why the DXO sensor tests are important (not the overall score, but the individual plots of the metrics).</p>

<p>It isn't just about small prints, either. When we look at a large print on a wall, typically we look at the whole image from a distance. How smooth are the tones, how rich the colours, how muddy the shadows? These questions are what the DXO sensor metrics can shed light into. A lot of the time we are looking at the big print as a whole image, to take in the message of the print and look at its overall composition, instead of looking at it up close with 10cm distance to print, inspecting details. The latter is also sometimes done, I guess, but the question of detail is already answered by other sites.</p>

<p>When it comes to testing the whole imaging system, including the lens, then I think it's simply something that we can test ourselves more easily by simply taking pictures with the setup and evaluating them visually, see what we like and don't like (that's the ultimate test really, using it in the real world). There are also plenty of websites which test lens+camera combinations for resolution etc. but all of them have some limitations in that it's difficult to find quantities which thoroughly characterize the quality of the imaging system. Or there would be so many numbers that readers would be lost as to what is important and what is not. </p>

<blockquote>

<p>What is your opinion about the scores?</p>

</blockquote>

<p>They're <em>utterly</em> irrelevant to the Real World experience.</p>

 

<p>The scores are interesting, and prone to cause me gear envy. I've used Canon 20D's, 40D's, 5D's, and 5DII's. Their scores seem to range all over the center of the DXO chart: 5DII #17, 20D #86. But in real life looking at large prints (16"x24"+) experienced photographers seem to have a very hard time picking which of my photos were made with which camera. </p>

<p>Dxo rates the consumer Nikon D5100 better than some medium format backs....I've leave it up to everyone to decided if they would want the back, or the Nikon D5100 for making prints. DxO is a complete joke.</p>

<p>I guess my ancient Nikon bodies (D80, D60, D50 and D40) are a bunch of junk; how much can I get for the group?</p>

<p>It is easy to demonstrate that DxO's DR estimates have no connection with reality by simply testing a few bodies with a Stouffer transmission step wedge. I haven't performed tests to specifically debunk their other metrics, but I do find it laughable that they rate several small format DSLRs as superior to medium format digital backs.</p>

<p>Sorry, but I consider DxO to be a joke, and give no consideration to their claims. The two sites I give the most consideration to for technical evaluations are DPReview and Imaging Resource. I can replicate their tests, and the test results are very much in line with real world experience.</p>

<blockquote>

<p>There is <em><strong>published controvers</strong></em>y regarding DxOMark's Sensor Overall Score<br>

<br /></p>

</blockquote>

<p>Published where? Forum ramblings or a reasoned analysis? </p>

<p>Forum ramblings and reasoned analysis. The above respected commenters now included.</p>

<p>The only article with any kind of authoritative analysis of the work DxO publish I have seen is <a href="http://www.luminous-landscape.com/essays/dxomark_sensor_for_benchmarking_cameras.shtml">here</a>, it seems well written and even though I am a qualified engineer, it is outside my realm of expertise, having said that it is written in such a way that I trust the points Mr van den Hamer makes. I don't believe his article is controversial, but it does put into perspective the results DxO publish. Do we really need performance figures down to 1/15 of a stop? And how much should that influence a purchasing decision?</p>

<p>Considering the sensor performance of practically every camera made in the last few years, the constant improvements, and the comparatively minor increases per generation being achieved, sensor performance is barely in the top ten list of purchasing priorities when entertaining a new body purchase for me. To give any credence, or importance, to such a finely focused analysis of one aspect of a cameras performance seems a strange thing to do, particularly when there are question marks about the testing methodology and the way results are arrived at.</p>

<p><em>To give any credence, or importance, to such a finely focused analysis of one aspect of a cameras performance seems a strange thing to do, particularly when there are question marks about the testing methodology and the way results are arrived at.</em></p>

<p>On the contrary, I think DXO gives extremely valuable information that no other test site does. For example, what if I need to know which camera will give the most headroom for doing single-image "HDR" for landscapes where water or trees move between shots preventing multiple images from being used? I look at the base ISO DR at DXO. What if I want to know which gives the most headroom for exposure and white balance corrections in night club lighting conditions? I look at the DR and SNR for the required ISO settings at DXO. This information is very important to me and I need to have it without having to buy each camera to test it myself.</p>

<p><em>Dxo rates the consumer Nikon D5100 better than some medium format backs....I've leave it up to everyone to decided if they would want the back, or the Nikon D5100 for making prints. DxO is a complete joke.</em></p>

<p>Unfortunately there is a problem with doing the DXO raw data analysis for medium format cameras (pattern noise). They admit it but for some reason publish the scores (with a caution message) anyway. :-( To me it's regrettable as the scores they give for these cameras are useless. But this doesn't prevent me from benefiting from their data on small-sensor cameras, which are excellent and I find them to be quite accurate in the case of cameras that I have owned. If I were to buy a MF system I would have to do my own testing.</p>

<p><em>The two sites I give the most consideration to for technical evaluations are DPReview and Imaging Resource.</em></p>

<p>Alas, both sites focus their studio scene images all over the place. Their quality control regarding focusing is atrocious. Photozone is much, much better in this respect.</p>

<blockquote>

<p>"For example, what if I need to know which camera will give the most headroom for doing single-image "HDR" for landscapes where water or trees move between shots preventing multiple images from being used? "</p>

</blockquote>

<p>DxO results won't show you that, they will show you the information that comes directly off the sensor, but not the RAW file you can actually download from your camera and manipulate, real world image files can show differences that DxO do not test and are not reflected in their results.</p>

<blockquote>

<p ><a name="00aBKG"></a><a href="../photodb/user?user_id=1774085">Keith Reeder</a>, Mar 24, 2012; 08:55 a.m.</p>

</blockquote>

 

<blockquote>

<p><em>What is your opinion about the scores?</em><br>

They're <em>utterly</em> irrelevant to the Real World experience.</p>

</blockquote>

<p>I agree. If you need further proof, consider this: the 'overall results' are not in relation to other cameras that are available, nor are they in relation to what is possible at the time of testing.</p>

<p>For instance, the G11 and G12 both received a 66% score - barely a passing score, to an American high school student. Fair enough - it's the same sensor. But at the G11's release, there was nothing on the market that had that quality sensor, aside from Sigma's DP1, which had a prime lens. The G11 was an A+ camera, with no real competition on release. If the G11 had the best sensor available for a small camera with a zoom - which it did - then why the D grade? The G12 is competing with several other cameras, using now-older technology, that can do the same thing with more zoom or using a slightly smaller form factor. In relation to other cameras, the G12 is much worse than the G11 was, which makes the D grade more appropriate.</p>

<p>And if we're grading on a curve, and the numbers just keep going up for new models (as they appear to), what do we do when something hits 100%? Do we completely rework the grading system then? Or will we be faced with a list of cameras that are all 95-100%? The latter seems to be where DxO is going, but it gives a very false impression of quality and capability.</p>

 

<p><em>what do we do when something hits 100%? </em></p>

<p>If I understand the rating system correctly, that would be a sensor that records every photon and adds no noise to the signal. No real-world sensor can ever do that (they never record all photons and they always add some noise), so DXO doesn't have to rework anything. </p>

<p><em>DxO results won't show you that, they will show you the information that comes directly off the sensor, but not the RAW file you can actually download from your camera and manipulate, real world image files can show differences that DxO do not test and are not reflected in their results.</em><br>

They do, to the degree that I need to know. Results of subjective analysis of real world images changes from day to day with a human observer. Without actually buying all the cameras you want to compare and shooting them side by side in various conditions it's impossible to get reliable estimates of the comparative performance. I don't care to carry around multiple cameras for testing purposes. When I'm in the field I shoot real images and do not waste time testing. I get the comparative data from competent sites and this saves me time and money. If you cannot interpret the data given by dxomark and predict real world results based on it then it is your limitation, but please do not tell others what their limitations in ability are.</p>

<p>Zack,</p>

<p>That just shows a complete lack of understanding of what they are doing and the figures they label a camera with. Besides, the G11 and G12 both received a 47 score, however, a score of 66 is not 66%, it is 10 (or 2/3 stop) better than a sensor that scored 56. There is no pass or fail and 100 is not a maximum achievable, as sensors get better you should expect all scores to go up. </p>

<p>Compare them to the same generation Nikon P6000 with a score of 35 and you should expect the Canon G11 RAW files to be around 2/3 stop better.</p>

<p>Having said all that, unless your usage fits in with their specific formula of scoring the three categories and you can ignore all other camera and system functionality for a possible 1/3 stop difference then the scores might have some merit. </p>

<p>Ilkka,</p>

 

<blockquote>

<p>If I understand the rating system correctly, that would be a sensor that records every photon and adds no noise to the signal. No real-world sensor can ever do that (they never record all photons and they always add some noise), so DXO doesn't have to rework anything.</p>

</blockquote>

<p>Your understanding is not correct. The scale is open ended as the linked quote below says.</p>

<blockquote>

<p><a href="http://www.dxomark.com/index.php/About/Sensor-scores"><strong>Sensor Overall Score</strong><strong> is open and it is not a percentage</strong>.</a> This score has been computed so that the current set of cameras, from low-end DSCs up to professional DSLRs and medium-format cameras, show results within a range from 0 to 100. However, new technologies may well lead to higher performance models.</p>

</blockquote>

<p> As for this comment.</p>

<blockquote>

<p>If you cannot interpret the data given by dxomark and predict real world results based on it then it is your limitation, but please do not tell others what their limitations in ability are.</p>

</blockquote>

<p>As my understanding of their ratings seems to be more accurate than yours I would not presume I am too limited. Anybody that doesn't understand the scores can't possibly be looking at anything other than a number. High number= good, lower number= not so good, well it doesn't work like that for a multitude of complicated reasons, I understand some of those reasons so find the DxO scores to be of extremely limited value.</p>

<p><em>On the contrary, I think DXO gives extremely valuable information that no other test site does. For example, what if I need to know which camera will give the most headroom for doing single-image "HDR" for landscapes where water or trees move between shots preventing multiple images from being used?</em></p>

<p>You're kidding yourself if you think they are showing you this. Their DR results have no relation to real shots and are literally worthless.</p>

<p>Most funny thing about IR and dpreview is, they don't even use same exposure for comparing different cameras. If for the same t stop, same shutter speed and same scene lighting, one camera needs ISO 6400 and another camera needs ISO 12,800 , they just change the exposure so that both camera use the same ISO value. How can that be a fair comparison? At least dxo knows these things and takes care of.<br>

But dxo's single number "rating" is very misleading, you have to look at the graphs to really understand and compare cameras. And you also have to understand that they do not have any weightage on resolution (where medium format cameras specifically shine).</p>