DxOMark's Sensor Overall Score

[zack_zoll]

<p>Scott, you're correct - I misread the posts about the G11/G12. Frankly, it was just too long a list for me :) So the ratings are 47, and not 66. However, that changes nothing.</p>

<p>DxO says that the rating is not a percentage. That's not true at all. They rate them 1-100, with 100 being a perfect, flawless score. 100 is the maximum allowable score. DxO can say whatever they like about their ratings system, but I think you'll find that that is the very definition of a percentage. Pretending that the definition, or the math, is different than what it truly is shouldn't fool anyone.</p>

<p>Despite my misreading of the chart, I understand the concept that a higher rating equals better ISO/IQ etc. But let me ask you this: as sensor technology improves, ISO and IQ performance increases. This is a given. Based on DxO's grading system, all ratings will go up over time. Say three years from now we get a pro model that scores a 99%, since a 100% is impossible. What happens then? Does the Rebel with the same sensor and a worse processor get a grade of 99% as well? Does the grading system mean anything by then?</p>

<p>The problem with limiting the scores to 100 (and yes, this means it is a percentage) is that once scores climb to a certain extent, they become meaningless. Since we don't know what the maximum quality that this technology will eventually be capable of is, any scale that we create without that knowledge is arbitrary. For all we know, sensors may one day be capable of 4 million ISO with no visible noise. If a million ISO gets a score of 99%, how do we rate 4 million ISO? We have to give it the same score of 99%, even though its two stops better. The only way to give them meaning is to rate them based on what is currently possible.</p>

<p>The fact is that DxO grades cameras based on a system that, because of our lack of knowledge about what the future will hold, is destined to eventually become worthless. It's just a matter of when - not if. Until then, it is dodgy at best.</p>

[philip_wilson]

I am surprised how emotional the reaction to DxO are. I find the scores to be mainly useless as the weight several test

results to create this score. Their tests and weights do not necessarily reflect my own needs. However, that said they do

produce some data in there reports that some may find useful. From my own experience their ISO sensitivity

measurement is consistent with my Leica M8 performance and their Signal to noise ratio tests do show where the sensor

tends to fall off in performance. Obviously these are not reasons for camera purchase decisions by themselves but they

do provide information which we can use or ignore at our discretion. While I have never read their scoring criteria I have

always assumed that MF backs and Leica do poorly due to a strong emphasis on high ISO performance. Indeed their

rating system seems a little over simplistic as it suggests portrait shooter only need Colour depth, landscape shooters

dynamic range and Sports shooters high ISO. All of these things are rather over generalized - I shoot B&W portraits in

low light for example or landscapes at dawn where colour depth and higher ISO are useful

[scott_ferris]

<p>Zack,</p>

<p>Sorry you are wrong again.</p>

<blockquote>

<p>DxO says that the rating is not a percentage. That's not true at all. They rate them 1-100, with 100 being a perfect, flawless score. 100 is the maximum allowable score. </p>

</blockquote>

<p>100 is not the maximum score, the number is open ended, cameras will score over 100. There is nothing to stop a camera getting a 140 score it just has to perform 3 stops better than the D800 with a 95. The scoring is linear not an ever steepening curve, there will be no 99.8-99.9 mess, it will just go over 100. No change of the rules of maths, no recalculations for everything, just a continuation of exactly what they have done up to now.</p>

<p>Quite how Nikon have managed to make the D800 score better than the D4 just illustrates that the weighting in the final calculation is skewed. Sure they are both supremely capable cameras, but seriously, two brand new sensors from the same company with the same R&D, with the result that a $3000 camera beats the off sensor signal of a $6,000 camera.</p>

[zack_zoll]

<p>Scott, from your own writing it states that cameras can currently score <em>up to </em>100. This means that based on the current grading system, 100 is perfect, as it is the maximum allowable grade within the current stated system. Unless you know something personally going on at DxO that runs counter to their stated grading system, there should be no argument there.</p>

<p>Now, the entire purpose of a grading system is not to make a list - it is to let people know how well something performs in relation to how well it <em>could</em> perform. In this case, DxO has picked an arbitrary number (100), but it could just as well be 200, or 347. The number given to a camera sensor indicates how well it does on certain things vs. the maximum, which is 'captures every photon perfectly.' I realize I am oversimplifying.</p>

<p>DxO's claimed purpose is to let users know how well a camera performs by grading. By changing the grading scale, they are no longer letting people know the same information. It is as much a grade as a college professor that decides that since he has too many students getting As, all tests will now be out of 140, rather than 100.</p>

<p>I'm not saying that their system cannot change. I am saying that what a <em>grade</em> is: an idication of score in relation to maximum possible score - is no longer shown if the scale is changed, unless <em>all</em> models are re-evaluated according to the new scale. Which won't happen. As an arbitrary number, all we know is that newer models score higher; which, frankly, should be obvious to anyone that has bought multiple cameras.</p>

<p>DxO is producing numbers that show light sensitivity. They are not grading cameras, nor are they doing anything else that the words that they use would imply. They have chosen incorrect terms for what they are doing, and it leads many people to believe that these numbers mean something different than what they mean.</p>

[scott_ferris]

<p>Zack,</p>

<p>What is it about this comment I have already posted direct from DxO themselves (with a link) that you don't understand?</p>

<blockquote>

<blockquote>

<p><a href="http://www.dxomark.com/index.php/About/Sensor-scores" rel="nofollow" target="_blank">Sensor Overall Score<em><strong> is open</strong></em> and<strong> <em>it is not a percentage</em><em>.</em></strong></a> This score has been computed so that the <strong>current set of cameras</strong>, from low-end DSCs up to professional DSLRs and medium-format cameras, show results <strong>within a range from 0 to 100</strong>. However, <strong>new technologies may well lead to higher performance</strong> models.</p>

</blockquote>

</blockquote>

<p>You will notice it does not say up to 100 anywhere, just that current technology falls between 0 and 100, but newer tech will take it higher.</p>

<p>As for this assumption</p>

<blockquote>Now, the entire purpose of a grading system is not to make a list - it is to let people know how well something performs in relation to how well it <em>could</em> perform.</blockquote>

<p>No it isn't. In this instance the DxO score is only meant to illustrate performance against others, there is no theoretical perfect score.</p>

<blockquote>The number given to a camera sensor indicates how well it does on certain things vs. the maximum, which is 'captures every photon perfectly.' I realize I am oversimplifying.</blockquote>

<p>No, again, that is not what the DxO number is, there is no maximum. If you drop your idea of 100 being a perfect score, which is wrong, and understand the scoring is open, which it is, then you will realize nothing changes when sensors start scoring over 100. The system does not need to change and nothing needs to be re-evaluated, all scores stay where they currently are. A score of 100 does not mean it is perfect, it will mean a sensor is 1 stop "better" than a sensor that scored 85, a sensor that scores 115 will be one stop "better" than one that scored 100 and two stops "better" than one that scored 85, it is <strong>an open linear scale</strong>.</p>

[keith reeder]

<p>I find it interesting that some of the most fervent advocates on this thread of the accuracy and relevance of DxOMark data and information flatly refuse to accept the premise they document <a href="http://www.dxomark.com/index.php/Publications/DxOMark-Insights/More-pixels-offset-noise!">here</a>.</p>

<p>Wonder why? Surely if DxOMark is right about the relevance of the data they pull off a sensor to the end result, they're going to be right about the relationship between pixel count and noise at the image level too? </p>

<p> </p>

[scott_ferris]

<p>Kieth,</p>

<p>Who is a <em>"most fervent advocate"</em> for <em>"the accuracy and relevance of DxOMark data"</em> on this thread other than Ilkka?</p>

[zack_zoll]

<blockquote>

<p ><a name="00aBbl"></a><a href="../photodb/user?user_id=657840">Scott Ferris</a> <a href="../member-status-icons"><img title="Frequent poster" src="../v3graphics/member-status-icons/3rolls.gif" alt="" /></a>, Mar 25, 2012; 12:26 p.m.</p>

</blockquote>

 

<blockquote>

<p>Zack,<br>

What is it about this comment I have already posted direct from DxO themselves (with a link) that you don't understand?</p>

<blockquote>

<p><em><a href="http://www.dxomark.com/index.php/About/Sensor-scores" rel="nofollow" target="_blank">Sensor Overall Score<strong> is open</strong> and<strong> it is not a percentage.</strong></a> This score has been computed so that the <strong>current set of cameras</strong>, from low-end DSCs up to professional DSLRs and medium-format cameras, show results <strong>within a range from 0 to 100</strong>. However, <strong>new technologies may well lead to higher performance</strong> models.</em></p>

</blockquote>

<p>You will notice it does not say up to 100 anywhere, just that current technology falls between 0 and 100, but newer tech will take it higher.</p>

</blockquote>

<p>I understand the concept. I don't understand how you seem to have a disconnect between the words that they use and what you think they mean. In the words of Inigo Montoya, "I do not think that word means what you think it means."</p>

<p>Before you remind me AGAIN that the scores can change, let me point out that, unless I'm reading it wrong, the current top score is 95%. If DxO were attempting to create a 'living' scoreboard, then they would claim instead that 'all scores are 0-95.' Because they are. Since they state that "all cameras are 0-100" even though NO cameras are 100, this means that <em>someone</em> at DxO perceives 100 to be the top of the chart. The top of the chart that theoretically has no top.</p>

<p>Clearly the number 100 is relevant to DxO, or else they wouldn't say that all models fall between 0 and 100 when they do not. DxO says that the scale is open-ended, but that current models fall between 0 and 100. Their charts max out at 100. This means that *right now* the score is 0 to 100. It cannot be 0 to 100 AND be open-ended. Those maths just plain don't work - I don't care how you try to explain it. Something has a top end, or it doesn't. This isn't a Schrodinger's Cat thought experiment where we can imagine that it exists in two states at once. The scale is NOT open-ended. It is, by the very definiton of the words, a closed scale that they are planning on changing.</p>

<p>What keeps all this from being a rant about semantics is this: regardless of the claims that DxO makes, the words that they choose to use imply so strongly as to practically <em>state with conviction</em> that their results mean something that they do not. Their results, in reality, are describing a variety of ways in which sensitivity to light improves with sensor design. The 'true' results will show each progressive model improving at the exact rate at which the technology improves, meaning that the actual results are really just an engineer's timeline.</p>

<p>The words that they have chosen to use, and the fact that they continue to use a 0-100 scale despite obvious confusion and years of no models being anywhere near 100%, lead the reader to believe - wrongly so - that they are producing a relevant 'grade.' In order to have a proper grade, you need a top limit. Regardless of what you claim, they currently have one. It is out of a maximum 100, just like a grade. All the indications on their page point to it being a grade, aside from a short little sentence in the 'about' section that says, 'This is not a grade.'</p>

<p>The fact is that regardless of how you spin it, the language and the system that DxO uses is misleading and self-contradictory to the point of total worthlessness to the reader. I've pointed out what it needs to be a grade and what it needs to be open-ended - the system currently resides in some nebulous area in the middle, where it says one and does the other.</p>

<p>I hope this post explains why the 'grading system' is extremely flawed. If not ... well if we couldn't see eye-to-eye by now, another post from me isn't going to help.</p>

 

[scott_ferris]

<blockquote>

<p>"I understand the concept."</p>

</blockquote>

<p>Clearly you don't, I am sorry I have failed to explain in a way you comprehend, DxO scores are of extremely limited value, but not because of your understanding/misunderstanding of them but clearly you are madder at me than you are interested in learning, so be it.</p>

<p>We will just have to wait until the first cameras go over 100 and we find the sky doesn't fall on our heads. By DxO figures it will take a camera that performs 1/3 stop "better" than the D800 to achieve 100, anything over 1/3 stop "better" than a D800 and we go into numbers higher than 100. I wonder if the 5D MkIII or the 1Dx will get there?</p>

[museebfoto]

<p dir="LTR">I have used</p>

<p dir="LTR">Canon ( 450D, 20D, 40D, 5D, 7D )</p>

<p dir="LTR">Konica Minolta 7D</p>

<p dir="LTR">Nikon ( D50, D70s )</p>

<p dir="LTR">Panasonic FX3</p>

<p dir="LTR">Sony NEX C3</p>

<p dir="LTR">From my point of view the Sensor Overall Score given by DxO is a very convincing.</p>

<p dir="LTR">One thing which has puzzled me is the very close score for Canon EF 24-105mm f/4L IS USM and the score for Canon EF 28-105mm f/3.5-4.5 USM, ( in which the degree of zoom close to some ).</p>

<p dir="LTR">The fist cost about $1250 while the second cost only $250.</p>

<p dir="LTR">( I have Only EF 28-105mm f/3.5-4.5 USM and it is really a good lens )</p>

 

<p dir="LTR"> </p>

[abbas_haider]

Hi all, I have seen DXO scorning I can see it is market advertising and not real evaluation or they have gone for points

that suite them. As if I am telling that T2i is better than 1D mk4 because it is lighter, cheaper, yes these points are

right but do actually it is, no so the photographer is the judge to DXO marking and not the other way around. As our

friends mention some entry level cams are marked better than MF in term of sensor preference which is totally awful

measure.

[museebfoto]

<p>DXO Sensor Overall Score dose not compare prices or weights, it goes to a more important thing like dynamic range, signal to noise ratio and color depth. We must not wonder that the modern cameras even if it is from lower-level excel the performance of an old camera which is in a higher level. If the case remains as it is, it means there is no technical progress and principally there is no digital photography, and if I have money I will buy Nikon D800 today and not tomorrow.</p>

[Brad_]

>>> What is your opinion about the scores?

 

Eh... I just don't have an opinion on lists like that.

 

With respect to my opinion about creating compelling photographs in general, and how "sensor scores" factor in... Putting energy into seeing, seeking nice light, imagining the

possibilities, and thinking about composition with the goal of creating an image evoking

emotional pull, hugely trumps differences in sensor scores.

 

And it's free - meaning you're not constantly spending $$$ to own the absolute best and arguing about

minutia...

[eric_duncan3]

<p>Lies, damn lies, and statistics!<br>

In short, I subscribe to the point-of-view of those who argue that DxO comparative scores between cameras are based upon arbitrary manipulation of data, introducing substantial artifacts into the comparisons, therefore rendering them of little validity.<br>

Objective, repeatable measurements form the basis of scientific conclusion. Arbitrarily weighting criteria in such a way that may vary from photographer to photographer forms the basis of a subjective opinion blog.</p>

[dan_south]

<p>I'm sure that the DxO test is regimental, repeatable, and accurate in what it measures. However, the score doesn't predict how well a camera will work for a particular application.</p>

<p>I spent two years shooting a D700 (currently ranked 11th) when I purchased my 5D Mark II (currently ranked 16th). From that day forward, I rarely used the D700 and I eventually ended up selling it. The 5D Mark II, inferior on the DxO scale, was clearly and demonstrably SUPERIOR for the requirements of my shooting.</p>

<p>The D700 is a nice camera, and it does some things better than the 5D2. But the 5D2 does some things better than the D700. The dimensions in which the 5D2 is superior might not be weighted heavily on the DxO scale, but they are more important to me than what DxO measures.</p>

<p>It's important to gather as much information as possible AND consider how that information will be of benefit to YOU. The DxO Mark ranking is only one piece of information. Despite the fact that it's very meticulously calculated, it does not reflect what's of importance to me in my image making projects.</p>

<p>I keep waiting for them to come up with a DxO Dan score. I don't know who this Mark fellow is, but he has different priorities than I do. ;-)</p>

<p> </p>