Film vs Digital - Dynamic Range

<p>Rishi.... basically you are reducing real exposure (Tv, Av) by 4 times, and increasing effective exposure (ISO) by 4 times. You shouldn't have any more or less highlights blown. It should be the same amount. What will change as Helen (I think) mentioned is that the DR will reduce as ISO 400 will have the same saturation, but will have 4 times the noise.</p>

<blockquote>

<p>Couple of points though: DR won't be infinity. No pixel will register 0 signal. Noise will ensure it has some value. In my 5D the darkframe noise is around level 128 out of 4095. The other point is that your point still doesn't change the scenario that Helen and I are on about. You can't make an 8 bit ADC display more than 8 stops of scene DR. A 10 stop printer could make a print with a physical DR of 10 stops, but it would still only translate 8 stops of scene DR.</p>

 

</blockquote>

<p>Agree with your first point that DR won't be infinity. Of course not.</p>

<p>Disagree with your second. I can make pixels 0 - just write a simple piece of code that goes changes every pixel value in the file that is less than 128 to 0. Really easy to do - just a little C programming needed. Or you can simply edit a raw file in emacs in hexl mode. Of course this assumes that you know which pixels to reset to 0.</p>

<p>Now what happens to that file when printed by a printer with 10 stops of DR? 0 will map to its blackest and FFF will map to its white. What did you get? 10 stops of DR. This will happen even with an 8-bit ADC - something in the printer will map a range of 8-bit, or 14-bit values to whatever the bit-depth of the DAC is; then it will place 0 at black and its own max value at white. If the file contains only nonzero values, then the printer simply won't put out its blackest black.</p>

<p>And I'm surprised that you're only getting 128 - 4095 from your 5D: this is a DR of only 5 stops; and it is the max DR your camera can deliver too, given that darkframe noise is at 128! And I'm quite certain you are capturing scenes with far greater luminance range than 5 stops, and still getting deep blacks and bright whites. If you take a lightmeter and measure the blacks and whites from prints made with your own camera, you'll probably find that the print DR is more than 5 stops. You shouldn't be finding more than 32 different gray tones, though; or more than 32k colors or so.</p>

<p>And if what you have been arguing about is true, you should never get a pure black or pure white from your camera (if you don't postprocess in Photoshop). Just a dull light gray for white and a dark gray for black. Is this what you see?</p>

<p>Dude, you got a 5-bit camera for all intents and purposes! No wonder you see banding and other problems in images. Is your 5D still under warranty?</p>

<blockquote>

<p>Rishi.... basically you are reducing real exposure (Tv, Av) by 4 times, and increasing effective exposure (ISO) by 4 times. You shouldn't have any more or less highlights blown.</p>

</blockquote>

<p>No... because by reducing the 'real exposure' by 4 times, you are much <strong>less likely</strong> to reach the well capacity of any given photocell. So with a 'smart amp' applying an extremely non-linear gain (i.e. hyperbolic with a significant shoulder on the top end), you'll increase noise in shadows, sure, but you won't be blowing out highlights (or you'll do so to a lesser degree) because there were less photocells that reached full charge.</p>

<p>Essentially, this is what Highlight Tone Priority is, correct? Well, why isn't it just <em>normal behavior</em> for the camera at higher ISOs? Why do the amps blow out pixels that weren't blown to begin with??</p>

<p>Like I asked before, isn't that just stupid amplifier/algorithmic design?</p>

<p>Vijay? Bernie? I didn't understand Helen's response b/c it seemed sarcastic & rhetorical and, well, yeah I just didn't get it.</p>

<p>Anyway, am I understanding this correctly?<br>

Rishi</p>

<blockquote>

<p>Now what happens to that file when printed by a printer with 10 stops of DR? 0 will map to its blackest and FFF will map to its white. What did you get? 10 stops of DR. This will happen even with an 8-bit ADC - something in the printer will map a range of 8-bit, or 14-bit values to whatever the bit-depth of the DAC is; then it will place 0 at black and its own max value at white. If the file contains only nonzero values, then the printer simply won't put out its blackest black.</p>

</blockquote>

<p>Just like when you take an 8-bit file that has a bunch of pixels of value 254, 254, 254, and then convert the file to 16-bit, you get 32639,32639,32639. You basically scale the values. If you actually had a device capable of displaying 16-bit output, then you might end up seeing banding starting from an 8-bit source.</p>

<p>Of course, none of this changes the dynamic range of the scene that was originally recorded.</p>

<p>Not sure why you two (three?) are arguing... you all seem to be in agreement.<br>

Rishi</p>

<p>And here's another example:</p>

 

<ol>

<li>Suppose I have a printer that can print 10 stops of reflectance range, such that the brightest white reflects 1024 times more light than the blackest black. Lets say that the printer outputs its whitest white when it sees a pixel with code 0, and its whitest white for a pixel with code 3FF (1023). Agree/disagree? </li>

<li>Now suppose I write a little program to write some values to a postscript file, which I then send along to this printer - the values I write are 0 and 3FF. It will faithfully print pure black and pure white with a reflectance range of 10 stops. Agree/disagree? </li>

<li>Now suppose I write a program to generate the postscript file with either 0 or 3FF based on a single bit input that I get from a device, say a 1-bit DAC. Agree/disagree? </li>

<li>Now suppose that I feed this DAC from the output of a 1 bit ADC whose input voltage range is 0 - 2000 mV. This ADC has a threshold at 1000 mV so that 0 - 1000 mV is encoded as 0, and 1000 mV - 2000 mV is encoded as 1. Agree/disagree? </li>

<li>Now suppose that this ADC is being fed by a photodiode that generates 2000 mV in direct sunlight, and goes down to its noise floor of 1 mV peak as the light decreases. This is approximately an 11 stop dynamic range of the sensor. Agree/disagree? </li>

<li>What happens when I take 20 million of these photodiodes and put them on a chip? I get a sensor that has extreme contrast - it has blacks and whites, but no grays. Agree/disagree? </li>

<li>If I take a picture of a scene with luminance range of say 15 stops with this sensor and print it using this system that I described, what will I get? A print with a reflectance range of 10 stops. Agree/disagree? </li>

<li>But my ADC and DAC in the path were only one bit wide. Agree/disagree?</li>

</ol>

<blockquote>

<p>I didn't understand Helen's response b/c it seemed sarcastic & rhetorical and, well, yeah I just didn't get it.</p>

</blockquote>

<p>Helen has this really sardonic sense of humor that makes her sound that way - I don't think she meant offense. And yeah, sometimes her sense of humor eludes me as well.</p>

<blockquote>

<p>Not sure why you two (three?) are arguing... you all seem to be in agreement.</p>

</blockquote>

<p>I don't know; and I don't even know if Bernie or Helen agree with me. I just gave an example of an entire system - scene to print working backwards using a single bit ADC/DAC in the path. Heck, CD players actually have just a single bit ADC/DAC in reality. The modulation is different, but that doesn't matter.</p>

<p>Maybe that will serve to make Bernie or Helen understand what I'm saying, but we don't have seem to have explicit agreement yet.</p>

<p>And what do you think of Bernie's 5D? Defective?</p>

<blockquote>

<p>And what do you think of Bernie's 5D? Defective?</p>

</blockquote>

<p>Not telling until you answer my question about bad amp/algorithm design & highlight tone priority yada yada -- I just want to make sure I was understanding all that correctly! Just read the post before my last post :)</p>

<p>Sorry to be petty, but this is really bothering me.<br>

Cheers,<br /> Rishi</p>

<p>Man, I can't keep up. My internet connection has been throttled again (thanks to arguing with you guys all day everyday ;) ), and my computer is crapping itself in the ridiculous heat we are experiencing here. Here's a response to Rishi I composed earlier. Will try to get to all points raised by Vijay as well, but it's hard at the moment.</p>

 

<blockquote>

<p>Rishi, with 'smart amp' technology, it might be ok, but i think they only have 'dumb amp' technology now ;)</p>

<p>Basically what happens is this: (Exposure/4)*4. That is, the net result is the same (of course I'm sure that is a pretty simplified version of what really happens).</p>

<p>Highlight tone priority is just 1EV underexposure and then a custom 'fill' curve to boost the shadows. ie. You think you are shooting at iso 200 and it sets Tv and Av as such, but in reality it is using iso 100.</p>

</blockquote>

<p>Thanks Bernie.</p>

<p>But then I would argue that there should be a mode where WHENEVER you shoot at 'non-native' ISOs [hopefully you understand what I mean by that], one should ALWAYS apply a non-linear gain (like fill-curve), to preserve highlight detail.</p>

<p>Sure, this means that images would look different exposed at 'non-native' vs 'native' ISO... but at least it's yield a more useful dynamic range -- if you're already sacrificing noise to gain some speed, may as well put it to some better use and recover some potentially blown-out highlights!</p>

<p>Don't ya think?</p>

<p>Funny, someone probably had this exact train of thought I've been outlining for the last few hours before developing the idea of 'highlight tone priority' :)<br>

-Rishi</p>

<blockquote>

<p>Disagree with your second. I can make pixels 0 - just write a simple piece of code that goes changes every pixel value in the file that is less than 128 to 0.</p>

 

</blockquote>

<p>Yes, but that isn't what happens. If only you'd look at some linear raw data and then we wouldn't have to waste time on this.</p>

<p>The noise floor gets set to zero in the raw converter. Also the white point gets set, so photodiode saturation equates to digital n^2-1.</p>

 

<blockquote>

<p>Now what happens to that file when printed by a printer with 10 stops of DR? 0 will map to its blackest and FFF will map to its white. What did you get? 10 stops of DR. This will happen even with an 8-bit ADC - something in the printer will map a range of 8-bit, or 14-bit values to whatever the bit-depth of the DAC is; then it will place 0 at black and its own max value at white. If the file contains only nonzero values, then the printer simply won't put out its blackest black.</p>

 

</blockquote>

<p>I agree with all this. Why do you keep bringing it up?</p>

<blockquote>

<p>And if what you have been arguing about is true, you should never get a pure black or pure white from your camera (if you don't postprocess in Photoshop). Just a dull light gray for white and a dark gray for black. Is this what you see?</p>

 

</blockquote>

<p>As explained, in linear raw yes. But when it goes through the raw converter, the black and white point is set then.</p>

 

<blockquote>

<p>Dude, you got a 5-bit camera for all intents and purposes! No wonder you see banding and other problems in images. Is your 5D still under warranty?</p>

 

</blockquote>

<p>Ha ha. I'll have to have a bit more of a think about this. It certainly sets the black at 127 or so, and it clips at about 3700.</p>

<p> </p>

<blockquote>

<p>The noise floor gets set to zero in the raw converter. Also the white point gets set, so photodiode saturation equates to digital n^2-1.</p>

 

</blockquote>

<p>(You meant 2^n - 1 right? I assume that this was a typo.)</p>

<p>And this isn't an example of mapping some codes to some arbitrary dynamic range? I mean if you assign 127 to 0 and 3700 to say the 16 bit max - 65,535; then either the raw data wasn't linear, or if the raw data was linear, then the generated file (post raw converter) isn't linear. I mean how can you linearly map a range of 128-3700 to a range of 0 to 65,535?</p>

<p>What am I missing?</p>

<p>Vijay, it's not linear after the raw processing. But that's not what we are talking about. We are saying that the ADC is more-or-less linear, and therefore, this limits how many stops can be meaningfully captured. What happens after this is another discussion (which is what you have actually been doing for the last week or so now).</p>

<p>What are you missing? The crux of this debate.</p>

<blockquote>

<p>But then I would argue that there should be a mode where WHENEVER you shoot at 'non-native' ISOs [hopefully you understand what I mean by that], one should ALWAYS apply a non-linear gain (like fill-curve), to preserve highlight detail.</p>

 

</blockquote>

<p>This sounds like a good idea. I'm not sure why this isn't done, but i figure there probably is some electronics issue that makes it not viable. Vijay would be the man to ask about this. By the way, I assume (and i'm pretty sure I've seen it written) highlight tone priority has a 1 stop reduced dynamic range, so there is a trade off with this particular approach.</p>

<p>I've had a similar line of thought as yours Rishi with the issue of raw converters and white balance scaling. The act of white balancing a raw file can cause one or more of the channels to clip where there isn't clipping in the raw data. This leads a lot of non-technical people to think that if it is clipped in the jpg then it must be clipped in the raw. They then proceed to dump on people who shoot raw, and berate them for not getting exposure right in the first place. But the reality is that the default jpg is not always getting access to the full sensor data. What happens is that when white balancing is done the green channel is (always?) set to unity, and the red and blue channels are scaled up in the correct proportion by values greater than 1. This can lead to pixels clipping even though they aren't really clipped on the sensor. DCRAW has a function which allows you to do the scaling with unity the highest value and the other channels scaled by less than 1. This is good, but you can wind up with an image which has dull highlights in the case where the green channel wasn't clipping and the red and/or blue were. When I have time to relearn C, I want to write a procedure that will scale the whitebalance multipliers up just enough to just clip the most saturated channel.</p>

<blockquote>

<p>Vijay, it's not linear after the raw processing.</p>

</blockquote>

<p>OK, but that certainly means that your 5D is capturing only 4.86 stops of light; 29 grayscale tones and around 25k colors. I don't own a 5D; so I can only go by specifications - yours must be defective.</p>

 

<blockquote>

<p>What are you missing? The crux of this debate.</p>

</blockquote>

<p>Then enlighten me, educate me. Simply insisting that I'm ignorant doesn't help.</p>

<p>And just to refocus: I'm saying that if I use a 1-bit ADC, I can only capture two tones; these two tones could be spaced by 20 stops in the scene and 8 stops on the photodetector but when they get reproduced on print or a monitor, they will be separated by whatever the maximum dynamic range of that medium is (say 10 stops).</p>

<p>I've always agreed that all information between the two tones is lost, so it is impossible to know <em>for sure</em> what was in the original scene. As you go to higher bit depths, you get more information; a 4 bit ADC will tell you that there were at least four stops in between; but the black and white on the final medium will still be separated by the same 10 stops.</p>

<p>You go to an 6 bit encoder, you know "for sure" (Helen's detective question) that there were <em>at least </em> 6 stops in the original scene, but the black and white on the final medium - still separated by the same 10 stops.</p>

<p>I get all that. I also get the part that Helen was saying - that if I were to interpret the codes as <em>linear</em> and <em>integral</em> then I would know that the data represented at least six stops; and I could make a printer place the code 111111 at white and 000001 at six stops darker. This is what you were saying as well.</p>

<p>(A side note was that the codes don't have to be linear, or integral or a power of 2 even: I could have a prime number of codes, each code being an arbitrary 128-bit prime number itself, and the system would work the same. This would be difficult to design, and software algorithms would be hellishly difficult to apply but the theory and mechanisms of operation would be the same.)</p>

<p>My point is that as soon as you place the 000000 code, it will extend the range to the final step - if the printer's dynamic range is 10 stops, then the 000000 code and 000001 codes (on the linear, integral system) will be separated by 4 stops, so the full set of 6-bit codes does encode a larger dynamic range than 6 stops.</p>

<p>If you and Helen agree with this last statement, then this debate is over from my end. The other stuff we agree about (and have for a while now).</p>

 

<blockquote>

<p>Then enlighten me, educate me. Simply insisting that I'm ignorant doesn't help.</p>

 

</blockquote>

<p>You're not ignorant. You're blinkered due to too much specialisation and not enough generalisation. Helen and I have elucidated what we are talking about innumerable times. If you do a search for anytime I have used the word 'crux' or 'crucial point', i have usually explained the difference. The first paragraph of my last post to you basically states the difference in our two arguments. Now I thought we had basically agreed that a 4 bit processor can only convert a meaningful 4 stops of scene data at max. And I think i basically agree with your premise that a 10 stop printer could print out a 10 stop paper dynamic range. But what you aren't seeing is that those 10 physical paper stops do not translate to 10 scene stops of DR. You've still only got 4 stops of scene dynamic range in there (although I agree that they have been translated non-linearly, but that isn't the point of our argument).</p>

<p>Now to the 5D issue. This has been a good process we have gone through as it has forced me to sit down and work this sort of thing out. The answer to this 5D question, is essentially something I have been avoiding because I couldn't have been bothered thinking it through (this is true a lot of the times in these DR and resolution threads, because when I comes down to it, I don't actually care about the absolute numbers between the two systems (film and digital). I know that negative film has more DR than digital and hence I just accept it and take photos cognisant of the limited DR). I previously mentioned the affect noise would have on this linear relationship of input -> output, saying that it would be more pronounced in the shadows (ie. less linear). This is the answer. If read noise was 127, at max signal the percentage of noise would be 127/3700 = 3%. At mid signal it would be 127/2000 = 6%. In the shadows it might be 127/256 = 50%. You get the picture. So if we incorporate the noise signal into the output value from the ADC we get something like this:<br>

input : output<br>

stop 12 : 3700 (3573 signal + 127 noise)<br>

-1EV : 1914 (1787 signal + 127 noise)<br>

-2EV : 1020 (893 signal + 127 noise)<br>

-3EV : 574 (447 signal + 127 noise)<br>

-4EV : 350 (223 signal + 127 noise)<br>

-5EV : 239 (112 signal + 127 noise)<br>

-6EV : 183 (56 signal + 127 noise)<br>

-7EV : 155 (28 signal + 127 noise)<br>

-8EV : 141 (14 signal + 127 noise)<br>

-9EV : 134 (7 signal + 127 noise)<br>

-10EV : 130 (3 signal + 127 noise)<br>

-11EV : 129 (2 signal + 127 noise)<br>

-12EV : 128 (1 signal + 127 noise)</p>

<p>Now I know this is highly simplified and there are other types of noise present other than read noise, but I am guessing read noise is the major type of noise. So you can see, 12 stops of data are recorded, but clearly the lower stops are awash with noise (this isn't something we didn't already know). And this goes back to my point about <em>usable </em> dynamic range. Clearly some of the lower stops it will be impossible to distinguish signal from noise. They technically have been captured, but they are unusable.</p>

<p>I'll try and do a 12+ stop exposure test on the 5D today and see exactly what the linear raw shows. I'll post the results when I get the time.</p>

<p>OK, here's what I got. Keep in mind this isn't the full story (particularly at the brighter stops) for the following reasons: <br /> <br /> 1. It's really hard to set your top stop. Unless the scene is consistently uniform in brightness, you get a wide histogram. For example, what I chose as my top stop has a range from full saturation (3692) down to about 1150. Now, the VAST majority were fully saturated, so it was good enough for me. <br /> <br /> 2. The next stop down likewise had a wide histogram and had a strong peak in one particular range of levels. It was this peak that I chose to judge exposure changes from as it is the only way I could get an accurate level number reading out of the software I was using. From this stop down to the lower stops, the comparison should be pretty accurate as this peak was represented in all stops. But the comparison of the peak in this stop and the levels in the top stop is not possible as the peak wasn't there in the top stop (coz it was fully saturated). The best way to do this would be to compare the most saturated pixel in each stop. But as I mentioned, the software I was using wouldn't allow me to do this.<br /> <br /> 3. At the brightest stop, I was forced to use a Tv of 20 seconds to grossly overexpose. This length of shot on digital introduces more noise into the mix. I think this is due to rising temperature on the sensor and electronics, but Vijay could probably explain this better.<br /> <br /> <br /> Keeping these three points in mind, here are my results:<br /> <br /> stop => integer level<br /> <br /> 12 => 3692<br /> <br /> 11 => 1287<br /> <br /> 10 => 669<br /> <br /> 9 => 399<br /> <br /> 8 => 263<br /> <br /> 7 => 196<br /> <br /> 6 => 162<br /> <br /> 5 => 145<br /> <br /> 4 => 136<br /> <br /> 3 => 136<br /> <br /> 2 => 132<br /> <br /> 1 => 129<br /> <br /> 0 => 128<br /> <br /> <br /> Keep in mind the technique I used to get this. Shooting at a flat creamish wall, defocussed, hand held. The hand-held part may also account for a few small variations. Cheers.</p>

<p>Something doesn't quite jibe with the data, and I'm just pointing it out: I don't know how to interpret these codes (and so I make no assumptions about them).</p>

<p>1. The 5D has around 8.2 stops of DR, right? I'm going with the dpreview test, and these are "usable" stops. The 5D is a 12 bit camera, right? So its theoretical ADC saturation value is 4095. 12 bits with 8 bits of DR allows for a max value of only 15 for noise (assuming linearity and integral data).</p>

<p>2. In your calculation, Bernie - I could assume my noise floor is at 2047; and get the following table:</p>

<p>input : output<br /> stop 11 : 3700 (1653 signal + 2047 noise)<br /> -1EV : 2874 (827 signal + 2047 noise)<br /> -2EV : 2460 (413 signal + 2047 noise)<br /> -3EV : 2254 (207 signal + 2047 noise)<br /> -4EV : 2150 (103 signal + 2047 noise)<br /> -5EV : 2099 (52 signal + 2047 noise)<br /> -6EV : 2073 (26 signal + 2047 noise)<br /> -7EV : 2060 (13 signal + 2047 noise)<br /> -8EV : 2054 (7 signal + 2047 noise)<br /> -9EV : 2050 (3 signal + 2047 noise)<br /> -10EV : 2049 (2 signal + 2047 noise)<br /> -11EV : 2048 (1 signal + 2047 noise)</p>

<p>which means that in the best case the noise is (2047/3700) = 75% of the signal, but I have 11 stops of DR. If something's wrong with this math, the same thing is wrong with yours too.</p>

<p> </p>

<blockquote>

<p>Something doesn't quite jibe with the data, and I'm just pointing it out: I don't know how to interpret these codes (and so I make no assumptions about them).</p>

 

</blockquote>

<p>The output is the level value corresponding to the majority of pixels in the image. This is the value that is output by the ADC.</p>

 

<blockquote>

<p>which means that in the best case the noise is (2047/3700) = 75% of the signal, but I have 11 stops of DR. If something's wrong with this math, the same thing is wrong with yours too.</p>

 

</blockquote>

<p>Your maths looks fine. You could set your noise at 2047, but in the case of the 5D it is only 128. The point of these figures is that clearly noise overwhelms signal at some point. If DPREVIEW was finding 8.2 stops, then I assume that after this point signal could not be distinguished from noise anymore. One thing to keep in mind, is that after raw conversion, the darkframe noise is subtracted from the file. So those pixels in my example which register say 130, they would register 2 after darkframe subtraction. So those pixels with a level of 2 theoretically have no noise in the signal anymore. But noise/pixel isn't always exactly 128. In the linear raw histogram you can see it varies in a normal like distribution around 128. So there is an error margin inherent in this darkframe subtraction, and it is probably this error which makes a certain number of lower stops unusable for identifying dynamic range.</p>

<p> </p>

<p>No, the math <em>has to be</em> wrong. If the ADC is linear and its output can be interpreted as integers, then 2048-4095 can represent only one stop of dynamic range, not eleven, noise or no noise.</p>

<p>The ADC doesn't know (ha, ha - sentience again) that the voltage corresponding to the first 2047 values is noise. It just encodes what is given to it and for a linear ADC whose output is integers, a 2048 to 4095 range of values can represent <em>exactly one stop of dynamic range -</em> from the top stop to a stop down.</p>

<blockquote>

<p>The ADC doesn't know (ha, ha - sentience again) that the voltage corresponding to the first 2047 values is noise.</p>

 

</blockquote>

<p>You and machine intelligence.... you've clearly watched too many sci-fi's as a kid :)</p>

<p>No, of course the ADC doesn't know how much is noise. But that doesn't mean that the noise doesn't exist. If 2047 was the noise floor, it would work as you state. But consider that the normal distribution I talked about around that value of 2047 would be a larger absolute value than the case of a noise floor of 128. So the error margin for a darkframe subtraction of a noise floor of 2047 would be far greater, and presumably would result in far fewer stops being discernible as true signal. I truly don't see any problem with the maths you presented. It could work, just not very well photographically speaking.</p>

<p> </p>

<p>Very interesting post! But... please, take a break. Stop looking at your cameras and go back to look through your cameras... You (and me) are loosing a lot of pictures and time in front of the computer screen...<br>

See you on the street, my friends!<br>

(By the way... life is irregular like film grain, not like CCDs)</p><div></div>

<p>Vijay!</p>

<p>Check out my last post here:<br>

http://www.photo.net/canon-eos-digital-camera-forum/00RewZ</p>

<p>It was dedicated to you :)<br>

-Rishi</p>

<p>Mauro -- I know it's been a while, but back when you did this test & we had these discussions, for the 40D file, did you ever try setting the black clipping to zero & then using Fill Light (Lightroom is great for this)? I wonder if you can pull out significant shadow detail from the 40D file as long as you shot RAW.</p>

<p>Perhaps we already talked about this in this thread but the search capabilities within a thread are non-existant here on photo.net :(</p>

<p>Thanks,<br>

Rishi</p>