Film vs Digital - Dynamic Range

[bernardwest]

<p>Vijay, get a grip.  How was I being "boneheaded"?  I presented numerous examples and made a real effort to explain my point of view.  I openly invited people to show me how my examples were wrong.  Did you do that, no, you just kept insisting that I was wrong because you deemed it so.  No examples, nothing.  The fact that I am open to new ideas and will gladly change my mind if shown proof, is best shown by the fact that it only took two posts from Ryan to show me where I was in error (although I now need more discussion on that with him or others).  You wrote a small novel of posts and couldn't achieve what Ryan achieved in 2 posts.  Hence why I owe you no apology (and particularly so because I wanted to stop debating this a number of times).  You know, all those things you labelled me with in your previous post, have been labelled at you well before this thread.  Maybe it says something about me, but I generally tend to descend to the lowest common denominator in a debate.  If my opponent wants to play dirty, then I'm happy to oblige.  If you can't take what you dish out in spades Vijay, then honestly that's your problem. </p>

[vijay_nebhrajani]

<blockquote>Bernie: "I generally tend to descend to the lowest common denominator in a debate."</blockquote>

<p>But I don't. Goodbye, Bernie.</p>

[helenbach]

<p><em>"That means that you can't get infinite DR in respects to integer output from an ADC (is this where the floating point math comes in?). "</em><br>

<br /> Yes. That's correct. Integer linear output from an ADC sets an intrinsic relationship between the bit depth and the maximum dynamic range (of the incoming signal) that can possibly be represented by those bits. If the analogue sensor itself has a linear response to whatever it is measuring then it should also be the dynamic range in terms of what is being measured, noise apart.<br>

<br /> If the ADC output is linear, but floating point, some precision is lost but the dynamic range that can be represented is increased. The floating point operations would have to happen after the ADC itself, however, because there's really no such thing as a floating point ADC. What might be described as a floating point ADC is really something that has floating point operations immediately after the digitisation of the analogue signal. A device could write files that are (say) floating point 8-bit linear files that represent a wider dynamic range than integer 8-bit linear data could represent, but the original digitisation would have had to be at a higher bit depth.<br>

<br /> As it has been referred to, dcraw (a Raw converter) uses floating point maths internally, then does integer rounding prior to writing the linear output file. There is a version that allows you to apply a gamma conversion before the integer rounding. Again, this is all post-ADC.<br>

<br /> Some HDR formats use floating point so that they can represent very high dynamic range linearly without using excessive numbers of bits (unless you call 32 bits per channel per pixel excessive).<br>

<br /> Vijay, you say that you are more used to explaining things to technical people. I find that when I discuss these matters with my technical peers I have less of a need to go into the nitty-gritty, because they quickly latch on to what I am saying (and they probably know me better than you do, so they tend to be more willing to pause and consider what I am saying instead of quickly dismissing it by ill-considered, apparently wilful misunderstanding). Thankfully this isn't a big deal, and I can't get very worked up about a few internet musings.<br>

<br /> Best,<br /> Helen</p>

[ryan_drake_stoker]

<blockquote>

<p>Bernie West - Any thoughts on this?  I wonder whether I am missing something simple from this calc.</p>

</blockquote>

<p>No, you are not missing anything, the formula for recording an 8 stop scene in 4bits would be 68.3 times X (where X is your 4bit number).  What you are seeing is a lack of precision, the difference between the minimum and maximum is still 8 stops.  The linear record is absolutely accurate and covers the dynamic range perfectly, it is our perception that values ratios (stops, a non linear scale) over linear difference, i.e. the human eye cares whether the 1mV difference is at the top of the scale or at the bottom. </p>

<p>Leica actually used this trick in the file encoder for the M8, by biasing the encoding to the bottom end of the scale to produce 8bit RAW files that they claimed had 16bits of information. What they meant was that the 8bit RAW represented a non-linear precision that provides 16bit precision at the low end and less than 8bit precision at the upper end.</p>

 

<blockquote>

<p>Helen Bach - A device could write files that are (say) floating point 8-bit linear files that represent a wider dynamic range than integer 8-bit linear data could represent, but the original digitisation would have had to be at a higher bit depth.</p>

</blockquote>

<p>Sorry in this you are missing an important aspect of floating point encoding, namely that it uses the same 8-bits as the integer, the bits just mean different things.  This is important because an equivalent bit length floating point value has <strong>less</strong> precision than the integer representation using the same number of bits.  Using the standard IEEE-754 floating point encoding a 32bit floating point value has the same precision as a 24bit integer value.  Floating point is used primarily because it makes the maths significantly easier.</p>

[mauro_franic]

<p>James, "Maybe we should start a thread of doing DR testing, techniques and  suggesting only???"</p>

<p>Your technique of cutting the shutter speed in half and using natural light seems ok to me.   Don't use a grey card, use something with features that can let you determine whether detail is clipped or not.</p>

<p>Do you have a doll handy?</p>

[helenbach]

<p><em>"Sorry in this you are missing an important aspect of floating point encoding, namely that it uses the same 8-bits as the integer, the bits just mean different things.  This is important because an equivalent bit length floating point value has <strong>less</strong> precision than the integer representation using the same number of bits. "</em></p>

<p>No, I didn't miss that point. I wrote earlier in the same post <em>"...but floating point, some precision is lost but the dynamic range that can be represented is increased." </em> I have made the distinction between precision and dynamic range a few times in this thread.<em><br /> </em><br>

Best,<br>

Helen<em><br /> </em></p>

[igor_jarm]

<p>Rishi said:<br>

<em>One thing that confuses me though -- sometimes it seems like the direction of the banding changes from image to image, by 90 degrees (and no, not because the image was portrait vs. landscape). I'm confused...</em><br>

<em><br /> </em><br>

I tested my Canon 20d for low light performance and made an average of 16 black frames containing only read noise. While individual shots contained horizontal bands, the average contained only vertical bands with random noise averaged out. So it seems vertical bands are always present but horizontal are more obvious.</p>

<p><a href=" exposure</a><br>

<a href=" of 16 exposures</a></p>

<p>Detail of 1 exposure :<br>

<img src="http://farm4.static.flickr.com/3244/3095714886_288f3c9e3a_o.jpg" alt="Detail of 1 exposure" width="800" height="800" /><br>

Detail of average of 16 exposures:<br>

<img src="http://farm4.static.flickr.com/3235/3094873179_5fda1afd21_o.jpg" alt="Detail of average of 16 exposures" width="800" height="800" /><br>

This does not explain what you're seeing but it is what 20d and perhaps all canon sensors record.<br>

The value of majority of this noise was around 4 in recorded raw at all ISOs.</p>

 

[ryan_drake_stoker]

<blockquote>

<p>No, I didn't miss that point. I wrote earlier in the same post <em>"...but floating point, some precision is lost but the dynamic range that can be represented is increased." </em> I have made the distinction between precision and dynamic range a few times in this thread.</p>

</blockquote>

<p>Fair enough I misunderstood the point you were trying to make, you clearly understand that a FP number of equal bit length contains less information than an integer, but I cannot see why you think that FP values would have an impact on recordable dynamic range.</p>

<p>From a purely mathematical point of view how does using a smaller number of discrete decimal values as opposed to a larger number of integers have any impact on the minimum and maximum of what is being measured.  Any dynamic range can be represented by just two values, 0 and 1.  0 is a single photon in the visual range and 1 is direct exposure to the sun at close range, my dynamic range is huge, my precision is terrible.   So I am at a loss to see how using discrete decimal values has any relevance to dynamic range.</p>

[vijay_nebhrajani]

<p><em>Helen: Vijay, you say that you are more used to explaining things to technical people. I find that when I discuss these matters with my technical peers I have less of a need to go into the nitty-gritty, because they quickly latch on to what I am saying (and they probably know me better than you do, so they tend to be more willing to pause and consider what I am saying instead of quickly dismissing it by ill-considered, apparently wilful misunderstanding).</em><br>

<br /> I am rather astonished at this hostility from you, Helen. If you are a technical person, like an engineer, then please use engineering terminology and explain why dynamic range has anything to do with the bit depth used for encoding it. I don't know if you have an engineering background; at least I tried to explain my point of view to you in non-engineering terms. If it came across as too simple, then by all means, complicate things a little.</p>

[vijay_nebhrajani]

<p>Ryan is exactly right; a dynamic range is just two numbers. We generally represent it as a ratio, but we could equally well represent it as an ordered pair (min, max). It takes one bit two represent two states. That is about the depth of the matter.</p>

[rishij]

<p>Igor,</p>

<p>Nice work! Thanks for that test. Perhaps we should move our conversation over to <a href="../canon-eos-digital-camera-forum/00Rjyp">http://www.photo.net/canon-eos-digital-camera-forum/00Rjyp</a></p>

<p>In the meantime: What did you use to average the exposures? Some layering technique in Photoshop?</p>

<p>Also, your example is precisely what I was talking about & I think it explains exactly what I'm seeing. Berg had previously posted the following example as a way to get rid of this variation:<br>

<a href="http://www.pbase.com/terrylovejoy/image/50670364">http://www.pbase.com/terrylovejoy/image/50670364</a></p>

<p>I don't quite understand how. Any ideas? Anyway let's move this discussion over to the other thread I linked to, as it's sort of irrelevant here :)</p>

<p>Thanks,<br>

Rishi</p>

[bernardwest]

<p>Nice work Igor.  Clever experiment.  In your other topic, Rishi, find out if the 1 series has this phenomenon as well.</p>

[bernardwest]

<blockquote>

<p>No, you are not missing anything, the formula for recording an 8 stop scene in 4bits would be 68.3 times X (where X is your 4bit number).</p>

 

</blockquote>

<p>Ryan, I'm not following.  What's 68.3?  Is that the decibel equivalent of 8 stops?<br>

Regradless, looks like I need to swat up on floating point math.  Ryan I hear your point about the lack of precision, but does it actually go further than this?  In that example I gave, say we photographed a 25 stop scene instead.  In that case, the first 4 stops would be accurately recorded, the 5th stop not very accurately recorded, and the 6th to 25th stops would all be squashed into level zero.  Basically there is ZERO precision in those stops.  So, the reality is that you are really only getting 4 and a bit stops, and everything below that is zero precision, and is therefore useless.  Thoughts on this?</p>

[rishij]

<p>Igor,</p>

<p>Could you by any chance repeat the same test with the lens detached? There are some blokes out there that think that the USM motor on the lenses causes the banding.</p>

<p>Thanks,<br>

Rishi</p>

[helenbach]

<p>I'll take this step-by-step, and number the steps so that if you disagree with anything it should be easy to pinpoint the disagreement.</p>

<p>1) We usually enumerate dynamic range as a ratio rather than an absolute difference. Examples are '12 stops'; 63 dB; 2.7 optical density; 1:1000. The first three are logarithmic differences (equivalent to ratios) and the fourth is a simple ratio.</p>

<p>1A) Zero cannot be the lower limit of a dynamic range, because dynamic range is a ratio.</p>

<p>2) We are discussing a linear relationships between the detected luminance and the numeric output of the ADC. We've already all agreed (I think) that if it isn't linear, there is no fixed relationship between dynamic range and bit depth.</p>

<p>3) My statement was with respect to the existence of a relationship between bit depth and the maximum dynamic range that could be represented by that bit depth if an integer representation was used, disregarding noise.</p>

<p>4) A linear relationship implies that the output is directly proportional to the input. For example halve the input and you halve the output.</p>

<p>5) If you use a certain number of bits to represent unsigned integers there is a limit to the greatest ratio that can be obtained from the maximum and minimum values that those bits can represent. For example, if you have 8 bits you can say that the non-zero values range from 1 to 256, or from 2 to 512, or from 4 to 1024 etc. In each case the ratio of the highest to the lowest is 256:1. Break that rule, and you are no longer dealing with a linear system. You could multiply, then subtract, of course - ie use 4 to 1024 and subtract 3 to give 1 to 1021, or an apparent 1021:1 dynamic range. It's no longer strictly linear because you have just destroyed the proportionality between input and output by introducing an offset.</p>

<p>6) Having established that the number of bits sets the greatest ratio that can be represented by integer values, we can now combine that with assumption 4). If the greatest output ratio is limited, so is the greatest input ratio, and the ratios are the same, because we have restricted ourselves to a linear system.</p>

<p>7) Now suppose that we use those same number of bits to represent floating point numbers, and keep the same strict proportionality between input and output. I'll use a version of 8-bit minifloat as an example, even though there is a waste of a bit in the first, sign bit. The largest positive value that a minifloat can hold is 0 1111 111 and the minimum non-zero positive is 0 0000 001 (one sign bit, four exponent, three mantissa).</p>

<p>If 0 0000 001 is set to 1, then 0 1111 111 is 114688. The ratio between them is now 114,688:1 instead of 256:1. That is an extreme example because of the nature of the minifloat (and the precision is extremely low). A non-standard floating point format could be 00 000000 (two exponent bits and six mantissa bits, no sign necessary) which would give a maximum ratio of 1024:1. There's a direct trade between dynamic range and number of levels in this case, because of the absence of a sign bit. You quadruple the dynamic range, but quarter the number of levels.</p>

<p>Remember that the floating point part of this is in an attempt to answer Ryan's question - I've already covered the practical (or impractical) aspect of it.</p>

<p>Best,<br>

Helen</p>

 

[bernardwest]

<p>Thanks for that description Helen.  Upto and including point 6, this is what my argument was from the start.  Now I am really doubting whether I should have doubted my initial assessment :)  Point 7, I need to swat up on.  But am I right in interpreting this point as saying that with 8bit minifloat you could output a range of 1 to 114688?  If that is the case, then my understanding would be that this isn't what is being done in current dslr ADC's.  When plotting linear raw data, the range is definitely within the standard integer expansion of the ADC bit-depth.  ie.  the 14 bit 40D outputs from 0 or 1 (don't know, is it really 1 but is called zero by convention??) to 16383(or 4) (of course, the maximum value is only about 13850 due to saturation of the photocells).</p>

[vijay_nebhrajani]

<p>Helen, your point 1A is in error. Formally, a range is a numeric pair. A ratio is just one way to express it. A numeric pair can include the number 0 in it. This <em>could</em> result in an infinite ratio; for instance in a perfectly noiseless transducer. That such a thing doesn't exist in practice is immaterial, there is only one set of definitions that may apply to both noisy and noiseless devices.</p>

[helenbach]

<p>OK Vijay, please could you give an example of a practically useful and relevant dynamic range that uses zero as one of the bounds, or, for interest, a relevant dynamic range that is not a ratio.</p>

<p>Thanks,<br>

Helen</p>

[vijay_nebhrajani]

<p>The output voltage of a lightswitch. If the switch is open, it is 0. If it is closed, it is 110V.</p>

[bernardwest]

<p>Helen, I've just reread one of your posts and see where you stated that dcraw uses floating point math and rounds to integers later.  Is this the reason why the linear raw output is only between 0 and (2^n-1)?  If the raw file itself contains the values of 1 to whatever the 14bit floating point value is, we should be able to confirm this somehow shouldn't we?  Infact I am wondering if this point is correct.  When you output in dcraw using the -D switch, you are getting undemosaiced raw data, and the results are definately between 0 and 16383.  If I could get Panopeeper's "Rawanalyze" to work on my machine, we should be able to verify this as I believe it can show you pre-demosaiced data as well.  Have you investigated any of these approaches?  Cheers.</p>

[helenbach]

<p>Bernie,<br>

I was only giving the minifloat as an example of the maximum ratio that could be achieved by using floating point instead of integer representation. Nobody in their right minds would use 8-bit minifloat for high dynamic range images because of the lack of precision.<br>

(I've just noticed an error in my calcs for, and explanation of, the rough-and-ready ad-hoc floating point example - I can do better than that. I was in a big rush, and still am. More later.)<br>

Best,<br /> Helen</p>

<p>PS This post was started ages ago and other things took priority, so it is out of order...</p>

[helenbach]

<p><em>"The output voltage of a lightswitch. If the switch is open, it is 0. If it is closed, it is 110V."</em></p>

<p>Well, that's a pair of values, and it is a useful pair of values, but it's not really a dynamic range - it's an absolute range. If you think that it is a dynamic range, then I can see why there's a lot of misunderstanding here.</p>

<p>Best,<br /> Helen</p>

[vijay_nebhrajani]

<p>A dynamic range doesn't have to be a continuously varying range - nothing requires it to be so. But if you want to negate that and state that the output voltage range of a lightswitch isn't a dynamic range, then so be it. Then we run into this difficulty:</p>

<p>Each output bit of an ADC is also a switch. So if it is improper to talk of the dynamic range of a lightswitch, then by the same token it is improper to talk about the dynamic range of the output of an ADC as well.</p>

[rishij]

<p>Man, now you guys are just boring the crap outta me :)</p>

<p>Now let me throw something in to spice things up...</p>

<p>Here are some negative characteristic curves for different development times, from Baines:<br>

<img src="http://staff.washington.edu/rjsanyal/Photography/NegCharacteristicCurves.jpg" alt="" width="800" /><br>

<a href="http://staff.washington.edu/rjsanyal/Photography/NegCharacteristicCurves.jpg">Link to Full-Size Image</a></p>

<p>As you can see, depending on the development, it can have quite a non-linear response (well, the region of linearity can be rather small).</p>

<p>I mean, sure, perhaps development is such that the response is mostly linear (past the 'toe' or 'foot' anyway). It surprises me that the thinking back then was to expose near the 'foot'... don't we do the exact opposite?</p>

<p>Also nice to see how the density can vary so wildly based on exposure times. That in itself indicates that the dynamic range the film can exhibit is rather high... but I guess doesn't say much, on its own, about the dynamic range of a scene that can be recorded -- correct Vijay? (I single out Vijay because he'd probably be the first to jump out at me :)</p>

<p>Rishi</p>

[helenbach]

<p><em>"A dynamic range doesn't have to be a continuously varying range - nothing requires it to be so. But if you want to negate that and state that the output voltage range of a light switch isn't a dynamic range, then so be it. Then we run into this difficulty:<br /> <br /> Each output bit of an ADC is also a switch. So if it is improper to talk of the dynamic range of a light switch, then by the same token it is improper to talk about the dynamic range of the output of an ADC as well."</em> <br /> <br /> <br /> Vijay,<br /> <br /> <br /> This is very frustrating. You have changed the subject and you are arguing against something that I haven't said. We were discussing whether dynamic range was a ratio, and therefore whether or not zero could be the lower bound. I didn't say that the lightswitch example was not dynamic range because it was not continuously varying. This has been the problem few times in this thread - you have argued against something you imagine that I have said, not against what I have actually said.<br /> <br /> <br /> I'm not sure what the purpose of this discussion is. If you believe that zero can be the lower bound of a dynamic range then you clearly have a different conception of dynamic range from the technical meaning of the term in engineering and photography. Zero can be the lower bound of a range of absolute values, of course. If you think that zero can be the lower bound for dynamic range then the entire discussion becomes meaningless (if it isn't already), and a completely new definition and method needs to be given to the engineering world so that meaningful, unambiguous technical communication can resume.</p>

<p>Rishi,<br /> <br /> <br /> Are you asking for comment of the film/developer curves? As an aside, I think that <em>"Also nice to see how the density can vary so wildly based on exposure times."</em> is a typo for <em>"... based on development times."</em> and that we usually refer to the "dynamic range" of the developed film itself, as in 'dynamic range the film can exhibit' as the density range of the film, also quoted conventionally as a ratio (DMax-DMin, where DMax and DMin are logarithmic densities, and hence the difference between them is a ratio).<br /> <br /> <br /> With silver-image B&W negative film the graininess is usually lowest at the toe (unlike dye-image film), so there's one reason for keeping exposures down at the toe. There's also the aesthetic preference of using the soft toe to slide the shadows gently into darkness. It's all a question of what you judge the 'optimum exposure' to mean. There is no one single answer to that question.<br /> <br /> <br /> The shape of the curves you show is, of course, a product of the development process. Developing to finality with a more energetic developer would probably result in a similar density range but longer linear section (I don't know for sure, because we don't know the details of the materials used in the original experiment). This is what happens in the first developer of a reversal process - the film is developed to a high density range with quite a long linear section, quite unlike the very soft shouldered results shown here.<br>

<br /> <br /> Best,<br /> <br /> Helen</p>