the rule of - expose "right" but avoid clipping

[bernardwest]

Andy, I had two quick attempts with your small web image. Here's a 2 minute curve job. There's a slight cast, but that could be easily fixed. This gets back to my point about it being easier if you have something to compare it against when performing the curves adjustment.<div></div>

[bernardwest]

Here's an adjustment just using Lightroom's exposure slider (-1EV). This was even simpler than the curves job. But to be honest, this scene was much simpler than similar attempts I have made on portrait photos.<div></div>

[colinsouthern]

If you want a digitally captured image to have maximum headroom for editing, and minumum noise then one needs to ensure that the highlights take the sensor right up to the limit - NO EXCEPTIONS.

 

Unfortunately we can't tell what the limit is by looking at the histogram, although we can learn a lot more by taking a variety of shots (and repeated for each ISO) - evaluate the histogram whilst still in the RAW converter - and then learning the corelation between the jpeg-conversion based histogram that our camera presents to us and the "real" one that we see in ACR.

 

If you're using a typical 12 bit A/D camera (say 20D) and you end up 2 stops to the left (EVEN IF THE PICURE LOOKS JUST FINE) then you've thrown away 3072 of the 4096 possible levels - but it gets worse - the above figures apply to linear gamma - we're converting to a non-linear gamma - the that means that the few remaining levels describing shadow detail have to get stretched even further - so noise is amplified - posterisation can easily occur - and you're also compressing the highlight range - which can lead to banding. On some photos you'll get away with it - on others (even though you're shooting RAW) they can still fall apart under even light to moderate editing.

 

All covered in just the 1st 2 chapters of Bruces book - and that's not even getting to the good stuff yet!

[van_camper]

Micheal Reichman and Thomas Knoll started this expose to the right idea. If the histogram is mainly to the left, then I agree sliding it over towards the right without washing out the higlights makes sense to improve shadow detail and reduce noise. However, the remainder of their theory is wrong. I know, Thomas Knoll wrote Photoshop, but he is a programmer, not a mathematician. They argue ? the information is recorded in the first stop, each one after that receiving ? of the previous one. This is ridiculous thinking because they applied a mathematical progression (2,4,8,16?2048,4096) to represent ONLY a simple way of showing a doubling of exposure as when describing a 1 stop change in shutter speed, and they then took a giant leap by arguing they can then apply the same scale to represent tonal values (without any proof, just expecting us to assume this, and if you believe this, then there strange ideas do make sense?..hogwash). Think for a moment, if ? the information is recorded in the first stop, and ? again in the next stop, then the histogram would have to be hevily skewed towards the right (where they argue the brightest first stop gets ? the tonal values). We DO NOT see skewing in any of our histograms, they are in fact TOTALLY RANDOM! Some are normal looking, some flat, some narrow, all the histogram does is record the scene. Remember, the histogram shows tonal values from 0-255 on the x-axis, while the y-axis records total numbers for each tone.

 

Exposing to the right works for improving shadow detail, and noise. But when he implies the further reason to expose to the right is because1/2 the tones are represented in the first stop, so you will be having superior quality?.that is wrong. Using his idea that the first brightest stop records 4096 tones, the next only 2048, then if you follow this idea through from zone 8 down to zone 3?..you would be seeing pixilation.

[van_camper]

Sorry fellows, for some reason a ? mark is being introduced at random that I am unable to control. I wrote it in WORD, then transferred it to the photonet window for entering comments, but when clicking on SUBMIT it goes funny.

[colinsouthern]

Hi Van,

 

I'm afraid that you've got a couple of things a bit skew, and as a result, you've added 1 + 1 and got 3.

 

First thing to remember (re: Histogram clumping): Output level is equal to input level x gamma. Or put another way:

 

If you take a normal output, divide that by the gamma (2.2 typically) you get back to linear input. You need to keep in mind that the histograms we look at in our cameras or in photoshop are all gamma corrected - for linear data it is expected that they will be clumped up to the left, not to the right.

 

The 2048 levels in the first stop, 1024 in the next stop down etc thing comes from the linear way a sensor counts protons - fire twice as many protons at a particular photoreceptor site for a given shot (say compared to the previous shot) and the A/D converter will return a number twice as big - but our eyes don't work in the same linear way, (and as a result the scene would seem only fractionally brighter. )

 

As an example, if a photoreceptor cell will be maxed out by the arrival of 4096 photons then BY DEFINITION because it's a linear device, when you only capture 2048 photons in the same time frame then you've halved the light entering the device. From that point down, when you've captured 1048 protons you have BY DEFINITION halved the light again.

 

But throughout all of this you have to remember that this is LINEAR data, not gamma encoded data - our eyes aren't very sensitive to large changes in bright light - they are however very sensitive to changes in low light (ie shadow areas) - so the camera captures with incredible precision, all of the stuff that doesn't make much difference to us, and captures with the least precision (typically 5 or 6 stops down) what we're most sensitve to.

 

Sorry, I'm not particularly good at explaining this :(

[colinsouthern]

*Grrrrrr - too late at night ... Photon, not proton!

[bernardwest]

Colin of course meant photons not "protons".

 

Van, I'm with Colin, I suspect you are a bit confused. Unfortunately it's been too long and there's been too many beers since I last did any hardcore maths for me to be able to explain this properly. But I will have a go at answering a couple of your points.

 

Firstly, the histogram doesn't measure f-stops, it measures equally spaced tonal values, and hence is a linear plot, so there is no need to expect it to be skewed towards the right side. However, the relationship between successive f-stops and exposure is logarithmic, like for example the Richter scale for earthquakes. That is, each successive number on the scale represents a doubling of the magnitude of the previous one. Now, the image histogram is a linear plot, not logarithmic. If, however, you plotted the histogram in a logarithmic fashion (ie. in successive f-stops, not equally spaced tonal divisions) you would indeed find it stacked to the right. I think this is where you are getting confused. In a linear sense, the 'expose to the right' technique doesn't require the final adjusted result to show any skewing to the right.

 

Finally, your concern about seeing "pixelation", is not very clear. I guess you mean that there would be so few levels actually left in the shadows to show anything meaningful? If that's the case, I think the answer goes something like this (someone please correct me if I'm not right here): If you apply the f-stop doubling exposure rule, in a 12-bit histogram, you would have a dynamic range of 12 stops. Does this mean that digital SLRs can capture 12 stops? No. Because the number of quanta of photons being captured in the lowest few stops is so very few, these lower stops have almost no detail and very low signal to noise ratios. Hence, they are unusable and don't contribute to an image's dynamic range.

[van_camper]

"Firstly, the histogram doesn't measure f-stops, it measures equally spaced tonal values, and hence is a linear plot, so there is no need to expect it to be skewed towards the right side."

 

Bernie and Colin, thanks for your views, the top comment tells me we were off track from the start(I didn't say it measures f/stops). I can tell were talking about the same things, but none of us are getting our points across. I somewhere missed my point, but to try to sort out both your views and mine, and understand where it went wrong, well I just am not up to it. It would take too much time and a very long article or few. This is where it would be nice if we could sit down for a few beers, and the instant feedback between us would sort things out quick. I am sure we then would all be in agreement, and a little drunk. LOL.

 

I fully understand what you said regarding logs, linear relationships, histograms, and understand all aspects of sensitometry (H&D curve), plotted many, etc, that is not the problem. The problem is to explain things sometimes you have to show it on paper, charts, and get instant feedback from others so we know were on the same wavelength before you move on to the next step. Reichman and Knoll I am sure also spent some time at this. If you read carefully what they say, you will see they have taken the exponential scale (2,4,8,16) to represent a doubling/halving of something (which is all it represents) and they APPLIED it to refer to the number of tonal values. So that according to them 1/2 the information is recorded in the first brightest stop. Ther error begins here, the rest takes time to explain. So I give up. Remember in the end film and dslr have to behave similarly, our handheld meters work for film and digital media, the theory is similar. Somewhere we are misunderstanding eachother. Good luck.

[colinsouthern]

Hi Van,

 

If you can accept that a digital sensor counts photons in a linear manner (which really isn't debatable) then one really has to accept that 1/2 the information is being recorded in the first stop because it's simply by definition (assuming linear gamma, and a maxed-out sensor to start with).

 

If we have a 12 bit A/D converter (ie 4096 possible levels - and it's linear (which it is) then when the light is cut (assuming maxing out the sensor to start with) by 1 stop then you're down to level 2048 by definition - any intensity between 1 stop down and full intensity must fall between levels 2048 and 4096, requiring 2048 levels to describe it. And so on and so forth for the next stop down.

 

Stopping down a lens 1 stop at a time halves the light passing through at each 1 stop reduction (1/2, 1/4, 1/8, 1/16th etc) - when you're processing that digitally in a linear manner then with each stop of reduction (from a stopped lens for simply because it's lower intensity light to start with; eg Shadow or mid-tone data) you must - again by definition - halve the value of the number that's assigned to represent the actual values (4096, 2048, 1024, 512, 256).

 

If you stop down 3 stops then you've cut out 7/8 of the light that could have passed through - and you've also cut out 7/8 of the A/D resolution that's available to describe it.

 

I'm wondering if the bit that's confusing you is that fact that stopping down is not a linear thing.

 

"Remember in the end film and dslr have to behave similarly, our handheld meters work for film and digital media, the theory is similar."

 

Perhaps this is part of the problem - a digital sensor has a response curve that's NOTHING like a film camera. Film responds in a very similar manner to our eyes (responsive to low intensity changes, relatively insensitive to large high intensity changes) - so what comes out is very "normal" looking to us. On the other hand a digital sensor with it's linear response curve produces output that looks hideously dark to our naked eyes - it's not until we massage the data by applying a Gamma correction does it start to look normal. Same result, vastly different ways of achieving it.

 

Best I can do at 1am I'm afraid.

[emre]

That's as clear as you can get without a diagram.

[obakesan]

Hi

 

its also late here, but it occurs to me that it would be silly to quantise the light data the way that is being explained. I suspect that Van is trying to express that point.

 

While I understand that single stop increments are 1/2ing the light it seems that given the quantisation of any given (say) 8 stop band by putting so much emphasis on the brightest components is perhaps not ideal.

 

Its all been very good food for thought, and I think I'll be doing some more research.

 

Ohh ... BTW Colin, rather than pass what could easily be interpreted as cynical and olbique remarks about my finances and my choice of what I buy and what I look at in the library before I decide to by, perhaps if it was "dirt cheap" you could buy it for me :-)

 

however in this instance I went to amazon and bought it myself, as its not available easily for me here in Finland. But thanks for the kind advice anyway mate.

[colinsouthern]

Hi Chris,

 

It's not "silly" per sec - it's just the way digital sensors work (ie in a linear fashion). We do suspect that Canon are fiddling the A/D conversion process a little with Highlight Tone Priority, but as of this writing, it's all still mostly linear capture. Perhaps on day they'll make a sensor that imitates the way our eyes work?

 

I think that many people (for whatever reason) seem to think that "stopping down, say, 7 stops" means you're cutting out something like 7/8ths of the light passing through - whereas in reality its being cut to 1/128th - welcome to the non-linear world!

 

I apologise if my "dirt cheap" remark came accross as being a bit flippent - I just tend to think along the lines of it seeming a bit out of whack when people invest literally thousands of dollars on equipment and fail to invest even a few hundred dollars on educational material that would help them get a lot more out of it. For me, that's the problem with forums like this - often we can answer a question, but most of us aren't professional writers - and many even have the wrong idea about many things (possible because they learned it from others with wrong ideas about things). When you spend perhaps $40 on a book written by the likes of Bruce Fraser you get a text from someone who (a) knows what they're talking about, and (b) can take all the time and space they need to lay the necessary foundations and explain things thoroughly - so in the end we get "an education" rather than "a question answered" - that's got to be good value for money :)

 

Cheers,

 

Colin

[obakesan]

Colin

<P>

The reason that I put forward that its silly is that irrespective of the

sensor being linear or not in its output there is no reason why these levels

need to be encoded in a linear fashion when being transcribed as DATA (<I>

IE the digitisation of that signal</I>)

<P>

The camera has (perhaps) 4096 discrete numbers that can be associated with a

light level. As an encoder designing an algorithm for that I see no benefit

in giving such a high weighting to the bright levels. Perhaps this is what

Van was trying to express. if so much differential of light exists in the

single ftsop of recordable data, is it worth recording at such fine levels of

quantification? Surely this would result in a loss of quantity of the midrange

and low levels?

<P>

If this was the case, then I would expect to see banding occur to these levels

when curves were applied. If the numbers differentiating these levels were indeed

so close together then (when 'expanded' by curves) steps would become larger and

tonal differences would become more like 16 steps than 128 or even 256.

<P>

Right now I don't know enough detail on how quantisation occurs, and I would like

to understand this more to know how the essentailly analog signal at the sensor

is digitised to help my know how to expose more appropriately for this media.

<P>

Just like slide is different to negative, sensor is different again.

[obakesan]

Folks, <P>for the record I have done a little more ferreting, and found some interesting tool that allows the analysis of RAW files. It leaverages off dcraw and the information it presents is quite interesting. <P> This software is available <A HREF="http://ruevs2.tripod.com/rawhistogram/rawhistogram.html" target="_blank">here</A> and allows you to have a look at the numerical data comming off your RAW file. <P> It seems to break this into 15 bit values and more or less gives you what you'll see in the histogram. The files produced can be loaded into EXCEL and if you restrict your graphing to valuse upto 4096 you'll get a graph that's usable. <P> as has been expressed, this is the linear representation of the data and so is biased towards the left rather than what we see in the camera (which seems to have the X axis scale non-linear) <P> This goes some way towards explaining my questions and may also prove useful for anyone wishing to comprehend their histograms better. <P> the learning continues

<P><IMG SRC="http://home.people.net.au/~cjeastwd/images/smile.gif" ALT="?" BORDER=0>

[bernardwest]

<i>as has been expressed, this is the linear representation of the data and so is biased towards the left rather than what we see in the camera (which seems to have the X axis scale non-linear)</i><br><br>

 

The X axis is still linear. The reason it doesn't look like the linear raw data is that this data has had a gamma correction applied to it, essentially remapping all tones to output lighter. I would be interested in knowing exactly when and how this gamma correction is applied.

[bernardwest]

Another interesting thought on this gamma correction is the issue of exposing to the right and then pulling back the exposure in post production. When you apply say -2EV in a raw converter, the levels in the histogram don't adjust in a linear fashion (ie. you don't see a 12-bit histogram move to the left by 3072 levels). Is this because the exposure compensation is factoring in issue of gamma correction? Or is something else going on here?

[kevin_schoedel]

<blockquote><i>The reason that I put forward that its silly is that irrespective of the sensor being linear or not in its output there is no reason why these levels need to be encoded in a linear fashion when being transcribed as DATA</i></blockquote>

 

Yes, there is -- the output of the ADC (analog-to-digital converter) hardware is linear. If you store it otherwise, you waste space and/or throw away information.

[rishij]

<p>My biggest problem with all this ETTR business is this: ETTR basically means that you *will* lose apparent saturation of certain colors (that are 'overexposed', not necessarily blown out) b/c as you increase luminosity of any color beyond a certain point, you approach white -- exemplified by traveling upward along the luminosity axis in this HSL diagram:</p>

<p><img src="http://upload.wikimedia.org/wikipedia/en/thumb/d/d3/Color_cones.png/564px-Color_cones.png" alt="" width="564" height="600" /><br>

Now, this would be fine, b/c all you need to do is to pull those particular pixels that are overexposed back down again... as Emre suggested above, you can do that by decreasing exposure. But then you decrease the shadows as well which, in many cases, you may not wish to do.</p>

<p>Hence, what you really need is a slider (algorithm) to just decrease the exposure of 'highlights'... and perhaps a slider that allows you to choose the 'cut-off point' for what to consider higlights vs. non-highlights.</p>

<p>Luckily, many software developers have already figured this out. Aperture has this very tool. So does Photoshop (Shadows/Highlights). Yet Adobe's newest software, Lightroom, written by the very guys who promulgate this ETTR philosphy, doesn't have any way of doing this.</p>

<p>This is <strong>inane</strong> . It <strong>irks</strong> me to no end. As it should you. Maybe if enough of us complain, LR devs will do something about it. You can't decrease exposure of highlights in LR. 'Highlight Recovery' doesn't do it; it was designed for something different (personal communication with Mark Hamburg on LR Beta fora). Decreasing 'highlights' in the Tone Curve tool also doesn't do it, b/c it geniusly (& by that I mean 'stupidly') decreases saturation of the very highlight pixels it operates on when you move the 'highlights' slider to the left.</p>

<p>Would love to hear from others about their favorite way of darkening highlights. Round-tripping to PS from LR is the way I do it right now, but, damn what a waste of time/space.</p>

<p>Rishi</p>