K-7 (K20D) D-Range Setting - how does it actually work?

[laur1]

<p>Hi,<br>

I am interested in getting a better understanding of what's going on when the D-Range options are turned on on the K-7 (Highlight Correction and Shadow Correction). I understand these options were available in some form on the K20D as well, so if you have information on that, it would help me too. I'm also interested in whether this mode changed between from K20D to K-7, but that's secondary.<br>

Note that I am not asking about how to set it or what the results are - I'm just trying to understand what goes on under the hood. I've searched the Pentax forum but only found a couple of mentions with the particular observation that noise in shadows is increased when using these settings. And the manual isn't very illuminating either. I understand how HDR works, but I haven't the faintest idea of what these options do what they do.<br>

I've found a couple of reviews that discuss the output of these settings, but they're short on details about the inner workings:<br>

<a href="http://www.imaging-resource.com/PRODS/K7/K7IMAGING.HTM">http://www.imaging-resource.com/PRODS/K7/K7IMAGING.HTM</a><br /><a href="http://www.trustedreviews.com/digital-cameras/review/2009/09/03/Pentax-K-7-Digital-SLR/p7">http://www.trustedreviews.com/digital-cameras/review/2009/09/03/Pentax-K-7-Digital-SLR/p7</a><br>

Anyone has more information?<br>

Thanks</p>

[andrewg_ny]

<p>My understanding is that when D-Range is turned on, the camera deliberately underexposes, perhaps by as much as a stop, then applies a curve to make exposure "correct" for the ISO being used. This means that 18% grey should still be 18% gray but the overall representable range is stretched a little. By underexposing, they can treat the highlights with a more gentle roll-off at the cost of additional noise in the darker areas. So a little more noise, but handles higher contrast and highlight clipping a little better. Whatever they do is done at low level as it is not a JPEG-only feature, the RAW file appears to respond to use of this feature too. If anyone has more knowledge or a better description, I wouldn't mind hearing more either.</p>

[laur1]

<p>It cannot be a simple exposure adjustment, as that does not change the dynamic range. I've found these settings useful in situations where underexposure would have led to additional data loss in dark areas, which no curve adjustment would be able to recover, while overexposure would also have led to additional data loss in bright areas. These settings seem to indeed improve the dynamic range of the camera.<br>

Also, the D-Range settings result in reduced contrast, not in higher contrast - at least for my shots so far.<br>

The noise in dark areas may also be due to the highlight correction setting which kicks the ISO to 200. Still, why is ISO 200 helpful - wouldn't the dynamic range be the same at that setting?<br>

Are there similar features available in other brands like Canon/Nikon?</p>

[zamazal]

<p>I've read in some magazine that on the K20D it works by reading highlight pixels from the sensor sooner, before they reach their maximum saturation. I don't know whether it's true nor how they knew it.</p>

[laur1]

<p>Milan, that begins to make sense, especially combined with this comment I just found:</p>

<blockquote>

<p>The response of the sensor isn't entirely linear and so-called "highlight recover" takes advantage of the fact that the red channel is typically half as sensitive as the green channel. So typical CFA sensors really have a double exposure that allows you to add one extra stop of DR to the "theoretical" DR using so-called "highlight recovery".</p>

</blockquote>

<p>From: http://www.jayandwanda.com/photography/DRtesting/drtest.html.<br>

By monitoring the red pixels, they could get an early warning of when an area will "burn". Not sure how this works for shadows though.</p>

<p> </p>

[andrewg_ny]

By 'handles higher contrast' I meant it can render a higher-contrast scene in the available space. Another way of putting

that is that it reduces scene contrast to fit. I recognize it is probably doing more than what you could probably do if you

attempted to do the same thing I describe, possibly because they can process the full data in the expanded space? My

theory is also consistent with the elimination of ISO 100 when D-Range is turned on; essentially with D-Range + ISO 200,

the sensor is used at 100 for the highlights to provide a more gentle transition to clipping but the midtone and shadows

need boosting so that they have the luminence expected w/ISO 200. I acknowledge there is probably a little more to it

than this tho.

 

I believe some other makers have been employing similar w

tecniques but not always with quite the same goal. One approach (design trick?) I think some have used to improve high

ISO performance is to use a sensor with a base sensitivity of 200, and only make 100 available as an extended mode,

possibly via some creative processing to pull the slightly overexposed readings down to where expected at 100. These

cameras performance doesn't necessarily improve at 100 rather than 200.

[michael_kuhne]

<p>My assumption is that "expanded dynamic range" tradeoff is compressed contrast. I leave it off now, or if using I try not to forget to turn it off, unless it will be especially needed.</p>

<p>Nikon's in- camera "D-lighting" feature, similar to the "fill flash" feature I find in my PSE, is often a handy way of dealing with this problem. I would like to see that feature added to an upcoming Pentax model. </p>

<p>There was a very good example shown of what this feature can do in dpreview's test of the D80. I'd form a link, but I have never been successful trying that.</p>

<p> </p>

[jgredline]

<p>I have no idea about the techy stuff regarding this feature, but I can say that in my own findings, it works very well on the K-7 and I do not notice much difference in the K20D.</p>

[laur1]

<blockquote>

<p>My theory is also consistent with the elimination of ISO 100 when D-Range is turned on; essentially with D-Range + ISO 200, the sensor is used at 100 for the highlights to provide a more gentle transition to clipping but the midtone and shadows need boosting so that they have the luminence expected w/ISO 200.</p>

</blockquote>

<p>Andrew, note that on the K-7, the D-range option is broken in two parts: highlight correction and shadow correction (with 3 levels) - only the highlight correction setting will bump the ISO to 200 - shadow correction will work on ISO100. This contradicts your theory, because ISO 200 is needed for highlights, not for shadows. See the D-Range section in the first link I included in my initial post. From what I read about the K20D, it only has an option and maybe some levels for it and it will always bump the ISO to 200. I actually tried to look for how this setting looks in the K20D menu on dpreview, but I couldn't find a screenshot showing it.<br>

Javier, thanks for confirming that something appears to have changed in the effectiveness of this setting. I was wondering if the K20D was as effective and whether the K-7 just added finer tuning by splitting this setting in two.<br>

From my quick use of these options on the K-7, they did allow me to reduce the over/underexposed areas significantly. D-Range feels like a mini-HDR mode with the advantage that it works in one shot, so you could use it for moving subjects as well. But I'd like to understand how this works, at least to the basic level in which I understand HDR.</p>

[andrewg_ny]

<p>My experience is only with K20D, where there's just the one setting. It sounds like the shadow correction on K-7 may be a little more like the D-Range on Nikon which I've been gathering is sort of like like an in-camera 'auto fill light'. Does the shadow correction generally take effect on K-7 raw files? Any idea if only certain RAW processors respect these directives?</p>

<p>With K20D I've found it to be OK, the extra noise not so worrisome. I suspect that if you use it in brighter-light, higher-contrast situations, where ISO is likely to be fairly low anyway, the benefits may outweigh the minor noise increase. In lower-constrast, lower-light situations where you're more likely to use higher ISO, there is less need and the noise would be more intrusive. On K20D the effect seems fairly subtle, I suspect K-7's may be a little more powerful and flexible.</p>

[laur1]

<p>On the K-7, these settings can be used with RAW files, meaning that a RAW file with the expanded dynamic range is produced. Is it not the same with the K20D?</p>

[andrewg_ny]

<p>I believe it does work with RAW. It is my understanding that some special in-camera (possibly Nikon D-Lighting?) including in-camera correction of falloff or maybe CA or distortion may sometimes be implemented as directives in the RAW file for the RAW processor to render later--some of these directives may not be implemented by all RAW processors.</p>

<p>It just occurred to me that D-range = derange. :-)</p>

[zamazal]

<p>So I tried another experiment. I took three pictures of a high contrast scene with my K20D and ISO set to 800 to increase noise: 1. D-range enabled ("D-range"); 2. same exposure, D-range disabled ("0 EV"); 3. half the shutter time, D-range disabled ("-1 EV"). I converted the pictures from RAW using `dcraw -4 -H 1' and applied gamma 2.2 on the result. "D-range" and "-1 EV" were very similar as for brightness and the histogram, with "D-range" shadows slightly brighter (less clipped?). There was a difference in some highlights, but the primary difference was much worse noise in "-1 EV" shadows. "D-range" shadows were much less noisy and with no less detail. "0 EV" was brighter, I applied gamma 1.25 to both "D-range" and "0 EV" to bring their shadow brightness to about the same level as "0 EV" and then compared the results. Highlights in "0 EV" were clipped (no surprise). In shadows, "D-range" was clearly more noisy than "0 EV", but the difference was significantly lower than between "D-range" and "-1 EV".<br>

My conclusions: 1. "-1 EV" is no substitute for "D-range". 2. If "D-range" is just a trick, it's a very good trick (no idea how to reduce shadow noise significantly without visible loss of detail; I'd believe more to the double exposure hypothesis than to the plain underexposure one). 3. D-range can be safely enabled by default for my needs as clipped highlights are much more visible and ugly than somewhat increased noise in deep shadows. 4. I should disable D-range only in those low contrast scenes where shadow noise really matters and when using high ISO.</p>

[laur1]

<p>Nice experiment Milan - thanks for sharing your observations.<br>

I am drifting off topic now, but any reason why you're using dcraw? I've never been able to use it so I can reproduce the output of Pentax's software, which I find better. Not sure if I'm doing something wrong, but I get darker images with it and details are less sharp. I'm going to attach a small image with details from a photo processed with both Pentax Digital Camera Utility and dcraw (used both in the simplest way possible, without specifying any additional settings) - the crops are tiny because to see the differences more clearly, you should magnify them to something like 800%.</p><div></div>

[zamazal]

<p>Laurentiu, I use dcraw because it's free software, it's easy to use for me, it's integrated into Digikam and it runs well on my GNU/Linux machine. But it's important to keep in view that dcraw is just a raw converter performing the tricky low level processing and nothing more. Further processing, such as brightness and contrast adjustments or USM, is to be done using other tools, preferably on the 16-bit output from dcraw. I guess the Pentax software (I've never seen it) tries to create a final image. It seems that in your sample Pentax software has applied some additional operations such as clipping and/or contrast increase, saturation increase, perhaps USM and noise reduction. You should be able to receive a similar result from the dcraw 16-bit output, but you have to use an image editor as an additional tool.</p>

[laur1]

<p>Milan, your K20D should have come with a software CD that includes Pentax Photo Browser/Laboratory. PDCU is just the new name these have in the K-7 bundle. So this is free in the sense that it's included in the price of your Pentax camera.<br />I think that that tricky low level processing is significantly different between raw converters. I've done some comparison between the Pentax converter and stuff like dcraw and Raw Therapee and it looks like each one uses a different algorithm. The results are pretty significant and can't be fixed with further processing, although they may not always be that noticeable in prints. But looking at high magnification, dcraw produces different images than Pentax - and Raw Therapee is the most interesting - its fine detail looks like it's painted with a brush. Also, all three extract a different resolution from the DNG file. In general, I find the Pentax converter to produce the best colors and details (with just the settings from the camera), but I'd be interested in exploring alternatives that can provide better results.<br />I use GIMP for post processing, but it doesn't yet handle 16bit files - what free software do you use?</p>

[zamazal]

<p>Laurentiu, by free software I mean software that comes with source code and that can be freely studied, modified and distributed. AFAIK Pentax software (unlike dcraw) doesn't satisfy any of these conditions and moreover it can't run on my GNU/Linux machine at all, so it's completely useless to me. I'm not saying it's bad, it may be very good, but I simply can't run it.<br>

I use Digikam (www.digikam.org) for postprocessing and managing my photo collection, it's a very nice software. I use Krita for special editing operations the Digikam editor can't do (mostly retouching), Krita isn't that great and user friendly, but it handles 16bit colors and does the job. Indeed, GIMP isn't suitable for photo editing, if you work with 8bit colors, it may be the reason you can't get good results.<br>

Are the Pentax converter results similar to in-camera JPEGs or are they significantly different?</p>

[laur1]

<p>Forgot about your GNU/Linux setup not working with the Pentax software. Got it now.<br>

I only shoot DNG and I never did a comparison between the in-camera JPEGs and the ones I get out of DNG files. But, if you are interested, take some test shot, send me the DNG, and I'll send you the JPG or TIFF that I can get out of it with the Pentax software. Email me here or message me on facebook, if interested.<br>

I didn't say I'm not getting good results when processing 8-bit files - at least, they look fine to me. I just said I couldn't get results as good as those I get from the Pentax software when using the free tools. I think the next major release of GIMP will add 16-bit support. I'll have a look at the other apps you mentioned - thanks for the suggestions.</p>

[zamazal]

<p>I tried to develop a raw image first to a JPEG image in the camera and then using dcraw + a photo editor. The results are comparable (none of the pictures being clearly better than the other), I had to adjust brightness, increase saturation a lot and apply some USM in the photo editor.</p>

[les_lammers]

<p>Laurentiu,<br>

I am just getting started with RAW and am using Silkypix. I think the Pentax software is based on it. I like it better than Adobe Camera Raw. You might want to try a free demo of SP. I will try Raw Therapee too. The the RAW processors I have tried are a little different.</p>