Illustration of Limitations of Nikon Compressed Raw Format?

[cjfraser]

Over the past few months, a number of people have asked if there is

any difference in quality between NEF and compressed NEF formats, and

several have suggested that they've yet to see any practical examples

of the difference. I think I've come across several examples recently,

which show that, as Thom Hogan and others suggested, compressed NEF

results in a limited ability to post-process highlight detail.

 

Attached is a 100% crop of a D2X compressed raw file shot at ISO 800.

The crop shows the edge of a person's arm lying against an

ivory-colored pillow. Notice that the bright part contains both ivory

colored regions and what appear to be blown-out highlights. In Nikon

Capture 4.4, if I reduce the "Exp Comp" by 0.67 EV, the blown-out

highlights vanish -- that is, in the "highlights" view, the image goes

totally black. According to the "Information" palette, the RGB value

of the washed-out areas seen in this crop is R228, G231, B228 --

nowhere near blown out.

 

However, whether in Capture or in PS-CS2, those areas continue to

*look* blown out, and no processing technique I'm aware of can bring

out any color or texture there. (I am not a PS expert, so if you know

of such a technique, I'd be grateful to hear about it.) As a color

photo, this capture is effectively ruined (maybe it might still work

as a B&W, I'm not sure).

 

I've recently processed at least a half-dozen images with similar

problems.

 

My hypothesis is that there was nothing wrong with my original

exposure, and that if I'd used uncompressed NEF, the image would have

recoverable highlight detail. But because I shot this in compressed

NEF, those highlights really are blown, even though the average RGB

value is only 229.

 

If that's correct, then the lesson is: Use compressed NEF only for

low-contrast images without bright highlights.

 

Do you agree that NEF compression explains what's going on here, or is

there another explanation?<div></div>

[arthuryeo]

I think I experienced loss of partial pixels when I look a the crop at 100% of some shots I have taken with compressed RAW.

 

I have learned my lesson well.

[cjfraser]

Arthur, in principle the loss should be confined to the highlights -- is that what you've found? Compressed NEF is a big convenience, but I'm turning it off and investing in more storage space and memory for my computer.

[fourfa]

"My hypothesis is that there was nothing wrong with my original exposure, and that if I'd used uncompressed NEF, the image would have recoverable highlight detail.

 

Do you agree that NEF compression explains what's going on here, or is there another explanation?"

 

What? Shoot it again uncompressed, and find out! Likely to yield a better answer than asking us.

[lex_jenkins]

Have you compared highlights in compressed vs. uncompressed NEFs at ISO 200? I would think that the limited dynamic range at 800 would make it even more important to shoot uncompressed NEFs when maximum quality is needed.

 

I've tried various highlight recovery utilities but after a certain point all they do is make the highlights gray. They can only recover just so much detail - if none was captured it can't be created.

 

Since I don't have a high powered PC my main reason for avoiding compressed NEFs, after a couple of experiments, is because they take for-freakin'-ever to open. I tried to time it but my watch died of old age.

 

It might not matter with a P4 and 2 GB RAM, but on a P3 with 128 MB RAM it's painful. (And forget about adding RAM - this is only a backup PC and adding RAM would cost more than buying a new low end P4.)

[jc5066]

With out an example of an uncompressed photo of the same image and same conditions, your just guessing.

 

Set up an example and a couple test shots, then it won't be a bunch of people guessing whats going on.

[david_moore13]

Would you mind posting the original raw file somewhere? We can examine it to see if the detail in the pillow was actually blown out or not. If it was truly blown out, the only solution would have been to lower your exposure compensation at the time the shot was taken.

[cjfraser]

Thanks for the comments. The friend in the photo would prefer that I not post the original file. I could post one with a similar problem, though. But in any case I can confirm that when I move the "Information" pointer over the area shown in the crop above, the RGB values are all around 228-231, not 255. So the RGB tells me the highlights are not blown, but they look blown and reducing the EV doesn't help.

 

Lex may be right about the impact of high/low ISO. I just did a few test shots at home on a white towel (detailed surface texture) placed next to a bright window to compare highlight recovery from NEF and compressed NEF at ISO 100. In a shot intentionally overexposed by 1.7 EV, highlight recovery from both was impressive. Areas for which RGB were initially all 255 could be brought down to 220-230 and looked good. I'll repeat the test shots at ISO 400 and 800 another day (don't have time at the moment). But it could be that compressed nef is fine at low ISO and a difference emerges only at high ISO (if that's what's really going on here).

[cjfraser]

Sorry, my previous post was confused. Let me correct. The highlights in the *original capture* shot at ISO 800 really are blown: they're at 255. Adjustment in post-processing indicates that they have been recovered: The RGB value given for those pixels is no longer 255, it's about 230. Visually, however, there's no data there. So the oddity here is that Nikon Capture indicates those pixels are not at 255, but in fact they do seem blown out.

 

The question I'll look into further is whether, in a similar shot at ISO 800, using uncompressed NEF instead of compressed will make a difference. I'm guessing it probably will. I'll do some test shots later to see.

[ilkka_nissila]

If you have pixels at 255,255,255 and then you adjust the brightness of the image in post-processing, you will of course get darker values as you make stronger adjustments. However, there is no detail as the detail was clipped in the original exposure.

[cjfraser]

Thanks Ilkka. What confuses me is this -- perhaps this is a Dumb Question but here goes: Typically, when a channel is clipped, the raw processing software (in my case, Nikon Capture) shows it as clipped even if we reduce the EV by -2. That is, in the highlight view, the highlights are white and they stay white even after the EV is adjusted. In the image I posted about, and in a few other images, after EV adjustment, the highlight view is fully black. Under normal circumstances, this would indicate that the highlights have been recovered; the channels were not totally clipped after all. But in these images, visually the highlights are still blown out. So Nikon Capture indicates that they have been recovered, but in fact they have not, because it turns out there really is no data there.

 

I have attached an example. This is a 100% crop of a highlight in a compressed NEF shot at ISO 1600 with an exposure comp adjustment of -0.6 EV in post-processing. (Why ISO 1600? It was night, and I didn't have a tripod with me, but I saw an interesting color and pattern and thought I'd take a shot, not really expecting it to succeed.) After the adjustment, the highlight view in Nikon Capture is fully black and the histogram does not touch the right side. Yet visually the highlights are blown out. So my questions are (1) why is there this discrepancy between what Capture indicates and what we see, and (2) is it related to using compressed NEF instead of uncompressed NEF? I think probably it is, though high ISO is probably a factor, too, as Lex suggested.<div></div>

[cjfraser]

~ "If it was truly blown out, the only solution would have been to lower your exposure compensation at the time the shot was taken." ~

 

Thanks, David. Of course you're right: at some point, blown out is blown out -- there's just no data there to be recovered. But in the test shots I did earlier today at ISO 100, even though the RGB channels were all at 255, the highlights were pretty much fully recoverable, presumably because the overexposure was less than 2 stops. That's why I expected to recover detail in my original shot -- I thought overexposing the highlights by only half a stop wouldn't give me a problem.

[walterh]

Chris you confuse me -> going half an f-stop over 255 is 255. going two stops over 255 is ? exactly: 255. Did you talk about this or what was your point?

 

If your data are 255,255,255 and your software "recovers" the highlights and lets you know that you got e.g 234 in each channel it simply pulls the values down. I can follow this. but if you get soething like 234,212,351 then this is a miracle to me that only the software developer can answer.

 

There is a difference between lossless compression and "almost" lossless compression like e.g. NEF. The difference is that we know what we get in the first case. In the second case we only know if the software producer lets us know. Nikon choose to hide this information from us. So we can just speculate. The best way to speculate is to shoot identical images with a camera that allows both lossless and "almost lossless" images. We then can try to trace the loss of the almost lossless compression. Anything else like shooting one day an apple and the next day an orange with different conditions and both only with compression make sno sense at all. Of course it can lead to longer threads than doing it the right way :-P

[cjfraser]

~ "Of course it can lead to longer threads than doing it the right way :-P" ~

 

Thanks, Walter, that one made me laugh out loud! My apologies for being so clumsy with all this, in both understanding and expression. My reference to a half stop overexposed was an awkward way of saying that pulling the highlights down about 1/2 to 2/3's of an EV in post-processing seemed to produce a satisfactory image.

 

As to the apples and oranges, I shot a pair of apples today at ISO 100 and found they tasted about the same. I'll try comparing a pair of oranges at ISO 800 tomorrow.

[walterh]

Chris no harm ment - if you shoot the apples and the oranges with and without compression under identical conditions that would be something everybody (well almost) would appreciate. My D70 will only allow compressed NEF - so my apples and oranges will not help at all .-P

[john schroeder]

All this talk about Apples and Oranges is makeing me hungry for Waldorf salad. I've never noticed a problem with highlights on my D70 while shooting NEF. I don't have anything to compare it to though.