In-camera RAW histograms

[frans_waterlander]

RAW pixels contain only luminosity data. Color is assigned later, based on coordinates of the pixel. If you view them in color, it means they have been processed. You could map these levels into an histogram, even color the chart to taste, but would it be of value?

The luminosity values of the photosites doesn't have to be processed to show them in a histogram; they can be directly read and presented. Yes, it would be valuable because the histogram would tell you exactly if you are close to saturation or go beyond and in what color channels.

The real breakthrough may be instituting 32 bit floating point data in image files. Depending on the way it is implemented, 32F can virtually eliminate overexposure

Overexposure occurs when more photons hit the photosite than is needed to fill the well. How has that anything to do with how the number is represented?

Clipping is an artifact of the interpretation, not the underlying data.

Clipping can happen in both.

[samstevens]

Omnipotence is not knowing how everything is done. It’s just doing it.

—Alan Watts

Knowledge is power.

—Francis Bacon

Take your pick.

 

OR

To attain knowledge, add things every day. To attain wisdom, remove things every day.

—Lao Tzu

Good luck with your photos ...

[andylynn]

Really? With ETTR you capture the maximum number of steps in the brightness range, resulting in the smallest difference between steps, minimizing the chance of visible posterization, especially important when you do post-capture editing. If you want me to, I can give you an example with some numbers thrown in to illustrate this. You also might want to read MIchael Reichmans's article: Expose Right - Luminous Landscape

All respect to Michael, but that article should be put out to pasture. He wrote it in 2003 for people using a 2002 model camera. In those days a camera could produce noise at low iso in daylight and you had to use every trick in the book to get as much out of the sensor as possible. I don’t understand why people still treat it as a necessity that you shove the histogram as far to the right as possible. If you have a decent camera from the last decade or so, the only reason to do that is if you are very concerned about shadows in a high contrast scene. Otherwise, your best bet is to expose for the shot you want, with adjustments when needed to keep from blowing the highlights. (Or just say to hell with convention and blow the highlights.)

[frans_waterlander]

All respect to Michael, but that article should be put out to pasture.

The science behind the article is as valid today as it was then. Noise may be less of an issue now, but by your own admission it still is a factor, for instance in high contrast scenes. ETTR also helps reduce posterization and that issue is the same for older or newer cameras.

[Matthew Currie]

I shy away from solutions that seem more theoretical than practical. If you can't see a difference in the form you're outputting to, it seems unecessary.

 

But in answer to the original question, I believe there is a way you can, at least, simulate a raw histogram in the camera, but doing so makes it very difficult to evaluate an image in review, because it requires a substantial fudging of white balance, as there's no way to separate the displayed histogram from the displayed JPG.

 

I have not tried it, but if one really wants to give it a whirl, I suggest you look up "UNIWB" to start with. This is a tweak of white balance that drastically increases the green in the image replay. Further adjustment can then be made with a program called "rawdigger." The whole procedure looks pretty cumbersome, and the page I found that outlines it suggests that if you really need to expose to the right, it would probably be easier to spot meter bright spots and use the zone system. But anyway, here's a page you can start on, if you feel so inclined, though others may also exist.

 

Making the in-camera histogram closely represent the raw histogram

 

Needless to say, if you use a raw reading program like the Nikon software, which uses the JPG as a starting point, I imagine that Uniwb would be very inconvenient.

[andylynn]

The science behind the article is as valid today as it was then. Noise may be less of an issue now, but by your own admission it still is a factor, for instance in high contrast scenes. ETTR also helps reduce posterization and that issue is the same for older or newer cameras.

Maybe if you process the photo beyond reason. If you really do find yourself in a situation where Michael’s ideas about number of levels on the right side of the histogram would help (as opposed to, say, Nasim Mansurov’s version of ETTR, which can be summarized as “Just don’t blow the highlights and you’re good to go,” or mine, which is “Sometimes use the high exposure from a bracket with the highlights pulled down if you really want to prioritize shadow detail”) I’d love to see examples.

 

But you’re talking about needing to ETTR a Nikon to avoid posterization, which makes me think your problem actually lies elsewhere. I haven’t run into such problems since I had a D60 and was a complete noob at raw file handling. What sort of scenes are you shooting, with what camera, where you’re seeing posterization? And are you sure it’s being caused by the raw file and not, say, display settings? Can you post some examples?

[paddler4]

ETTR also helps reduce posterization and that issue is the same for older or newer cameras.

 

I still don't understand this. You referred to having fewer "steps" if you don't ETTR. Where you place the histogram (what EV you select) affects the number of steps only if one of the placements clips. Otherwise, it seems to me, the issue is only noise. Yes, noise is lower if you ETTR. The physics behind that are straightforward. With modern cameras, the effect isn't always noticeable, but it's there.

 

Re posterization: I can't recall the last time I noticed any posterization in any of my images. I don't shoot JPEGs, and even fairly extreme edits of images rendered in 16-bit ProPhoto don't show posterization, even if they are noisy.

[frans_waterlander]

andylynn and paddler4,

 

My interest in having an in-camera RAW RGB histogram is not because I'm running into a certain problem right now. Do I have noise problems with my Nikon D70? You bet! Have I run into posterization issues? Only a few times. However, all I get in-camera right now is a jpeg histogram with all its serious limitations and misinformation. I'd like a RAW RGB histogram and wonder why in heaven's name the only camera that has this feature costs a whopping $52,000.

[samstevens]

My interest in having an in-camera RAW RGB histogram is not because I'm running into a certain problem right now.

Translation: None of us is going to get the examples we requested. Which tells me, perhaps wrongly but nonetheless somewhat securely, that this is merely a theoretical trolling expedition.

[andylynn]

Okay, now I understand a bit better. A D70 is difficult, it's from the era when Reichmann's article still had practical application and gives you probably worse data than my D60 did. You've got 12-bit raw and a CCD sensor that maxes out at 1600 ISO.

 

Unfortunately, nobody was really doing Nikon hacking back then like the Magic Lantern people did for Canon. There is a Nikon firmware hacks group but they don't have anything for a D70 or that does raw histograms or would add a histogram by color to older models. I very much doubt that you'll ever find an opportunity to add histogram improvements. I think you have a few good options if your goal is to cure (or at least reduce) posterizing:

 

1. Upgrade to a newer Nikon DX camera. Even a D7000, which is very reasonable on the used market, would produce much better imaging data than what you're using. It has that 16 MP Sony CMOS sensor that everybody was using for years, 14-bit raw, and color histograms (so you can get more useful feedback if you're still not satisfied and want to try the ETTR route). A D7200 from a reputable source is about $500 and it's even better. With these cameras you can just shoot, and not have to fiddle with things like hacking your exposure.

 

2. Sidegrade to a Canon with a version of Magic Lantern firmware available that does what you want.

 

3. Ditch all that and go mirrorless. The raw file you can get from an APSC Sony or Fuji these days beats the pants off of even an FX DSLR from a few years back. It's to the point where I only pull out my D800 if I want to use the 105mm macro of the 24-70, because the X-H1 beats it and weighs less. If you want to stay in the Nikon system, a Z50 should be just as good and works with the FTZ adapter, I just don't have experience with it so I can't vouch for it.

[markwyatt]

I think what might be useful is a largely unprocessed cumulative histogram of relative pixel readout (voltage, amperage, whatever) ordered on the x-axis from minimum ("0" on y-axis, i.e., no reading) to max ("1" on y-axis, i.e., clipped). Then you could see how much of the dynamic range of your sensor array is being used in a given range. By "largely unprocessed) I man unprocessed other than dividing each pixel's value by the largest value the pixel can read before clipping.

[paddler4]

However, all I get in-camera right now is a jpeg histogram with all its serious limitations and misinformation. I'd like a RAW RGB histogram and wonder why in heaven's name the only camera that has this feature costs a whopping $52,000.

 

I have no idea what capabilities the D70 has. However, with my Canon equipment, if I set the picture style to faithful and set the camera to show all three histograms (because you can clip one channel even if the averaged luminance histogram looks fine, I never run into appreciable problems. That is, it is close enough to what I get when I render the images in my raw processor (Lightroom). It tells me whether I have substantial clipping and if I have missed exposure by a sizeable amount. That's all I need it for. I'm guessing that if you have a reasonably neutral setting on your camera and the ability to show separate R, G, and B histograms, using them will get you most of what you need.

[frans_waterlander]

Translation: None of us is going to get the examples we requested. Which tells me, perhaps wrongly but nonetheless somewhat securely, that this is merely a theoretical trolling expedition.

There is a big difference between theoretical (concerned with or involving the theory of a subject or area of study rather than its practical application) and technical (relating to a particular subject, art, or craft, or its techniques). If looking for better tools (histograms) is trolling, yes, then I'm guilty as charged. If you can't envision what I'm talking about without seeing an example, then I have to wonder if you can contribute in a constructive way.

[frans_waterlander]

Okay, now I understand a bit better.

Wish I could justify buying a D7500 right now, but even then I would want an in-camera RAW RGB histogram.

[frans_waterlander]

I think what might be useful is a largely unprocessed cumulative histogram of relative pixel readout (voltage, amperage, whatever) ordered on the x-axis from minimum ("0" on y-axis, i.e., no reading) to max ("1" on y-axis, i.e., clipped). Then you could see how much of the dynamic range of your sensor array is being used in a given range. By "largely unprocessed) I man unprocessed other than dividing each pixel's value by the largest value the pixel can read before clipping.

Mark, you are basically describing a RAW histogram.

[frans_waterlander]

I have no idea what capabilities the D70 has. However, with my Canon equipment, if I set the picture style to faithful and set the camera to show all three histograms (because you can clip one channel even if the averaged luminance histogram looks fine, I never run into appreciable problems. That is, it is close enough to what I get when I render the images in my raw processor (Lightroom). It tells me whether I have substantial clipping and if I have missed exposure by a sizeable amount. That's all I need it for. I'm guessing that if you have a reasonably neutral setting on your camera and the ability to show separate R, G, and B histograms, using them will get you most of what you need.

Close enough and getting most of what you need would be a lot closer and a lot more if you had RAW histograms.

[andylynn]

Wish I could justify buying a D7500 right now, but even then I would want an in-camera RAW RGB histogram.

A D7500 is worse than a D7200. But with either one, you would not see these problems. A raw histogram wouldn’t be needed.

Edited January 7, 2020 by andylynn

[markwyatt]

Mark, you are basically describing a RAW histogram.

 

What I am quoting is much more basic and requires a lot less processing, plus I posited a cumulative histogram.

[paddler4]

Close enough and getting most of what you need would be a lot closer and a lot more if you had RAW histograms.

 

 

my point is that if what you have can do what you need -- mine can -- there isn't much to be gained by worrying about what you can't have.

[frans_waterlander]

A D7500 is worse than a D7200.

If you mean dynamic range, then they are very close: DxOMark DR's are 14.0 and 14.6.

[frans_waterlander]

a cumulative histogram

I'm not sure what you mean by that; how would it differ from usual histograms, apart from the scale, yours going from 0 to1, usual ones going from 0 to whatever value?

[samstevens]

There is a big difference between theoretical (concerned with or involving the theory of a subject or area of study rather than its practical application) and technical (relating to a particular subject, art, or craft, or its techniques).

There is, indeed. But technical questions, which yours is, can be either theoretical or practical. Yours is a theoretical approach.

 

Let's consider hammers. Someone wanting the ideal hammer is speaking about a technical matter in theoretical terms. Someone wanting a hammer that can do what no hammer is yet able to do is also speaking about a technical matter in theoretical terms, though there's a practical aspect to wanting a hammer that can do something no hammer can yet do? The practical aspect would come into play, however, when the carpenter wanting this hammer no one's yet wanted to or been able to create can make evident in real terms what use he would have for the features of the hammer he's proposing. So, something like showing those he's trying to convince of the need for this hammer a particular nail in a particular situation that can't be dealt with by the current crop of available hammers and how the hammer he's proposing could be effective in this situation would be a practical approach.

 

Sometimes the most constructive way to contribute is not necessarily to provide an answer but rather to point to the lack of clarity and actual purpose of the question or proposal.

 

In any case, you'll be delighted to hear I've spent about all the energy on this I'm going to and will leave you to it. You can determine in due course just how constructive the conversation will have been for you ...

[frans_waterlander]

Yours is a theoretical approach.

Not theoretical at all. When I figure out roughly how much headroom I have based on the jpeg histogram, then that works for many shooting conditions, but not all. If, for instance, I use an IR filter, all bets are off and I would need to figure out another value for the headroom when using my IR filter. In my particular case, I have about 0.7 stops of headroom for most shooting conditions, but about minus 1 stop when using IR. IF I had in-camera RAW RGB histograms, then I would know right after my first shot how I need to adjust exposure.

While other shooting conditions might be less taxing, they all would benefit to some extent.

[andylynn]

If you mean dynamic range, then they are very close: DxOMark DR's are 14.0 and 14.6.

The sensor is fine, but the D7500 lacks AI lens meter coupling, dual card slots and vertical grip option.

[Matthew Currie]

I doubt there will be a good chance of getting a practical raw histogram in the camera. But might it be possible to set up various difficult exposures as samples, and then get raw histograms from Rawdigger or the like? You would then have at least a guideline for various conditions. How finely you can cut it will depend, anyway, on what settings your camera allows. If you have 1/3 stop increments, that's as close as you can come anyway.

 

I do think that much of your problem would be solved in a practical, if not theoretical, way if you could save up some cash for, say, a D7200, or even a D7100. Both have good dynamic range, and color histograms, as well as a highlights "blinkie" screen with single color selections. I suspect that if you ETTR for JPG on one of these, the difference between that and the raw would be hard to spot. But if you're looking for the very last drop of information, you could also simply bracket. Both these cameras allow a one-direction sequence of three shots, with the metered shot as #1, with increments starting at 1/3 stop. The D7100 allows up to 5 shots in a bracket, two under and two over. The D7200, though it also allows only three in a single direction, can also do as many as 9 ( first the metered shot, then four minus, then four plus). So if you were willing simply to throw away four, you could get the metered shot, and four + exposures, one of which would be pretty sure to hit the edge of the theoretical raw limit.

 

Or if you were adventurous, you could set your initial exposure above the JPG right limit, and bracket both sides of it. Either way, with a nine shot bracket, and increments from 1/3 stop to 2 stops, I think you'd be pretty likely to hit the ideal exposure.