Linear Processing of RAW Image Files .

[roger_smith4]

Frans, do you not like the linear result? What do you not like about it?

 

Did you see the result I did using Lightroom which would be "normal" processing?

[bernardwest]

Hi Luis... I haven't read every last thing posted in here, so I apologize if what I mention is already explained above.

 

I've got a couple of points of confusion. 1. I was of the understanding that gamma is applied towards the end of the image processing pipeline. This is based on my recollections of similar discussions over at Luminous Landscape. Unfortunately I haven't been posting there much for a while now, and my memory may be a bit fuzzy. Tim, I know you hang out there a bit, what is your understanding of this issue?

2. You say you are importing a 16bit linear image to photoshop, and then doing your tonal adjustments there. It's my understanding that Lightroom/ACR/DCRAW and many others operate with floating point calculations (when working on a raw file), but photoshop (at least up to the early CS versions) doesn't calculate this way. Hence, you are better off doing your adjustments in your raw converter than doing so in photoshop and suffering rounding errors. This might explain why Rodger was able to get a very good rendering also of your raw file.

 

Let me know if I've got some of that wrong, or if I missed something from earlier on in the discussion. Cheers, Bernie.

[frans_waterlander]

Roger,

 

It's not an issue of what I like or dislike; I haven't seen the subject so I can't judge whether the "normal" processed image or the linear processed image (the first two side-by-side images that Luis posted) is closer to reality and that also is not the issue. The issue I have is that those images are so radically different in colors and tonality that it looks to me like either Luis' "normal" process or his linear process or both are severely flawed.

[blumesan]

Luis,

 

I have watched your linear processing video. While it makes some things clear, it leaves a couple of questions in my mind:

 

 

1. Apparently the Mac version of dcraw-X does not allow you to choose an Output Colorspace for the converted file, while the MS version does so (parameter -o [0-5]). What is the colorspace tag of the lineal file produced by dcraw-X, or is it simply untagged (raw or no profile)?

 

 

2. I note from your screen shot of Photoshop Color settings that you have selected "preserve embedded profile" and that you have not checked to request a warning when the file is untagged (missing profile). I note that you only change the PS working space to Lineal-ProPhoto (gamma=1.0) but you do nothing to assign this space to the file just opened. Is this because the file has no embedded profile and the default behavior of your setup is to assign the working space profile?? I am guessing this is the case because, with a file having an embedded profile already open, simply changing the PS working color space would not alter the appearance of the file.

[stephen_werst2]

Luis,

 

Through your testing and experience you’ve established that you obtain better results when you convert raw images

using dcRAW-X’s 16-bit Output option rather than 8-bit. This is not entirely surprising if you look beneath the

covers a bit and read the FAQ for dcraw.c that dcRAW-X references as the basis for the program. Specifically, the

16-bit options store the image using a color space containing a gamma of 1.0 (i.e. a linear file), though the

color space is not defined with an embedded profile. Presumably it would just use the native color space of the

camera sensor given the comment on the dcRAW-X page that “the right way to do it, is to assign a custom camera

color profile to the linear file.” The 8-bit option appears to do additional processing that would explain the

differences between the images you posted above, but this is just a hypothesis that I’m unable to test as I can’t

run the program.

 

You then extrapolate your finding to ask the more general question about whether the advanced technique of using

“Linear processing workflows” is preferred by others and later assert that “when utmost image quality matters,

linear processing is the way to go”. This is quite a leap. Just to be clear, what you’re asking here is whether

it’s better to digitally represent an image using a color space with a gamma of 1.0 vs. a color space with a

gamma of something other than 1.0 – 2.2 being the most common.

 

To explore this topic requires an understanding that a color space is basically a way of interpreting the numbers

in an image file to a specific color as viewed by an observer. It is analogous to a ruler. An engineering drawing

of an object with associated numerical dimensions would be meaningless without an indication of the units of

measure and a standard providing physical meaning to the units. In other words, to build what the diagram is

depicting requires knowledge that the numbers represent inches, mm, etc. and a standard bar (i.e. ruler) with

markings showing the distance of a standard inch, mm, etc. A color space provides the equivalent for the numbers

in an image file, so asking if one is better than another is analogous to asking whether one ruler is better than

another. It depends.

 

You mention that sRGB is “the most restrictive color space”. However, if the image being stored doesn’t contain

any colors outside of the gamut that can be represented by sRGB, then not only is it sufficient but is actually

preferred to a wider color space because it would be able to record smaller variations in color assuming the same

number of bits per pixel/color were used to record the data. In other words, if the 256 possible values of red

don’t have to cover as wide of a range of reds as within a wider color space, then more subtle variations in red

may be recorded in the file.

 

In this example the gamma of the color space defines how “equidistant” each recorded value of red is from its

neighbor. In other words, are the 256 recorded points established after the same increase in intensity of red, or

does the change in intensity vary between points from one end of the color definition to the other. How these

values get mapped to color intensities in the recorded file is what you’re essentially asking about.

 

My experience with recording scanned film using a gamma of 1.0 was that the level of detail in the shadows was

horrible. This seems also to be evident in the file you show being processed in the first movie you’ve uploaded.

After defining the color space to have a gamma of 1.0 the dark sections are very splotchy – something very

similar to what I saw when experimenting . After you apply the auto tone correction the darks are better but

still speckled. You refer to it as noise but the effect appears to impact a larger group of pixels than what

would be seen with sensor noise. It’s hard to tell without the file. Have you had someone process it using

another raw converter and compared the amount of detail that can be extracted from the shadows? For this image

you’re okay masking this effect using a levels adjustment to pull more pixels to black, and perhaps this is

acceptable for most of your images, but this seems to be a required step of the “linear workflow”.

 

[william l. palminteri]

Oh dear....... First, the luma space color space of all films and ccd sensors are far less inclusive than the human

eye, so the data is 'compressed' from the outset. Then, the various RAW formats compress that data even more. I

know most people don't want to hear that, but to my knowledge, there are no true 16 bit RAW file types.

If that has changed, I'm sure a poster here will update me.

 

Bill P.

[roger_smith4]

"It's not an issue of what I like or dislike; I haven't seen the subject so I can't judge whether the "normal" processed image or the linear processed image (the first two side-by-side images that Luis posted) is closer to reality and that also is not the issue. The issue I have is that those images are so radically different in colors and tonality that it looks to me like either Luis' "normal" process or his linear process or both are severely flawed."

 

This doesn't make much sense to me. To me a raw file is like a color negative (something I spent years balancing before switching to digital). Sure you can do a relatively neutral proof sheet for color negatives but there are no absolute correct renditions as you can map the tremendous brightness range in many ways for viewing in a monitor or print. Ditto for raw files.

 

I think the goal of raw conversions is to be able to render files in a way that matches your vision of the scene. Whether or not it is "real" is irrelevant as no versions are "real" and literal conversions often look far flatter and duller than the original scene. I personally like conversions that look realistic and don't have viewers screaming "fake." I don't think Luis' linear renderings are literal renditions of the scene at all. I don't have a problem with that. You can download the raw file and see what it looks like for yourself.

 

A couple of things I did with my Lightroom (ACR) rendering- I started by reducing exposure until artifacts began to show up around bright edges, and then restored brightness with curves. I increased black point and then used fill light to bring up shadows. I used the highlights slider to impart a warm cast to the highlights and make the flower closer to Luis' version.

[Tim_Lookingbill]

Well my AT&T DSL Motorolla modem just died yesterday and I'm using my local library's Dell machines that won't let me view the video tutorials. All the systems here are slow as hell for some reason and waiting 18 minutes to download would take up my allotted time allowed by the library.

 

I'm still with Frans and now William on this. The two different linear raw output samples from two converters confirms what William stated.

 

Besides that notice in Luis's previous thread of his PS Color Settings screenshots showing the differences between assigning sRGB, ProPhoto RGB and finally 1.0 gamma ProPhoto RGB to the dcRAW-X image and see that he got similar results in color (magenta highlight on the tree trunk) but not density contrast than my linear ProPhoto RGB version which required a strong correction curve for mine where his did not. Same OS, Mac computer, calibrated display and raw converter settings but different results.This is assuming he assigned his monitor profile and converted to sRGB for posting here. If he didn't then this might explain the density differences.

 

But now look at his final image which is noticebly processed with an added vignette, the tree trunk is nolonger magenta, the leaves are a different green hue and the highlighted flower petals have lost all the fucia pink shown in the assigned linear ProPhoto RGB screenshot. It's this additional processing that on top of the differences in his screenshot comparison that is not explained. And if it is explained in the video then I stand corrected.

 

But the hole point of this better way of processing linearly from a pro photographer's standpoint is to determine if it is faster requiring less processing to get desired and better results and is it consistent and predictable. The answer from what's been demonstrated here is NO.

 

And those that get blotchy shadows assigning linear ProPhoto RGB to the dcRAW-X linear file, to get rid of it just convert to regular ProPhoto RGB or any other working space of choice, but then that defeats the "don't apply gamma to the untouched linear data" way of doing things. I can't remember from past Adobe forum discussions why this occurs but I think it has something to do with assigning profiles with unknown black point and Photoshop's slope delimiting algorithm screwing up previews. I could be wrong.

[luis-a-guevara]

<p>This seems to be the root cause of so much disension . I hope it helps because I cannot keep up reacting individually to everybodys

posts:</p>

<p>Gamma by definition is the slope of a curve.The problem is that Curves , unlike Straight Lines ,do not have ONE Gamma Value that

defines them , they have an infinite number of them . One for each point along the curve , so to define a Gamma value for a CURVE the

Gamma of the Midpoint is used , because ,in the case of human vision we have better vision at the Mid Tones than we do at Highlights or

Shadows.</p>

<p>Because of this, Color Balance is also done at the Mid point , which ,on an image that has been Gamma Encoded ,corresponds

roughly

with its average midpoint in the Histogram .consequently COLOR BALANCE BECOMES OPTIMUM AT THE MID TONES AND

DETERIORATES TOWARDS THE WHITE AND BLACK POINTs . Dont confuse this whith White point , which is just the neutrality of the

whites.</p>

<p>For the same reason Tone Separation in GAMMA ENCODED images is LARGEST around THE MIDPOINT AND COMPRESSES

TOWARDS THE END POINTS. This gives posterization of the Highlights and Shadows .</p>

<p>LINEAR GAMMA , instead , is defined by a Straight Line of uniform, constant slope , so it does not introduce any compression of

tones , but leaves the tone distribution at the mercy of Binary numbers , that are strongly compressed at low Bits values and very spread

apart at the high bit values . The end result is excellent separation of the high tones with possible posterization of the low tones.</p>

<p>Neither of this processes is better per se , they are just tools in the toolbox of the resourceful artist . If your image has delicate

Highlights that you want to preserve or enhance ,then Linear processing will be your best tool , but if it is a dramatic low key image with

abundance of low tones , definitely Gamma encoded RAW processing is called for .</p>

<p>If your image has Both (HDR)Delicate , expresive highlights and interesting detail in the Shadows, then Double Processing ,

combining Linearly generated images to non linear ones is what it needs to be done.</p>

<p><img src="http://www.sigmacumlaude.com/6548803-md.jpg" width="679" height="354" /></p><BR><P>LUIS A GUEVARA</P>

[frans_waterlander]

"The problem is that Curves , unlike Straight Lines ,do not have ONE Gamma Value that defines them , they have an infinite number of them . One for each point along the curve , so to define a Gamma value for a CURVE the Gamma of the Midpoint is used"

 

Nothing could be farther from the truth for the type of curves we are discussing here, where in your image editor you pull the straight-line curve up or down at one point only. Mathematically this new curve is defined as: new brightness value = old brightness value x p to the power of gamma, where p is the percentage of where you are on the tonality scale (anywhere between 0 and 100%). So the gamma involved applies to any and all points on the curve.

 

So all your reasoning about color balance deteriorating and tone separation compression is incorrect.

 

And Luis, you still haven't answered my question as to why your side-by-side examples are so drastically different.

[frans_waterlander]

Roger,

 

You totally miss my point; I don't care about realistic one bit in this discussion. My point is that the two side-by-side examples are so radically different that one or both processes are introducing horrendous tonality and color shifts. I don't need linear processing to do that: Photoshop is chockful of tools to do that. So what the heck is the advantage of linear processing? Sorry to say, but Luis hasn't answered that question or demonstrated it with meaningful side-by-side comparisons.

[roger_smith4]

" I don't need linear processing to do that: Photoshop is chockful of tools to do that. So what the heck is the advantage of linear processing? "

 

I am impressed by the look of Luis' first image and in the delicate tones of the flower in the second. I don't care if the images have tonal and color shifts (Relative to what? None of us have seen the original scene!) and as you say there are a variety of tools which produce different looks. This is one more. If you think the approach is worthless and do not think any of the examples have any redeeming value why are you posting in this thread to begin with? You clearly know all you need to already so why waste your time?

 

I'd like to see your conversion of the file Luis posted for download using your preferred tools and techniques. What do you think of my ACR/Lightroom conversion?

[frans_waterlander]

Roger,

 

I like to understand how exactly this linear process is different/better than the tools already available and I haven't seen any of that clearly defined/demonstrated. Luis claims a number of specific advantages/characteristics of this process over "normal" processing but hasn't delivered on demonstrating that.

[tim_brown26]

A quote from the beginning of the thread.

 

"Hi Frans . Those curves that you are refering to are really just describing the behavior of the Midtones of the image, in fact the end points are fixed and your options are limited to defining the Slope of the curve at the Mid Point , which as you described , by default is lineal , meaning no transformation takes place."

 

Not true. The end points of a Curves adjustment layer are movable just like any point you make along the curve. They are also not restricted to midtones, each section of the Curves layer corresponds to shadows, midtones and highlights.

[mendel_leisk]

The ongoing unsettling thought I have: the majority of images processed in this manner are going to look way too dark. I gather that's a given, and for the sake of this exercise is ok. But from a practical point of view, I don't think this is, well, practical ;)

[frans_waterlander]

Well, it looks more and more like a "new toy syndrome". The advantages, if any, haven't been clearly described or demonstrated. The only sure things as far as I can see are that linear processing makes your images look way too dark (I would have thought that somewhere later in the linear process you would apply a gamma of 1/2.2 or 0.45 to compensate, but that's apparently too sophisticated), that different people get different results and that there is a lot of confusion and misinformation.

 

Oh well, maybe somebody will clarify it all and make sense out of it, but for now it looks like a very unscientific "look what I just did" exercise that could easily be done with your regular image editing software (but don't ask me why you would want to do it in the first place) at no additional cost and no need to try to figure out a new piece of software (and apparently that is not easy given the comments here).

[william l. palminteri]

Frans, I couldn't agree more!

 

You said...."Well, it looks more and more like a "new toy syndrome"..... and...." Oh well, maybe somebody will

clarify it all and make sense out of it, but for now it looks like a very unscientific "look what I just did" exercise that

could easily be done with your regular image editing software (but don't ask me why you would want to do it in the

first place)....."

 

That pretty much sums it up for me. It's easier for me to get the lighting and exposure right in the first place. If that's

not possible, a quick "tune up" in a small editing program should do the trick. If I needed to do more than that, it

would be time for me to re-evaluate my artistic skills!

 

Bill P.

[luis-a-guevara]

Please dont blame me for your failure to apply this process properly . The instructions , quoted here from my tutorial

posted earlier in this Thread , and from the first video , clearly say:

 

"SO , WHAT IS THIS LINEAR PROCESSING AGAIN?

Simple. I am sure that you will agree that is simple. Linear processing is avoiding the introduction of this Inverse

Gamma at the point where it hurts the most ,to reintroduce later, where it hurts the least ,as one of the last steps of the

Image Optimization , rather than at the Image Creation. A big substantial Difference.

 

Of course we must reintroduce it , to achieve again the necessary overall System Gamma of 1.0. If we don't , we will

end up with an image that possess the same Gamma as the display does , that is G2.2 , which for most people is VERY

DARK AND CONTRASTY and TOTALLY UNUSABLE. ( Please take a mental note of this , because when you process

Lineal Images and you forget to reintroduce the compensating Gamma of 0.45 , your image will be like that , dark and

concentrated at the lower tones.)"

 

If you dont get get it , nobody can take care of your inability . It might look like like a "new toy syndrome"..... for you ,

that are clearly disconected from what is going on , but Linear Processing has been around since the early 2000 ,

Canon has it ,as part of its RAW processing options , and so does Adobe in Lightroom and Camera RAW Converter. They might have a

good reason to be spending time and money in extra lines of code.

 

Linear processing is all over the place . Jonathan Wienke , in 2004 posted an excellent article on Hibrid Linear

processing, at Luminous Landscape here: http://www.luminous-landscape.com/tutorials/hybrid-conversion.shtml, there is many others ,if

you care to do some Google research.

 

 

"When you point to the moon with your finger , the fool looks at the finger.. " Old saying.

[frans_waterlander]

Luis,

 

The only thing you have clearly demonstrated is that linear processing can result in really dark images with vast color shifts, things you can easily do in almost any image editing program you already own, if you are so inclined. Besides, some of your assertions are severely flawed.

 

And yes, I have read Wienke's article and have some opinions on it as well. And googling hasn't resulted in any useful information. The only possible advantage of linear processing may be the potential for reducing posterization in the highlights. How much of an issue this is and how much linear processing would help hasn't been addressed at all in this thread.

 

If I may be so bold, you seem to use linear processing not as a technical tool to improve the image but as an artistic tool to express yourself. Nothing wrong with using artistic tools to express yourself, but there are way simpler, more intuitive and cheaper (free) tools to do that.

[william l. palminteri]

Oh dear.....

 

Gamma correction has been used in the television industry forever. In the early days, the video camera was very

nonliear, less so today, but still nonlinear. I keep my monitor calibrated to something close to the NTSC standard, I

assumed everybody ran tightly calibrated monitors. Yes, I know the gag, "When you assume....."

I watched "The Odd Couple" too.

 

Bill P.

[luis-a-guevara]

Cheaper than a free Program? Doubt it.

 

Frans .You are clearly in denial . You don't seem to take the time to digest what you read and , therefore you are stuck

in old technology . As one that knows what it feels to have the fingers wet in Pinacryptol ,developing film negatives by visual

inspection , in search for those elusive tones in the highlights , I welcome the technical advantage of doing that very

same thing with my "Digital Negatives" in both a quest for , both , improve my images , and second and inseparably

from the first one, express myself.

 

Too bad that you are not listening.

[luis-a-guevara]

By the way , Where are your images?

[frans_waterlander]

Luis,

 

You have demonstrated that linear processing can darken images and distort colors. Linear processing may have real advantages, but you have not defined nor demonstrated those. See, I'm very interested to learn about ways to improve images, but you haven't convinced me one bit yet that linear processing does that. And come on now, Luis, do you really need to boast about your wet darkroom experiences and questioning my qualifications?

[luis-a-guevara]

Just show your images.

[frans_waterlander]

Luis,

 

How does that have to do with anything we are discussing here? YOU claim that linear processing has advantages but fail to define and demonstrate those and now you want to see MY images? Where is the logic in this?