logarithmic bitmaps

[edgar_njari]

How come non of the scanners support log bitmap files? There are huge

benefits in using logarithmic bitmaps

[steven_clark]

Because it's an obscure file format that's impossible to find software to wrok with? That and it bears no resmblance to the actual data coming from the sensor?

[jbq]

Perhaps because ADC are linear by construction?!

[gordonr]

Do you mean gamma encoded? Almost all images once scanned are converted to a gamma space: <a href="http://www.poynton.com/notes/colour_and_gamma/GammaFAQ.html">http://www.poynton.com/notes/colour_and_gamma/GammaFAQ.html</a>

[edgar_njari]

Well, i know it is not used much on standard computers, but it would be nice if it was a standard.

 

First of all, film negative records light logarithically, and it uses digital space more efficiently

For example, if you use 10-bit per chanel for log image, you are actually presenting the same qualitty to the human eye as 14-bit linear (because human eyes allso don't see linear)

It would take 12-bit log files to have the same qualitty of 16-bit linear files (smaller filesizes, more space)

 

Plus the greates of all things is that you can have the full latitude of the neg in the digital file.

When you have linear files, you are choosing which part of the films latitude you are going to use (highlights are cliped)

like when you are printing.

 

But with log files, you record everything, nothing is clipped, you don't have to throw anything away.

 

Motion picture industry uses log files as a standard. It would be a great archiving format for still images, you could archive all the latitude from the negative without cliping anything.

(later you convert to linear and clip off what every you want and print it)

[jbq]

>First of all, film negative records light logarithically

 

I'd like to know which film you're talking about. Looking at the characteristic curves of NPS and TMX (which are drawn in a log-log space), they look fairly "linear" to me, meaning that the response of those films is a power curve.

 

>It would take 12-bit log files to have the same qualitty of 16-bit linear files.

 

Claude Shannon would have enjoyed to read a proof of that.

 

-Obviously, 16-bit files can record more different values than 12-bit ones (regardless of linear vs. logarithmic issues).

 

-Assuming that you use "exactly" the same range - to simplify, let's say 0 log = 1 linear, 4096 log = 65536 linear. We have 16 stops of range with the 16-bit data, i.e. we'll use 256 log values per stop

 

Here's the start of the table: 1 linear->0 log. 2 linear->256 log. 3 linear->405 log. 4 linear->512 log.

 

Here's the end of the table: [65095-65270] linear ->4094 log. [65271-65447] linear ->4095 log. [65448-65536] linear ->4096 log.

 

Notice that are the start of the table lots of values are entirely unused, wherease at the end of the table the quantization noise becomes fairly big - it alone accounts for a S/N ratio of approximately 750, which is probably not enough (a good digital camera can reach 400 or more, so I wouldn't be surprised if a multi-pass desktop scanner was able to reach more than 1000 in the highlights).

 

>Plus the greates of all things is that you can have the full latitude of the neg in the digital file.

 

On my planet, negative film can't really record much more than 10 stops and by the time you really use the full range of negative film the low contrast and the high grain in the shadows will create a lot of noise, whereas the quantization noise of a 16-bit scan still leaves 36dB of S/N. Worse, since in photography your filan media is paper, a picture with 10 stops of range is essentially unprintable "as is" and will need to very severely dodge and burn, which will lose more shadow detail than the 16-bit quantization.

 

>But with log files, you record everything, nothing is clipped, you don't have to throw anything away.

 

Without even going through extremes, let's suppose that you keep enough values for an extra 4 stops, i.e. that you use approximately 200 log values per stop instead of 256. The quantization that you've introduced makes your S/N fall from about 750 to about 500, at which point you may even start losing some of the values that are created by digital cameras. Do the same with 16-bit files, and only use the bottom 12 bits for normally exposed pictures, and 12 bits is probably enough precision to work in.

 

I'm starting to wonder if you're really talking about log files. I'm thinking that in reality you may be talking about floating-point files. I've thought about the problem in the past, and reached the conclusion that a 10+5+1 linear floating-point format (10 bits of mantissa, 5 of exponent and 1 of sign) would be very appropriate for pictures, possibly even as an intermediate format (though with today's memory sizes the IEEE standard 23+8+1 would probably make programmers' lives easier, since it's natively supported by all CPUs).

 

It is possible in theory to create CCDs with non-linear responses (it's possible to create non-linear ADCs), but my instinct as a software engineer wouldn't trust an analog circuit to do a linear->log conversion.

 

I'd advocate that user power curves with a gamma around 2 is probably the best compromise between the fact that the natural behavior of a CRT is a power curve with gamma 2.5 and the curve where the quantization noise matches the statistical noise is a power curve with gamma 1.5 (the curve whose derivative is a square root).

[gordonr]

You probably want to read Bob Atkins' article on <a href="http://www.photo.net/learn/drange/">Dynamic Range</a> (http://www.photo.net/learn/drange/). This gives some basics about scanning, with some followup thoughts by me on numeric coding, and signal/noise ratios.

[edgar_njari]

Jean..

 

I am not talking about dynamic range. I am talking about latitude (the range you can lighten or darken the image to get normal qualitty images)

 

If you look at a log file without a LUT correction you can see that

both deep shadows and highlights are presented without cliping. The image has a flat washed out look. If you convert it to linear, you loose highlight detail because log had more image detail beyond the white point. (685 out of 1023 is white point when you convert to linear)

 

I am not talking about float images, but log images have this in common with float images.

I am talking about logarithmic images like Cineon or DPX

 

And as for the number of colors:

Log files do not map shades linearly. They put more shades of gray in the dark areas, and less in lighlights. It is because human eye can resolve more light steps in dark areas.

So why do i say that you need 14-bit linear to match 10-bit log?

Because the largest amount of steps in a given light interval

mathces that of 14-bit linear.

The only difference is that 14-bit linear, devides light changes equally across all the range (from black to white), and log uses more steps in the dark areas, and less in light areas.

(theoretically 14-bit linear has more color resolution than 10-bit log, but to the human eyes the effect is the same)

 

And excuse me, but film does react to light logarithmically, as oposed to linear electronic sensores. This is why film images have more latitude (digital images usually clip the highlights)

[jbq]

Edgar - back to the original question: scanners output linear data because that's how CCDs work.

 

How much you can lighten or darken an image has nothing to do with how it's encoded. If I add "1" to all the pixels (in log2 coding) or multiply all the values by 2 (in linear coding) or multiply by 1.36 (in a gamma-2.2 space), all I've done is lighten the image by one stop - it doesn't matter which representation is used.

 

How much detail is held beyond the white point depends purely on where you put the white point, and not on the whether the data is represented logarithmically or linearly or anything else. It's purely a convention.

 

How many bits "in linear space" match how many bits "in log space" depends on the base of your logarithm. If you don't specify it, you can't compare anything. Let's assume that you use 64 values per stop, you have 16 stops of range with a 1% precision. You'll need 23 bits in linear space to do the same thing. Conversely, if you have 14 bits of linear space, that's 14 stops with precision that goes as high as 0.006%, and you'll need 17 bits in log space to do the same thing. Each time you convert from one space to another, you need extra bits in order not to lose any data.

 

The sad part is that you're dealing with light here, which is an inherently stochastic process. As such, for each stop you get darker, you have 40% more noise, so there's no point having that much precision in the dark areas. As I said in my previous post, a power curve with gamma 1.5 would be ideal if that was the only factor.

 

I'm looking forward to seeing the characteristic curves of the film(s) you mention. If getting a logarithmic response mattered that much, I'd expect to see the chararectistic curves to be in a log-linear space, so that you could use a ruler to see how "logarithmic" they behave. All the curves I've seen so far were plotted in a log-log space, which is how you normally plot power curves (because power curves show as straight lines in a log-log space).

[emre]

<blockquote>And excuse me, but film does react to light logarithmically, as oposed to linear electronic sensores. This is why film images have more latitude (digital images usually clip the highlights)</blockquote>

 

Film owes its latitude to its non-linear response at the tails (the so called "S" shape.) Manufacturers are beginning to incorporate similar abilities into digital cameras (cf. <a href="http://www.photo.net/bboard/q-and-a-fetch-msg?msg_id=005GJe">HDRI</a>.)

[edgar_njari]

Jean

 

Linear files show nothing beyond the white point.

white point in linear images is full 255 code value.

As for log files, white point is 685, and they have more room beyond it to 1023 for overexposed parts of the image that appear totally white in prints.

Printing negatives is all about limiting yourself to a density interval. For example, if you print a high contrast sunny-day image, you can never print everything from the negative, you have a given interval of densities that you can put on the print or in linear files. Your choice is what interval will you choose. Ok so now,

log files are a way of not having to choose.

Log files capture image information below the blackpoint and above the white point of linear files. And when you convert them to linear files you are cutting off everything beyond the whitepoint and throwing that image information away (just like when you are printing)

 

You didn't understand me about the color resolution.

 

The amount of shades is 1024 either way for 10-bit, but they are not evenly distributed in log format.

If you devide that range into some 10 stops, you can say that log format puts a lot more density steps in the lower stops, and puts less steps in the highest stops.

And now to match the number of steps for the "riches" stop of light (in lower levels) you have to use 14-bit linear

 

In the high light levels, 14-bit linear will have a LOT more steps

that 10-bit log but that will not make any difference because you can't percieve that much steps in highlights. In the lower densities, 14-bit will have about the same number of steps as 10 bit log

 

the qualitty of shadows in 10-bit log therefore matches that of 14-bit linear, and the qualitty of highlights is compromised.

[jbq]

Edgar - I spent 8 years in college studying math, physics and computer science, don't worry, I know very well what a logarithmic distribution is, What its advantages and disadvantages are. And, yes, I know that it's the ideal distribution if you want to keep the quantization noise at a constant S/N ratio. I very much doubt that such a goal is compatible with photography - where the process inherently has a better S/N ratio in the highlights than in the shadows.

 

You are strongly confusing the issue of the range of values that are possible to represent vs the distribution of values inside that range. Deciding that 10-bit log files have a white point at 685 is as arbitrary as saying that linear files have a white point at 21756 (I made up that value). And you still haven't told me what the base of the logarithm is (which is another arbitrary value for log files which has no counterpart in linear files).

 

Let's take some values for the sake of having numbers (not the kind of stuff that my French education recommends, but I'm not in France).

 

Let's imagine that we're going for a normal exposure, going for about 5 stops of "nominal" dynamic range, where the dark areas are exposed at 0.01 lux.second (l.s) and the bright areas are exposed at 0.32 l.s (that's about the normal exposure for ISO 100).

 

I can decide to have a 7-bit linear scale, where 0 is 0 l.s, 1 is 0.01 l.s, 2 is 0.2 l.s and so on, where 32 is 0.32 l.s and 127 is 1.27 l.s. The white point is 32. This scale has enough range for 2 stops of overexposure.

 

I can device to have a 7-bit logarithmic scale with a step size of 1.027665 where 0 is 0.01 l.s, 1 is 0.01027655 l.s, 2 is 0.01056093 l.s, 3 is 0.01085312 l.s, up to the point where 127 is 0.32 l.s. The white point is 127. This scale doesn't have any range for overexposure.

 

The above example shows that there is no inherent property that makes a logarithmic scale have more latitude for overexposure than a linear scale. It's only the fact that you arbitrarily decide where to put the white point on the scale that makes a scale have (or not have) any such ability.

 

If you want to have more fun, you can probably figure out the pixel size for which 1 l.s is 100000 electrons. The dark areas will have 1000 electrons (31 electrons of statistical noise), the bright areas will have 32000 electrons (179 electrons of statistical noise). I'll let you figure out with a logarithmic scale at which point the quantization noise in the highlights is as big as the statistical noise, and how many values you waste in the shadows with such a method. I'll let you figure out what is the shape of the optimal curve where the quantization noise matches the statistical noise at each value.

 

>[...] because you can't percieve [...]

 

Of course - but you weren't talking about perception before this, you were talking about scanners. Yes, the human eye has the ability to distinguish two gray brightness level approximately 1% apart, both in the dark and in the bright areas, and the smallest scale of values that can be distinguished is going to be a linear scale. No doubt about that. And, yes, a 10-bit logarithmic scale with steps of 1.005 percent will be very good at representing gray levels in as few bits as possible while not showing any visible quantization noise. No doubt about that. But we weren't talking about perception so far, and that's a totally different discussion. You'll notice that the various modified power curves used in the video/computer world are also reasonably good, and that a floating-point 7.3 linear representation also does the job pretty well (and has the great advantage of being linear, which is a blessing for software engineers). And, yes, I agree that a 10-bit log scale with a well-chose step size will be able to reproduce the same perceptual richness as a 14-bit fixed point linear scale. But the way your human eyes perceives light is different from the way a scanner perceives it, which was your original question.

 

 

 

Now, for the film part of things (which has litte to do with scanners, and even less to do with perception). Reading approximate values from the blue channel of the characteristic curve of Superia 100:

 

for log H=-2 (H=0.01), D is about 1.2, i.e. 0.0631 transmittance.

 

for log H=-1.5 (H=0.0316), D is about 1.6, i.e. 0.0251 transmittance.

 

for log H=-1 (H=0.1), D is about 1.95, i.e. 0.0112 transmittance.

 

for log H=-0.5 (H=0.316), D is about 2.4, i.e. 0.00398 transmittance.

 

for log H=0 (H=1), D is about 2.8, i.e. 0.00158 transmittance.

 

I claim that the transmittance is approximately 0.00167*(H^-0.7875), you can check the numbers, it's accurate within a few percent (this gives Superia 100 a gamma of about -0.8 - it's a negative film). I don't think that any logarithmic scale would come close to being that precise an approximation of those measurements.

 

That being said, I have much better things to do than arguing on photo.net...

[gordonr]

I didn't spend as many years at university as Jean-Baptiste, but I too have studied mathematics and numerical analysis, and spent many years on computer programming, and I have better things to do than argue on photo.net. Read the links I posted above, and you might eventually realise why you are wasting your time.

[edgar_njari]

Yes, i was talking about perception , and as for the white and black points, i was talking about the standards used in motion picture scanning (95 and 685) for 10-bit log cineon format.

 

I don't know what the base is, i supose it is 10 because it is never mentioned anywhere (for cineon files)

[jbq]

Edgar - if we're talking about perception, sure, I totally agree with you, a logarithmic representation is very appropriate.

 

Here's a fairly thorough description of the cineon data format: http://www.dotcsw.com/doc/cineon2.pdf

 

For a nominally exposed cineon negative, 2% black is set at code 180, 90% white is set at code 685, with 90 steps per stop of density (this makes the base of the logarithm 1.00773, i.e. 2^(1/90), in density space). This is a space of negative film densities, not a space of original luminance, i.e. the curve is not corrected for the toe and shoulder of the curve.

 

The model for this negative is a gamma 0.6, with Dmax below 2 (code 1024 is D=2). This seems to be representative of a motion picture negative, but not of a photography film. I measured a gamma of (negative) 0.8 on a sample film, and its Dmax could be as high as 3 or even higher (the shoulder is so far that it's not used for normally exposed film).

 

So, if I may reach a conclusion:

 

-More than 10 years ago, storage space in the movie industry was a severe design issue. It made much sense to try to squeeze out every last bit of data while not altering the perceived quality, even if it had an impact on the processing time. From that point of view, the 10-bit log format, with 90 steps per stops, is probably the most appropriate (note, as I said, that the same format wouldn't work for photography negatives). I'd be very surprised if the conversion from linear light levels to analog densities was done in the analog world. Most probably the scanners have 14- or 16-bit DACs, and perform the inversion and log conversion in the digital world.

 

-Right now, storage space isn't such a huge deal. Even if you scan with a dimage 5400 and store the data "raw", it's only 233MB per frame. Safely storing that much data costs around $1 (two copies on high-end CD-Rs, or 4 copies on hard drives). This ain't a big deal. 1GB of RAM costs $200, which is enough to photoshop those images. Once again not a big deal. Photographers would much rather be able to store the data "raw" (i.e. unprocessed, kept exactly as it was when it entered the digital world), for the peace of mind that it will be possible to apply heavy transformations to it and still keep the scanner as the limiting factor (notice that the cineon format, with 90 steps per density stop and gamma 0.6, has 54 steps per stop of exposure, i.e. 1.3% per step, which is close to the limit where eyes can perceive a difference (some will argue it's already above). Change the mid-point or the contrast just a tiny bit (things that photographers love to do), and you'll quickly be able to see the steps. Do 2 or 3 such operations in a row and the picture will be totally ugly.

[edgar_njari]

Perhaps 12-bit or 14-bit log would be better (and it would still be more efficent than 16-bit linear)

 

Anyway, 10-bit cineon is still used (well DPX is now used more, but it is basicly the same old thing) in motion picture postproduction.

 

And there is one more standard that captures my attention, and it's openEXR. I read the specs for it, but i don't really understand what are the benefits of float bitmaps (can you explain to me in a few words?) openERX has a much wider dynamic range than cineon 10-bit

and it has better color resolution.

But what is the difference between 16-bit integer and 16-bit floating-point images?

[jbq]

Edgar - the point of floating point is to get the best of both worlds (linear and logarithmic), using a logarithmic representation for full stops and linear representation between stops. The great advantage is that operations on floating-point numbers are easy and cheap to implement - all CPUs support them in hardware. The disadvantage is that floating point can't give you an arbitrary precision, and that the precision isn't quite as uniform as with logarighmic numbers (the relative precision varies by 6dB across the range). To reach a similar range as the 10-bit log format you were mentioning, you'd need 11 bits of floating point - 4 bits to cover 16 stops of range, and 7 bits to have 128 steps per stop.

[edgar_njari]

The float bitmap format that ILM is using is 16-bit (they even mentioned 32-bit, but i am not sure was that really a color space)

 

How good would that be compared to log and linear bitmaps (how many bits?)

 

And what does it mean that you don't have the precision? I am used to the fact that in digital there is no such thing as unprecise (a 0 is a 0 and a 1 is a 1), so what do you mean by that?

[jbq]

The number of bits are comparable, with a slight advantage to log, but I would say that the advantage is less than 1.5 bits, with 2 bits being a worst case. 16-bit floats would be significantly better than 10-bit logs, without a single doubt (they could e.g. cover 32 stops of dynamic range, with 2048 steps per stop, whereas cineon bitmaps cover 11 stops of dynamic range with 90 steps per stop).

 

The problem with floating point is that you can't make them cover e.g. 90 steps per stop, the only choices are powers of 2. That's why I said that you can't have an arbitrary precision. Similarly, the number of stops that can be covered (assuming a linear representation) can only be a power of 2 (I won't mention the special cases of infinite and denormal values).

 

Digital values are not precise, they are just reproducible - a pixel value of 435 today will still be 435 in 10 years. But it doesn't mean that the original film has a perfectly uniform square where the density could be coded exactly with the value 435 - some approximation has to be done.

[edgar_njari]

No, i didn't say that digital is acurate. I am saying that it is precise as in "no alternatives", i mean every number is well defined

(if it is not there are errors. I am not talking about the digital reproduction of film images here, just about the way digital technology works (no matter what is involved)