Embedded jpg in the RAW file?

<p>I keep reading that there is a jpg file embedded in the RAW files and that's how web browsers can render RAW files. Some even claim a full size jpg.<br>

I'm frequently wrong, but I don't think this is correct.<br>

I know I can export RAW and RAW + jpg and jpg, depending upon my settings. It's the embedded jpg I'm curious about.</p>

<p>yes, it's usually a fullsize lower resolution jpg. That is generally what you see in computer Finder/Explorer things, etc</p>

<p>this will extract it (as other software can too) ... <br>

http://michaeltapesdesign.com/instant-jpeg-from-raw.html</p>

<p>It is also what you see on the camera LCD when you preview an image shot in RAW.</p>

<p>I disagree Dan. The jpg seen on the camera viewer is a low resolution jpg created separately, but from the raw file.<br /> Unless you set your camera up for export of both RAW and jpg, the jpg is discarded.</p>

<p>I may disagree with you as well Howard. I'm guessing that IFJR is based on David Coffins dcRaw or similar.<br /> I have asked IFJR and am awaiting their answer. I will return here as soon as I have their answer.<br /> Tom</p>

<p> </p>

<p>Well, I got an answer from Michaeltapesdesign.<br /> Hi,<br /> First it is not a converter. We extract the embedded JPEG. So nothing is done with or to the raw image. And yes, it is based on DCRaw, but again, not using the conversion aspects at all.<br /> ~michael<br>

It would indicate that I am wrong, but I am still a little confused.<br>

Why would it be based on David Coffins DCRaw if there is an embedded jpg in the RAW file? I am familiar with Davids DCRaw and have used it in own programming. Run with default settings it will produce a jpg image, but that image is converted to jpg from the RAW file.</p>

<p>Not all cameras store a full-size JPEG embedded in the RAW file; my Sony A7 and NEX 6 only contain JPEG of dimensions 1616x1080 pixels. Nikon stores a full-size high-quality JPEG (from the size of the extracted JPEG it's the equivalent of saving a JPEG in photoshop with quality setting of about 7; the size is about 1/6 of a JPEG saved at maximum quality setting of 12). I am using the above mentioned instant-jpeg-from-raw tool as well a BreezeBrowser Pro which also allows to extract the embedded JPEG from the RAW file.</p>

<p>If you use an EXIF viewer like Jeffrey's EXIF Tool on a Nikon NEF file, you will find the PreviewImageStart and PreviewImageLength of the embedded JEPG; I believe - though have not tried - you can use EXIF Tool to extract the embedded JPEG to.</p>

<p>OK, I'm proven wrong:) Thanks to Dieter Schafer' hint. <br>

I use Phil Harvey's Exiftool and there it is, a thumbnail using the -b switch. <br>

Thanks to all for educating me. That's the value of Photo.net and the corporate knowledge in the fora ( I'm stubborn and prefer the latin fora to forums.)<br>

I'm really glad I found this place.</p>

<p>Thanks again.<br>

Tom</p>

<p>My hunch is that Mr. Tapes uses dcraw so that he has a known, fast, proven method of parsing the raw binary data and can easily extract it by using the -b flag. It's a simple free utility, why would he go to the trouble of re-inventing the wheel by writing a raw parser?</p>

<p>When I wrote 'as other software can too', I was, in fact, referring to exiftool and there may be still others.</p>

It isn't rocket science: there is a fully fledged JPEG inside the raw file, with pointers in the file (section) headers that tell where the JPEG starts and where it ends. The specifications of the different raw file formats are available, so it's not hard to navigate through the different file types. Not that much trouble.<br>Why anyone would do that could be because they enjoy doing that, and because it gives an opportunity to market something (which isn't necessarily the free give-away).

<p>I have changed my mind. I no longer believe there are embedded jpgs i RAW files!<br /> I sent an email to Olympus:<br /> ---------------------------------------------------------------------------<br /> I am discussing the RAW format online. It has been my understanding that the orf format does not contain a jpg preview/thumbnail.<br /> I know I can view the orf with Picasa (and other), but I thought that this was trough the use of a RAW converter, like David Coffin's dcRaw. <br />I am familiar with the ORF codec, which allows MS Vista and later OS' to render the orf file.<br /> Does the orf contain a jpg thumbnail in addition to the RAW data?<br /> Rgds<br /> Thormod<br /> ----------------------------------------------------------------------------<br /> I have received a response form Oly today 05 05 2015:<br /> ----------------------------------------------------------------------------<br /> Dear Thormod,<br /> Thank you for your email regarding ORF files.<br /> I'm able to confirm that ORF files don't contain any JPG thumbnails.<br /> <br />Should you require any further assistance, please do not hesitate to contact me and I'll be happy to help you.<br /> Kind regards,<br />Jaroslav Zika<br /> Technical Support Specialist<br /> ---------------------------------------------------------------------------------------------<br /> QED<br>

PS. Check out the Nikon or Cannon RAW Codecs.</p>

 

There is no universal raw file format, Thormod, and embedding a JPEG is an option some makers go for, others don't, and some have done in some of their cameras' raw files but not in some of their other cameras' file format.

<p>I know that there is no universal raw file format. RAW file formats are proprietary and not disclosed to "the public". <br>

With Win XP, it was impossible to view RAW files from the operating system. With the exception of proprietary viewers and converters, they could only be viewed through third party applications like PS, GIMP (using UfRAW), Picasa, anything based upon dcRaw.<br>

Since there was a requirement, the manufacturers have created codecs that allow the OS (from Vista on) to render an image from their RAW files. Not by extracting an embedded jpg, by converting the RAW data to jpg. It does not make sense to me to bloat the RAW file unnecessarily with a jpg. <br>

I do take your point though, and I will send the same question to Nikon and Cannon.<br>

Tom<br>

</p>

<p>I have received the following today from Canon. Its in Norwegian and I am providing the translation.<br>

--------------------------------------------------<br>

Hei Tormod<br>

Takk for at du kontaktet Canon.<br /><br />Jeg kan bekrefte at ikke heller Canons CR,CR2 inneholder en thumbnail og krever RAW codec pack for fremvisning. <br>

Tjenester som feks Picasa har ofte integrerte løsninger som tillater de vanlige RAW formatene. <br>

<br />Med vennlig hilsen,<br /><br /><br /><br /><br /><strong>Ole Buer Askeland</strong><br /><strong>Canon Services & Support</strong> <br /><br>

<br>

Thank you for contacting Canon.<br>

I can confirm that Canon CR, CR2 do not contain a thumbnail either, and require a RAW codec pack for display.<br>

<br>

Applications like Picasa often have integrated solutions allowing the common RAW formats.<br>

Best regards<br>

<strong>Ole Buer Askeland</strong><br /><strong>Canon Services & Support</strong> </p>

<p>I have also received an answer from Nikon:<br>

-----------------------------------------------------------<br>

Dear Thormod Nordahl,<br>

<br />Thank you for contacting Nikon. In regard to your question, Nikon NEF (RAW) file does not contain a JPEG thumbnail.<br /><br />Regards<br />Laurence<br />Nikon Support.<br>

--------------------------------------------------------------<br>

Q.G. de Bakker wrote: on May 5.</p>

<blockquote>

<p>There is no universal raw file format, Thormod, and embedding a JPEG is an option some makers go for, others don't, and some have done in some of their cameras' raw files but not in some of their other cameras' file format.<br>

</p>

</blockquote>

<p>Do you know of any maker who either has an embedded jpg, or has one in some of their camera's RAW format?</p>

For what it's worth, have a look at the analysis of <a href="http://lclevy.free.fr/cr2/">Canon's CR2 format</a> or <a href="http://lclevy.free.fr/nef/">Nikon's NEF format</a>.

<p>Thanks for the link to iclevy.free.fr<br>

A treasure trove of RAW information:)<br>

As for the embedded jpg. The RAW format is TIFF like. It may be that the author of the site just copied the IFD content from a TIFF document and therefore the reference to the jpg remains.<br>

Remember, a few posts ago, I checked my old RAW files(E-500 uncompressed) and found the reference to a jpg. Well, I never bothered to check if there was a jpg between the start and stop byte. <br>

Anyhoo, I may as well send the info from Oly, Nikon and Canon to the guy, and see what he says.</p>

<p>Tom</p>

As far as i can see, raw formats do indeed use the TIFF format a lot. The TIFF format is just an empty container with a few rules of how to fill it without creating a mess.<br>Popular because you can stuff any other file format inside a TIFF and because it can contain multiple formats at once. As many and as proprietary or universal as you like.<br>It doesn't have a 'native content' that can be cleaned out leaving remnants that hint at what had been in the container before. No prior content that was replaced by the raw data a camera produces. Everything that is in there has been put there by whatever created the file.<br>I don't know what the author of the site did, nor why he would 'project' what he found in a raw file onto a TIFF template containing a map of a file that is not the raw file. Nor if he did why he would not have removed from the description something he had not found in the raw file. So unless demonstrated that it is not the case (and again: i don't know how the analysis was done or how the description was drafted) i would think that the description given is of what was found in the raw file.

<p><em>It doesn't have a 'native content' that can be cleaned out leaving remnants that hint at what had been in the container before. No prior content that was replaced by the raw data a camera produces. Everything that is in there has been put there by whatever created the file.</em><br>

<br /> You misunderstand me again, remember - English is not my first language.<br /> I was trying to refer to the visual presentation of the TIFF format on the web page. The actual words, headings and frames used to organize the information, the black and white pixels on my monitor. Not the meta data within the Exif area of a file.<br /> I simply said that maybe he forgot to remove the letters jpg from the text in the relevant IFD. <br /> Instead of recreating the textual presentation of Exif content, I consider it plausible - he may have cut and pasted from a explanatory TIFF document.<br /> As to how the analysis was done, I can only speculate. My guess is that he hacked til files, the same way I hacked an older version of the ORF file (the only one of interest to me), but has done a far better job of it - than I.<br>

<br /> Working on the assumption that technical employees at the camera manufacturer's possess a more reliable access to propitiatory information; He may have made a Boo-boo.</p>

What i was trying to say, Thormod, is that there is no reason why someone would use a description of a particular TIFF file used to contain (among other things perhaps) JPEG encoded data, remove everything that is not in the other particular TIFF file and forget about removing a header that describes JPEG encoded data.<br><br>A TIFF file is a simple container format, with a main header and sub headers describing the formatting used in sections in the TIFF container and where to find those sections. The formatting used in such sections are file formats in their own right, and differ from 'stand-alone' versions of those file formats usually only in that they are encapsulated in a TIFF envelope, often together with data chunks formatted in a different way,(else it would not make much sense to put the data inside a TIFF file).<br><br>Now if you take the trouble of examining a particular TIFF file, it is much easier to simply write down what you find, instead of writing down what you find inside a template that itself was a description of another, different particular TIFF file, with headers describing things that aren't in the examined file, appearing at the wrong place, containing wrong information. There is no need to begin with the wrong description. And since it only would make the task more complex and more work, it would not be a sensible thing to do. So i don't think it a likely scenario, and it's a lot more likely that what is described is what was found, as it was found, inside the examined file.<br>He may have made a mistake, i don't know. But if so, i don't think it plausible at all that it was the particular mistake you think it was.

<p>Just to be sure: regardless of a JPEG being embedded in a raw-file (and I am 100% convinced the answer Nikon sent you is complete rubbish), web browser can not and will not display raw files, period. Neither can they officially display TIFF files. On web pages, it's GIF, JPEG, PNG - all other formats have to be handled via extensions on the browser. Trying to fake a browser to believe a raw file is a JPEG - won't work, the file structure is sufficiently different (as discussed - QG knows more about it than me by a long mile).</p>

<p>Why I am convinced NEF-files do contain a JPEG is actually 'thanks' to Nikon's own products. CaptureNX2 saved its edits directly into raw files; the edits were saved as basic 'interpreters' of the changes, yet filesizes for a 12MP raw file went from ~13MB to ~17-18MB after saving a file. According to all sources I could find at the time: CaptureNX2 saved a full-res JPEG with low compression into the NEF file.<br>

I've extracted the JPEG files using the exif tool discussed before, and found this to be correct. The codec used in Windows for NEF files uses this jpeg preview (and hence will show the changes made by CaptureNX2), and the higher quality of the CaptureNX2 previews is noticeable in programs that use the (WIA) codec. Other raw tools (lightroom, captureone and so on) ignore the embedded JPEG, though, and create their own previews. </p>

<p>I've tested whether my camera embeds a jpeg into the Raw file by taking two shots of the same scene, same lighting using the same WB preset and exposure, one in Raw format and the other in jpeg high quality BUT cranking up contrast and saturation and switching to Vibrant rendering for both shots.</p>

<p>I extracted the jpeg out of the PEF using IJFR and embedded the color space I chose incamera which was AdobeRGB and did the same with the incamera jpeg version. Both jpegs show identical previews where the Raw preview viewed in ACR is quite different since it doesn't know a thing about incamera Vibrant rendering, maxed out contrast and saturation levels and certainly will interpret incamera WB preset differently (mine was Daylight Fluorescent).</p>

<p>I suggest you conduct the same test to check that IJFR is extracting the jpeg embedded by the camera manufacturer in the Raw file. Don't even have the OEM Raw converter open just to rule out any influences.</p>

<p>If it's DCRaw creating the jpeg from the Raw it can't possibly know anything about the incamera settings used to render incamera jpegs in order to produce identical previews. The previews should be different between the two jpegs.</p>

<p>Q.G: <br />I thank you for your efforts to educate me:) I'm just not good at explaining my self any more.<br>

Wouter: <br />The Browser, actually the OS, <strong>was</strong> unable to render RAW files. They were limited to the well known image file formats. This has changed <br />through the introduction of the <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ee719654%28v=vs.85%29.aspx">Windows Imaging Component</a> (WIC) and the codecs developed by various manufacturers of image files.<br>

Your point about growth in size is good. The only answer I can think of is if the RAW editing data are not kept in a .xmp sidecar, but stored within the RAW it self. But, my Oly .ORFs are 13MB and the sidecars only 6KB - unless Nikon is not as smart as Oly and uses a larger font for metadata? ;)<br>

I would like to send a copy of your post (without the "rubbish" bit) to Nikon. It may clarify if the individual who answered my question knows what <br />he is talking about. OK?<br>

Tim:<br>

Good test proposal, if IFJR doesn't use WIC. ( I know, I asked and Michael says he doesn't)<br />I will try something else. <br />I will open a RAW file in HexEditor and replace a radom block of data, in the image area, with &H00. (I can think of a dozen different reasons this will not work, but I will try anyway)<br />If it works, I should see a black (or white) line when opening the ORF in ACR, but not in Windows explorer.<br>

Tom<br>

PS. Thank you all for your patience with me. Its not that I am overly stubborn, or have an inflated ego (Wife may disagree with both), but I need to be convinced I am wrong when I am wrong. Or else I cannot sleep!</p>

 

<p>Correction to my previous.<br /> If it works, I should see a black (or white) line when opening the ORF in ACR, but not in <strong>Windows explorer</strong>.<br /> Should be IFJR.<br /> I could not make a block. Had to limit my experiment to a single line, and the line became sort of pinkish. I replaced the data useing the space bar, which of course is not &H00 but &H20.<br /> Opening the file with ACR and Picasa the line is visible.<br /> Now I have to download IFJR.<br /> I shall return<br /> Tom</p>

<p>Darn! <br>

I do not see the line using IJFR. This almost conclusive evidence of an embedded jpg.<br>

I say almost because my test line is only one pixel wide, and the IJFR image is of lower resolution than my ORF.<br>

I need to figger out how to make a block</p>

<p>Tom</p>

<p>I forgot to add on my previous post about testing incamera jpeg rendering presets in thumbnails that my 6MP Raw PEF extracted jpeg using IJFR is full resolution meaning 3000x2000 pixels.</p>

<p>I had "Native Size Only" option selected where I had the option of choosing Native and smaller with pixel dimension selector provided. My camera is a 2006 6MP Pentax K100D DSLR.</p>

<p>Newer cameras may have been engineered to not provide an embedded jpeg but then how would the thumbnail and main preview be generated viewing in non-OEM image viewers that don't provide Raw data editing tools.</p>