Embedded jpg in the RAW file?

Discussion in 'Beginner Questions' started by thormod, Apr 28, 2015.

  1. I keep reading that there is a jpg file embedded in the RAW files and that's how web browsers can render RAW files. Some even claim a full size jpg.
    I'm frequently wrong, but I don't think this is correct.
    I know I can export RAW and RAW + jpg and jpg, depending upon my settings. It's the embedded jpg I'm curious about.
     
  2. yes, it's usually a fullsize lower resolution jpg. That is generally what you see in computer Finder/Explorer things, etc
    this will extract it (as other software can too) ...
    http://michaeltapesdesign.com/instant-jpeg-from-raw.html
     
  3. It is also what you see on the camera LCD when you preview an image shot in RAW.
     
  4. I disagree Dan. The jpg seen on the camera viewer is a low resolution jpg created separately, but from the raw file.
    Unless you set your camera up for export of both RAW and jpg, the jpg is discarded.
    I may disagree with you as well Howard. I'm guessing that IFJR is based on David Coffins dcRaw or similar.
    I have asked IFJR and am awaiting their answer. I will return here as soon as I have their answer.
    Tom
     
  5. Well, I got an answer from Michaeltapesdesign.
    Hi,
    First it is not a converter. We extract the embedded JPEG. So nothing is done with or to the raw image. And yes, it is based on DCRaw, but again, not using the conversion aspects at all.
    ~michael
    It would indicate that I am wrong, but I am still a little confused.
    Why would it be based on David Coffins DCRaw if there is an embedded jpg in the RAW file? I am familiar with Davids DCRaw and have used it in own programming. Run with default settings it will produce a jpg image, but that image is converted to jpg from the RAW file.
     
  6. Not all cameras store a full-size JPEG embedded in the RAW file; my Sony A7 and NEX 6 only contain JPEG of dimensions 1616x1080 pixels. Nikon stores a full-size high-quality JPEG (from the size of the extracted JPEG it's the equivalent of saving a JPEG in photoshop with quality setting of about 7; the size is about 1/6 of a JPEG saved at maximum quality setting of 12). I am using the above mentioned instant-jpeg-from-raw tool as well a BreezeBrowser Pro which also allows to extract the embedded JPEG from the RAW file.
    If you use an EXIF viewer like Jeffrey's EXIF Tool on a Nikon NEF file, you will find the PreviewImageStart and PreviewImageLength of the embedded JEPG; I believe - though have not tried - you can use EXIF Tool to extract the embedded JPEG to.
     
  7. OK, I'm proven wrong:) Thanks to Dieter Schafer' hint.
    I use Phil Harvey's Exiftool and there it is, a thumbnail using the -b switch.
    Thanks to all for educating me. That's the value of Photo.net and the corporate knowledge in the fora ( I'm stubborn and prefer the latin fora to forums.)
    I'm really glad I found this place.
    Thanks again.
    Tom
     
  8. My hunch is that Mr. Tapes uses dcraw so that he has a known, fast, proven method of parsing the raw binary data and can easily extract it by using the -b flag. It's a simple free utility, why would he go to the trouble of re-inventing the wheel by writing a raw parser?
    When I wrote 'as other software can too', I was, in fact, referring to exiftool and there may be still others.
     
  9. It isn't rocket science: there is a fully fledged JPEG inside the raw file, with pointers in the file (section) headers that tell where the JPEG starts and where it ends. The specifications of the different raw file formats are available, so it's not hard to navigate through the different file types. Not that much trouble.<br>Why anyone would do that could be because they enjoy doing that, and because it gives an opportunity to market something (which isn't necessarily the free give-away).
     
  10. I have changed my mind. I no longer believe there are embedded jpgs i RAW files!
    I sent an email to Olympus:
    ---------------------------------------------------------------------------
    I am discussing the RAW format online. It has been my understanding that the orf format does not contain a jpg preview/thumbnail.
    I know I can view the orf with Picasa (and other), but I thought that this was trough the use of a RAW converter, like David Coffin's dcRaw.
    I am familiar with the ORF codec, which allows MS Vista and later OS' to render the orf file.
    Does the orf contain a jpg thumbnail in addition to the RAW data?
    Rgds
    Thormod
    ----------------------------------------------------------------------------
    I have received a response form Oly today 05 05 2015:
    ----------------------------------------------------------------------------
    Dear Thormod,
    Thank you for your email regarding ORF files.
    I'm able to confirm that ORF files don't contain any JPG thumbnails.

    Should you require any further assistance, please do not hesitate to contact me and I'll be happy to help you.
    Kind regards,
    Jaroslav Zika
    Technical Support Specialist
    ---------------------------------------------------------------------------------------------
    QED
    PS. Check out the Nikon or Cannon RAW Codecs.
     
  11. There is no universal raw file format, Thormod, and embedding a JPEG is an option some makers go for, others don't, and some have done in some of their cameras' raw files but not in some of their other cameras' file format.
     
  12. I know that there is no universal raw file format. RAW file formats are proprietary and not disclosed to "the public".
    With Win XP, it was impossible to view RAW files from the operating system. With the exception of proprietary viewers and converters, they could only be viewed through third party applications like PS, GIMP (using UfRAW), Picasa, anything based upon dcRaw.
    Since there was a requirement, the manufacturers have created codecs that allow the OS (from Vista on) to render an image from their RAW files. Not by extracting an embedded jpg, by converting the RAW data to jpg. It does not make sense to me to bloat the RAW file unnecessarily with a jpg.
    I do take your point though, and I will send the same question to Nikon and Cannon.
    Tom
     
  13. I have received the following today from Canon. Its in Norwegian and I am providing the translation.
    --------------------------------------------------
    Hei Tormod
    Takk for at du kontaktet Canon.

    Jeg kan bekrefte at ikke heller Canons CR,CR2 inneholder en thumbnail og krever RAW codec pack for fremvisning.
    Tjenester som feks Picasa har ofte integrerte løsninger som tillater de vanlige RAW formatene.

    Med vennlig hilsen,




    Ole Buer Askeland
    Canon Services & Support


    Thank you for contacting Canon.
    I can confirm that Canon CR, CR2 do not contain a thumbnail either, and require a RAW codec pack for display.

    Applications like Picasa often have integrated solutions allowing the common RAW formats.
    Best regards
    Ole Buer Askeland
    Canon Services & Support
     
  14. I have also received an answer from Nikon:
    -----------------------------------------------------------
    Dear Thormod Nordahl,

    Thank you for contacting Nikon. In regard to your question, Nikon NEF (RAW) file does not contain a JPEG thumbnail.

    Regards
    Laurence
    Nikon Support.
    --------------------------------------------------------------
    Q.G. de Bakker wrote: on May 5.
    There is no universal raw file format, Thormod, and embedding a JPEG is an option some makers go for, others don't, and some have done in some of their cameras' raw files but not in some of their other cameras' file format.​
    Do you know of any maker who either has an embedded jpg, or has one in some of their camera's RAW format?
     
  15. Thanks for the link to iclevy.free.fr
    A treasure trove of RAW information:)
    As for the embedded jpg. The RAW format is TIFF like. It may be that the author of the site just copied the IFD content from a TIFF document and therefore the reference to the jpg remains.
    Remember, a few posts ago, I checked my old RAW files(E-500 uncompressed) and found the reference to a jpg. Well, I never bothered to check if there was a jpg between the start and stop byte.
    Anyhoo, I may as well send the info from Oly, Nikon and Canon to the guy, and see what he says.
    Tom
     
  16. As far as i can see, raw formats do indeed use the TIFF format a lot. The TIFF format is just an empty container with a few rules of how to fill it without creating a mess.<br>Popular because you can stuff any other file format inside a TIFF and because it can contain multiple formats at once. As many and as proprietary or universal as you like.<br>It doesn't have a 'native content' that can be cleaned out leaving remnants that hint at what had been in the container before. No prior content that was replaced by the raw data a camera produces. Everything that is in there has been put there by whatever created the file.<br>I don't know what the author of the site did, nor why he would 'project' what he found in a raw file onto a TIFF template containing a map of a file that is not the raw file. Nor if he did why he would not have removed from the description something he had not found in the raw file. So unless demonstrated that it is not the case (and again: i don't know how the analysis was done or how the description was drafted) i would think that the description given is of what was found in the raw file.
     
  17. It doesn't have a 'native content' that can be cleaned out leaving remnants that hint at what had been in the container before. No prior content that was replaced by the raw data a camera produces. Everything that is in there has been put there by whatever created the file.

    You misunderstand me again, remember - English is not my first language.
    I was trying to refer to the visual presentation of the TIFF format on the web page. The actual words, headings and frames used to organize the information, the black and white pixels on my monitor. Not the meta data within the Exif area of a file.
    I simply said that maybe he forgot to remove the letters jpg from the text in the relevant IFD.
    Instead of recreating the textual presentation of Exif content, I consider it plausible - he may have cut and pasted from a explanatory TIFF document.
    As to how the analysis was done, I can only speculate. My guess is that he hacked til files, the same way I hacked an older version of the ORF file (the only one of interest to me), but has done a far better job of it - than I.

    Working on the assumption that technical employees at the camera manufacturer's possess a more reliable access to propitiatory information; He may have made a Boo-boo.
     
  18. What i was trying to say, Thormod, is that there is no reason why someone would use a description of a particular TIFF file used to contain (among other things perhaps) JPEG encoded data, remove everything that is not in the other particular TIFF file and forget about removing a header that describes JPEG encoded data.<br><br>A TIFF file is a simple container format, with a main header and sub headers describing the formatting used in sections in the TIFF container and where to find those sections. The formatting used in such sections are file formats in their own right, and differ from 'stand-alone' versions of those file formats usually only in that they are encapsulated in a TIFF envelope, often together with data chunks formatted in a different way,(else it would not make much sense to put the data inside a TIFF file).<br><br>Now if you take the trouble of examining a particular TIFF file, it is much easier to simply write down what you find, instead of writing down what you find inside a template that itself was a description of another, different particular TIFF file, with headers describing things that aren't in the examined file, appearing at the wrong place, containing wrong information. There is no need to begin with the wrong description. And since it only would make the task more complex and more work, it would not be a sensible thing to do. So i don't think it a likely scenario, and it's a lot more likely that what is described is what was found, as it was found, inside the examined file.<br>He may have made a mistake, i don't know. But if so, i don't think it plausible at all that it was the particular mistake you think it was.
     
  19. Just to be sure: regardless of a JPEG being embedded in a raw-file (and I am 100% convinced the answer Nikon sent you is complete rubbish), web browser can not and will not display raw files, period. Neither can they officially display TIFF files. On web pages, it's GIF, JPEG, PNG - all other formats have to be handled via extensions on the browser. Trying to fake a browser to believe a raw file is a JPEG - won't work, the file structure is sufficiently different (as discussed - QG knows more about it than me by a long mile).
    Why I am convinced NEF-files do contain a JPEG is actually 'thanks' to Nikon's own products. CaptureNX2 saved its edits directly into raw files; the edits were saved as basic 'interpreters' of the changes, yet filesizes for a 12MP raw file went from ~13MB to ~17-18MB after saving a file. According to all sources I could find at the time: CaptureNX2 saved a full-res JPEG with low compression into the NEF file.
    I've extracted the JPEG files using the exif tool discussed before, and found this to be correct. The codec used in Windows for NEF files uses this jpeg preview (and hence will show the changes made by CaptureNX2), and the higher quality of the CaptureNX2 previews is noticeable in programs that use the (WIA) codec. Other raw tools (lightroom, captureone and so on) ignore the embedded JPEG, though, and create their own previews.
     
  20. I've tested whether my camera embeds a jpeg into the Raw file by taking two shots of the same scene, same lighting using the same WB preset and exposure, one in Raw format and the other in jpeg high quality BUT cranking up contrast and saturation and switching to Vibrant rendering for both shots.
    I extracted the jpeg out of the PEF using IJFR and embedded the color space I chose incamera which was AdobeRGB and did the same with the incamera jpeg version. Both jpegs show identical previews where the Raw preview viewed in ACR is quite different since it doesn't know a thing about incamera Vibrant rendering, maxed out contrast and saturation levels and certainly will interpret incamera WB preset differently (mine was Daylight Fluorescent).
    I suggest you conduct the same test to check that IJFR is extracting the jpeg embedded by the camera manufacturer in the Raw file. Don't even have the OEM Raw converter open just to rule out any influences.
    If it's DCRaw creating the jpeg from the Raw it can't possibly know anything about the incamera settings used to render incamera jpegs in order to produce identical previews. The previews should be different between the two jpegs.
     
  21. Q.G:
    I thank you for your efforts to educate me:) I'm just not good at explaining my self any more.
    Wouter:
    The Browser, actually the OS, was unable to render RAW files. They were limited to the well known image file formats. This has changed
    through the introduction of the Windows Imaging Component (WIC) and the codecs developed by various manufacturers of image files.
    Your point about growth in size is good. The only answer I can think of is if the RAW editing data are not kept in a .xmp sidecar, but stored within the RAW it self. But, my Oly .ORFs are 13MB and the sidecars only 6KB - unless Nikon is not as smart as Oly and uses a larger font for metadata? ;)
    I would like to send a copy of your post (without the "rubbish" bit) to Nikon. It may clarify if the individual who answered my question knows what
    he is talking about. OK?
    Tim:
    Good test proposal, if IFJR doesn't use WIC. ( I know, I asked and Michael says he doesn't)
    I will try something else.
    I will open a RAW file in HexEditor and replace a radom block of data, in the image area, with &H00. (I can think of a dozen different reasons this will not work, but I will try anyway)
    If it works, I should see a black (or white) line when opening the ORF in ACR, but not in Windows explorer.
    Tom
    PS. Thank you all for your patience with me. Its not that I am overly stubborn, or have an inflated ego (Wife may disagree with both), but I need to be convinced I am wrong when I am wrong. Or else I cannot sleep!
     
  22. Correction to my previous.
    If it works, I should see a black (or white) line when opening the ORF in ACR, but not in Windows explorer.
    Should be IFJR.
    I could not make a block. Had to limit my experiment to a single line, and the line became sort of pinkish. I replaced the data useing the space bar, which of course is not &H00 but &H20.
    Opening the file with ACR and Picasa the line is visible.
    Now I have to download IFJR.
    I shall return
    Tom
     
  23. Darn!
    I do not see the line using IJFR. This almost conclusive evidence of an embedded jpg.
    I say almost because my test line is only one pixel wide, and the IJFR image is of lower resolution than my ORF.
    I need to figger out how to make a block
    Tom
     
  24. I forgot to add on my previous post about testing incamera jpeg rendering presets in thumbnails that my 6MP Raw PEF extracted jpeg using IJFR is full resolution meaning 3000x2000 pixels.
    I had "Native Size Only" option selected where I had the option of choosing Native and smaller with pixel dimension selector provided. My camera is a 2006 6MP Pentax K100D DSLR.
    Newer cameras may have been engineered to not provide an embedded jpeg but then how would the thumbnail and main preview be generated viewing in non-OEM image viewers that don't provide Raw data editing tools.
     
  25. The only answer I can think of is if the RAW editing data are not kept in a .xmp sidecar, but stored within the RAW it self. But, my Oly .ORFs are 13MB and the sidecars only 6KB - unless Nikon is not as smart as Oly and uses a larger font for metadata? ;)
    Thormod, the growth is solely and purely due to the much higher quality JPEG being embedded - there is no XMP sidecar with this program, and as stated, the solution is rather unique to Nikon, so there is little point in comparing to Olympus. As said, I used the open source command-line tool ExifTools to extract JPEGs, and verified myself. With all respect, it seems you still oppose the idea there is a JPEG embedded in there, but do reconsider what I wrote - by far the most likely explanation is a JPEG being embedded. Occam's Razor.
    Indeed I mixed up WIA and WIC - close relatives but not identical. I think your use of browser pertains to Windows Explorer, while I usually reserve that word for internet browsers (IE, FF etc.) only. Hence my other point.
    I'll drop you a PM for the other question.
     
  26. I don't oppose the existence of a embedded jpg in a RAW file! I question it. My knowledge of RAW stems from the literature of Bruce Fraser, and I have credible information from three major camera manufacturers supporting my opinion.
    On the other hand, with all respect, I have some people on the internet claiming the opposite. Occam's Razor right back at you:)
    However, I am getting close to the answer. I now know how to "force" a colored block into a RAW file without impacting upon a embedded jpg. A single line was simple, a block requires some math to avoid destroying the raw data
    Wouter, I will await your PM, but in the mean time,
    there is no XMP sidecar with this program​
    What program are you referencing? If it is a RAW editor, there are( to my knowledge) two options for keeping the etiting meta data. Either within the RAW file it self or a separate .xmp file. If the meta data is saved within the RAW file, I thought the RAW file would grow in size. Anyhoo, actually the statement about size in my answer was a joke. Did you miss my ridicoulous claim that the size might grow due to the use of a larger font! You do know that the meta data in the.xmp is ASCII text?

    Tom
     
  27. I refer to CaptureNX2, to which I refered earlier. It saves into the raw file, but given the increase in filesize (4MB on a 12MP photo), it's not only metadata. Metadata is pure txt, 4MB of pure text would be quite a novel. I did send you the PM via the internal messaging system of this site, it'll give you the possibility to see for yourself.
     
  28. Thanks Wouter, PM received. I have a feeling that this may convince me:)
     
  29. It did and didn't:)
    Wouter, thanks for the files. I have looked at them closely, but they do not answer whether the RAW file from the camera contains a embedded jpg or not.
    Your image no.:2 (BW) proves the existence of a preview file embedded in the edited cnx2edit.nef.
    After installing the Nikon codec, I could view both files in windows Explorer. The cnx2edit.nef thumbnail was BW.
    I then opened both files in Adobe ACR where both images were in color. Unquestionable proof that WinExplorer renders a thumbnail. from the cnx2edit.nef. Albeit strange that the thumbnail/preview does not look like the raw image.
    My conclusion is that you are right. A nef file may contain an embedded .jpg, but only if it is edited and saved using the CaptureNX2 software. (and any other Nikon SW I don't know about).
    I continue to believe that the RAW file, unedited, from the camera, does not contain an embedded .jpg.
    I continue to believe the reason the unedited RAW file renders an image in WinExplorer, and other, is the WIC and the Nikon Codec.
    It seems that you are all right. You'all, and the guy at Nikon.
    My error is not considering the possibility of a jpg being added during editing along with the editing meta data. Maybe even Oly does that, when using their SW. I use PS.
    Tom
    Wow, major headache: Does ACR add a jpg to my edited .orfs?
     
  30. I am going to disagree once more, simple test:
    Run exiftools (exiftools -b -JpegFromRaw input.raw > output.jpg) on a system without ANY WIC codec installed (i.e. XP, Vista, 7 without additional download, or Linux): it creates a JPEG that looks identical to what you see in camera. If you have your camera set up to shoot monochrome, the extracted JPEG will be monochrome. So, obviously, it is not using dcraw or similar to 'create' a jpeg on the fly - it's using something the camera already put there. Maybe a more manufacterer agnostic approach would be "exiftool -a -b -W %d%f_%t%-c.%s -preview:all dir" (ref.: http://www.sno.phy.queensu.ca/~phil/exiftool/exiftool_pod.html). As said, I only have Nikon files to play with, and on those -JpgFromRaw works fine.
    Exiftools is open source, if you're a bit PERL-savvy you can dig into the code to figure out how it works. A nice place to start would be this: http://www.sno.phy.queensu.ca/~phil/exiftool/under.html.
    You're probably aware, but for the completeness of discussion: there is no such thing as 'the raw image'; the raw data needs interpretation of some sort to render an image. There are different algoritms in different programs to do this (in camera, ACR, manufacterer software, CaptureOne, dcraw/ufraw, and so on - they all use different engines to render the final result). There is no single correct way to do this, and so it's a false assumption to believe one is a correct interpretation, and the other not. They're all equally valid, in their own ways.
    Does ACR add a jpg to my edited .orfs?​
    No. ACR never changes the raw file. It keeps previews (=jpeg files) in its cache, and its edits in either xmp sidecars, or in a database (depending on the configuration you choose).
    If you use an Olympus WIC codec, your thumbnail in Windows Explorer should not change after editing the image with ACR, nor standard filedata (i.e. Date Last Modification) - hence, the file didn't change in any way.
     
  31. No. ACR never changes the raw file. It keeps previews (=jpeg files) in its cache, and its edits in either xmp sidecars, or in a database (depending on the configuration you choose).​
    I actually hunted down Bridge CS5's cached previews of my edited (with xmp sidecar) Pentax PEF Raw files in my OS X directory. Pretty easy to find in a logically named folder structure in my user's Library.
    I copied one of the 1024 resolution versions to my desktop and opened in Photoshop and got the color management warning that it was embedded with sRGB profile. Adobe thinks of everything.
    00dHin-556728784.jpg
     
  32. Thanks Wouter for you patience. I dug up an old XP laptop No WIC no codec. no MS codec pack. I can not see a thumbnail in WinExplorer, as expected.
    Had an old version of Exiftool on it, the one where you drop imaged om the desktop icon (the camel), with the -k -w txt switches. (Creates a txt file on the desktop with the Exif data). Started updating Exiftool and wife unplugged power supply during installation. The old suffering battery has about 10 sec charge and Laptop crashed. I get the option to boot in safe mode - which it doesn't:(
    Tim: Interesting. Found the cache but very strange. The thumbnails seem to have the same size as the small jpgs I load together with the RAW file. I used to save a 640x480 jpg together with the RAW until I heard of the WIC and Oly codec. One day, I saved larger jpgs, to do a QD (quick and dirty) comparison with a Canon lady. The thumbs from that shoot are larger.
    I think it's about time to pull the plug on this thread. I declare my question for answered. Sheer probability indicates that I, Canon and Nikon are wrong.
    Thanks to everybody who have participated. I will at leisure continue investigating and should I find conclusive evidence to the existence of a thumb/preview in the unprocessed RAW file straight from the camera, I will return here and make a statement to that effect.
    Tom
     
  33. This was quick.
    I have proven myself to be wrong. I hacked an Oly .orf RAW file with a Hex editor and created a block of junk data within the RAW data. dCraw renders the Raw data containing my junk, while ExifTool renders a preview unblemished. I consider this conclusive proof of a jpg thumbnail/preview in thee Olympus RAW file.
    I am very disappointed with Oly support. We will have words!

    Tom
     
  34. Ok, final update. I have had words with Oly, and it turns out that Oly was right. A .Orf file does not contain a .jpg thumbnail/preview. The .Orf does contain a miniature of the "RAW" image in a proprietary format.
    The WIC and the Oly Codec allows third party rendition of the miniature, like WinExplorer.
     
  35. I have to stop claiming "Final"!
    Therefore: New update.
    I am now 99% convinced (again) that the Oly Raw thumbnail is a .jpg.
    I have called Oly support, again, and they confirm it, with 99% certainty.
     
  36. The thing with proprietary file formats is that the one who invents it can do anything he likes. It would appear that many raw formats are based on the idea of a TIFF-like container. Because the OS doesn't know all these proprietary formats, it requires a little helper to be able to know and show what's in them.<br>That does however not mean that what is hidden in those file could not be a plain and universally readable format like JPG. The OS just doesn't know about it, nor where to find it.<br>That Windows Explorer needs a 'codec' to be able to show the embedded miniature says nothing about the nature of the miniature. If that miniature itself is indeed encoded in a well-known well documented way, and if the information how to find it inside th eraw file is available, any programmer would have no trouble at all to write a little programm to extract it. Would not be difficult, nor much work. Just find where it starts and ends inside the container, and perhaps prepare a separate header, and write it to disk.
     
  37. Quote: Just find where it starts and ends inside the container, and perhaps prepare a separate header, and write it to disk.Unquote.
    Thats what I did. Didn't even need to prepare a header. My problem was Oly supports claim that the thumbnail was not JPEG. They have changed their mind now.
     

Share This Page