DXO Optics and DNG format

<p>I’ve just been informed by DXO that DXO Optics doesn’t read/open DNG files created by other software. I find this a little odd, because it was the intention that DNG was to be the “universal negative” RAW file. It just doesn’t make sense to me that software developers would still want to be locked into proprietary file formats, which make their product less attractive to those using competitor’s products.</p>

<p>DNG was created by Adobe in the hopes of establishing an open file format that all could access, regardless of their software source. Unfortunately, it hasn't happened yet, since the NIH attitude seems to be alive among other developers. Maybe in time, but I'm not holding my breath...</p>

<p>What DxO says doesn't make any sense if they are following the DNG openly published format. I don't use the product but I recall hearing something in the past about them not following it hence this may be the issue. It's like saying they can't open a TIFF created by another package. DNG and TIFF are close cousins, openly documented and there should be zero issues with other software products IF DxO is following the spec. </p>

<p>That's sort of what I was thinking, Andrew.</p>

<p>http://dpbestflow.org/DNG</p>

<blockquote>

<p>Other popular imaging software that does not support DNG includes DxO (can’t read a DNG, but it can create a linear DNG) and Bibble, which can’t read or write DNG.</p>

</blockquote>

<p>One thing I do know is if you wish to start with DxO, you can't get a true raw (non demosaiced) version out, even using DNG. It's a rendered image in the DNG package. So passing a DNG from DxO back to say Lightroom is kind of pointless. Might as well send LR a TIFF! It appear's DxO's "support" for DNG is simply as an output format and not a good variant at that, but they can't do anything but process the data and save it out as a non raw file.</p>

<p>Nothing stops them from opening a DNG. They choose not to. </p>

<p>For what it's worth, <a href="http://forum.dxo.com/index.php/topic,7360.msg42542.html#msg42542">DxO's reasoning</a> for "not DNG":</p>

<blockquote>

<p>We understand the issue with this format and here is an explanation: a DNG file cannot be used as just another RAW input file, since it does not contain all the calibration data Optics Pro uses when processing RAW files (especially, but not only, for the denoising algorithm).<br /><br />Therefore, even if Optics Pro could process DNG files without the data it requires, it could not achieve the same quality level as with an original RAW file, and we do not believe people who shoot RAW would be interested in such a substandard solution...<br /><br />This is the reason why Optics Pro only supports the DNG files generated by the cameras we calibrated in our labs, and for now we’d rather focus on extending our camera coverage than working on generic support for DNG files.</p>

</blockquote>

<p>It sounds <em>plausible </em>on first blush, but I don't know if it's <em>true</em>. I can't imagine what "calibration data" are provided in (say) a CR2 that wouldn't also be provided in a DNG copy of the same CR2; or which data in a Raw file "drive" Optics Pro's (excellent) NR functionality that would not be in the DNG too.</p>

<p>If it's proprietary "Maker Notes" data, that stuff is usually only fully readable by the camera manufacturer's own software.</p>

<p> </p>

<p>Not “plausible” because the changes made to DNG are non-destructive – they can be reset, just as NEF files can be reset by deleting the “xmp” sidecar file.</p>

<blockquote>

<p>It sounds <em>plausible </em>on first blush, but I don't know if it's <em>true</em>.</p>

</blockquote>

<p>Still makes no sense to me. What so called calibration data are they talking about and from where? They process the camera maker's proprietary raws don't they? </p>

<blockquote>

<p>If it's proprietary "Maker Notes" data, that stuff is usually only fully readable by the camera manufacturer's own software.</p>

</blockquote>

<p>And that data can be stored in a DNG even if everyone else couldn’t access it. There are private tags available for DNG conversion to store such proprietary info. Kinds of sounds like BS but you know who to ask, Peter Krogh who wrote the piece I referenced. </p>

<p>It would be interesting to see Adobe's opinion of DxO's reasoning...</p>

<p>LOL. I think Adobe’s opinion would be, “Stick with Photoshop and/or Lightroom, and don’t buy from our competitors.” If DXO wants to use the current situation to improve their market share, they should take a look at their own reasoning.</p>

<blockquote>

<p>Not “plausible” because the changes made to DNG are non-destructive – they can be reset, just as NEF files can be reset by deleting the “xmp” sidecar file.</p>

</blockquote>

<p>I think we're at cross purposes, Stanley. I'm well aware of DNG (or any other Raw format) changes being non-destructive - that's not the issue. </p>

<p>The issue is whether there's something in a native Raw file that Optics Pro <em>needs</em>, that would not be present in a DNG copy of the same Raw file.</p>

<blockquote>

<p>The issue is whether there's something in a native Raw file that Optics Pro <em>needs</em>, that would not be present in a DNG copy of the same,</p>

</blockquote>

<p>I'm checking with Peter and he's checking even higher up. Hopefully we'll have an answer soon.</p>

<blockquote>

<p>"The issue is whether there's something in a native Raw file that Optics Pro <em>needs</em>, that would not be present in a DNG copy of the same Raw file."</p>

</blockquote>

<p> <br>

Maybe Andrew will tell us what data gets discarded when we convert to dng?</p>

<blockquote>

<p>Maybe Andrew will tell us what data gets discarded when we convert to dng?</p>

</blockquote>

<p>I did get an answer but I need to hear back if I can use a direct quote and from whom the quote came from! The bottom line is this: as I assumed above, all metadata is preserved in the DNG file converted Adobe software. Even the proprietary metadata. I'm told this is for nearly every raw format that can be converted (read that as Nikon and Canon among others). <br /> So if someone wants to call DxO on this, by all means do so.</p>

<p>DxO works in a way that both the camera and the lens must be supported and it uses those modules to get all the adjustments.<br>

For instance, if you have a not supported camera it will not perform the conversion of the RAW files, therefore it doesn't look that surprising it doesn't read DNG files not produced by a supported camera.<br>

As an example, check their supported equipment list and look for Fuji X-Pro1. As this camera is not yet supported you have to check "Other models" to see what the program can do, and going down the list you'll confirm that it will not convert the RAW files:<br>

http://www.dxo.com/intl/photography/dxo-optics-pro/supported-equipment</p>

 

<blockquote>

<p>DxO works in a way that both the camera and the lens must be supported and it uses those modules to get all the adjustments.</p>

</blockquote>

<p>Fine but the question is if this data set is recorded by the proprietary raw file, it should also be recorded in the DNG. If DxO and read and understand it, it's not proprietary and should also be accessible in the DNG. IOW, nothing is thrown away, even data only the manufacturer could understand. </p>

<blockquote>

<p>"I did get an answer but..."</p>

</blockquote>

<p> <br>

You've been specific in past dng threads on what "useless proprietary data" gets discarded when converting to dng.</p>

<blockquote>

<p>You've been specific in past dng threads on what "useless proprietary data" gets discarded when converting to dng.</p>

</blockquote>

<p>I think you misunderstood! The point about being <strong>useless proprietary data</strong> is that <strong>only</strong> the manufacturer can understand it. That's what makes it proprietary! And DxO would have as little use for data they can't understand as well everyone else. But that's not what they are saying or rather implying. Further, my source made it very clear that if one converts using Adobe software, the proprietary data isn't throw away, it is stored in private tags. Data that again is unusable to anyone else but still there. ALL the metadata including the camera maker's private metadata data is preserved in a DNG. Who can understand and access it is another story. </p>

<p>OK, thanks to Peter for contacting Thomas Knoll and Thomas for allowing me to copy and paste exactly what he wrote back:</p>

 

<blockquote>

I don't know what they are talking about.

 

For nearly all raw formats (the main except being medium format camera backs), all the metadata (including the camera maker's private metadata data) is preserved in the DNG file created by Adobe software at default compatiblity settings. This is certainly true for Canon and Nikon raw files.

 

If Adobe warned to commit the engineering resources to it (which we don't at present), we could write a reverse converter. For example (this would work for Canon also):

 

NEF -(nikon raw software)-> TIFF

 

NEF -(adobe dng converter)-> DNG -(adobe reverse converter)-> NEF -(nikon raw software)-> TIFF

 

The two resulting TIFF files would be exactly (bit for bit) the same.

 

The main reason the NEF files would be not exactly the same as we discard the camera generated JPEG previews in the original NEF file. We could generate very similar ones using the Nikon SDK, but I don't think the Nikon SDK is bit-for-bit identical to the firmware in the camera as far as rendering goes.

 

Thomas

 

</blockquote>

<p>Oh and Thomas wrote back:</p>

<blockquote>

I would be interested in hearing DxO's reply. I would like to hear exactly what bit of private metadata they are talking about.

</blockquote>

So, anyone here who's perhaps a DxO customer want to ping them to explain?

<p>I have been doing some test with Nikon NEFs and DNG conversions, using the latest version (8.0.1.43) no compression, compatibility for Camera Raw 7.1 and later.</p>

<p>Opening both .NEF and .DNG files with <a href="http://www.rawdigger.com/">Rawdigger</a>, it seems that the DNG version does not keep the masked pixel area present in the .NEF files, which is useful for noise characterization.</p>

<p>According to the DNG specification, there is a tag for masked pixels, but I'm not sure why they do not appear in Rawdigger, so this is either a bug in Rawdigger or the free DNG converter strips away that data. </p>

<p>If data such as masked pixels is thrown away in the DNG conversion, then the argument from DXO is reasonable.</p>

<p> </p>

<blockquote>

<p>So passing a DNG from DxO back to say Lightroom is kind of pointless. Might as well send LR a TIFF!</p>

</blockquote>

<p> <br>

I don't agree with this statement. DxO DNG are "Linear DNG" meaning the they are demosaiced (no longer RAW) but they:<br>

</p>

<ul>

<li>Are not color space encoded, and it is actually the only way to preserve colours outside of AdobeRGB (the default workspace for DXO)</li>

<li>In LR you can apply the DNG profile of your choice and continue with the edits</li>

<li>You can apply white balance later as in a Raw File (the channels are still linear)</li>

<li>You can generate a DNG profile with the Xrite tool</li>

<li>You can open it in Rawdigger and do the analysis to the data</li>

</ul>

<p>This workflow had more validity in the past, when older versions of LR which did not have optical corrections.</p>

<p>DxO has also reduced the compatibility with LR. Before you could select images from a LR catalog, not anymore.</p>

<blockquote>

<p>According to the DNG specification, <strong>there is a tag</strong> for masked pixels, but I'm not sure why they do not appear in Rawdigger, so this is either a bug in Rawdigger or the free DNG converter strips away that data.</p>

</blockquote>

<p>I can't answer that but according to the spec and Thomas, that data <em>should</em> appear so it looks like a bug in RD to me. But again, that's an assumption. I could ask Thomas, but I suspect he'd say the same and RD could say otherwise. </p>

<blockquote>

<p>I don't agree with this statement. DxO DNG are "Linear DNG" meaning the they are demosaiced (no longer RAW) but they:<br /></p>

</blockquote>

<p>None of the items in your list could be encoded into a TIFF? I'm not saying that's necessary and my point was, you don't have raw data anymore. I suspect a good majority of DxO users are not as savvy about this as you are and assume: <em>It's a DNG, it is raw. </em>Based on the data you point out and based on expected workflows, makes sense it be saved as a DNG but I don't know that a TIFF couldn’t be used as well. And while this is interesting, it doesn't answer their lack of DNG on the input side which is the topic at hand. </p>

<blockquote>

<p>None of the items in your list could be encoded into a TIFF?</p>

</blockquote>

<p>I guess you could. I should have said "The current implementation of DxO..." </p>

<blockquote>

<p>it doesn't answer their lack of DNG on the input side which is the topic at hand</p>

</blockquote>

<p>Maybe it is just playing safe from their side as they only consider files straight from cameras. They actually do support DNG's from cameras (such as some Leica models)</p>

<p>Just for the record, I don't subscribe to the idea that "something" about native Raw files compared to converted DNGs somehow makes or breaks Optics Pro's ability to convert and process a a file properly... </p>

<p>This has been an interesting discussion, since I know so little about these file structures.</p>

<blockquote>

<p><br /> "The point about being useless proprietary data is that only the manufacturer can understand it."</p>

</blockquote>

<p><br /> This would suggest that, in theory, the manufacturer’s software would be best able to convert and process their own RAW file, assuming all other controls are present and work equally well. If I remember, Capture NX and View NX were Nikon’s software, but was sold off to another company.</p>