# RAW File vs DNG



## Paul_DS256 (Jul 25, 2021)

I have been getting confused by the many references to DNG as a RAW file especially with tools which promote "Convert to RAW" when the output is DNG,.

A RAW file contains sensor data which needs to go through a converted into an image format like DNG or TIF.

I know that Adobe promotes DNG as a Digital Negative, but it is based on TIFF, and needs no conversion. My understanding was that  DNG files did not contain sensor data however, from that Wikipedia reference (which I felt may be more objective) it states:

"The latter (non-raw) format is known as "Linear DNG".[36] Linear DNG is still scene-referred[37] and can still* benefit from many of the operations typically performed by a raw converter,* such as white balance, the application of a camera color profile, HDR compositing, etc. All images that can be supported as raw images can also be supported as Linear DNG. Images from the Foveon X3 sensor or similar, hence especially Sigma cameras, can only be supported as Linear DNG."
"*DNG can contain raw image data from sensors with various configurations of color filter array (CFA*_)_"
So, I still DNG is not a RAW format but is more like a TIFF+"Linear DNG"+"some sensor data".

Am I missing something?

Thanks


----------



## Roelof Moorlag (Jul 25, 2021)

This is a great resource for comparising RAW with DNG: DNG | dpBestflow
It describes similarities and differences and no, it is not just a TÏ"FF+some sensor data


----------



## clee01l (Jul 25, 2021)

Yes, you are missing something .  DNG is a very extensible file format and is in the public domain.    The file contains a Header block and now or more data blocks.
What is contained in the data block very much determines what kind of DNG file you have.  
A RAW file contains raw sensor photo site data  If you convert a RAW proprietary file like an NEF  to DNG , you are essential taking the NEF header block data an putting it into the DNG header block fields.  You then can take the raw sensor photo site data  and put it into one of the designated data blocks. 
There can also be stored a JPEG thumbnail in the proprietary RAW file format, This can be stored in a DNG data block designated as a RGB thumbnail. 
If you demosiac the proprietary RAW  data from the NEF or from  a DNG, then you create a set of RGB data that an app like Lightroom can use.  This can be saved in a DNG file format with a DNG header bloc and an RGB data block. 

If you  create a Panorama or HDR in Lightroom, this is saved as an RGB image file in a DNG file format.   Smart Previews are another type of DNG file   The information in the file header describes the number and types of data blocks. 

The purpose of converting proprietary RAW file formats is to extract the proprietary information and RAW data out of the Proprietary format and put it into the non proprietary DNG container.   The Difference between at TIFF file and a RGB DNG file is minimal and is defined by the information in the respective header block.   The Header block describes and enumerates the data blocks that follow.  It tells if the data is compressed, what kind of compression was used and  other information about each data block


----------



## Paul_DS256 (Jul 26, 2021)

Thanks @Roelof Moorlag and @clee01l . Very informative link and reply. I learned a lot which is what I wanted to do.


----------



## Linwood Ferguson (Jul 27, 2021)

So one thing I still cannot find written down clearly -- if you convert to DNG, and do NOT check the option to "embed original raw file" (I think the default is not)... 

Does it convert to linear data (referring to the definitions at the top of Roelof's posted article)?


----------



## Hal P Anderson (Jul 27, 2021)

No. It stays raw.


----------



## Linwood Ferguson (Jul 27, 2021)

Hal P Anderson said:


> No. It stays raw.



I would hope so, but have you ever seen that written down by Adobe? 

I've always been frustrated with an "open" standard but the actual process Adobe uses in the conversion is rather opaque. 

My real question is this: if I have specific RGGB sensor site values  do they remain unchanged in the conversion?


----------



## clee01l (Jul 27, 2021)

Ferguson said:


> I would hope so, but have you ever seen that written down by Adobe?
> 
> I've always been frustrated with an "open" standard but the actual process Adobe uses in the conversion is rather opaque.
> 
> My real question is this: if I have specific RGGB sensor site values do they remain unchanged in the conversion?



Written down by Adobe? Since Adobe developed the file format, all you need to do is look at the Header fields. When you create a DNG file, the header block that defines each data block that follows will specify that the data in the block is RGB or RAW, Compressed or Uncompresssed, Lossy or Lossless. 

It could not be more clear if you only look at the file spec. 


Sent from my iPad using Tapatalk


----------



## tspear (Jul 27, 2021)

Hal P Anderson said:


> No. It stays raw.


Hal,

I thought some formats are converted to RGB (I think it was X-Trans but not sure) And all sensor data is converted to a standard structure, but is stays the raw data.

Tim

Sent from my HD1907 using Tapatalk


----------



## Linwood Ferguson (Jul 27, 2021)

To Tim's point, I am not asking about the odd sensor type (though that's an interesting question) but more or less standard bayer matrixes. 

There's a difference in my mind between all those words and "original".   My question is this -- if sensor site X has (800,900,850,300) for RGGB, do those exact values appear in the same location in the DNG?   Or have they been processed, scaled, stretched, normalized or something else? 

Here is why I ask: 

It has always APPEARED to be that the debayer process was being tweaked for different cameras by Adobe and others, so it may be done slightly differently for a Nikon D5 vs a Canon 1DX.   Witness some of the early issues with Z sensors and color variations that Adobe worked out.  And the decades long debate about debayer algorithms and improvements over time that seem to help different sensors differently.

Yet a big benefit of using DNG conversion is the software no longer cares which camera -- an example is people using the DNG converter to make new cameras compatible with old Lightroom.

This SEEMS to imply one of two things:  (1) that CFA data debayer does NOT vary between cameras, in fact, or (2) that Adobe is somehow normalizing (but not yet demosaicing) the CFA data to a common format that no longer is sensor specific. 

Which makes me wonder if they are normalizing the CFA data. 

Now there is a third possibility that is somewhat convoluted: That all the algorithm variation between sensors is entirely data driven, that the reason for new programming for new sensors is to derive these data from sensor awareness and existing metadata, not because of programmatic variations in how the data is processed, and that in the DNG conversion this sensor metadata gets exposed in an open format so the unchanged CFA data can be processed sensor-specific without being coded for that sensor. 

But... I still would like to know -- is the CFA data literally unchanged?


----------



## tspear (Jul 27, 2021)

@Ferguson 

This was how it was explained to me. 

The standard DNG raw that Adobe uses is based on a Bayer (I think) pattern. The one used by 90% on the cameras made.
Some cameras include a base reading and each pixel is either a plus or minus from that value. Others are zero based. Others store the data as length followed by value. Others use delimiters.... The result is pixel row 1 col 1 has a mathematical value of 1.2234455
What the DNG conversation does it convert all these variable formats to store that 1.2234455 at position 1 by 1. 

The embed raw data option just keeps a copy of the original data in a second block.


Tim

Sent from my HD1907 using Tapatalk


----------



## Linwood Ferguson (Jul 27, 2021)

Tim, your description would not much concern me, so much as everything you described is reversible.  5 offset 10 is still 15.  What concerns me is anything they do that is not reversible, for example any kind of stretch applied, or any math applied to relative color channel (pre-debayer) values.  For example, let's pretend that two different sensors have relatively different green sensitivity.  If the DNG process normalized this by scaling green to bring them into some theoretical "normal" relative strength, to me that is no longer the original raw.  (I'm making this up out of whole cloth I have no idea). 

However, the underlying question that concerns me is this: 

I have seen over years that Adobe changes their processing algorithm to improve rendering of specific sensors.  The early Z series from Nikon is one.

So suppose that I did a DNG conversion before that change occurred, and process that DNG afterwards with lightroom, what happens?  Does it see the improved algorithm?


----------



## tspear (Jul 28, 2021)

That I cannot answer. I only researched this for the Canon 6D I used to have. The data changes are not reversible, only because Adobe does do not store the original offset Canon has in the raw file; and the offset changed for each shot. However this does not matter because the first step of any conversion process is to change everything to absolute value, which is what is stored in the DNG.

So the process improvements Adobe has done for Canon I used to see when I opened older files and compare the new rendering to older published images.

For Nikon, I have never looked into how it works or when/how values are converted, so I have no idea if the new process code could be applied to older DNGs 

Tim


Sent from my HD1907 using Tapatalk


----------



## Linwood Ferguson (Jul 28, 2021)

Thanks.  One day I may find a program to look at the details inside and look more closely.  But I still lament that Adobe's open standard lacks openness in terms of the actual conversion process.  It just worries me that somehow it can allow non-supported cameras to magically be supported, yet purports to preserve the 'rawness' of the original.  Those two concepts seems at odds.


----------



## Hal P Anderson (Jul 28, 2021)

Ferguson said:


> it can allow non-supported cameras to magically be supported, yet purports to preserve the 'rawness' of the original.



Linwood,
The way I understand things, is that the same logic that allows LRC to support the raw format from a new camera gets built into the current DNG converter. You cannot use a DNG converter that was written before the new camera came out to convert that camera's raw files to DNG.

DNGs contain the sensor data from the raw file _plus_ the colour values of the filters of the photo sites. Getting those colour values is what is involved in reverse-engineering a new raw format. There's nothing magic happening.


----------



## Linwood Ferguson (Jul 28, 2021)

Hal P Anderson said:


> Linwood,
> The way I understand things, is that the same logic that allows LRC to support the raw format from a new camera gets built into the current DNG converter. You cannot use a DNG converter that was written before the new camera came out to convert that camera's raw files to DNG.
> 
> DNGs contain the sensor data from the raw file _plus_ the colour values of the filters of the photo sites. Getting those colour values is what is involved in reverse-engineering a new raw format. There's nothing magic happening.



Exactly.   My question is whether anything is baked into that conversion.   Let's pretend for a moment that Adobe has a deficiency in their interpretation of a new camera's sensor.  Not a bug exactly, some weakness.  They fix it a year later.  In that year you converted 1000 raws to DNG.  Did it bake in that deficiency into the DNG data? 

It's one reason I am keeping with regular raw -- what you don't process, you don't change (well, bit rot excepted).   I guess I just don't trust adobe enough to let them go.


----------



## tspear (Jul 28, 2021)

@Ferguson

Ask Adobe what they do; when I looked into it for Canon they took a few days but sis get back to me.


----------



## clee01l (Jul 28, 2021)

Ferguson said:


> It's one reason I am keeping with regular raw -- what you don't process, you don't change (well, bit rot excepted). I guess I just don't trust adobe enough to let them go.


I've always thought the RAW data block was copied as is from the Proprietary File.  And in the header was information about the data contained in the data block.  Then the Demosaic process can handle the data just as it would with a proprietary file.  The data blocks in a proprietary file from one manufacturer should vary little except for size of the sensor and the photo sites .    DNGs do not do anything with the manufacturers notes.  There may be future technologies that can handle the data based upon the manufactures notes field.  Converting to DNG would for me eliminate the possibility of applying future technology from the manufacturer.


----------

