# Converting RAW to DNG - opinions?



## Luc (Dec 20, 2014)

I'm looking for some opinions about converting RAW to DNG.. I've been doing some research and know about the pro and cons... One of the pro arguments i like is that DNG stores everything in one file so you don't have to deal with side files... But on the other hand this means that if a file gets corrupt you loose everything.  I know it's a personal choice but I was wondering if most of you do convert to DNG and if not why... 

Luc


----------



## clee01l (Dec 20, 2014)

Storing everything in one file is not IMO a "pro".  First each time you make a change, your backup program must make a new full backup of a relatively large file. If there is an XMP file, it is quite small and backs up quickly.  More importantly, there does not need to be an XMP file at all.  Your backup software backs up the master original once and it stays untouched on the backup drive until and IF it is needed.  Everything related to your master image is stored in the LR catalog.  Not everything related to your LR efforts with your master original RAW file are stored in the XMP sidecar or XMP header of a DNG.  And if the catalog is backed up regularly with the LR backup copy function and those copies are regularly backed up by your backup software, then you have everything that your need to recover the catalog and the master original images. 

The advantage that some (not me) see in a DNG is the checksum value embedded in the DNG to validate the file is intact.  In order to take advantage of this feature you have to periodically validate every DNG to ensure that they are not corrupt. 

I keep master original RAW file copies that came out of the camera.  I see no benefit in compounding my data storage requirements with a DNG copy of the same data. I see  negative benefit of destroying my master original RAW files that came out of the camera in favor of a DNG replacement.  So, I keep master original RAW files which are imported into my master LR catalog and I keep redundant copies of the catalog backup files.  All of this is backed up with TimeMachine locally and Crashplan with unlimited Cloud storage.


----------



## johnbeardy (Dec 20, 2014)

My long term experience is that storing everything in one file is a "pro". I first started using DNGs around the time of CS2, and having all the metadata inside the file meant I was able to migrate my images and metadata from one image management system to another. Sidecar XMP files are an ugly workaround. 

Secondly, you do not need to keep backing up the large files each time you make a change (and I wish people would stop repeating that nonsense  ). Apart from not including all your adjustment and metadata work, it does not matter if LR writes to the files which are recorded in the catalogue. That's because the true backup of your images is the backup you make when the DNGs are first created. 

If you keep raw files too, you have the advantage of additional backup in a different file format. Space is cheap, and 4 backups are better than 2.


----------



## clee01l (Dec 20, 2014)

johnbeardy said:


> Secondly, you do not need to keep backing up the large files each time you make a change (and I wish people would stop repeating that nonsense.


 While you are correct in a sense, this means that you need to specifically instruct your backup software to exclude the DNGs from backup after the backup software has backed them up once. And continue to exclude new DNGs that are added regularly. That means that any data recorded in the XMP section of the DNG is not going to be backed up. The supposed benefit to XMP (whether in DNG or as a sidecar) is to provide an additional copy of some of the metadata recorded in the LR catalog. If so, this benefit is lost and the only disaster recovery source for the data contained in the XMP section is going to be in the LR catalog file backup. If that is the case, why write metadata back to the XMP section of the DNG? 

Now, which is nonsense? Writing metadata to XMP and not backing it up or simply not writing metadata to XMP at all and relying upon backups of your LR backup files instead of a copy of some metadata in a DNG?
My contention all along is that you do not need to instruct LR to write metadata to the XMP. By eliminating that from your LR workflow, you free up CPU and I/O cycles in LR and there should be less overhead and some improvement in LR's performance (although I doubt that it will be detectable).


----------



## johnbeardy (Dec 20, 2014)

It's not exactly difficult to exclude the catalogued DNGs from your backup routines. Stick them on a separate drive, for example. That's a non-issue. 

It doesn't matter that the DNG's XMP section is not backed up - you routinely back up your catalogue, and that includes all your work. So I'd agree with your contention about not writing XMP, but if you do so it doesn't then mean you need to back up these DNGs. If you then want to back them up, that's only your choice, but it's misleading to claim you "must make a new full backup of a relatively large file". You really don't.


----------



## clee01l (Dec 20, 2014)

johnbeardy said:


> It's not exactly difficult to exclude the catalogued DNGs from your backup routines. Stick them on a separate drive, for example. That's a non-issue.
> 
> It doesn't matter that the DNG's XMP section is not backed up - you routinely back up your catalogue, and that includes all your work. So I'd agree with your contention about not writing XMP, but if you do so it doesn't then mean you need to back up these DNGs. If you then want to back them up, that's only your choice, but it's misleading to claim you "must make a new full backup of a relatively large file". You really don't.


While I want my master images copies backed up (once is fine), I don't want to have to manually manage that process outside of my Backup app  Sticking them on a separate drive does not back them up once although it does make it easier to manage the backup software.  Saying it is a Non-issue does not make it a non-issue  any more than saying that constantly changing DNGs will always get a new full copy when included in the backup scheme is nonsense. 

John, I don't see a need for Writing XMP to a file (sidecar or otherwise) and I don't find merit in converting perfectly usable proprietary RAW files to DNG.   Telling me that this is nonsense or a non issue is not going to change that, Why do you persist?   Luc asked for opinions.  I gave hime mine and you gave hime yours  Let's leave it at that and leave off with the evangelism.   We are all sinners in someones eyes.


----------



## johnbeardy (Dec 20, 2014)

There's a big difference between what's opinion and what's repetition of a tired assumption, Cletus, and I pointed very precisely at what I called nonsense : "you do not need to keep backing up the large files each time you make a  change (...that nonsense)". How one handles backup routines really makes little difference. If you use a DNG workflow, you just tweak your backup procedures. I suggested one physical approach, or you can tell your backup software to ignore modified files. Knocking down straw man arguments is not evangelism....

DNG then provides the long term benefit of your picture archive being in a publicly-documented file format, an adjusted preview that's visible in other apps, embedded metadata that's easier to read in other apps, data validation, embedded camera profiles, and usually-smaller file sizes.


----------



## Luc (Dec 20, 2014)

Hey guys,  take it easy lol. As I stated,  i know it's a personal choice so there is no "one size fits all" answer. 

I can relate to John's opinion but truth be told,  as I'm a lazy bum,  I'm not to keen on doing backups where I need to remember what and what not to backup. My backup system is fairly simple since my pix and LR files are stored on an EHD and my automated backup system also puts them in the cloud without thinking about what or not to backup. Since I'm not a pro and don't depend on my pics for my lively hood  I'm not paranoid on having backups on a zillion servers,  HD tucked away in a secure volt lol. 

On the other hand,  Cletus has me confuzled a bit because i was under the impression that RAW does need the XML side car because the XMP file contains all the changes you have made to the file in Lightroom. Which means if the Lightroom catalog gets corrupted, lost or for whatever reason is not working right all you will need to do is reimport the images into a new catalog and Lightroom will automatically read those XMP files in the same folder and apply those changes immediately to the files. So,  if LR stores everything in the catalog and it gets corrupted,  it's just a matter of using your backup?


----------



## clee01l (Dec 20, 2014)

Luc said:


> On the other hand,  Cletus has me confuzled a bit because i was under the impression that RAW does need the XML side car because the XMP file contains all the changes you have made to the file in Lightroom. Which means if the Lightroom catalog gets corrupted, lost or for whatever reason is not working right all you will need to do is reimport the images into a new catalog and Lightroom will automatically read those XMP files in the same folder and apply those changes immediately to the files. So,  if LR stores everything in the catalog and it gets corrupted,  it's just a matter of using your backup?


If the LR catalog gets corrupted, you use a back up catalog to recover.  No need to reimport.  You are being foolhardy if you aren't redundantly backing up your master LR catalog file.  The XMP sidecar fils does contain the last set of adjustments and it can be used by other Adobe apps. AFAIK, no other photos Apps besides those from Adobe make use of the XMP data.  It does not contain the adjustment history or the membership in collections or Publish Services or virtual copy adjustments.  The only way you can recover everything is with a good recent catalog backup.

So, the XMP data should not be confused for a backup strategy.


----------



## Luc (Dec 20, 2014)

Cletus, indulge me to sketch  a scenario:

I'm doing post-processing in LR and suddenly LR crashes or my machine shuts down... 

At that moment there is no backup ofcourse,  so if I didn't enable the XML setting then what? 

Greetz 

Luc


----------



## clee01l (Dec 21, 2014)

Luc said:


> Cletus, indulge me to sketch  a scenario:
> 
> I'm doing post-processing in LR and suddenly LR crashes or my machine shuts down...
> 
> ...


If LR/Windows crashes it won't likely destroy the catalog.  Everything but the last operation will be in the catalog or in the Journal file.  And LR will pick up where it left off once you remove the *.lock file.  

I back up once a week.  If my catalog suddenly gets corrupted. I have a backup that is not more than one week old. I replace my corrupt catalog with the most recent backup. I might have a few images to reimport and regenerate my work efforts on these images.  I am comfortable with that level of risk.  If you are not, you can backup the catalog everytime LR shuts down or once a week, your choice.  
More often, you will need your backup catalog to correct "Stupid User Mistakes".  I have gone back 3 or more months to find a backup catalog that did not have a user induced keyword error on about 2500 images.

Different people are comfortable with differing levels of risk.  At one extreme, there are many that could not be bothered with any backup. When their HDD crashes they lose all of their data.  For some this is a conscious decision for others it is a decision made through ignorance. At the other extreme, there are those that insist on a backup every step of the way. This too can be a conscious decision or through an improper understanding of  how the digital environment works. I have had a long career in the IT industry and fall into the "belt and suspenders" group. I've probably been burned about every way that you can in regards to data security.  I think my approach is reasoned.  I used to shoot DNG and I used to write metadata back to the DNG and did not consider the overhead of backing up a full DNG everytime I made a change.  When I realized that I had everything that I needed in the LR catalog I also realized that I could turn off metadata update to the master file.  When I moved to Nikon, NEFs were my only choice and I was not needing a sidecar file anymore conversion to DNG was not reasonable to be for the checksum benefit that I mentioned previously. 

In the end, you'll have to weigh the level of risk that you are willing to take and respond accordingly.   I worked out my workflow through trial and error.  I expect you will too.  I streamline it as much as possible and creating DNGs at import is an extra step and additional overhead.  It takes time and processing capacity to create a DNG on import.  It also take time and processing capacity to automatically write XMP data back to an XMP file or DNG section.   For me I insert a camera card in the card reader, choose an import preset and hit the import button. I want to be looking and culling newly imported images as soon as the first image appears in the Previous Import special collection.  Writing out a DNG file just makes that wait a little longer.


----------



## Denis de Gannes (Dec 21, 2014)

Cletus, good post I agree everyone needs to adopt a workflow that they are comfortable with.

Lightroom is my raw converter of choice and its design by default allows me to work without xmp sidecars or the need for DNG files.

That said I am an amateur and not working with client files.


----------



## Luc (Dec 21, 2014)

Cletus, 

Your answers opened up some LR techniques for me that I'm sure going to follow. I didn't knew that in my scenario the catalogue would be saved for about 99.9999%. And also the fact that a XML sidecar in reality does not add much since the catalogue already stores everything.  

Besides,  as a hobbyist I'm not much concerned if I should loose a bit of metadata... 

To sum up,  my work flow will be: no sidecar,  keep the RAW files,  import and get culling. As for backing up,  I'm inclined to backup each time LR closes.  Doesn't take long is good for my peace of mind lol. 

Anyway tnx for the great advice. 

Greetz 

Luc


----------



## camner (Dec 30, 2014)

And using the plugin TPG LR Backup will automatically zip the LR backup files to a small fraction of their original size so there's virtually no overhead to keeping lots of backed up catalogs.


----------



## davidedric (Dec 30, 2014)

My take is that I rely on the catalogue, though I am one of those who backs up daily on exit, then periodically cull the backed up catalogues so that I have them at what seems sensible time points.

Cletus has set out his good reasons.  The other one for me is that after any significant shoot, my RAW images are copied into my preferred back up storage(s).  That's it.  Period.  The RAW images never change, and I then only have to think about the catalogue.

Dave


----------



## ToomasV (Mar 3, 2015)

Hi
I didnt want to create new thread cause I have question generally on the same theme - converting RAW to DNG.
In the LR5 The Missing FAQ on the page 76 there is MakerNotes part where I can read, that _'secret data'_ of Nikon and Canon RAW files will be carried over to the DNG file (when converting). So with especially Nikon and Canon files NO info is lost and Adobe copies all info to DNG (even if Adobe doesn't understand it). Can I have comment from the author? Thank You! Ofcourse all comments are welcomed.


----------



## clee01l (Mar 3, 2015)

ToomasV said:


> Hi
> I didnt want to create new thread cause I have question generally on the same theme - converting RAW to DNG.
> In the LR5 The Missing FAQ on the page 76 there is MakerNotes part where I can read, that _'secret data'_ of Nikon and Canon RAW files will be carried over to the DNG file (when converting). So with especially *Nikon and Canon files NO info is lost and Adobe copies all info to DNG* (even if Adobe doesn't understand it). Can I have comment from the author? Thank You! Ofcourse all comments are welcomed.


While this data is preserved, It is only accessible in Nikon/Canon proprietary software when it is  stored in its original format. It is not that the data is secret, it is that the data is proprietary and useful only to Nikon/Canon processing Nikon data.  Makers Notes is a single EXIF standard data structure.  Inside this "blobish" data structure are Nikon's/Canon defined data structures useful to Nikon/Canon for data processing.  The only way to keep all of your future options open is to keep the Proprietary RAW file AND the DNG.  There is even a provision to wrap the DNG around a Proprietary RAW file block which can be extracted later if need be.  Of course this results in a file that is approximately twice as large as the original NEF/CR2.   Once you convert the proprietary file to DNG you can not convert it back.


----------



## rob211 (Mar 7, 2015)

I'd like someone to clarify for me what gets stored in XMP in sidecars and DNG files.

Is there anything done in LR or PS that is NOT stored in XMP in either format? (I assuming either manually or automatically writing to file.)

Is there a practical difference between the XMP in a sidecar and the XMP in a DNG file, aside from the one-file difference. I'm asking more about content and accessibility.


----------



## johnbeardy (Mar 7, 2015)

Missing are - flags, stacking, virtual copies (inc soft proof), history steps, assignment to collections. I've probably forgotten one or two others. Manual or auto make no difference.

The xmp is identical whether it's in a sidecar or embedded in a file. In the past some photo editing programs didn't read sidecars, but nowadays the only "content and accessibility" difference is you can open the sidecar in a text editor.


----------



## rob211 (Mar 7, 2015)

johnbeardy said:


> Missing are - flags, stacking, virtual copies (inc soft proof), history steps, assignment to collections. I've probably forgotten one or two others. Manual or auto make no difference.
> 
> The xmp is identical whether it's in a sidecar or embedded in a file. In the past some photo editing programs didn't read sidecars, but nowadays the only "content and accessibility" difference is you can open the sidecar in a text editor.



Thanks tons; all intuitive I guess since they relate to LR itself. Rating is included, however?


----------



## johnbeardy (Mar 7, 2015)

rob211 said:


> Thanks tons; all intuitive I guess since they relate to LR itself. Rating is included, however?



Yes, it is. It illustrates how the primary purpose of saving xmp is communication with other apps, not backup. Other apps understand ratings or colours, but no other app understands flags, stacks etc.


----------



## clee01l (Mar 7, 2015)

johnbeardy said:


> Yes, it is. It illustrates how the primary purpose of saving xmp is communication with other apps, not backup. Other apps understand ratings or colours, but no other app understands flags, stacks etc.


Long before there was a Lightroom, there was a "committee" to define the standard fields to be included in the EXIF/IPTC Header.  Rating (*) and Labels (not color labels but a text field that LR associates with the Color Label Set) were two of the fields that make up the fields in the Header.  DNG, TIFF and JPEG include in addition to EXIF and IPTC sections, a provision for an XMP Section to extend the metadata to include non standard data structures.  If the file standard has a provision for XMP, it is included in the file structure, otherwise it is added to a separate "sidecar" XML text file.


----------



## johnbeardy (Mar 7, 2015)

Well, I keep pointing out that the X in XMP stands for extensible, and nothing prevents Adobe adding other fields like flags to the metadata saved to the files, if they wanted a backup. But they only save information like adjustments that other Adobe apps will read, or fields like ratings which are widely read by third parties. So at least in the context of LR, saving metadata to XMP is for communication.


----------



## rob211 (Mar 8, 2015)

I love the whole system of file annotation with images; I wish it was more common with other file types. The Mac and Windows OS's can tag, but at least with Mac it's rather limited, and is part of the file system and not really the file itself. And although not primary, it does facilitate the storage of the info in a format that makes it amenable to backup. I've even had some interesting instances of where a sidecar could be used as a sort of versioning backup. And PhotosInfoPro eg makes interesting use of it to pass info from iPads without having to transfer big image files.

Mr. Beardsworth has a nice plugin called Big Note that can be used to enter custom data. I don't suppose LR itself is going to have any provision for custom fields in the new version?

And it's kind of amusing that Apple's new Photos doesn't understand its predecessor's star ratings, and converts to keywords. Which really aren't the same ("<" doesn't really work with the keyword text "2 stars", which is what I think it uses, not a string of "*").


----------

