# Suggestions about automatically writing changes within XMP sidecar files (or DNG)



## Slaytanic (Aug 23, 2013)

Hi,
some months ago I've setup my Lightroom 5 Catalog in order to automatically write changes within XMP sidecar files (which in my cases means within DNG files, since I always convert RAW files in DNG when importing in Lightroom) and only after a while I've realized that this is a problem for my incremental backups, since even a small change (such as updating geolocation information or adding a keyword) forces the backup software (Carbon Copy Cloner, in my case) to copy the whole file.. So, I'm asking the following to you experts:
1) Do you use this option? Is it safer to have develop and metadata changes information stored within each image or, since I constantly backup the lightroom catalog, is it better not to store this redundant information within the files (also to speed up Lightroom)? The only advantage that I see is the ability to reimport the DNG files in lightroom, maybe in a new catalog, and having all the develop settings applied to the photos (which again is useful if the Catalog is lost or damaged), am I right?
2) If I want to avoid storing this redundant information, what should I do, aside disabling the automatic save, with the "old" photos that I have in DNG? They have the redundant information embedded within them due to the use of this setting in the past months.. Can simply tell Lightroom to remove such information from the DNGs? How should I do it?

Thank you in advance for any help


----------



## Linwood Ferguson (Aug 23, 2013)

I do use the option, but I do not use DNG, so for me it only means the XMP is backed up, which is fast and appropriate.  To the "also to speed up lightroom" however is generally not true, keeping the XMP file updated is generally slower than not (true for XMP, not sure about how DNG interacts with this, but if it would otherwise not rewrite the whole DNG I think it's much faster off than on).  To (2) I can't answer about DNG, not familiar.  For XMP if I turned it off, I would just delete all the XMP files and be done.  I keep them for redundancy in case lightroom catalog ever goes bad and I don't notice in time to have a current backup (I'm not sure this is likely enough to matter, but still I keep them).


----------



## johnbeardy (Aug 23, 2013)

I do use the option, and with DNGs. However, I have backups of the original DNGs in their virgin state, and backup my catalogue file. That means that in a disaster, I'd be able to restore all my pictures and all the work I've done on them. So that's 100% coverage - without continually backing up the DNGs in the catalogue to which xmp may be written. Don't forget too that this xmp is mainly for communicating with other apps, not backup, and it doesn't include all your Lightroom work. Why bother with automatically writing xmp - every now and then it's useful, but not worth triggering a backup of the entire fie.

So can your backup software be set to target new files only?


----------



## Slaytanic (Aug 23, 2013)

johnbeardy said:


> I do use the option, and with DNGs. However, I have backups of the original DNGs in their virgin state, and backup my catalogue file. That means that in a disaster, I'd be able to restore all my pictures and all the work I've done on them. So that's 100% coverage - without continually backing up the DNGs in the catalogue to which xmp may be written. Don't forget too that this xmp is mainly for communicating with other apps, not backup, and it doesn't include all your Lightroom work. Why bother with automatically writing xmp - every now and then it's useful, but not worth triggering a backup of the entire fie.
> 
> So can your backup software be set to target new files only?


Yes, I think that Carbon Copy Cloner can be told to backup only new files and not modified ones, but then it would be the same for me to remove this info from DNG files and keep my current backup settings. Thank you for the answer.


----------



## clee01l (Aug 23, 2013)

I started out with the automatic write to XMP setting on DNGs just like you. I discontinued it for the same reasons you have found. All of the metadata including what is never saved to XMP is stored in the master catalog. LR backups of this master catalog are redundantly backed up with Time  Machine and Carbonite as are the master image file originals. Since LR is the only app that I use that can make use of the data contained in the XMP, it makes little sense to me to write the metadata to,XMP, even though my current camera does not create DNGs and a smaller side car file could easily be accommodated by the back up processes.


----------



## Slaytanic (Aug 23, 2013)

clee01l said:


> I started out with the automatic write to XMP setting on DNGs just like you. I discontinued it for the same reasons you have found. All of the metadata including what is never saved to XMP is stored in the master catalog. LR backups of this master catalog are redundantly backed up with Time  Machine and Carbonite as are the master image file originals. Since LR is the only app that I use that can make use of the data contained in the XMP, it makes little sense to me to write the metadata to,XMP, even though my current camera does not create DNGs and a smaller side car file could easily be accommodated by the back up processes.


Also my camera (Canon 650D) does not produce DNG files, but in my first Lightroom hours I've "learned" that it is better to convert your raw files in DNG format, so I keep doing that... maybe I could abandon the DNG format and directly store the RAW+XMP files, which would alleviate the backup issue by re-copying only modified XMP files, but I think I'll stick to the DNG format (shouldn't it be slightly smaller than the original RAW files? this is a good point to stick with the DNG format for me...).


----------



## johnbeardy (Aug 23, 2013)

My suggestion would be to initially import the raw files and only convert the keepers to DNG. Then just archive the raw files and treat them as another backup. You can never have too many backups.


----------



## johnbeardy (Aug 23, 2013)

By the way, smaller file size is only a side effect of using DNG, not a reason!


----------



## Slaytanic (Aug 23, 2013)

johnbeardy said:


> By the way, smaller file size is only a side effect of using DNG, not a reason!


Ok, so do you think that I should not use DNG format and keep storing RAW + XMP files? I understand that smaller file size is a side effect of using DNG format, but it is quite important for me, unless there is a good reason to prefer using RAW files and avoid DNG conversion..


----------



## johnbeardy (Aug 23, 2013)

I wouldn't say I think you should do one thing or the other, just that you should decide on more solid grounds than saving space, which is cheaper than ever. I prefer DNG for a number of reasons (non-proprietary  and publicly-documented format, embedded metadata, updated previews, data integrity control through hash validation) and space saving is only a happy side effect which only applies to some raw file types. I know of  people with similar views who don't keep their raw files, but I don't see why one can't have DNG's advantages while also keeping the raw files. So what I do is have a point when I'm editing a shoot when I  decide which pictures to keep. I then generate the DNGs and archive the raws with the xmp sidecars. At tht point I remove the raws from Lightroom and  may never see them again - but you never know....


----------



## clee01l (Aug 23, 2013)

Slaytanic said:


> Ok, so do you think that I should not use DNG format and keep storing RAW + XMP files? I understand that smaller file size is a side effect of using DNG format, but it is quite important for me, unless there is a good reason to prefer using RAW files and avoid DNG conversion..


Whether you convert to DNG or not is really a personal decision.  There are not offsetting positives or negatives that can make the decision clear.  

If you convert to DNG, do you keep or toss the master original RAW format?  If you keep the original RAW,  does storing essentially two copies of the same image make a reasonable case for the extra HD space.  If you keep both, do you get any advantage if there is a smaller DNG file size? If you toss the original RAW, do you give up any future technology advances from the camera mfg?


----------



## Slaytanic (Aug 23, 2013)

clee01l said:


> Whether you convert to DNG or not is really a personal decision.  There are not offsetting positives or negatives that can make the decision clear.
> 
> If you convert to DNG, do you keep or toss the master original RAW format?  If you keep the original RAW,  does storing essentially two copies of the same image make a reasonable case for the extra HD space.  If you keep both, do you get any advantage if there is a smaller DNG file size? If you toss the original RAW, do you give up any future technology advances from the camera mfg?



I personally do not store the original RAW files.. I usually convert in DNG format and then backup my entire laptop drive (i.e. images + catalog) on a Time Machine volume and on a bootable drive with Carbon Copy Cloner (so, I keep two copies of my data).


----------



## johnbeardy (Aug 23, 2013)

clee01l said:


> If you toss the original RAW, do you give up any future technology advances from the camera mfg?



That'll be the day .....


----------



## Slaytanic (Aug 23, 2013)

johnbeardy said:


> That'll be the day .....


I'm sorry, I've missed the point (maybe because I do not know what does mfg stands for) and I do not understand the possible problems with future technology advances related to the camera if I do not keep old RAW files.. can you clarify?


----------



## clee01l (Aug 24, 2013)

Slaytanic said:


> I'm sorry, I've missed the point (maybe because I do not know what does mfg stands for) and I do not understand the possible problems with future technology advances related to the camera if I do not keep old RAW files.. can you clarify?


Mfg. is the standard abbreviation for Manufacturer.  but I see i have been using it incorrectly. 
http://answers.yahoo.com/question/index?qid=20070125163839AA8gDvh

Mit freundlichen Grüßen


----------



## Slaytanic (Aug 24, 2013)

clee01l said:


> Mfg. is the standard abbreviation for Manufacturer.  but I see i have been using it incorrectly.
> http://answers.yahoo.com/question/index?qid=20070125163839AA8gDvh
> 
> Mit freundlichen Grüßen


Ok, so what do Manufacturer's upgrades have to do with my missing raw of old images? I think nothing to do, right?


----------



## clee01l (Aug 24, 2013)

Slaytanic said:


> Ok, so what do Manufacturer's upgrades have to do with my missing raw of old images? I think nothing to do, right?


it is future advances advances in that you should worry about.  You know those slightly OOF photos that you are probably tossing because the camera moved?  What if in the future there was some technique developed by the camera manufacturer that could capture that jiggle and sharpen those images.  You could go back and process those originals and make sharp images wher now you only see a blur.  Most big name Mfrs. include an in camera setting for a dynamic ISO where ISO is adjusted at the pixel level to boost shadows and prevent clipping on bright spots. Adobe ignores this and recommend that it be turned off so that ACR will have a level playing field. While this information is preserved in the DNG, the Mfr. will only read their proprietary RAW formats to extract and use this. There is more information recorded like this in the Makers Notes field.  If the Mfr. develops new software to handle some of this data recorded in the Makers Notes field, it will only be of benefit if the original Proprietary RAW format file is around to run through the Mfr's advanced software.   It is this Makers Notes field that makes the Proprietary RAW format proprietary. Most Mfr's RAW (Canon and Nikon) base there proprietary RAW format on the TIFF/EP6 standard.  So does Adobe with DNGs.  
ACR will always be able to read the proprietary RAW formats. I think I save myself some time on import by not converting to DNG. I don't use other Adobe software that could take advantage of the DNG.  I see no need for creating the lossy DNGs when ultimately I want a lossless RAW (DNG or NEF or CR2).  No one has every convinced me that DNGs offer advantages over the proprietary RAW and no one will ever convince me to throw away the master original RAW file and keep only a DNG copy of it.


----------



## Slaytanic (Aug 24, 2013)

clee01l said:


> it is future advances advances in that you should worry about.  You know those slightly OOF photos that you are probably tossing because the camera moved?  What if in the future there was some technique developed by the camera manufacturer that could capture that jiggle and sharpen those images.  You could go back and process those originals and make sharp images wher now you only see a blur.  Most big name Mfrs. include an in camera setting for a dynamic ISO where ISO is adjusted at the pixel level to boost shadows and prevent clipping on bright spots. Adobe ignores this and recommend that it be turned off so that ACR will have a level playing field. While this information is preserved in the DNG, the Mfr. will only read their proprietary RAW formats to extract and use this. There is more information recorded like this in the Makers Notes field.  If the Mfr. develops new software to handle some of this data recorded in the Makers Notes field, it will only be of benefit if the original Proprietary RAW format file is around to run through the Mfr's advanced software.   It is this Makers Notes field that makes the Proprietary RAW format proprietary. Most Mfr's RAW (Canon and Nikon) base there proprietary RAW format on the TIFF/EP6 standard.  So does Adobe with DNGs.
> ACR will always be able to read the proprietary RAW formats. I think I save myself some time on import by not converting to DNG. I don't use other Adobe software that could take advantage of the DNG.  I see no need for creating the lossy DNGs when ultimately I want a lossless RAW (DNG or NEF or CR2).  No one has every convinced me that DNGs offer advantages over the proprietary RAW and no one will ever convince me to throw away the master original RAW file and keep only a DNG copy of it.


Oh, now I understand, thank you for the detailed answer! I think that I can throw away original RAW, since I'm not a professional photographer and so I can risk to miss this future enhancements of manufacturers.. If there are no clear disadvantages in using DNGs, apart from having to wait for the conversion, the decrease in size (even if minimal) is a good point for me.


----------



## sizzlingbadger (Aug 24, 2013)

Slaytanic said:


> Oh, now I understand, thank you for the detailed answer! I think that I can throw away original RAW, since I'm not a professional photographer and so I can risk to miss this future enhancements of manufacturers.. If there are no clear disadvantages in using DNGs, apart from having to wait for the conversion, the decrease in size (even if minimal) is a good point for me.



I made this MISTAKE when I first started. Keep a backup of your raw files somewhere you may regret it one day if you don't.


----------



## Slaytanic (Aug 24, 2013)

sizzlingbadger said:


> I made this MISTAKE when I first started. Keep a backup of your raw files somewhere you may regret it one day if you don't.


I'm getting a bit confused now  I can't keep a backup of both DNGs and RAWs, too much space is needed.. so, maybe I'll revert back to RAW+XMP files and completely avoid DNGs, which you all don't seem to like too much.. maybe I'll waste some MBs but I won't regret missing RAWs in the future, as you say.. BTW I must read the LR5 Missing FAQs about DNGs to see if I learn something I don't know (almost sure) and then, maybe, I'll have clearer ideas


----------



## clee01l (Aug 25, 2013)

Slaytanic said:


> I'm getting a bit confused now  I can't keep a backup of both DNGs and RAWs, too much space is needed.. so, maybe I'll revert back to RAW+XMP files and completely avoid DNGs, which you all don't seem to like too much.. maybe I'll waste some MBs but I won't regret missing RAWs in the future, as you say.. BTW I must read the LR5 Missing FAQs about DNGs to see if I learn something I don't know (almost sure) and then, maybe, I'll have clearer ideas


I think you'l find that Victoria likes DNGs. And they have a purpose.  It is just that not everyone needs that purpose. i don't.  Almost everyone will (or should) tell you not to erase the master original RAW format.  With everything in the LR catalog including metadata never written to the XMP, there is little justification for sidecars or constantly updating DNGs.  Few if any programs outside of Adobe Products read or use the metadata that LR writes into the XMP.   So, Whay make the extra effort to pull this stuff out of the catalog. 

If you have redundancy in your LR catalog backups. you have all of your XMP data in one securely backed up file. 

My workflow is to shoot NEFs, Import NEFs and keep everything except the original RAW NEF data in the catalog.  Imports are faster (no DNG conversion) and LR keeps up with the master original and I don't havbe to waste HD space with DNGs or XMP sidecar files.


----------



## darky (Nov 5, 2013)

Very new in this good forum I like to report about a terrible experience with DNG. My new DSLR Pentax K-30 does produce only DNG format and optionally JPG but no PEF raw files. So I have no chance to use RAW and XMP sidecars and low success with setting read-only-for DNG files, because most applications like to modify the DNG (I dislike software which modifies my original files and prefer fast backups of XMP sidecars only). In addition the DNG files of my camera couldn't be processed from some raw editors in the beginning. It took e.g. one year for CaptureOne to support the embedded image information correctly. The color and brightness in a couple of raw editors was very very different and in C1 the colors were totally wrong until this year (expensive SW upgrade + expensive 64-bit HW upgrade). May be the DNG format is standardized like a container, but the real image content is not standardized and still proprietary. With this every old or new DNG from a camera requires support from the used software.  

Currently I replace the embedded large preview in my camera DNGs with mid size preview through AdobeDNGConverter. Before doing so I export the big preview as separate JPG file with ExifTool. Through this trick I do not need a JPG from the camera itself (saves time and memory). The processed DNG files are reduced in file size and I can immediately see, when a image software is using small preview instead of full size DNG rendering for display. In addition the exported large preview has lens correction and the new mid size preview not. That is good for fast comparisons. Finally my DNG files are set write-protected. This read-only attribute causes some trouble for DAM software and Lightroom, but I see no other chance to protect DNG modifications. I even tried to modify the file extension DNG to something else (e.g. *.rwx or *.rah). The result is funny. Some image software is still able to show the DNG preview for such files and even reads and writes XMP sidecar files in this case. But Lightroom is more "intelligent" and does detect, that such files are still DNG and tries to write the XMP data directly in the file. So the modification of file extension does work but does not help.  

I wonder how I can convert my original camera DNG files back to "real" RAW format which does allow XMP sidecar files? Is there a way to convert DNG to some RAW format? Back conversion with AdobeDNGConverter is not possible, because there is no original RAW existing/embedded.  

I really like DNG, JPG, ... but I dislike any software which tries to modify embedded metadata without my authorization. I think, there should be options to force reading and writing XMP sidecar files for ALL file formats/extensions in Lightroom and other image software. To my knowledge only JPhotoTagger and GeoSetter do respect sidecar files for DNG and JPG. It seems that users who share my opinion are quite rare and alone in the world. And the MetaDataWorkingGroup seems to have also other opinions...


----------



## Denis de Gannes (Nov 6, 2013)

I just love the ability of LR to store all the info in the Catalog as apposed to ACR which stores the info in .xmp

I can now avoid the total confusion of using .xmp sidecars or DNG files. requiring the continual backup of these files.

I make backups of my original raw files. (two copies is considered necessary). ONCE only no further backups.

Make regular backups of my catalog file. (two locations if considered necessary)

Avoid the possible corruption of .xmp files if you are using LR and ACR with your raw files or if several users accessing the raw files at different times and saving changes to the xmp files. (unlimited opportunities for corrupting/confusing your xmp data.)

You can have several users using the same raw "original" files at different times using their own "Catalog" as long as they are not writing data to xmp.


----------



## sizzlingbadger (Nov 6, 2013)

darky said:


> Very new in this good forum I like to report about a terrible experience with DNG. My new DSLR Pentax K-30 does produce only DNG format and optionally JPG but no PEF raw files. So I have no chance to use RAW and XMP sidecars and low success with setting read-only-for DNG files, because most applications like to modify the DNG (I dislike software which modifies my original files and prefer fast backups of XMP sidecars only). In addition the DNG files of my camera couldn't be processed from some raw editors in the beginning. It took e.g. one year for CaptureOne to support the embedded image information correctly. The color and brightness in a couple of raw editors was very very different and in C1 the colors were totally wrong until this year (expensive SW upgrade + expensive 64-bit HW upgrade). May be the DNG format is standardized like a container, but the real image content is not standardized and still proprietary. With this every old or new DNG from a camera requires support from the used software.
> 
> Currently I replace the embedded large preview in my camera DNGs with mid size preview through AdobeDNGConverter. Before doing so I export the big preview as separate JPG file with ExifTool. Through this trick I do not need a JPG from the camera itself (saves time and memory). The processed DNG files are reduced in file size and I can immediately see, when a image software is using small preview instead of full size DNG rendering for display. In addition the exported large preview has lens correction and the new mid size preview not. That is good for fast comparisons. Finally my DNG files are set write-protected. This read-only attribute causes some trouble for DAM software and Lightroom, but I see no other chance to protect DNG modifications. I even tried to modify the file extension DNG to something else (e.g. *.rwx or *.rah). The result is funny. Some image software is still able to show the DNG preview for such files and even reads and writes XMP sidecar files in this case. But Lightroom is more "intelligent" and does detect, that such files are still DNG and tries to write the XMP data directly in the file. So the modification of file extension does work but does not help.
> 
> ...




You can't convert your dng to any other raw format. You could try setting the dng files to read only, I think this will force LR to create an xmp because it can't write to he dng file.


----------



## johnbeardy (Nov 6, 2013)

sizzlingbadger said:


> You can't convert your dng to any other raw format. You could try setting the dng files to read only, I think this will force LR to create an xmp because it can't write to he dng file.



That method works with Bridge, not Lightroom.

In my view there really isn't a problem here. Make a backup of DNGs immediately after importing (I call these my "virgin DNGs"), and then allow Lightroom to update the catalogued or "working DNGs" without worrying about backing them up after changes. The metadata written to these files does not contain all your Lightroom work, and it's obviously silly to keep backing up their image data. Your backup should the catalogue, and the virgins. This combination means that you can recover all your images and all your work.

John


----------



## Slaytanic (Aug 23, 2013)

Hi,
some months ago I've setup my Lightroom 5 Catalog in order to automatically write changes within XMP sidecar files (which in my cases means within DNG files, since I always convert RAW files in DNG when importing in Lightroom) and only after a while I've realized that this is a problem for my incremental backups, since even a small change (such as updating geolocation information or adding a keyword) forces the backup software (Carbon Copy Cloner, in my case) to copy the whole file.. So, I'm asking the following to you experts:
1) Do you use this option? Is it safer to have develop and metadata changes information stored within each image or, since I constantly backup the lightroom catalog, is it better not to store this redundant information within the files (also to speed up Lightroom)? The only advantage that I see is the ability to reimport the DNG files in lightroom, maybe in a new catalog, and having all the develop settings applied to the photos (which again is useful if the Catalog is lost or damaged), am I right?
2) If I want to avoid storing this redundant information, what should I do, aside disabling the automatic save, with the "old" photos that I have in DNG? They have the redundant information embedded within them due to the use of this setting in the past months.. Can simply tell Lightroom to remove such information from the DNGs? How should I do it?

Thank you in advance for any help


----------



## darky (Nov 7, 2013)

Thank you John for the description. I understand this, but my current workflow, version management and backup strategy has problems with this.
I like to keep ALL originals and derivate files together in the same directory, because they are managed as versions with my DAM and it is easy to have an overview about versions.
Separation of virgin DNGs from working DNGs would result in more complex syncronization and backup procedures. With real RAW files this separation isn't necessary.

I use XMP sidecars for most office, video, audio, raw, graphic, ... files with a couple of sidecar based tools and the backup process is very fast and simple. 
For me the sidecar files are mainly for keywords, categories, rating, labels, dates and other administrative DAM information, which needs to be shared between various programs.
The XMP sidecars really "saved my life" during a last year change from one DAM to another, because the XMP sidecar information could be used for transfer of metadata from one database to the other.
I do care about database-export and backup of the basic text information, but not about any raw editing information or any updated image previews.

But I will use your proposal to investigate the option to exclude modified DNGs from backup, when a target virgin DNG is already existing.
But with this any source file renaming or deletion needs special manual treatment of the target file. And the above basic XMP information embedded in DNG and JPG will not be part of the backup.
DNGs with read-only attribute would give basically the same result, but then no more XMP export with LR is possible. It is very strange that Adobe products are so different/inconsistent with this point.

It is really a pitty, that not all software is respecting, reading and writing sidecar XMPs for ALL file formats or at least for read-only files.
As a XMP sidecar file fan I see only drawbacks with DNG files instead of RAW files. 
May be I have to drastically modify my complete workflow, backup and XMP sidecar strategy... 

Damian


----------

