# more sidecar and catalog questions



## Stumped (May 21, 2020)

So there is a lot out there, and here, on the above.  Ive been reading, re-reading and re-re-reading to the point where im possibly more confused than I was before I started.  It began from another thread when I mentioned .dng and .cr2 files, convert or not to - another ball of wax itself....  So, if I may, lay out what I understand to be correct for confirmation and some questions.   I save all original files externally from local hard drive outside of LR.   I currently shoot raw .cr2 files and had been importing them to LR as such and have the "Automatically write changes into XMP"  checked in the catalog Metadata settings.  QUESTION - Doing such with that setting checked gives me .xmp sidecar files, right?

I have also, after reading, started to import .cr2 files as .dng files, saving the original .cr2 externally and have the same "Automatically write changes into XMP"  checked in the catalog Metadata settings.  QUESTION - Because i now have that setting checked w/ a .dng file, LR writes to the XMP section of the file and not a xmp sidecar file?

Ive read that metadata refers to develop settings, tags, labels, crops, etc. that are primarily stored in the catalog file.   QUESTION - This is the default whether the above setting is checked or not checked, right?  Where can I see that / those files?

To make sure I fully understand this and possibly redundant - jpeg, png, tiff and psd and dng files - allow for writing to the xmp section of the file by having the above option checked and proprietary .cr2 and HEIC will get xmp sidecar files when checked?

Some examples of other proprietary raw files besides .cr2, HEIC ?

Confirm: Ive read that all changes to pictures are always recorded to the catalog - yes or no?  If yes, when recorded to the catalog, does this imply to a xmp sidecar file or just the catalog?  

If saving to a sidecar file, will that picture be able to be opened in PS as is, w/o further changes?

Confused - Under the Catalog Settings window, Metadata, "Automatically write changes into XMP", is this what writes the sidecar file or is this referring to the "into xmp" section of the file?

Are .xmp sidecar files hidden by default?  Im on a Mac.

So now that ive probably asked the same question 18 different ways - all this reading as mentioned has led to confusion for me.  Additionally some places are written by those much more experienced and I think sometimes some things are taken for granted and maybe not explained very well and im just trying to make sure I fully understand this before getting too deep into it.

Additionally, from the other thread I briefly made reference to dng vs cr2 ---  this led to convert or not to - a whole other argument - file check vs no file check - in hind site, at this point, probably a non-issue for me.  Still trying to learn that just because i read it doesn't mean I have to try it.  Maybe just concentrate on basic workflow.

Finally, this is what id like to accomplish, and my understand, please correct if not - as mentioned, on a Mac and I use time capsule and store originals on external drive.  Its my understand(please confirm), that if I write to the xmp portion of a file, those changes are picked up as file changes by time capsule and I get a new back up copy with every change made, right?  This seems like a waste of space compared to if I were just writing to a sidecar file and that was the file being backed up and not the whole image file.

What settings should I set to use sidecar files and not back up the picture with every edit made?

Finally, If worried about space, and if I were to keep the settings as is  and seeing as I save originals external, Im safe to exclude this LR (picture only directory) from Apple backups and still be safe?  I am backing up my catalog in LR AND time capsule.

Thanks all for any help here in getting me straight!

Attached is what I have now.


----------



## Roelof Moorlag (May 21, 2020)

Stumped said:


> I currently shoot raw .cr2 files and had been importing them to LR as such and have the "Automatically write changes into XMP" checked in the catalog Metadata settings. QUESTION - Doing such with that setting checked gives me .xmp sidecar files, right?


Yes



> QUESTION - Because i now have that setting checked w/ a .dng file, LR writes to the XMP section of the file and not a xmp sidecar file?


Yes



> Ive read that metadata refers to develop settings, tags, labels, crops, etc. that are primarily stored in the catalog file. QUESTION - This is the default whether the above setting is checked or not checked, right? Where can I see that / those files?


This data is saved into the Lightroom database (SQLite)


----------



## Roelof Moorlag (May 21, 2020)

> Confused - Under the Catalog Settings window, Metadata, "Automatically write changes into XMP", is this what writes the sidecar file or is this referring to the "into xmp" section of the file?


Both! It writes XMP files for photo's wich i can not write into the XMP section of the file itself



> Additionally, from the other thread I briefly made reference to dng vs cr2 --- this led to convert or not to - a whole other argument - file check vs no file check - in hind site, at this point, probably a non-issue for me. Still trying to learn that just because i read it doesn't mean I have to try it. Maybe just concentrate on basic workflow.


Preserving the integrity of your files is probalby 'basic workflow' i think? In that case, the file validation option of DNG can be very valuable.



> What settings should I set to use sidecar files and not back up the picture with every edit made?


If you use sidecar files they are the only files that are changed. Only changed files are in the backup.
When you have backups of your files and from you catalog (.cat file) then there is no need for XMP however.



> inally, If worried about space,


Why? Storage is very cheap these days


----------



## Paul McFarlane (May 21, 2020)

Roelof Moorlag said:


> When you have backups of your files and from you catalog (.cat file) then there is no need for XMP however.


I agree with all your comments, including this one - there is no need. However, as an additional step (in event of catalog failure and inability to repair) or for reading by some other programs outside of Lr, they can have their place.


----------



## Roelof Moorlag (May 21, 2020)

Yes, i see what you mean and Lightroom does offer sidecare files, free to use.
However, my impression was that the OP does consider them as necessary


----------



## Paul McFarlane (May 21, 2020)

Roelof, totally agree with you! And great point-by-point breakdown of answers too!


----------



## Stumped (May 21, 2020)

Roelof Moorlag said:


> Yes
> 
> 
> Yes
> ...


So the data is saved to the lrcat file and is visible only with SQLite?  Thank you - I am trying to get this multi quote down....


----------



## Stumped (May 21, 2020)

Roelof Moorlag said:


> Both! It writes XMP files for photo's wich i can not write into the XMP section of the file itself
> 
> 
> Preserving the integrity of your files is probalby 'basic workflow' i think? In that case, the file validation option of DNG can be very valuable.
> ...



To clarify - so if I have the box check w/ .dng file, it writes to the XMP section of the file and if I have it checked w/ .cr2 files I get a .xmp sidecar file?

yes

when you refer to "XMP", do you mean XMP Metadata section of file or .xmp sidecar?  Nomenclature ?  Because I have .cr2 files mixed with .dng files in the same catalog I do not want to be backing up each .dng when I make a small change.  Thats why I wanted to know if I could unselect that directory from the time machine back up.


----------



## Johan Elzenga (May 21, 2020)

Paul McFarlane said:


> I agree with all your comments, including this one - there is no need. However, as an additional step (in event of catalog failure and inability to repair) or for reading by some other programs outside of Lr, they can have their place.


Please consider two things. Catalog failure is best dealt with by making regular catalog backups,  not by writing changes to XMP. Too many catalog stuff is not written to XMP. And writing changes to XMP will trigger your image backup application if you use DNG. The entire DNG file will be copied again.


----------



## Stumped (May 21, 2020)

Roelof Moorlag said:


> Yes, i see what you mean and Lightroom does offer sidecare files, free to use.
> However, my impression was that the OP does consider them as necessary


Huh?


----------



## Stumped (May 21, 2020)

Sorry for the confusion - I didnt mean to say, if I did, that anything was not necessary.   I'm just trying to figure out the above and what happens when, so that I have a full understanding of all the settings.  As far as I can see now, as a new user, seems redundant if I have an original unmolested file in offsite storage to keep making backups when I might just be making simple edit (by writing into the XMP section of a file).  So I'm seeing what I need to have set to only have the sidecar file, that the changes be made there and that would be the file that would be included in the backup by Time Machine and how to accomplish that and any additional provisions, if necessary(another question), that would need to be done to have the file to be be able to be used by PS or another edit program while using sidecar file.....


----------



## Roelof Moorlag (May 21, 2020)

> As far as I can see now, as a new user, seems redundant if I have an original unmolested file in offsite storage to keep making backups when I might just be making simple edit (by writing into the XMP section of a file)


1. Writing to XMP is not necasary for a dam backup
2. If you decide to write into XMP then only the XMP sidecarfiles are changed so in the backup. When the file has a XMP section, there wil not be written a sidecar file. So, in these situations the whole file wil be in the backup
3. There is no option in Lightroom to force XMP sidecar files on files with a XMP section (as DNG)


----------



## Stumped (May 22, 2020)

Roelof Moorlag said:


> 1. Writing to XMP is not necasary for a dam backup


Ok, you are a "Power User" with thousands of posts and I am new trying to understand this - there's no need to curse.  I never said writing to XMP is necessary for a backup.


Roelof Moorlag said:


> 2. If you decide to write into XMP then only the XMP sidecarfiles are changed so in the backup. When the file has a XMP section, there wil not be written a sidecar file. So, in these situations the whole file wil be in the backup


Tks


Roelof Moorlag said:


> 3. There is no option in Lightroom to force XMP sidecar files on files with a XMP section (as DNG)


Tks


----------



## Stumped (May 22, 2020)

If saving to a sidecar file, will that picture be able to be opened in PS as is - showing edits, w/o further changes?

Or do the changes have to written to the file and then opened in PS?

What settings *should I set* to use sidecar files and not back up the picture with every edit made?


----------



## Roelof Moorlag (May 22, 2020)

Stumped said:


> there's no need to curse. I never said writing to XMP is necessary for a backup.


Sorry! English is not my native language and i meant to write 'sound'...


----------



## Paul McFarlane (May 22, 2020)

Stumped said:


> If saving to a sidecar file, will that picture be able to be opened in PS as is - showing edits, w/o further changes?


Saving to XMP is irrelevant if you open Ps directly from Lr - see page 336 of your Classic book for the options.


----------



## Jim Wilde (May 22, 2020)

Paul McFarlane said:


> Saving to XMP is irrelevant if you open Ps directly from Lr - see page 336 of your Classic book for the options.


And while reading that page, you could skip forward a few pages to page 345 to the section called "Saving Metadata to the Files"....there you'll find the answers to most of the questions you've been asking here. If there's anything that you don't understand then tell us what that is.



Stumped said:


> What settings *should I set* to use sidecar files and not back up the picture with every edit made?


It's not a question of *settings*, it's a question of which file-type you are using....sidecar files will only be created when you save XMP to proprietary raw files (CR2/3, NEF, ARW, etc.). For ALL other file-types in your catalog saving XMP will update the actual files, and so would cause your backup software to back up those updated files. All this is covered in your book as specified above.


----------



## Stumped (May 22, 2020)

Roelof Moorlag said:


> Sorry! English is not my native language and i meant to write 'sound'...


Got it - tks for the help - sometimes that spell check can get you - dam is close to sound......



Paul McFarlane said:


> Saving to XMP is irrelevant if you open Ps directly from Lr - see page 336 of your Classic book for the options.


ok.  tks.  



Jim Wilde said:


> And while reading that page, you could skip forward a few pages to page 345 to the section called "Saving Metadata to the Files"....there you'll find the answers to most of the questions you've been asking here. If there's anything that you don't understand then tell us what that is.


So that's why I put all this up---  my very 1st sentence of the post says it all,  "So there is a lot out there, and here, on the above. Ive been reading, re-reading and re-re-reading to the point where im possibly more confused than I was before I started."  Ive read a lot in Victoria's book and Martin Short's, specifically pages 626-634 and more....  I was simply confused and trying to understand the details.  For instance, Should I write to XMP? section on 346...  Is very helpful and everyone's situation is different, but its not clear at all which are Pros and which are Cons...  Also, on that page at the end, another confusion to me and another question I asked but no answer I think - Which of Lightroom's data isn't stored in XMP - nomenclature - when its said, "stored in XMP", this will always refer to the XMP section of the file like the next section How do I write to XMP, also referencing the XMP section of the file?  Compared to when "xmp sidecar" is used will always refer to the sidecar file?

So, to confirm......  Catalog Settings, Metadata, ☑️ Include Develop metadata settings inside JPEG, TIFF, and PSD files when checked will always write to file  XMP and when not checked to XMP sidecar.  Same with the next setting, Automatically write changes into XMP - when checked the XMP section of the file will get updated.  If unchecked, one gets an XMP sidecar file?


----------



## Jim Wilde (May 22, 2020)

Stumped said:


> So, to confirm...... Catalog Settings, Metadata, ☑ Include Develop metadata settings inside JPEG, TIFF, and PSD files when checked will always write to file XMP and when not checked to XMP sidecar. Same with the next setting, Automatically write changes into XMP - when checked the XMP section of the file will get updated.  If unchecked, one gets an XMP sidecar file?


No, you still don't seem to get this. The only option it to write to XMP or not to write to XMP (the default is NOT to write to XMP as all that data, plus more, is always stored in the catalog database). If, however, you do choose to write to XMP then how that is done is determined (as I said earlier) by the type of files that you are dealing with.....for proprietary Raw files, the XMP data is stored in XMP sidecar files, but for all other file-types the XMP data is stored in the XMP section of the file header. That is automatically determined by Lightroom, the user cannot control how the XMP data is stored in the files.
The other option (Include Develop Settings in metadata....) only determines if the develop settings are additionally included with all the other metadata when writing to XMP.....if you don't write to XMP, having this option checked is irrelevant,


----------



## Paul McFarlane (May 22, 2020)

Stumped said:


> For instance, Should I write to XMP? section on 346... Is very helpful and everyone's situation is different, but its not clear at all which are Pros and which are Cons...


They are listed as Pros and Cons - the first three can be considered potential pros, the five that follow (that start 'on the downside) are cons.



Stumped said:


> Which of Lightroom's data isn't stored in XMP - nomenclature - when its said, "stored in XMP", this will always refer to the XMP section of the file like the next section How do I write to XMP, also referencing the XMP section of the file? Compared to when "xmp sidecar" is used will always refer to the sidecar file?


Page 346, the paragraph immediately before 'Should I write to XMP?' makes it clear from most file types this will be in the file header, but for proprietary files it is written to a separate sidecar file.



Stumped said:


> So, to confirm...... Catalog Settings, Metadata, ☑ Include Develop metadata settings inside JPEG, TIFF, and PSD files when checked will always write to file XMP and when not checked to XMP sidecar. Same with the next setting, Automatically write changes into XMP - when checked the XMP section of the file will get updated. If unchecked, one gets an XMP sidecar file?


No, nothing is written to any type of file unless you tell it to - by Ctrl/Cmd+S or ticking the Automatically write changes into XMP


----------



## Stumped (May 22, 2020)

Jim Wilde said:


> No, you still don't seem to get this. The only option it to write to XMP or not to write to XMP (the default is NOT to write to XMP as all that data, plus more, is always stored in the catalog database). If, however, you do choose to write to XMP then how that is done is determined (as I said earlier) by the type of files that you are dealing with.....for proprietary Raw files, the XMP data is stored in XMP sidecar files, but for all other file-types the XMP data is stored in the XMP section of the file header. That is automatically determined by Lightroom, the user cannot control how the XMP data is stored in the files.
> The other option (Include Develop Settings in metadata....) only determines if the develop settings are additionally included with all the other metadata when writing to XMP.....if you don't write to XMP, having this option checked is irrelevant,


Thanks Jim - I think im getting it. Sorry to be so dense here, but it may actually be a medicine thing too.....  So....  To beat a dead horse more, write to XMP refers more to a format/platform(as stated in the book) than location (unless a sidecar file)and format?  As I quoted earlier, all info is written to the catalog and you just confirmed - tks. 

Which are prioprietary formats and which are not?


----------



## Roelof Moorlag (May 22, 2020)

Stumped said:


> Which are prioprietary formats and which are not?


The RAW files that come out of the camera's like the NEF's from Nikon, CR2 from Canon, RAF from Fuji and ARW from Sony.


----------



## Stumped (May 22, 2020)

Im sorry - those are proprietary or are not?


----------



## Jim Wilde (May 22, 2020)

Yes, they are proprietary (i.e. raw files which are specific to each different camera manufacturer). Non-proprietary files are "open-standard" formats such as Jpeg, Tiff, PSD, DNG.


----------



## clee01l (May 22, 2020)

Stumped said:


> Im sorry - those are proprietary or are not?


ANY RAW file that is not in the open DNG format is considered proprietary to the camera manufacturer.   Writing into an XMP block in those RAW file leaders would destroy the format integrity of the RAW file format and therefore Lightroom will not write to those source files.   DNGs, JPEGs, TIFFs etc are  documented public formats and have rules to writing into the XMP section for the file header blocks.


----------



## Stumped (May 21, 2020)

So there is a lot out there, and here, on the above.  Ive been reading, re-reading and re-re-reading to the point where im possibly more confused than I was before I started.  It began from another thread when I mentioned .dng and .cr2 files, convert or not to - another ball of wax itself....  So, if I may, lay out what I understand to be correct for confirmation and some questions.   I save all original files externally from local hard drive outside of LR.   I currently shoot raw .cr2 files and had been importing them to LR as such and have the "Automatically write changes into XMP"  checked in the catalog Metadata settings.  QUESTION - Doing such with that setting checked gives me .xmp sidecar files, right?

I have also, after reading, started to import .cr2 files as .dng files, saving the original .cr2 externally and have the same "Automatically write changes into XMP"  checked in the catalog Metadata settings.  QUESTION - Because i now have that setting checked w/ a .dng file, LR writes to the XMP section of the file and not a xmp sidecar file?

Ive read that metadata refers to develop settings, tags, labels, crops, etc. that are primarily stored in the catalog file.   QUESTION - This is the default whether the above setting is checked or not checked, right?  Where can I see that / those files?

To make sure I fully understand this and possibly redundant - jpeg, png, tiff and psd and dng files - allow for writing to the xmp section of the file by having the above option checked and proprietary .cr2 and HEIC will get xmp sidecar files when checked?

Some examples of other proprietary raw files besides .cr2, HEIC ?

Confirm: Ive read that all changes to pictures are always recorded to the catalog - yes or no?  If yes, when recorded to the catalog, does this imply to a xmp sidecar file or just the catalog?  

If saving to a sidecar file, will that picture be able to be opened in PS as is, w/o further changes?

Confused - Under the Catalog Settings window, Metadata, "Automatically write changes into XMP", is this what writes the sidecar file or is this referring to the "into xmp" section of the file?

Are .xmp sidecar files hidden by default?  Im on a Mac.

So now that ive probably asked the same question 18 different ways - all this reading as mentioned has led to confusion for me.  Additionally some places are written by those much more experienced and I think sometimes some things are taken for granted and maybe not explained very well and im just trying to make sure I fully understand this before getting too deep into it.

Additionally, from the other thread I briefly made reference to dng vs cr2 ---  this led to convert or not to - a whole other argument - file check vs no file check - in hind site, at this point, probably a non-issue for me.  Still trying to learn that just because i read it doesn't mean I have to try it.  Maybe just concentrate on basic workflow.

Finally, this is what id like to accomplish, and my understand, please correct if not - as mentioned, on a Mac and I use time capsule and store originals on external drive.  Its my understand(please confirm), that if I write to the xmp portion of a file, those changes are picked up as file changes by time capsule and I get a new back up copy with every change made, right?  This seems like a waste of space compared to if I were just writing to a sidecar file and that was the file being backed up and not the whole image file.

What settings should I set to use sidecar files and not back up the picture with every edit made?

Finally, If worried about space, and if I were to keep the settings as is  and seeing as I save originals external, Im safe to exclude this LR (picture only directory) from Apple backups and still be safe?  I am backing up my catalog in LR AND time capsule.

Thanks all for any help here in getting me straight!

Attached is what I have now.


----------



## Stumped (May 24, 2020)

Thanks to all who helped me get this straight.  Think I have a better understanding now.  One of things I was trying to figure out too was the file backup process.  As mentioned above, storage is cheap, but I hate to waste space too...  As pointed out, I use Time Machine, and as mentioned in the book, when writing to the file it can cause a large file to be included again in the backups.   For instance - I looked at 2 files - each one has so far 6 back ups, 4 .dng and 2 .jpg.  The 8 .dng's use 172mb and the 4 .jpg 22mb. 

Just a real life example of what's mentioned in the book.  Thanks all for the help!


----------

