# Lightroom catalogue backup vs. XMP files



## Danagordon (Feb 11, 2014)

Hi,
just want to confirm if I understand this correctly - if I back up my Lightroom catalogue every time I exit Lightroom, I won't need to write changes to XMP files?  Because Lightroom catalogue backup already contains the same information as XMP would?
Which is a better way to go? Use XMP files and not back up my Lightroom catalogue? Or vice versa?
thanks. Lightroom is so confusing :hm:


----------



## Tony Jay (Feb 11, 2014)

Hi Dana welcome to Lightroom Forums.

This is a good question that you ask because a lot of people do get a bit confused about what the differences, and similarities, are between information stored in the catalog versus that found in XMP files.

XMP stands for eXtensible Metadata Platform BTW.

First up, information that actually applies to individual images such as EXIF and IPTC metadata, keywords, ratings, and develop settings can be copied to the XMP files.
This is good and can be very helpful.

However, information that will not be transmitted to XMP files includes things like collection membership, keyword list, saved presets like import presets, develop presets, metadata presets, print presets etc etc.

What this tells you is that backing up the catalog and saving XMP files are NOT the same thing and are NOT equivalent.
They do however both have complementary utility and you need, IMHO, to have backups of both.

A catalog backup allows one to a recent and hopefully functional catalog to fall back on if the most current catalog were to become corrupted or perhaps inadvertently deleted.

XMP files can be useful when you need other applications to read your RAW images - if they can read the XMP files then they will automatically pick up all the metadata, including the edits.
If you were to lose all the catalogs, including the catalog backups then it would be possible to partially restore your catalog by re-importing all your images into a new catalog - in this instance Lightroom would read the XMP files and populate the catalog with the relevant data.

Some people do not use XMP data at all and just make multiple backups of their catalog and images.
This can work but I would rather do the extra step as well - one can never be too careful!

The whole concept of Digital Asset Management (DAM) is a big one and is deserving of your attention but it is nowhere near as sexy and entertaining as developing your best images for print or display and so often gets neglected.

As, and when, there are matters arising from what I have shared please post further questions and we can delve into each issue separately.
Whole books are written about these topics so they are a bit too big for a single forum post!

Tony Jay


----------



## Jimmsp (Feb 11, 2014)

Tony Jay said:


> Hi Dana ....
> What this tells you is that backing up the catalog and saving XMP files are NOT the same thing and are NOT equivalent.
> They do however both have complementary utility and you need, IMHO, to have backups of both.
> 
> ...


Tony has given you a very good description of the details and some good advice.
I am of the old school - back up everything multiple times. This has paid off for me, and will again in the future. Storage devices will fail, we just don't know when.

Thus, I back up my photos and associated xmp files, and the LR catalog and the LR back up. I back these up to an external usb hard drive, as well as to another drive that I keep off site at my daughter's house.
I recommend the same to anyone who asks me.


----------



## Tony Jay (Feb 11, 2014)

Jimmsp said:


> Tony has given you a very good description of the details and some good advice.
> I am of the old school - back up everything multiple times. This has paid off for me, and will again in the future. Storage devices will fail, we just don't know when.
> 
> Thus, I back up my photos and associated xmp files, and the LR catalog and the LR back up. I back these up to an external usb hard drive, as well as to another drive that I keep off site at my daughter's house.
> I recommend the same to anyone who asks me.


Yes, indeed.
Excellent advice!

Tony Jay


----------



## BobMc (Feb 11, 2014)

The default location for the LR catalog backups is in a folder on the same drive as the active catalog.  Adobe's primary purpose for this backup is Data Base integrity.  If the active catalog is corrupted, you have a “backup” to use.
BUT, the active catalog and it’s backups on the same drive only works if they are then, also backed up to others drives as well.
To fully protect both your images and the work done on them in LR, requires that both have backups on multiple drives and as already suggested, at least one set (both images and catalogs) off site.


----------



## Tony Jay (Feb 11, 2014)

BobMc said:


> The default location for the LR catalog backups is in a folder on the same drive as the active catalog.  Adobe's primary purpose for this backup is Data Base integrity.  If the active catalog is corrupted, you have a “backup” to use.
> BUT, the active catalog and it’s backups on the same drive only works if they are then, also backed up to others drives as well.
> To fully protect both your images and the work done on them in LR, requires that both have backups on multiple drives and as already suggested, at least one set (both images and catalogs) off site.


That is correct!
In fact I back everything up to two internal drives and four external drives with at least one been offsite at any one time.
I keep all my Lightroom related stuff including images on one dedicated internal drive and then mirror this drive with the others regularly with bit-for-bit verification.
I just need to remember every now and again to delete the older catalog backups.

As an aside one should open the backup catalogs regularly to check that they indeed do work - it is possible to faithfully backup a corrupted catalog.

Tony Jay


----------



## camner (Feb 11, 2014)

I use the TPG LR Backup plugin to help with this.  It has two wonderful characteristics:
1.  It compresses the backed up catalogs into a zip file, which takes up a lot less space
2.  One can specify the desired destination of the zipped up catalog backup files

By doing this (which can be a "set and forget" item, once things are set up correctly), the backups of the catalog automatically are sent to a different drive than the location of the catalog itself.

I don't use this to replace a complete backup strategy, though.

_Edit:  I forgot to mention that this plugin also backs up the LR configuration files, which can make recovery from a hard drive failure easier_


----------



## Tony Jay (Feb 11, 2014)

This plug-in is a useful complement to a robust back-up strategy.

Tony Jay


----------



## Jimmsp (Feb 11, 2014)

Tony Jay said:


> This plug-in is a useful complement to a robust back-up strategy.
> 
> Tony Jay


Since the OP is on a Windows machine, he needs something different.
I use the free SyncToy in the echo mode.
That allows me to make changes to my catalog and photos after a backup, and it will change the backup drive to match. It is quick and works very well.
I backup after I load a set of photos. If I later delete a bad one, the echo mode also deletes it from the backup drive.

JIm


----------



## camner (Feb 11, 2014)

Jimmsp said:


> Since the OP is on a Windows machine, he needs something different.
> JIm



AFAIK, the plugin works on Windows machines as well as it does on Macs.


----------



## Jimmsp (Feb 11, 2014)

camner said:


> AFAIK, the plugin works on Windows machines as well as it does on Macs.


Thanks. I'll take a look at it. I like the idea of zipped files, esp for the off site bu, which is now 2 drives for all my stuff.


----------



## DaveS (Feb 11, 2014)

That plug in does in fact work like a charm on windows.   In addition to backing up and compressing to zip the catalog, it also backs up all of the lightroom Configuration scripts and what not as well (templates, develop presets, external editor configuration and all that other good stuff that is needed to get a new installation of Lightroom configured how it was before).

I used that feature when I built a new computer and installed Lightroom.  A matter of dragging the config folder out of the backup zip and overwriting the default lightroom settings folder.


----------



## Denis de Gannes (Feb 11, 2014)

Quote from Tony Jay
_"XMP files can be useful when you need other applications to read your  RAW images - if they can read the XMP files then they will automatically  pick up all the metadata, *including the edits*."_

This may be true as far as the Metadata is concerned but as far as I am aware the only other applications that can read and fully apply your develop edits from Lightroom are an equivalent version of Lightroom or Adobe Camera Raw.


----------



## Tony Jay (Feb 11, 2014)

Denis de Gannes said:


> Quote from Tony Jay
> _"XMP files can be useful when you need other applications to read your  RAW images - if they can read the XMP files then they will automatically  pick up all the metadata, *including the edits*."_
> 
> This may be true as far as the Metadata is concerned but as far as I am aware the only other applications that can read and fully apply your develop edits from Lightroom are an equivalent version of Lightroom or Adobe Camera Raw.


Denis you are spot on.
I did not make that detail explicit in the post you are referring to.

Tony Jay


----------



## coachejp (Feb 11, 2014)

Tony (etall)-So before I Get to involved with LR could you clear 2 things up for me.  1) The only place edits are kept is in the catalog,not written back to the folder?  2) What about virtual copies are they written back to the folder or do you have to manually copy them to your folder?  Thanks for the help.


----------



## Denis de Gannes (Feb 11, 2014)

coachejp said:


> Tony (etall)-So before I Get to involved with LR could you clear 2 things up for me.  1) The only place edits are kept is in the catalog,not written back to the folder?  2) What about virtual copies are they written back to the folder or do you have to manually copy them to your folder?  Thanks for the help.



This is a very good question, however the answer can be very simple but complicated to understand. Lightroom by *default* stores everything you do in the Catalog file. This is great because you no not need to have the info in a sidecar file and have to manage that file.
However Adobe also has one of its premier products known as *Photoshop* which uses a plugin called Adobe Camera Raw which also carries out the same functions as the Lightroom Develop Module. However the Adobe Camera Raw (ACR) plugin *cannot* read the info from the Lightroom Catalog file. For this to happen the info has to be written to an .xmp file.

Lightroom procedure of operation is know as "The info is in the Catalog" and when working with files Lightroom reads and writes to the Catalog. ACR on the other hand reads and writes its work to .xmp files or for DNG, tiff and jpeg files to the file header, this is known as "The info is in the file".

To assist in the sharing of the info Lightroom also has an *option* to auto write info to .xmp. Lightroom does not necessarily need this but Photoshop / ACR does.

The link below is to a video that explains how the Lightroom Catalog works and while it was created several years ago it is still relevant and help full in understanding how it works.
http://tv.adobe.com/watch/george-jardine-on-lightroom/the-lightroom-catalog/


----------



## DaveS (Feb 11, 2014)

Hi,

  My understanding of virtual copies is that they are that... virtual.  They are entirely a construct of the catalog, and don't actually have a file associated with them, so there is nothing to copy back into the folder.  It's just an alternate set of develop instructions.   If you export the virtual copy, you of course do get a file from it.     

 If you do an edit-in on a virtual copy though, a file is created (for the external program to use) and is usually created in the originating folder and will also exist in the catalog.


----------



## Sunny16 (Feb 12, 2014)

Tony Jay said:


> First up, information that actually applies to individual images such as EXIF and IPTC metadata, keywords, ratings, and develop settings can be copied to the XMP files.
> This is good and can be very helpful.
> 
> However, information that will not be transmitted to XMP files includes things like collection membership, keyword list, saved presets like import presets, develop presets, metadata presets, print presets etc etc.



Just to confirm this 100% in my mind.  

Say I completed all develop module adjustments to all of the RAW files in a folder and the folder contained the XMP files.  Then say my .IRCAT file and backups were all gone.  If I create a new catalog and import the RAW files into it, all of the develop module adjustments to the RAW files would still be retained.  Is that right?

Thanks,
Sunny


----------



## Denis de Gannes (Feb 12, 2014)

Yes if you had "Automatically write changes to xmp" selected see the screen shot. (the name of the catalog file is .LRCAT not .IRCAT)




However info that is not specific to the file would be lost, Collections, Virtual copies etc.


----------



## Tony Jay (Feb 12, 2014)

I would not personally use the 'Automatically write changes to XMP' option.
In many cases the result is a distressing slow down of the system.
It is much easier, IMHO, to just select the images that you have altered at the end of a session and manually ask Lightroom to write the changes to XMP then.
After that I do my disk back-ups as well.

Tony Jay


----------



## Jimmsp (Feb 12, 2014)

Tony Jay said:


> I would not personally use the 'Automatically write changes to XMP' option.
> In many cases the result is a distressing slow down of the system.
> It is much easier, IMHO, to just select the images that you have altered at the end of a session and manually ask Lightroom to write the changes to XMP then.
> After that I do my disk back-ups as well.
> ...


I use the auto write feature, as I have occasionally forgotten to do a manual write. I don't find it slows down my edits, though I may be a bit slower with my edits than you. The only slowdown I have seen is upon a shutdown of LR, and that only happens sometimes if I quit right after adding new key words. Then it asks me to wait or write later. It is usually done in a few seconds.


----------



## Sunny16 (Feb 13, 2014)

Jimmsp said:


> I use the auto write feature, as I have occasionally forgotten to do a manual write. I don't find it slows down my edits, though I may be a bit slower with my edits than you. The only slowdown I have seen is upon a shutdown of LR, and that only happens sometimes if I quit right after adding new key words. Then it asks me to wait or write later. It is usually done in a few seconds.


I too have LR setup to auto write XMPs and since I haven't done a ton of edits with LR yet, I haven't noticed any delays.  If I notice any slow downs in the future, I'll at least know what to look for in this regard.

Thanks both,
Sunny


----------



## Tony Jay (Feb 13, 2014)

If you don't have any issues with the 'Automatically write changes to XMP' activated thats great.
Just be aware that for many users that there could be a big performance hit.

Tony Jay


----------



## Danagordon (Feb 13, 2014)

Thanks guys for all the input and good discussion.

Reading all these brought up another question for me, and this may sound stupid... How do I back up my images?   I mean, is there a correct way?
For example, if I have a folder of photos plus XMP files. Do I just simply copy this folder and paste them into my external hard drive for backups? Does this mean in the case of my internal hard drive losing all the photos, LR can still link to the photo folder plus XMP in the external drive? Reason I ask, is because I know if I do anything outside Lightroom, such as moving photos around and creating folders behind its back, Lightroom won't be able to detect the change and won't find the photos. And then hell breaks lose and everything will be a mess. So this got me confused on backing up the images/XMP itself.

also, is this TPG backup plug-in aware of changes made to both drives and sync to it? If so I need to hook up the external drive to the computer whenever I use it right?

thanks. I also just got my first own computer, so I am not very computer savvy.


----------



## clee01l (Feb 13, 2014)

Danagordon said:


> ...How do I back up my images?   I mean, is there a correct way?
> For example, if I have a folder of photos plus XMP files. Do I just simply copy this folder and paste them into my external hard drive for backups? Does this mean in the case of my internal hard drive losing all the photos, LR can still link to the photo folder plus XMP in the external drive?


A Backup is a second copy located some where that fire, flood or other pestilence won't harm both it and the master.   All HDs will fail. Count on that and be prepared to replace your master(s) with a backup copy.   You need to do this with all of your critical files, not just your images and you LR catalog. but also you irreplaceable Word documents and spreadsheets.   Every time a file changes you need to make another copy.  You need several versions of the same file so that if you manage to screw up the master document, the backup is not a copy of the screwed up file.   There is software dedicated to do this And I recommend that you invest (yes invest) in one or more before disaster strikes, 
There is software that will backup to a local drive and software that will backup to the cloud.  The local drive is the most accessible but also the most vulnerable.  Fire or Flood can wipe out all of your HDs.  Cloud Backup is the most secure , but a complete backup can take a month or more over internet speeds and full recovery almost as long. 

Lately, I have been recommending CrashPlan. It will do both types of backup as well as provide version control  Free for local use and $60/year for unlimited cloud backup.


----------



## Danagordon (Feb 11, 2014)

Hi,
just want to confirm if I understand this correctly - if I back up my Lightroom catalogue every time I exit Lightroom, I won't need to write changes to XMP files?  Because Lightroom catalogue backup already contains the same information as XMP would?
Which is a better way to go? Use XMP files and not back up my Lightroom catalogue? Or vice versa?
thanks. Lightroom is so confusing :hm:


----------



## Sunny16 (Feb 13, 2014)

Danagordon said:


> Reading all these brought up another question for me, and this may sound stupid... How do I back up my images?   I mean, is there a correct way?



Hi Danagordon,

Here are backup strategies by Julieanne Kost that I think are good ones.

http://www.youtube.com/watch?v=QJ2bj1Jal2Q

Simple but effective.
Sunny


----------



## Tony Jay (Feb 13, 2014)

Danagordon said:


> Thanks guys for all the input and good discussion.
> 
> Reading all these brought up another question for me, and this may sound stupid... How do I back up my images?  I mean, is there a correct way?
> For example, if I have a folder of photos plus XMP files. Do I just simply copy this folder and paste them into my external hard drive for backups? Does this mean in the case of my internal hard drive losing all the photos, LR can still link to the photo folder plus XMP in the external drive? Reason I ask, is because I know if I do anything outside Lightroom, such as moving photos around and creating folders behind its back, Lightroom won't be able to detect the change and won't find the photos. And then hell breaks lose and everything will be a mess. So this got me confused on backing up the images/XMP itself.
> ...


Cletus has made a good suggestion in CrashPlan.
More generally what you want is a system that does a bit-for-bit copy and verifies that the copy has actually occurred.
Most systems will 'mirror' a folder/file/drive which means that only what has changed since the last time will be copied.
CrashPlan does all of this and more - it allows one to time backups and has several other features.

This is a relatively non-technical look at backup software more for the purposes of introduction.

Tony Jay


----------



## LouieSherwin (Feb 13, 2014)

Tony Jay said:


> If you don't have any issues with the 'Automatically write changes to XMP' activated thats great.
> Just be aware that for many users that there could be a big performance hit.



This can be a big problem if you use mostly DNG, JPG TIFF or PSD files. This is because all the metadata including develop settings (if you select that option) is included in the file with the image. So any change will cause the whole file to be rewritten. This can cause big performance problems when doing bulk metadata updates to many files. 

Also I do not know how frequently LR updates the metadata when making a series of edits. It could conceivably slow down the edit progress as the file is repeatedly written as you make a series of edits.

If on the other hand your images are mostly raw format (not including DNG) then Lightroom uses XMP sidecar files. These are short text files and are quickly and easily updated. I have done bulk metadata updates of several thousand images that completes in 10-20 seconds at the max. It is my experience that this also occurs in the background so continue working. 

I have heard some comments that you have to always wait for all the changes to complete before making additional changes. I personally have not seen any problems. I believe that the background queueing and processing of changes is quite robust and I have not experienced any problems with applying multiple changes in sequence.

-louie


----------



## johnbeardy (Feb 13, 2014)

LouieSherwin said:


> This can be a big problem if you use mostly DNG, JPG TIFF or PSD files. This is because all the metadata including develop settings (if you select that option) is included in the file with the image. So any change will cause the whole file to be rewritten. This can cause big performance problems when doing bulk metadata updates to many files.



It's only a big problem, Louie, if you fail to adapt your backup practice. The time to backup the DNG, JPG, TIFF or PSD is when it is created or after significant pixel editing - not when some metadata happens to be written to it. The backup value of these files would only be second rate as a range of LR work isn't included, and superfluous as the backup of your catalogue and the originals already provide 100% coverage of your images and work.

I'm not sure automatically writing xmp has been slowing down LR since version 2, but I prefer to leave it off and save metadata manually. I don't think it's a big deal either way. 

John


----------



## DaveS (Feb 13, 2014)

As for the question on the TPG backup plugin, no it's not a synchronization tool, and doesn't look at what is in the backup location at all.  It merely copies a backup of the entire catalog and all the config files to a new set of zipped folders in the target backup location.

You "could" just copy and paste your photos folder to an external drive... but there are a pile of other files and folders on your pc that really should be backed up anyways.  So a backup tool/program is generally a better plan.   There are numerous free/inexpensive options.


----------



## Danagordon (Feb 13, 2014)

DaveS said:


> You "could" just copy and paste your photos folder to an external drive... but there are a pile of other files and folders on your pc that really should be backed up anyways.  So a backup tool/program is generally a better plan.   There are numerous free/inexpensive options.







Do I need to have my Lightroom open, and copy and paste my photo folder to my external drive while in Lightroom??  I normally don't even have my external drive hooked up to my computer. 
 Just still a little unclear about the proper step by step when backing up into a new drive... I wanna start from the basic and then move on to software when needed.


----------



## LouieSherwin (Feb 13, 2014)

Hi John, 

I was just referring to the effect of writing metadata changes to these types of files. So maybe using the adjective "big" in this case was a bit hyperbolic. You would detect a noticeable difference in the time between updating the metadata in a 100 DNG file vs 100 XMP files. I do think that the fact that this operation is done in the background does a good job at minimizing the impact to overall performance. 

Your point about the optimal time to back up these files is well taken. I am sure this why the Lightroom designers choose as the default behavior all metadata is stored in the catalog and only pixel changes are put into the files. If I were to use DNG as my raw format I would turn off the automatic to write xmp option because I would not want to have my original raw files rewritten for every little metadata change. 


-louie


----------



## DaveS (Feb 13, 2014)

Danagordon said:


> Do I need to have my Lightroom open, and copy and paste my photo folder to my external drive while in Lightroom??  I normally don't even have my external drive hooked up to my computer.
> Just still a little unclear about the proper step by step when backing up into a new drive... I wanna start from the basic and then move on to software when needed.



If you are talking about doing a copy and paste to the external drive for making a backup... No,  you wouldn't do it from Lightroom (if you did, then Lightroom would assume that's where they normally reside).

If you are doing it to move the photos to the external drive and want lightroom to know that the external drive is where the photos live (i.e, their normal home, and where you want Lightroom to always access them), yes you would.


----------

