# Backup Strategy



## FredL (Mar 6, 2016)

Can anyone recommend sources of reference information on backup strategy?  I currently backup my raw photos and LR catalog to an external drive.  I would like to periodically swap this drive with a 2nd external drive and store it offsite, and rotate these drives monthly or so.


----------



## Hal P Anderson (Mar 6, 2016)

It sounds like you have a handle on things. If you want more, look here: Backup | dpBestflow


----------



## PhilBurton (Mar 6, 2016)

Fred,

There are different flavors of backup.  The "easy" one is to duplicate files on a second drive.  That's easy enough, but what if the file you want to restore is not the latest version,  but some prior version?  That leads to the "hard" form of backup, where you preserve old copies of files.  I take the second approach, which has saved my tuchis several times.  

I use a product call Retrospect, from a company of the same name. There are both Windows and Mac versions, for both desktop and server use.  www.retrospect.com.

Phil


----------



## Tony Jay (Mar 7, 2016)

I agree with Hal that your fundamental strategy is sound.

A couple of things to add though.
A major principle of backing up and archiving in digital-asset-management is redundancy.
Having several hard drives that are backed up is better than just one or two.
Consider having pairs of drives doing things - so two on-site backups and two off-site drives as well.

Also, consider using a utility that can do verified bit-for-bit backups - this way the chances of file corruption during the process is minimised.
There are plenty on offer (often the same vendor offers both free and paid-for versions of their product).

Don't forget about your catalog - my suggestion is to backup your catalog on the same drive as the images.
That way the catalog and images on any particular drive are complete to the last backup date.

I would also second Hal's suggestion of dpbestflow as a good reference to learn more.

Tony Jay


----------



## LouieSherwin (Mar 7, 2016)

Hi Fred,

Backup strategies are a multiple layer project. First and foremost is to have a reliable and consistent full system backup. Retrospect is an excellent system but since you have Mac I highly suggest that you get a large capacity external drive and take advantage of the excellent Time Machine backup system included with your Mac. Having a Time Machine backup saved my last year when my SSD system drive evaporated in a millisecond. It only took me several hours to completely restore from Time Machine.

Secondly adding an offsite backup as you describe is prudent. A rotating set of hard drives is not a bad idea if you can consistently keep it updated. I did that for many years by storing a set of drives at my office. But I found that I was not always as consistent as I needed to to keep it up to date. An excellent alternative is a cloud service such as CrashPlan or Blackblaze. Both of these install on you computer and keep all my images safely tucked far far away from any disaster that can occur locally. 

All I have to do now is monitor and make sure both systems are running. 

An excellent and comprehensive guide is the eBook: Backing Up Your Mac: A Joe On Tech Guide

-louie


----------



## clee01l (Mar 7, 2016)

I would suggest as a first effort to use the TimeMachine app that Apple delivers with OS X.  By default, It will backup and version files on your Primary drive, excluding any EHDs.  You can Exclude any folders and files that are not needed to be restored from a backup (like the LR Previews).  You can Include any EHDs that host critical user data. TM runs in the background quietly and without user intervention.  It backup us changes every 30 minutes Keeping daily, weekly, monthly and yearly versions until the backup device is full, then deletes the oldest versions as needed to make room for new. 
With Time Machine, you can specify multiple backup destinations.  TM will alternate between these automatically.  If you eject a destination EHD, TM will continue to backup to the destination(s) available. 
I use this approach with my mobile rMBP.  Normally at home it backs up to an Airport Time Capsule (NAS).  when I travel, I carry a 1TB Seagate portable EHD and TM backs up to it.  
Being a "belt & suspenders" guy, one backup is not enough.  As a Safety to the TimeMachine backup, I use CrashPlan.  It is a free app that lets you backup locally to an EHD or another computer.  For ~$60/yr, you can use CrashPlan on their Cloud storage.  This provides offsite storage which is a protection from local fire, flood or pestilence that might wipe out your computer and your local backup devices.


----------



## FredL (Mar 7, 2016)

Thanks to everyone for the excellent advice!  I will definitely cobble together a backup routine using most (if not all) of the methods you have suggested.
Thanks again!


----------



## shiraz (Mar 13, 2016)

I convert my photo's to DNG and keep the NEF's for the case of Corrupt DNG file. Anyone else doing this?


----------



## tspear (Mar 14, 2016)

I keep the RAW, but convert to DNG. Less for data corruption then I have the space and at some point in the future it is theoretically possible that someone will invent a process which depends on the original raw file. (Not likely, but it is harmless to me to keep them)


----------



## Roelof Moorlag (Mar 14, 2016)

shiraz said:


> I convert my photo's to DNG and keep the NEF's for the case of Corrupt DNG file. Anyone else doing this?


Yes i do! I'm a little parania if it concerns my images 
I'm using the 'Second Copy' option to put my RAW's on a seperate hard drive but i do not do anything with it. I'm working with my DNG 'originals' and only in case of problems i'm getting back to the NEF. Because the filenames are identical this is very easy. Thankfully i have never had to come back to my NEF's.


----------



## Linwood Ferguson (Mar 14, 2016)

shiraz said:


> I convert my photo's to DNG and keep the NEF's for the case of Corrupt DNG file. Anyone else doing this?


Frankly I think for most applications DNG's are a solution in search of a problem.

But a backup strategy is really about preserving bits -- it should not matter much what the bits are.  So whether you save as raw, DNG, TIFF, or combinations, you need your backup strategy to be able to recover them all from any appropriate time frame.

I think the three most frequently omitted issues with people's backups are: 

1) Failure to test that you can really restore
2) Failure to have off-site backups (and see #1 in that regard also, especially for cloud backups)
3) Failure to have versions, so in the event of a corrupt but not lost file, you can go back to a version not corrupt (and see #1)


----------



## Roelof Moorlag (Mar 14, 2016)

Ferguson said:


> But a backup strategy is really about preserving bits -- it should not matter much what the bits are. So whether you save as raw, DNG, TIFF, or combinations, you need your backup strategy to be able to recover them all from any appropriate time frame


Lightroom provides a solution to guard the bits with DNG validation. There is no other (easy) way to know if your files are getting corrupt.


----------



## Linwood Ferguson (Mar 14, 2016)

Roelof Moorlag said:


> Lightroom provides a solution to guard the bits with DNG validation. There is no other (easy) way to know if your files are getting corrupt.



I agree, though it only guards some of the bits, and only DNG's, so if one has edited and produced TIFF's, etc. that is not a complete solution.    I even wrote my own validation routine to fill those gaps (see here), but I really think LR is missing the boat here, for not including an actual "validate the image file regardless of type" feature.


----------



## Roelof Moorlag (Mar 14, 2016)

Ferguson said:


> but I really think LR is missing the boat here, for not including an actual "validate the image file regardless of type" feature


 That would be a nice option indeed!
I looked into your routine and it sounds very interesting! How do i use it? Is there any risk to damage my catalog?.


----------



## Linwood Ferguson (Mar 14, 2016)

Roelof Moorlag said:


> That would be a nice option indeed!
> I will look into your routine, it sounds interesting!



I'd welcome feedback.  I use it, but never really got anyone else interested in it, probably because it's an external routine as opposed to a plugin.  That's partly for performance (this will definitely find the bottleneck - disk or CPU -- on most systems for doing the validation).


----------



## Roelof Moorlag (Mar 14, 2016)

On Codeplex i see only a few downloads indeed, pity. 
How do i install and use it? Is there any risk on damaging my catalog?


----------



## Linwood Ferguson (Mar 14, 2016)

Roelof Moorlag said:


> On Codeplex i see only a few downloads indeed, pity.
> How do i install and use it? Is there any risk on damaging my catalog?


The download is a standard windows install kit, which will install a dot net application.

I believe it will not harm anything, but there is always a risk, so I urge you to back up.  It creates two new tables in the catalog.  I did this as opposed to adding any columns or changing any data in the existing LR tables.  That is why I think it is safe, LR will not "see" anything it changes.

LR and it cannot run at the same time, as LR takes out an exclusive lock on the database, not shared.

It also has an option to remove the additional tables (Cleanup button).

The basic flow is as follows: 

- Prepare the catalog (initialize) 
- Find new (it shows how many images it should find)

It is possible that this step will have errors, namely that it cannot locate the image in the location specified in Lightroom.  The "Fix" button will let you review those.

The later, any time in the future: 

- Revalidate (check all the ones previously found to ensure they did not change) 
- Find new (for images added since)

If you produce and edit TIFF's using the Edit Original, or if you write metadata updates to non-raw files, these operations will change the data and give an indication that the file changed.  And indeed it did change.  It is still useful if you are only validating those that are old and no longer being changed (as if they "change" it is probably corruption).


----------



## Roelof Moorlag (Mar 14, 2016)

First (minor) feedback. The installation did not allow me to install on a other location than default (c:\program files), i like to install programms on another drive than my system drive.


----------



## Linwood Ferguson (Mar 14, 2016)

Roelof Moorlag said:


> First (minor) feedback. The installation did not allow me to install on a other location than default (c:\program files), i like to install programms on another drive than my system drive.



Yeah, was lazy and didn't want to build the extra dialog boxes in WIX.    It's tiny though (I realize that size is not the only reason people separate programs and OS).


----------



## Roelof Moorlag (Mar 15, 2016)

Ok, the interface is quite logic and the 'tooltip' info is helping also.
I tried it on a test catalog with about 500 media files (it does more than only photo files or only the files supported by Lightroom?)

I made some changes on metadata and put that back into the (DNG and JPG) files. The application found these files indeed. 
The information about the changes looks basic in the first place (the list view) but i like the 'display more information' option! It gives you back graphical information about the photo 'before' and 'after'.
One suggestion, it would be nice to be able to copy the file name easily for further research.

I think this is a very workable solution. My files normaly don't change, all edits are kept in the Lightroom database only. 
I'm willing to test it on my main catalog with 73.000 images. How long does it take to initialize (about) do you tink?

Do You have any experience with a Lightroom update (major version). Does the altered database had any drawbacks?


----------



## Roelof Moorlag (Mar 15, 2016)

The Lightroom Validate application is crashing on DNG's which are crashing windows explorer
In another thread i'm discribing this issue: Windows Explorer is crashing on DNG (compatible Lightroom 4.1)
On the 'Display more information' the DNG is not only not showing, but the application is stop working:


----------



## Linwood Ferguson (Mar 15, 2016)

Roelof Moorlag said:


> Ok, the interface is quite logic and the 'tooltip' info is helping also.
> I tried it on a test catalog with about 500 media files (it does more than only photo files or only the files supported by Lightroom?)
> 
> I made some changes on metadata and put that back into the (DNG and JPG) files. The application found these files indeed.
> ...



With regard to file types, and with the caveat I am testing old memory (I would look more closely but have a volleyball shoot and can't until tomorrow), if I recall I wrote it so it would try to process all types of files, so video or anything else that is pointed to by the catalog entry.  It looks only at the file itself, though, not a sidecar file (if any).  And I don't recall whether when raw+jpg is selected as one if there are separate entries, but if it is a single catalog entry it will only see the one.

Now that said, I only paid attention to the raw really, and did not do much research on how things like the preview fetch would work (or not), the code looks like it just goes after the preview cache entry item without regard to media type.   So my guess is you just don't get a preview.

My catalog is a bit over 56,000 entries, so we are not that far apart.  I did the initial run over night, and do revalidate runs over night, so I cannot tell you how long.  My guess is hours.  The actual initialize (in the sense of creating the tables) is instant, but the Find New will be slow as it has to calculate the checksums.  On my system the slowest part of that is reading the actual image files -- that disk (which is spinning on mine) saturates, but the catalog does not (which is SSD for me).   

You can stop it at any point -- just click the button (which has changed to cancel) and it restarts where it left off.  It does run-down each thread so it has to finish the current image before it stops, if that happens to be some really huge image (like a video file) it may take a while, but it shows the thread status on the UI.  There's no penalty for stopping and restarting the find new.

The revalidate there is also no penalty except -- there's a somewhat cryptic time shown.  If you want to restart revalidation another day, by default it revalidates all again.  If you put a time stamp in there it will revalidate only those not validated after that date, so you can force it to pick up where you left off.

As to versions: I wrote this on LR 5.2 and it survived the version 6 update.  Based on watching the LR database from early version 4 I doubt that a major version will screw this up, because generally Adobe has been adding tables and fields, not restructuring existing ones, but of course there is no guarantee of that.   Now I'll show you how bad my memory is -- I can't recall when I did the upgrade from 5.x to 6.x, and it copied the catalog, whether it copied my tables or not.   If it copied the catalog FILE and did the upgrade, it would have; if it copied each data table individually it would not, but in that case the program would just offer to initialize again.  But I did it long enough ago I do not recall.  I do not have a version 5.x catalog handy to test (I can probably find one on a backup but no time today).


----------



## Linwood Ferguson (Mar 15, 2016)

Roelof Moorlag said:


> The Lightroom Validate application is crashing on DNG's which are crashing windows explorer
> In another thread i'm discribing this issue: Windows Explorer is crashing on DNG (compatible Lightroom 4.1)
> On the 'Display more information' the DNG is not only not showing, but the application is stop working:



I'll have to try to reproduce that error.  I have extremely few DNG's, and those are all new version compatible.  That it and explorer both are dying trying to display the preview is probably not a coincidence, I'm using some pretty vanilla Windows routines to display the original image.  It just reads it as a stream and let's windows try to display it (having said that I really do wonder what happens if I throw a MP4 at it).  I'll try to reproduce the situation tomorrow.

I did spend a lot of time trying to harden the program against any database issues from unexpected aborts, it uses transactions and should not be impacted by such aborts. 

One reason I'm not a fan of DNG's personally is I am a heavy user of cloud backups, and changing the "raw" file (such as it were) for metadata changes like keywords means the whole file is backed up again.   But as a side effect it also gets flagged here.


----------



## Roelof Moorlag (Mar 15, 2016)

Ferguson said:


> If you put a time stamp in there it will revalidate only those not validated after that date, so you can force it to pick up where you left off


Thats nice!



Ferguson said:


> I'll have to try to reproduce that error. I have extremely few DNG's


Here is a sample: Dropbox - RM_20160311_161039.dng



Ferguson said:


> One reason I'm not a fan of DNG's personally is I am a heavy user of cloud backups, and changing the "raw" file (such as it were) for metadata changes like keywords means the whole file is backed up again


I'm not pushing data back into the DNG. I treat them like 'real' RAW files. In my case there should be no DNG files in the list as changed. When they do, something is wrong. 
Until your Lightroom Validation application the only way to check on the validity of photo files was to use the Validation option of DNG's in Lightroom. Now there is an alternative option 

I think i'm diving in with my main catalog one of these days.


----------



## Linwood Ferguson (Mar 16, 2016)

Let's take the validation discussion to another thread, so as not to further hijack this one.

Here


----------

