# Testing backup and restore



## tspear

I no longer have spare computers to wipe and rebuild to test my backup processes.
Also, the amount of data is getting a little out of hand. 

So what methods are you guys using to test your backup process?

Tim


----------



## Roelof Moorlag

I'm only testing a part of my backup:
The online backup (Crashplan) and the offsite BluRay backups i don't test. The only test i do is the external harddisc copy but i don't do it often enough...
Occasionally i connect the both EHD (a onsite copy and offsite copy) to my bacbook, open the most recent catalog, reconnect the images and do some checks (comparison of numbers) and do a DNG validation.


----------



## tspear

I am debating standing up a virtual desktop on AWS for a few days and test by complete restore flow.

Tim


----------



## Roelof Moorlag

tspear said:


> I am debating standing up a virtual desktop on AWS for a few days and test by complete restore flow.


Including al your photo's, how much data is that?
My Crashplan took month's to finish.


----------



## Linwood Ferguson

I am doing more and more testing on VM's.  All it takes is a lot of storage to be able to test most anything, and Windows 10 Pro (not home) has absolutely terrific VM support.

I also keep an old laptop around, but lack of storage on it makes things harder.

But the main test I do is periodically go after a specific file, notably if I go to work on something for the first time in years, it will remind me to check that that area is still backing up, make sure I can retrieve a copy of that data from backup.


----------



## PhilBurton

tspear said:


> I no longer have spare computers to wipe and rebuild to test my backup processes.
> Also, the amount of data is getting a little out of hand.
> 
> So what methods are you guys using to test your backup process?
> 
> Tim


For local backup, I use an application called Retrospect, which is very nice because it backs up multiple versions of a file, instead of overwriting files the way file sync programs work.  More often than I would like, I have to restore a file or folder because of "user error."  Always works.   Retrospect backups go to a physically separate HDD.   I purge these backups after three years.

Whenever I do an import into Outlook, Edit: Lightroom, I make a Second Copy, on a different physical HDD.  Several times now, again due to user error, I have had to do a re-import from that Second Copy.


----------



## tspear

Roelof Moorlag said:


> Including al your photo's, how much data is that?
> My Crashplan took month's to finish.



I have just under a TB., it takes about two-three days on average. I have "high speed", as in 750mb sustained up/down.



Ferguson said:


> I am doing more and more testing on VM's.  All it takes is a lot of storage to be able to test most anything, and Windows 10 Pro (not home) has absolutely terrific VM support.
> 
> I also keep an old laptop around, but lack of storage on it makes things harder.
> 
> But the main test I do is periodically go after a specific file, notably if I go to work on something for the first time in years, it will remind me to check that that area is still backing up, make sure I can retrieve a copy of that data from backup.



Yeah, I was thinking the remote VM route since I do not have enough local space. Maybe I should just go get a cheap 2TB hard drive to use for a VM to test restore functionality on. 

Tim


----------



## tspear

PhilBurton said:


> For local backup, I use an application called Retrospect, which is very nice because it backs up multiple versions of a file, instead of overwriting files the way file sync programs work.  More often than I would like, I have to restore a file or folder because of "user error."  Always works.   Retrospect backups go to a physically separate HDD.   I purge these backups after three years>
> 
> Whenever I do an import into Outlook, I make a Second Copy, on a different physical HDD.  Several times now, again due to user error, I have had to do a re-import from that Second Copy.



Phil,

I have a personal script I have built up over time. So far it has served me well, but I can see the day where I am not gonna bother with it. 
At this point, I am more curious about ways people are testing the restore capabilities. Such as you actually installing Lr and making sure the catalog and images are there. What about plugins....

Tim


----------



## Linwood Ferguson

tspear said:


> Yeah, I was thinking the remote VM route since I do not have enough local space. Maybe I should just go get a cheap 2TB hard drive to use for a VM to test restore functionality on.


Test USB connected drives first, I'm not sure if they can be used, there are some limits to what connects through, but I think local VM's are a great way to do all sorts of testing.  EC instances are similar, but it always bothers me a bit that the meter is running while I'm experimenting.


----------



## PhilBurton

tspear said:


> I have just under a TB., it takes about two-three days on average. I have "high speed", as in 750mb sustained up/down.
> 
> 
> 
> Yeah, I was thinking the remote VM route since I do not have enough local space. Maybe I should just go get a cheap 2TB hard drive to use for a VM to test restore functionality on.
> 
> Tim


Tim,

Hard drives are getting ridicously cheap these days, especially the bigger 3.5" drives.  My GO TO brand is HGST, based on drive failure rates reported by Backblaze.  Check out HGST DeskStar NAS 3.5" 4TB 7200 RPM 128MB Cache SATA 6.0Gb/s High-Performance Hard Drive for Desktop NAS Systems Retail Packaging 0S04005 - Newegg.com

Throw that bare drive into this case:  ORICO Aluminum Alloy & ABS 3.5 inch USB3.0 External Hard Drive Enclosure for 9.5mm .5 inch SATA HDD / SSD 8TB Max Support UASP  Windows,XP,Linux,Mac OS 9.1 or above Tool Free -Gray (MD35-U3) - Newegg.com

I have one of these bare drive "readers" and it works great. Computer Parts, PC Components, Laptop Computers, LED LCD TV, Digital Cameras and more - Newegg.com


----------



## PhilBurton

tspear said:


> Phil,
> 
> I have a personal script I have built up over time. So far it has served me well, but I can see the day where I am not gonna bother with it.
> At this point, I am more curious about ways people are testing the restore capabilities. Such as you actually installing Lr and making sure the catalog and images are there. What about plugins....
> 
> Tim


Tim,

Retrospect has backup scripts, with more flexibility than I know what to do with.  Actually, one of my scripts backs up folder C:\Users\phil\AppData\Roaming\Adobe\Lightroom, which  has a subfolder for all plug-ins.  Now, I didn't say that Retrospect is exactly cheap. it's not, but for me it is totally worth it.   Retrospect: Windows > Overview


----------



## tspear

@PhilBurton 

cool product. If I switch, I am gonna choose something bullet proof for a dumb user that is set and forget. 
My current script has all sorts of intelligence; which is likely kinda stupid; and I would never do this now. It really is a waste of time, but I have built it up over the years, swapping in/out some technology as things change.  A few examples of the crazy stuff I have in the script
--  Local git repository for documents to handle version management that auto checks in local changes and replicates the change repository to my remote sync. 
-- Remote sync setup for photography that covers the images/catalog/settings with a date based intelligence. Basically it only deletes from the remote after 60 days.  
-- Remote git repositories for source code control, financial systems...
-- Auto prune of Lr catalog backups

Anyways, trying to see who else is paranoid enough to test his/her backup restore solution; and how they did it and verified it.

Tim


----------



## Linwood Ferguson

@PhilBurton that looks worth checking out, I use Cloudberry which seems similar - versions, cloud or hardware based -- but they changed their licensing some time ago to limit the cheap licenses to 1TB, which was really stupid.  Otherwise it's a terrific product which has been bug free for me for ages, I've used it on CIFS shares, USB drives, Backblaze B2, Amazon S3 and Glacier all without trouble (their Amazon Cloud Drive was not very robust when I tried it). 

I'm a big fan of (paid if needed) 3rd party tools as opposed to a cloud vendor's own -- there have been too many cases of cloud vendors short changing users in some fashion through their tools - throttling, limiting retention periods, etc.  Plus they know your encryption keys.


----------



## OogieM

tspear said:


> So what methods are you guys using to test your backup process?


I run several backup systems. First is Time machine that does incremental backups every hour alternating between our RAID server and an external hard drive. Then I do an automatic bootable backup every night of my main computer. Monthly bootable backups on additional hard drives. One of those is always swapped out to a fire safe in a location about 100 miles away so safe from our expected natural disasters. Another is in another media rated fire proof safe in another building on our farm and a third is in a file cabinet near my machine.

Laptop has Time Machine backing up on a partition on my main computer and an external hard drive. Bootable backups regularly but not automatic. 

LambTracker code  is in GitHub. A few Scrivener documents are in DropBox and a few thingsare in google drive but otherwise basically no cloud solutions.

Picture files are in a partition on the server. That partition is backed up nightly to an external hard drive. The monthly bootable backups of my main machine also include a partition that is a backup of the server partition with my photos.

I test the backups every month by booting both machines from their latest bootable hard drives and opening at least one or 2  files in my major software packages. 

I also have checksum verification on all files running on the daily backups and on my monthly backups to find corrupted and replace corrupted files.


----------



## PhilBurton

tspear said:


> @PhilBurton
> 
> 
> 
> Anyways, trying to see who else is paranoid enough to test his/her backup restore solution; and how they did it and verified it.
> 
> Tim


Yes, I am that paranoid, and for good reason.  To me, the main reason for backups is not hardware failure, it's operator error.


----------



## PhilBurton

OogieM said:


> I also have checksum verification on all files running on the daily backups and on my monthly backups to find corrupted and replace corrupted files.



Oogie, what do you use for checksum verification?


----------



## OogieM

PhilBurton said:


> Oogie, what do you use for checksum verification?


CarbonCopy Cloner's built in advanced option to find and replace corrupted files. 

For me the more automated the backups are the better.


----------

