# Slow backup and diagnosis



## debi.lightcraft (Aug 24, 2021)

I decided to begin a new post because my previous post was mixed in with another one.  I am new and learning the rules.  Thank you for your patience! Below is Jim Wilde's comment I am responding to:

"A couple of hours" is simply not right for a backup of a 1.7GB catalog. Can you tell from the backup process if any of the stages seems to take up most of that time?

Where does the catalog reside? If it's on the same internal drive that you are backing it up to, then that's something you should consider changing (a system drive crash means you've lost the catalog and the catalog backups).

Unplugging external hard drives shouldn't have any effect on the speed of the catalog backup."

Thank you Jim.   So I believe my catalog and backup both exist on the imac hard drive.  Most of my photos are on external drives which I import and work on in LR.  There are some still on the computer in various places, a hold over from many years ago when I was starting out.  Also, I store select photos in a special collection area in LR.  I believe the first checking of integrity takes the longest.   However, the computer also freezes up from time to time during the backup which is unsettling!    Actually, the whole computer is running slow.  I pulled out my old imac (I purchased this one a few years ago) and the old one boots up in a snap unlike my newer one!  Also, When I purchased the new imac, the transfer was done by the company.  I wonder if something happened in the transfer?  Looking at my old imac I notice that the backup file structure is a little different.  Would that jam things up?  Should I change the location of the backup to match the old one?  However,  I might relocate the backup to an external drive.   Do you think this might make things run faster? Also, when my files were transferred some photos went missing.  In fact, I had many photos in the old iphoto catalog which linked to LR and  and none of those were transferred over.  There are also other missing photos.  Could this be jamming up the works? Should my next step be to try to get the photos all linked up as a first step in figuring out what is going on?  What do you think might be a first step?  Thank you in advance!


----------



## Replytoken (Aug 25, 2021)

Two thoughts come to mind.  First, how full is your internal hard drive?  Do you have any free space on it?  If so, how much?  A drive that is too full can cause issues if the computer need space to swap files.

Second, have you tried to create a new catalog to see if it also exhibits issues?  Yes, it will be smaller than your catalog in question, but if it also exhibits problems, then the catalog alone may not be the issue.

--Ken


----------



## Gnits (Aug 25, 2021)

A few thoughts.

Every 18 -24 months I create a new Catalog from my existing Catalog.  It gives the database a chance to restart from a fresh start.  Who knows how many times a Lightroom process has aborted mid way (power cut, system crash) leaving an accumulation of broken indices within the catalog. This may also reduce the size of the database. I am sometimes prompted to do this by the sense that Lr or Lr backups are running slower than usual.  The internal housekeeping within the Lr Catalog has improved over the years, but a fresh catalog, with internal indices rebuilt from scratch is worth considering.

I second the suggestion to check the amount of free space on your drive.

Finally, as you have loads of external drives, consider backing up your catalog to an external drive. The obvious benefit is that you do not use your live and backup catalog if/ when your main drive fails.  Use your most recently purchased external drive on the basis that it might be the fastest and also least likely to fail or maybe it is time to get a new fast external  ssd drive for backup (faster and more resistant to various, but not all, forms of failure).  Check how much space your old catalogs are consuming and decide if it is time to delete your oldest backups ( are you really going to go back to a backup taken months or years ago).


----------



## debi.lightcraft (Aug 25, 2021)

Replytoken said:


> Two thoughts come to mind.  First, how full is your internal hard drive?  Do you have any free space on it?  If so, how much?  A drive that is too full can cause issues if the computer need space to swap files.
> 
> Second, have you tried to create a new catalog to see if it also exhibits issues?  Yes, it will be smaller than your catalog in question, but if it also exhibits problems, then the catalog alone may not be the issue.
> 
> --Ken


Thank you Ken.  My catalog is on the hard drive of my Imac.  It has 718.98 GB available of 1TB.  (My photos are all stored on external hard drives. ) So I don't think the problem is space.  I have not yet tried to create a new catalog.  Why would that be smaller than my current catalog?


----------



## debi.lightcraft (Aug 25, 2021)

Gnits said:


> A few thoughts.
> 
> Every 18 -24 months I create a new Catalog from my existing Catalog.  It gives the database a chance to restart from a fresh start.  Who knows how many times a Lightroom process has aborted mid way (power cut, system crash) leaving an accumulation of broken indices within the catalog. This may also reduce the size of the database. I am sometimes prompted to do this by the sense that Lr or Lr backups are running slower than usual.  The internal housekeeping within the Lr Catalog has improved over the years, but a fresh catalog, with internal indices rebuilt from scratch is worth considering.
> 
> ...


Thank you Gnits.  I don't think space on my hard drive is the probem as it has close to 719 GB available of 1 TB. (see above).   To back up to an external drive, do I simply select it when the backup prompt comes up? Is there a certain external drive you recommend? My photos are on a 2 tb g drive which has lots of space available  G-Technology 2TB G-DRIVE mobile USB 3.1 Gen 1 Type-C External Hard Drive (Gold).    I only have a few backups on my computer.  I've deleted most over the years.


----------



## Replytoken (Aug 25, 2021)

The test catalog I had suggested would be smaller as it would be created brand new and you would just add a number of images to it.  I was not thinking of trying to recreate your catalog in question and it is a quick test to perform to help rule out problem areas.  Alternately, if you have other working catalogs, you could try to back one of them up and see if the problem persists.

--Ken


----------



## Gnits (Aug 25, 2021)

Yes, select a new destination at the time of doing a backup. Any working external drive will do. As the root problem is the speed of your backup, then, if you have a choice, pick the fastest drive  and which has the fastest interface (eg Thunderbolt 3 is faster than Thunderbolt 2, Usb 3 is faster than USB 2, , ssd drive is faster than spinning disk, etc).

Great that space on your system drive is not an issue.

Exporting your full catalog as a new catalog may reduce the size of your catalog (in some cases significantly so) as there may be a build up of internal broken links and indices within the catalog database, which take up space and may be a root cause of slow backups .

This has solved issues for me in the past, less so now but I still regard this process as part of my long term maintenance of my catalog (which I do approx every 18 minths).  Earlier versions of Lr were more buggy, might have crashed more often in the past  so maybe less need for me to do this now……. but if I find anything unusual with Lr performance or length of backup, apart from checking the integrity and amount of available space on my backup destination drive I will create a fresh version of my catalog.


----------



## debi.lightcraft (Aug 25, 2021)

Replytoken said:


> The test catalog I had suggested would be smaller as it would be created brand new and you would just add a number of images to it.  I was not thinking of trying to recreate your catalog in question and it is a quick test to perform to help rule out problem areas.  Alternately, if you have other working catalogs, you could try to back one of them up and see if the problem persists.
> 
> --Ken


Ken--Thank you for the clarification!  I was confused between the backup and the brand new catalog!


----------



## Replytoken (Aug 25, 2021)

debi.lightcraft said:


> Ken--Thank you for the clarification!  I was confused between the backup and the brand new catalog!


No worries.  I was suggesting a very quick test.  @Gnits also had a good suggestion above.  Either should help you try and identify the source of your problems.

--Ken


----------



## debi.lightcraft (Aug 26, 2021)

debi.lightcraft said:


> Ken--Thank you for the clarification!  I was confused between the backup and the brand new catalog!


Thank you Ken and @Gnits .  I really appreciate your help.  So Ken, if I understand you, I should start a brand new catalog and import a few photos into it and then back it up as I always do to see how it backs up.   If this is correct, where should I locate the brand new test catalog?  Also, I an wondering if I should just start a brand new catalog. BTW, my computer only has 8 gb of ram (I should have known better).


----------



## debi.lightcraft (Aug 26, 2021)

Thank you Ken @Replytoken and @Gnits .  I really appreciate your help.  So Ken, if I understand you, I should start a brand new catalog and import a few photos into it and then back it up as I always do to see how it backs up.   If this is correct, where should I locate the brand new test catalog?  Also, I an wondering if I should just start a brand new catalog. BTW, my computer only has 8 gb of ram (I should have known better).

Also, @Jim Wilde, I started a new thread about my issue in this forum to respond to your post in another thread (please see quote at top) as I did not see then how to tag you.  Thank you so much for your help!    I also rechecked the size of my catalog and I believe it is 2.40 gigs, not 1.7 as I previously mentioned (which was the size of the zipped file).


----------



## Replytoken (Aug 26, 2021)

To keep things simple, save the new catalog on your main internal drive.  Copy a hundred or so photos to the new catalog, and then back it up.  It should all happen pretty quickly.  If not, then your problem is probably not catalog specific.

--Ken


----------



## debi.lightcraft (Aug 26, 2021)

Thank you Ken @Replytoken!  Does it matter where I save the new catalog on the main internal drive?  

 Also, re "copy a hundred or so photos to the new catalog"  Is this as opposed to importing them?  Do you know where I might find instructions on how to do this?  Currently, I add photos to external hard drives and then import the photos to lightroom.   I am nervous about making errors and screwing things up.


----------



## Hal P Anderson (Aug 26, 2021)

Just create a folder on your main drive. Create the new catalogue in that folder. You'll delete it when you're done testing. 

Ken meant "import" to the new catalogue.


----------



## debi.lightcraft (Aug 26, 2021)

Thank you for the clarification Hal (@Hal P Anderson) .  So Anywhere on the main drive?  Is the desktop ok?


----------



## Hal P Anderson (Aug 26, 2021)

Create a new folder on your desktop and create the catalogue in that folder.


----------



## debi.lightcraft (Aug 27, 2021)

Well, I created a new catalog with over a hundred photos and it backed up in a matter of seconds. Editing in it was speedy as well.   Is there a way to fix my regular catalog?  I have suspicions that the problem has something to do with the way it was transferred when I purchased my new imac.  Or perhaps it has to do with missing files?  Or all the edits I do on photos that have built up over the years?   Thank you in advance for your thoughts and advice!


----------



## Replytoken (Aug 27, 2021)

Hal P Anderson said:


> Just create a folder on your main drive. Create the new catalogue in that folder. You'll delete it when you're done testing.
> 
> Ken meant "import" to the new catalogue.


Thanks Hal.  Dealing with a small medical issue and multi-tasking is not my strong suit.

--Ken


----------



## debi.lightcraft (Aug 27, 2021)

I am sorry you are not feeling well Ken.  thank you for getting me going on this test.


----------



## Replytoken (Aug 27, 2021)

debi.lightcraft said:


> I am sorry you are not feeling well Ken.  thank you for getting me going on this test.


Thank you for the kind words.  Usually it is me.  In this case, it was my beloved.  She is doing better, but it was a very long couple of days.

--Ken


----------



## debi.lightcraft (Aug 27, 2021)

I hope she is better and you too, Ken.  --debi


----------



## debi.lightcraft (Aug 27, 2021)

Thank you to everyone on this forum who has been helping me get to the bottom of my slow backup with LR classic.  I have now created a test catalog on my desktop with over 100 photos that backs up in seconds.  I guess I could just create a new catalog for my new photos. But this still leaves me with the question as to why the other one backs up so slowly.  It has s 30,773 images and the catalog is 2.40 gb (although the zipped file is 1.7).  Should I try to back it up to an external disk and see if that is faster?   Also, there are many missing photos.  Could this be causing the slow backup?  Thank you in advance!


----------



## clee01l (Aug 27, 2021)

debi.lightcraft said:


> …. But this still leaves me with the question as to why the other one backs up so slowly. It has s 30,773 images and the catalog is 2.40 gb (although the zipped file is 1.7). Should I try to back it up to an external disk and see if that is faster? Also, there are many missing photos. Could this be causing the slow backup? Thank you in advance!


In your earlier thread, I suggested that slowness may be due to orphaned records in some of the internal tables. If you use 3rd party plugins (and maybe even Lightroom itself) doesn’t clean up properly Important records in related tables may be properly deleted and related records in other tables may get left behind.

That is why it has been recommended by me and others to use the import from another catalog (your slow master) to a new empty catalog. this will only bring over the good records and leave the detritus behind. 

As for the missing photos, these should be resolved soonest. The only thing missing in these records is the path to the source file field is not resolving. In theory, this should not slow the backup, but Lightroom might’ve be pausing to try to resolve the missing file.


Sent from my iPad using Tapatalk


----------



## debi.lightcraft (Aug 28, 2021)

Thank you Clee.  I left that thread because I had jumped in on it when I first joined and then felt it wasn't clear who people were writing to and thought to clarify by starting out anew. Apparently this did not have the intended effect! Sorry!   I wiil try to keep to one thread and learn the rules better.  Thank you for following up here!

So it seems from what you wrote I should start first to find missing photos.   Many went missing when I transferred to this new computer.  I have pulled out my old Imac which may help me track down some of the paths.  I will start this process and then start the second part suggested by you and Gnits and others.   I may run into  a snag in tracking them down.  Thank you so much!


----------



## debi.lightcraft (Aug 29, 2021)

I am in the process of finding missing photos in order to clean up the mess that may be contributing to the slow backup.  I have two external hard drives with the same name. The first is pretty old so I copied most of the photos to a newer drive which I keep plugged into my computer so I can use it while I am working.  The older one pretty much sits on my desk unused except now I am using it because it has photos that turn up missing when not plugged into LR.    I tell them apart on the computer by the color of the icon, not by the name which is unfortunately the same.  I seem to run into  snags when they are both plugged in at the same time.  Is this a coincidence or is it a bad idea to plug them in at the same time while tracking missing photos.  I'm very uneasy about renaming them.   Thank you so much in advance for your thoughts and advice!


----------



## debi.lightcraft (Aug 24, 2021)

I decided to begin a new post because my previous post was mixed in with another one.  I am new and learning the rules.  Thank you for your patience! Below is Jim Wilde's comment I am responding to:

"A couple of hours" is simply not right for a backup of a 1.7GB catalog. Can you tell from the backup process if any of the stages seems to take up most of that time?

Where does the catalog reside? If it's on the same internal drive that you are backing it up to, then that's something you should consider changing (a system drive crash means you've lost the catalog and the catalog backups).

Unplugging external hard drives shouldn't have any effect on the speed of the catalog backup."

Thank you Jim.   So I believe my catalog and backup both exist on the imac hard drive.  Most of my photos are on external drives which I import and work on in LR.  There are some still on the computer in various places, a hold over from many years ago when I was starting out.  Also, I store select photos in a special collection area in LR.  I believe the first checking of integrity takes the longest.   However, the computer also freezes up from time to time during the backup which is unsettling!    Actually, the whole computer is running slow.  I pulled out my old imac (I purchased this one a few years ago) and the old one boots up in a snap unlike my newer one!  Also, When I purchased the new imac, the transfer was done by the company.  I wonder if something happened in the transfer?  Looking at my old imac I notice that the backup file structure is a little different.  Would that jam things up?  Should I change the location of the backup to match the old one?  However,  I might relocate the backup to an external drive.   Do you think this might make things run faster? Also, when my files were transferred some photos went missing.  In fact, I had many photos in the old iphoto catalog which linked to LR and  and none of those were transferred over.  There are also other missing photos.  Could this be jamming up the works? Should my next step be to try to get the photos all linked up as a first step in figuring out what is going on?  What do you think might be a first step?  Thank you in advance!


----------



## clee01l (Aug 30, 2021)

debi.lightcraft said:


> I am in the process of finding missing photos in order to clean up the mess that may be contributing to the slow backup. I have two external hard drives with the same name. The first is pretty old so I copied most of the photos to a newer drive which I keep plugged into my computer so I can use it while I am working. The older one pretty much sits on my desk unused except now I am using it because it has photos that turn up missing when not plugged into LR. I tell them apart on the computer by the color of the icon, not by the name which is unfortunately the same. I seem to run into snags when they are both plugged in at the same time. Is this a coincidence or is it a bad idea to plug them in at the same time while tracking missing photos. I'm very uneasy about renaming them. Thank you so much in advance for your thoughts and advice!



Lightroom Classic will have set the catalog path to the old name. Presumably you have images that are also cataloged to a path on the new volume with the same name. It won’t hurt to rename the old volume because missing images in LrC that are on it will still be missing and can be found using the “Find missing folder/find missing file” process. 

By renaming the old volume and mounting it alongside the new volume, you will create less confusion in the file system and probably make it easier to relocate the missing images. 


Sent from my iPad using Tapatalk


----------



## debi.lightcraft (Aug 30, 2021)

clee01l said:


> Lightroom Classic will have set the catalog path to the old name. Presumably you have images that are also cataloged to a path on the new volume with the same name. It won’t hurt to rename the old volume because missing images in LrC that are on it will still be missing and can be found using the “Find missing folder/find missing file” process.
> 
> By renaming the old volume and mounting it alongside the new volume, you will create less confusion in the file system and probably make it easier to relocate the missing images.
> 
> ...


Thank you Clee.  On second thought would it be fine to buy a new external hard drive and transfer all the lightroom photos from the external old hard drive to the new one under a new name?  That way I don't risk messing up the old hard drive which is sluggish anyway and has other non-lightroom documents on it anyway.  ?


----------



## debi.lightcraft (Aug 31, 2021)

Ok.  I have ordered a 1 terabyte external hard drive (G Drive) due here on Wednesday.  I plan to transfer my lightroom photos from a very old clunky external hard drive to the brand new one but under a different name.  This will distinguish it from my other 6 month old external hard drive (the two had the same name which was confusing-- see above).   Hopefully this will make searching for missing photos simpler because I can have them both plugged in to my imac at the same time without confusing LR and myself!  (Thank you for the clarification clee)  So the identification of missing photos comes first.  Next I will see if the backup is faster.  If not I will tackle the suggestion by Clee and Gnits to use the import from another catalog to a new empty catalog which should  bring over the good records leaving the detritus behind.   However, it was also suggested earlier that I should move my backup so it doesn't exist on my computer next to the catalog which could be disastrous in the event of a hard drive failure.  To address this I am thinking of purchasing yet another hard drive, this time a one terabyte Sandisk (which is supposed to be really fast) as a designated place to store my catalog backup.  Is this a sensible plan?  Thanks!


----------



## debi.lightcraft (Sep 7, 2021)

Gnits said:


> A few thoughts.
> 
> Every 18 -24 months I create a new Catalog from my existing Catalog.  It gives the database a chance to restart from a fresh start.  Who knows how many times a Lightroom process has aborted mid way (power cut, system crash) leaving an accumulation of broken indices within the catalog. This may also reduce the size of the database. I am sometimes prompted to do this by the sense that Lr or Lr backups are running slower than usual.  The internal housekeeping within the Lr Catalog has improved over the years, but a fresh catalog, with internal indices rebuilt from scratch is worth considering.
> 
> ...





Gnits said:


> A few thoughts.
> 
> Every 18 -24 months I create a new Catalog from my existing Catalog.  It gives the database a chance to restart from a fresh start.  Who knows how many times a Lightroom process has aborted mid way (power cut, system crash) leaving an accumulation of broken indices within the catalog. This may also reduce the size of the database. I am sometimes prompted to do this by the sense that Lr or Lr backups are running slower than usual.  The internal housekeeping within the Lr Catalog has improved over the years, but a fresh catalog, with internal indices rebuilt from scratch is worth considering.
> 
> ...


Thank you Gnits--it has taken me so long to reply because I am dealing with multiple issues in LR including tracking down missing photos.  I have spent days at this and simply can't find them all (about 40 are left) and originate in the Photos library on my iMac.  But I have concerns about my hard drive as the whole system is slow--not just LR.  So I really want to backup my catalog now to an external drive for the reasons you suggest.  I will follow your suggestion in your earlier post!  Also, you suggest creating a new catalog from my existing catalog to clean up the broken indices.  Would you be able to explain how to do that?  Or point me  to the right place? (I have the missing facts and questions ebook).  

Also, do you think I should purchase a fast external drive -- I'm thinking of a sandisk with up to 1050 MB/s Read Speed or is my g drive with up to 130 MB/s Data Transfer Speed perfectly adequate?  I apologize for such basic questions and for any lack of clarity, but while I have learned a lot about the basic editing tools in LR, I jumped in years ago without understanding the catalog and am paying the price now.  Thanks!


----------



## Gnits (Sep 8, 2021)

Busy right now.  I will revert later today or tomorrow latest.  

Re Drive.  Choice of drive is always a compromise between price, performance and capacity.
Decide how much storage you need will help define the capacity question and then check price v performance to see if you can find your specific sweet spot.

Regarding external storage……. Check the max speed of the port you plan to use. If it is usb2 then a fast drive will not deliver immediate speed benefits, but will help future proof possible future computer upgrades. Thunderbolt 3 /4 is the fastest port type at the moment, with  speed diff which is orders of magnitude  compared to ,say, usb2.


----------



## debi.lightcraft (Sep 8, 2021)

Thanks a million for getting back so quickly Gnits!   From what I can tell, my 2017 iMac has 4 USB 3 ports (compatible with USB 2) and 2 thunderbolt 3 (USB-C) ports with support for thunderbolt (up to 40 Gbps) and USB 3.1 Gen 2 (up to 10 Gbps).    I will check with B&H tomorrow to see what they have available (while considering cost)  to make use of this.   I'm hoping a fast external hd will help speed up my LR backup and also protect my catalog in the unhappy event of a hd failure!    Meanwhile I'll wait for your explanation re how to create a new catalog from my existing one.   Thanks so much again!


----------



## Gnits (Sep 9, 2021)

I have created some notes to assist with creating a new version of a catalog,  leaving the images in their current location. 

I also take this opportunity to get Lr to rebuild previews  as required (ie I do NOT copy the existing previews), as there may be an accumulation of orphan preview files from previous Lr crashes, etc.

I am happy for anyone to critique the attached pdf or to add comments or suggestions. Please note there are 2 pages.


----------



## debi.lightcraft (Sep 9, 2021)

Wow!  Thank you so very much for taking the time and effort to create this excellent document Gnits!  I have read it quickly and will go back after deciding on a hard drive (possibly a new one--have to call b&H) and then read your instructions very carefully  and summon up the courage to move forward.


----------



## Gnits (Sep 10, 2021)

You can always create a new test catalog, with say 5 images in a folder on your desktop, use the instructions to create a new test catalog in a location of your choice.
You can then double click in this new test catalog and see that the images have remained in their existing desktop  location, even though your test catalog has been rebuilt from scratch in a new location.

Top tip. Give the new catalalog a short, sensible, recognisable filename.
Remember, you can also use the File Open Recent Catalogs to go back to open your main catalog. Write down the name and location of your current main catalog, so you do not confuse it with your test or previous c atalogs.


----------



## debi.lightcraft (Sep 10, 2021)

Gnits said:


> You can always create a new test catalog, with say 5 images in a folder on your desktop, use the instructions to create a new test catalog in a location of your choice.
> You can then double click in this new test catalog and see that the images have remained in their existing desktop  location, even though your test catalog has been rebuilt from scratch in a new location.
> 
> Top tip. Give the new catalalog a short, sensible, recognisable filename.
> Remember, you can also use the File Open Recent Catalogs to go back to open your main catalog. Write down the name and location of your current main catalog, so you do not confuse it with your test or previous c atalogs.


Thank you Gnits.  I'm really glad you suggested a way to test the process before working with the big catalog and really appreciate the thought and time you have put into this question!  Meanwhile,  I ordered a new fast(er) external hd and my first step will be to try my catalog backup to the new drive.  After seeing what happens with that I will take on the task of importing the old catalog into a new one and understand the steps and details you have so carefully described.


----------



## debi.lightcraft (Sep 13, 2021)

Gnits said:


> You can always create a new test catalog, with say 5 images in a folder on your desktop, use the instructions to create a new test catalog in a location of your choice.
> You can then double click in this new test catalog and see that the images have remained in their existing desktop  location, even though your test catalog has been rebuilt from scratch in a new location.
> 
> Top tip. Give the new catalalog a short, sensible, recognisable filename.
> Remember, you can also use the File Open Recent Catalogs to go back to open your main catalog. Write down the name and location of your current main catalog, so you do not confuse it with your test or previous c atalogs.


Hi Gnits--I have tested the process for creating a new catalog out of the old using a test catalog (from about a week ago) according to your excellent pdf.    The process was smooth and a new catalog was created.  Before I move ahead with my main catalog (over 30,000 photos)  I'm wondering about unchecking all the boxes (step 3) and if this might lead to losing some of the edits or ratings etc?   Thank you in advance--I don't want to make any mistakes!


----------



## Gnits (Sep 13, 2021)

Check boxes.
1. Export Selected Photos Only   (you want all photos selected .to be sure....to be sure) ... Leave unticked.
2 Export Negative Files.  You do not want to move 30000 image files. Leave unticked.
3. Build / Include Smart Previews .  Again, I want to start from scratch and not have lots of old /orphaned smart previews. Leave unticked.
4. Include Available Previews.  The objective is to start with as fresh a setup as possible. Leaving unticked means no orphaned / corrupt previews are migrated.

Sometimes I saw 3 options and sometimes 4.  I always go into All Photographs and Cmd/Ctrl A to insure all photos are selected.  You should then see the number of images and virtual images will be created in the newly built catalog.


----------



## debi.lightcraft (Sep 13, 2021)

Gnits said:


> Check boxes.
> 1. Export Selected Photos Only   (you want all photos selected .to be sure....to be sure) ... Leave unticked.
> 2 Export Negative Files.  You do not want to move 30000 image files. Leave unticked.
> 3. Build / Include Smart Previews .  Again, I want to start from scratch and not have lots of old /orphaned smart previews. Leave unticked.
> ...


Thank you for the explanation! Question--Re, "start from scratch"  -- where does that mean LR is starting from?  What happens to edits and edit history, ratings, etc?


----------



## clee01l (Sep 13, 2021)

debi.lightcraft said:


> Thank you for the explanation! Question--Re, "start from scratch" -- where does that mean LR is starting from? What happens to edits and edit history, ratings, etc?



Gnits might be on the other side of midnight. So, I just clarify what I think he meant. “Start from Scratch”. Refers only to the Preview images in the preview folder(s) By leaving that box unticked, you will cause Lightroom to build a new set of previews in the preview folders. It will use the edits stored in the catalog to be applied to the original images to create the preview images.


Sent from my iPad using Tapatalk


----------



## debi.lightcraft (Sep 13, 2021)

clee01l said:


> Gnits might be on the other side of midnight. So, I just clarify what I think he meant. “Start from Scratch”. Refers only to the Preview images in the preview folder(s) By leaving that box unticked, you will cause Lightroom to build a new set of previews in the preview folders. It will use the edits stored in the catalog to be applied to the original images to create the preview images.
> 
> 
> Sent from my iPad using Tapatalk


Thank you clee!  Ok, I will assume this also means that the edits, ratings, etc., will all be represented in the new catalog--so I'll give it a go and see what happens!


----------



## Gnits (Sep 13, 2021)

Thank you Cletus....


----------



## debi.lightcraft (Sep 14, 2021)

OMG!  It has worked!!  At least I think it has worked!  This is amazing!  I was able to import the old master catalog into a new one according to Gnits' crystal clear instructions.  The new catalog, on my hard drive opens up with the same number of photos as the old master catalog and as far as I have been able to tell, all the edits, ratings, presets I created are still intact.  It moves more rapidly than the old one.   I haven't really tested the editing yet, but I have an idea it will be faster too.  Then I backed it up to my new external ssd sandisk.  This took a total of 6 minutes (as opposed to an hour and a half on my old catalog, what with the computer freezing up etc.).   The majority of time backing up the new catalog was in checking the integrity and then optimization, with back up itself maybe seconds.   The new master catalog  (1.23 GB) is about half the size of the old master catalog ( 2.58GB).  Similarly, The new backup zip file is also much smaller (708.8mb) as opposed to the old backup (1.74 GB).  Question: Is this reduction in size ok?  What causes it?   I have other questions such as what to do with the old master catalog, but for now, assuming the reduction in catalog size is fine, I am delighted, as this problem has been with me for over a year.   Thanks so much Clee and Gnits for guiding me through missing photos, new hard drives and what to call them, step by step instructions on how to create a new catalog all the way to a backup and where to put it. I feel I am making progress.


----------



## clee01l (Sep 14, 2021)

debi.lightcraft said:


> …The new master catalog (1.23 GB) is about half the size of the old master catalog ( 2.58GB). Similarly, The new backup zip file is also much smaller (708.8mb) as opposed to the old backup (1.74 GB). Question: Is this reduction in size ok? What causes it?



“That is why it has been recommended by me and others to use the import from another catalog (your slow master) to a new empty catalog. this will only bring over the good records and leave the detritus behind. “

You had a lot of detritus in the old catalog. Zip the old catalog file up and save it in your backups folder. Hopefully you will never nee to recover from a backup as far back as this bloated catalog.


Sent from my iPad using Tapatalk


----------



## Gnits (Sep 15, 2021)

Glad that this has a successful conclusion.

This is an example which demonstrates the amount of orphaned data which can reside in a catalog.  While it is possible that some of this may have been left, at some stage, by errors within Lr, especially in the early versions, Lr has been running so long by so many users on so many platforms that Adobe will have optimised the stability of their database processing in so far as what is practically possible.

We tend to forget that Lr can be asked to complete complex database updates on lots of metadata fields across very large number of image records. These processes may be running in the background even as you are using Lr for some other task. It s not surprising that these tasks can be interupted by factors outside of Adobe’s control, such as power outage, memory issues, disk i/o problems, forcing Lr to shutdown, etc.

As part of this exercise, I completed the steps documented in the pdf above, so I could create the screen grabs, etc.  In my case, there was absolutely no difference in size between my old and rebuilt catalog. It is approx 18 months since I last completed such a rebuild, which indicates that nothing out of the ordinary has happened to my catalog in that period of time. This is reassuring for me.

However, there must be many many Lr users who have catalogs which may be accumulating orphaned data over many many years. At some stage, thus orphaned data will trip up other normal Lr processes. Not surprisingly, backing up a Catalog is probably one activity with a high likelihood of this happening.

I will continue to create a fresh version of my catalog every 18-24 months, as documented above, , just as a routine  maintenance task.


----------



## debi.lightcraft (Sep 16, 2021)

clee01l said:


> “That is why it has been recommended by me and others to use the import from another catalog (your slow master) to a new empty catalog. this will only bring over the good records and leave the detritus behind. “
> 
> You had a lot of detritus in the old catalog. Zip the old catalog file up and save it in your backups folder. Hopefully you will never nee to recover from a backup as far back as this bloated catalog.
> 
> ...





Gnits said:


> Glad that this has a successful conclusion.
> 
> This is an example which demonstrates the amount of orphaned data which can reside in a catalog.  While it is possible that some of this may have been left, at some stage, by errors within Lr, especially in the early versions, Lr has been running so long by so many users on so many platforms that Adobe will have optimised the stability of their database processing in so far as what is practically possible.
> 
> ...


I thank you Gnits for the time and effort you have put into helping diagnose and resolve the slow backup of my LR catalog and also for your patience and explanations as to the origins of these frustrating issues.  Once again, you have anticipated my questions.  I have long wondered about the cause of the the slow backup to my catalog--(eg., was it not enough ram,  something missing in the data transfer to a new computer, too many missing photos?). My understanding now is  that LR's ability to process data can be affected by many factors and "orphaned" data may be compounded over the years (probably originating way back in my LR3 catalog).   Finally here is a way to import the master into a new catalog that leaves the broken data behind and captures the solid working data.  I've been dealing with this for a long time (and probably before I was even aware of it) and have had no luck in resolving 'til now!  I thought about starting a new catalog, but am glad I held off as this appears to have solved the problem.   I'm eager to test it all out with some new photos.

I will follow your lead in repeating the process in 18 months or sooner if need be.  But do you have any suggestions as to how to prevent these errors in the first place?  I hope to get to the point where I can complete the steps in the pdf and say as you do there is no difference in the size of the two catalogs.

Thank you !!!

Debi


----------



## debi.lightcraft (Sep 16, 2021)

clee01l said:


> “That is why it has been recommended by me and others to use the import from another catalog (your slow master) to a new empty catalog. this will only bring over the good records and leave the detritus behind. “
> 
> You had a lot of detritus in the old catalog. Zip the old catalog file up and save it in your backups folder. Hopefully you will never nee to recover from a backup as far back as this bloated catalog.
> 
> ...


Thank you! I will put it aside and hope I never have to recover from it (the old catalog)!


----------



## debi.lightcraft (Sep 16, 2021)

debi.lightcraft said:


> I thank you Gnits for the time and effort you have put into helping diagnose and resolve the slow backup of my LR catalog and also for your patience and explanations as to the origins of these frustrating issues.  Once again, you have anticipated my questions.  I have long wondered about the cause of the the slow backup to my catalog--(eg., was it not enough ram,  something missing in the data transfer to a new computer, too many missing photos?). My understanding now is  that LR's ability to process data can be affected by many factors and "orphaned" data may be compounded over the years (probably originating way back in my LR3 catalog).   Finally here is a way to import the master into a new catalog that leaves the broken data behind and captures the solid working data.  I've been dealing with this for a long time (and probably before I was even aware of it) and have had no luck in resolving 'til now!  I thought about starting a new catalog, but am glad I held off as this appears to have solved the problem.   I'm eager to test it all out with some new photos.
> 
> I will follow your lead in repeating the process in 18 months or sooner if need be.  But do you have any suggestions as to how to prevent these errors in the first place?  I hope to get to the point where I can complete the steps in the pdf and say as you do there is no difference in the size of the two catalogs.
> 
> ...


“I thought about starting a new catalog,”. 

Imeaning a separate catalog with brand new photos


----------



## Gnits (Sep 16, 2021)

debi.lightcraft said:


> But do you have any suggestions as to how to prevent these errors in the first place? I


Some root causes can be completely outside your control, such as a power failure while Lr is updating the catalog, a bug in Adobe or 3rd party software, a disk drive which may be failing, faulty memory and lots of other events outside of your control.

That is why regular backups are so important.

You can positively work to minimise such events by making sure your Pc/Mac has a quality power supply, is operating within the recommended temperatures, components are not drawing more power than the power supply can deliver and other normal computer hygiene actions.  You cannot eliminate all such factors.

Overclocking Pcs may damage or cause internal component failure (cpu, memory, motherboard). Installing graphics cards with more power demands than the Pc motherboard can supply, incorrectly configured memory, etc., can be elements which are self inflicted.


----------



## debi.lightcraft (Sep 17, 2021)

Gnits said:


> Some root causes can be completely outside your control, such as a power failure while Lr is updating the catalog, a bug in Adobe or 3rd party software, a disk drive which may be failing, faulty memory and lots of other events outside of your control.
> 
> That is why regular backups are so important.
> 
> ...


Thank you for your  tips — i will try to stay on top of it.  It’s a wonder anything ever works at all.


----------



## paulcroft (Oct 9, 2021)

Gnits said:


> I have created some notes to assist with creating a new version of a catalog,  leaving the images in their current location.
> 
> I also take this opportunity to get Lr to rebuild previews  as required (ie I do NOT copy the existing previews), as there may be an accumulation of orphan preview files from previous Lr crashes, etc.
> 
> I am happy for anyone to critique the attached pdf or to add comments or suggestions. Please note there are 2 pages.


Hi Gnits, just wanted to say many thanks for creating this pdf.  Today I was getting exasperated with the slowness of my pc when closing Lightroom and I realised that it was primarily due to my never having updated my catalog in far too many years.  This was just what was needed and I've taken the liberty of downloading and saving this for next time.
Paul


----------



## debi.lightcraft (Aug 24, 2021)

I decided to begin a new post because my previous post was mixed in with another one.  I am new and learning the rules.  Thank you for your patience! Below is Jim Wilde's comment I am responding to:

"A couple of hours" is simply not right for a backup of a 1.7GB catalog. Can you tell from the backup process if any of the stages seems to take up most of that time?

Where does the catalog reside? If it's on the same internal drive that you are backing it up to, then that's something you should consider changing (a system drive crash means you've lost the catalog and the catalog backups).

Unplugging external hard drives shouldn't have any effect on the speed of the catalog backup."

Thank you Jim.   So I believe my catalog and backup both exist on the imac hard drive.  Most of my photos are on external drives which I import and work on in LR.  There are some still on the computer in various places, a hold over from many years ago when I was starting out.  Also, I store select photos in a special collection area in LR.  I believe the first checking of integrity takes the longest.   However, the computer also freezes up from time to time during the backup which is unsettling!    Actually, the whole computer is running slow.  I pulled out my old imac (I purchased this one a few years ago) and the old one boots up in a snap unlike my newer one!  Also, When I purchased the new imac, the transfer was done by the company.  I wonder if something happened in the transfer?  Looking at my old imac I notice that the backup file structure is a little different.  Would that jam things up?  Should I change the location of the backup to match the old one?  However,  I might relocate the backup to an external drive.   Do you think this might make things run faster? Also, when my files were transferred some photos went missing.  In fact, I had many photos in the old iphoto catalog which linked to LR and  and none of those were transferred over.  There are also other missing photos.  Could this be jamming up the works? Should my next step be to try to get the photos all linked up as a first step in figuring out what is going on?  What do you think might be a first step?  Thank you in advance!


----------



## Gnits (Oct 9, 2021)

Paul, 

Very happy this has been useful to you.  In a few weeks time I will review comments on this discussion and will add any useful comments or nuggets of info to the pdf.


----------



## dpirazzi (Oct 10, 2021)

Thanks for the write up, this would reduce the size of my catalog file by about 4% (910MB -> 870MB). 

You may want to add to your write up something regarding synced photos and synced collections. Since LRC can only sync a single catalog, the new catalog will have lost those connections to the data stored in the cloud.  Not sure if there is an easy way to restore those connections without deleting all synced photographs prior to export and then re-syncing in the new catalog.


----------

