# I have a catalogue MESS!



## domsueads (Apr 18, 2019)

i have over 300,000 photos in my catalogue. the catalogue (and backups) are on my internal HD on an IMAC.
i have the photos spread over 6 or 7 hard drives, each in a folder by date. 
here's the problem-as i ran out of room on one external drives, i originally "squeezed" new files that i imported into any drive that had space on it...
So, now i sit with some photos from 2017 spread over multiple hard drives, sometimes i have the same MO/DA/YY on different drives (not duplicate files but different photos from the same day on different drives).
I have a brand new 8TB drive that i'd like to use to reorganize my folders by year and month. 
i'd like to create a new catalogue on that drive and export from the current catalogue, import into it (to the proper mo/da/yy).
Will this work? if so, how do i do it? 
thanks.


----------



## PhilBurton (Apr 18, 2019)

domsueads said:


> i have over 300,000 photos in my catalogue. the catalogue (and backups) are on my internal HD on an IMAC.
> i have the photos spread over 6 or 7 hard drives, each in a folder by date.
> here's the problem-as i ran out of room on one external drives, i originally "squeezed" new files that i imported into any drive that had space on it...
> So, now i sit with some photos from 2017 spread over multiple hard drives, sometimes i have the same MO/DA/YY on different drives (not duplicate files but different photos from the same day on different drives).
> ...


I'm sure that other people will have better suggestions than mine, but here are my ideas,  just to get the ball rolling.

To clarify, you have only ONE catalog, but the photos in that catalog are spread out right now over 6-7 separate hard drives.  You now want to consolidate all those photos onto one new 8 TB hard drive.  

Here is how I would approach this problem.  The first thing to do is get all those photos onto the new HDD and have the catalog recognize the new drive location of all your files.  Then you can move files into the new folder structure of your choice.

1. I would start by assigning  names to each of your existing drives, as Drive1, Drive2, etc.  You don't need to actually change volume labels, just use "sticky notes" with these names.
2. On the 8 TB drive, create 7 top level folders.  Call them Drive1, Drive2, etc.  
3.  COPY the entire folder structure of Drive1 into the Drive1 folder on the new HDD.  (If you MOVE rather than copy, you risk losing files due to data corruption in the data transfer process.)
4.  Follow the helpful instructions that Victoria created  Lightroom thinks my photos are missing—how do I fix it? | The Lightroom Queen so that the catalog now recognizes the new location (on the new HDD) of all those photos.
5. Back up the catalog.
5. Repeat steps 3, 4, and 5 for all the remaining drives.

You now have all your files consolidated on the new HDD.  Set aside the old HDDs as backup, just in case you discover some corrupted files on the new HDD.

At this point, you should think about your new, preferred folder structure.  Some of us like the simple, date-based structure YYYY/MM/DD.  Some people like to add some sort of event or subject name to the DD, such as _DD-Phil's Birthday Party. _ That is entirely optional.  YYYY/MM/DD is preferable to DD/MM/YYYY because it's much easier to locate a folder for a given date, even if you live in the USA where we don't normally put year first in dates.  For example, I use /Photos as my top level folder and then put in YYYY/MM/DD folders underneath, so photos taken today would be in folder /Photos/2019/04/17.  (I use the forward slash in these examples, but if you are a Windows user, you will need to  use the backslash, \, to mark folder levels.

Now that you have all your files on one drive, create that new folder structure and move(drag and drop)  the files *within Lightroom *from the old Drive1/ folders to your new folder structure.  As long as you move photo files within Lightroom, the catalog will be kept up to date.

Don't move too many files at once, arbitrarily no more than 50 or 100.  _Also, be sure to back up the catalog frequently. _ If you are using a laptop, make sure that your AC or mains power adapter is plugged in.

Depending on how you have  named all your files, you might run into name conflicts, that is, 2 or 3 files with exactly the same date-based name.  Unlikely, but not impossible.  I'll leave it to others as to how you deal with that issue.

A few final points:

Be sure to back up both your Lightroom catalog AND your 8 TB drive.  
Get a second 8 TB drive for use as a backup.  Depending on how you like to approach this sort of backup,  you might find this utility very useful.  Scooter Software: Home of Beyond Compare.  I use this utility all the time.  Another useful utility for drive/folder synchronization is  Key Features of GoodSync.  Both of these utilities cost money, but they work well, save me lots of time and reduce the errors that would happen if I did backups manually.
Don't rely just on Lightroom for your catalog backups.  There have been too many reports of catalog corruption due to backups.  Back up the catalog in a Windows or MacOS folder operation using one of the utilities I just mentioned.
Hope this helps.  Keep us posted on your progress. This is a big "migration" and you can expect to need several days to complete this project.

Good luck.

Phil Burton


----------



## Rob_Cullen (Apr 18, 2019)

Phil's first advice should have been- DO NOT CREATE  A NEW CATALOG.!!
Next advice would be- DO NOT IMPORT ANYTHING.!!
Otherwise all other advice is 'spot-on' - Relinking the Folders of photos in their new location on the 8TB drive.


----------



## domsueads (Apr 18, 2019)

wow, thanks SOOOOO much for a logical, step-by-step approach, Phil...(my mind doesn't work that way, thus my mess!!). i've been slowly trying to move my files into my file structure (ie 2018 Photos and then within that parent folder i have separate folders by yy/mm/dd for each day). I thought it might be easier when i move forward. is there any downside to doing that BEFORE i move all to my 8tb? 
As of now, i don't think i have any more questions....just need to take my time and keep track of my steps, as i think this will take weeks (i have 389,000 photos).  on a side note, i have discovered a corrupt external drive (with photos from 3 months of travel in Newfoundland) . i had some photos backed up, but not all...very disheartening.   
I-See_Light: thank you for stepping up also....i will head your warnings and glad you posted, as i would have intuitively wanted to do both!!!!! 
keep your fingers crossed.
oh, just to let you all know--i'm also in the process of throwing all these photos up to the cloud (Backblaze) as another backup...this will probably take a year, based on my internet speed....oh what a tangled web i spun!


----------



## domsueads (Apr 18, 2019)

I have one more issue....my catalogue backup (through LR) takes well over a day to complete. i backed it up a few days ago and wanted to do a second backup last night to see if this one would be faster...NOPE....it's still moving very very slowly....any ideas?????  i am on an IMAC with almost 200gb free on my HD (where the catalogue and back ups reside)


----------



## PhilBurton (Apr 18, 2019)

domsueads said:


> I have one more issue....my catalogue backup (through LR) takes well over a day to complete. i backed it up a few days ago and wanted to do a second backup last night to see if this one would be faster...NOPE....it's still moving very very slowly....any ideas?????  i am on an IMAC with almost 200gb free on my HD (where the catalogue and back ups reside)


Try doing a backup outside of Lightroom, copying just the_ .lrcat_ file to a different drive.  How long does that take?  

Be aware that a  backup inside of Lightroom includes a catalog integrity test.  Also Lightroom compresses the .lrcat file into a ZIP file to save space, which also adds time.  I backup the catalog every time I close Lightroom and I also back up the catalog daily outside of Lightroom.

I also suggest that you write out your plan before you start on this process and that you keep notes of daily progress

Phil Burton


----------



## domsueads (Apr 18, 2019)

thanks again, phil. i will backup the catalogue by copying it to an external drive when this LR backup finishes. i was hoping that the integrity check and optimizing that took so long a few days ago would enable this next LR catalogue backup to go faster-oh well, i was wrong! i'm wondering if my catalogue is damaged somehow ?


----------



## Hal P Anderson (Apr 18, 2019)

Might as well add another thing:
NEVER BACK UP YOUR CATALOGUE TO THE VOLUME THAT THE CATALOGUE RESIDES ON. If that drive fails you'll have neither the catalogue nor its backup.


----------



## domsueads (Apr 19, 2019)

thanks, hap...great idea. I'll change that backup!


----------



## PhilBurton (Apr 19, 2019)

domsueads said:


> I have one more issue....my catalogue backup (through LR) takes well over a day to complete. i backed it up a few days ago and wanted to do a second backup last night to see if this one would be faster...NOPE....it's still moving very very slowly....any ideas?????  i am on an IMAC with almost 200gb free on my HD (where the catalogue and back ups reside)


Does your system have a spinning HDD?  if so, you can speed up all operations dramatically, including the catalog integrity test, by replacing that drive with an SSD.  IF you are comfortable doing a replacement SSD installation yourself, 2.5", Internal SSDs, SSDs, Components - Newegg.com is a good source.  You will need a 2.5" form factor drive,  not any of the other form factors.  Good brands include Samsung, WD, and Crucial.  DO NOT GET Mushkin.   Look for a brand that includes migration software.  

If you don't have a spare external drive case for the SSD for the migration operation, this sort of cable will do the job.  StarTech USB3S2SAT3CB USB 3.0 to 2.5" SATA III Hard Drive Adapter Cable w/ UASP – SATA to USB 3.0 Converter for SSD/HDD - Hard Drive Adapter Cable - Newegg.com.

If you are not comfortable doing this migration yourself, any competent repair shop can do the job.  Be clear on what SSD you will be getting.  

Phil Burton


----------



## LouieSherwin (Apr 19, 2019)

Just one more tip to make sure this whole process is completed successfully. You may already have thought of this but still seems worthwhile mentioning.

*Do not erase and/or reuse your original 6 or 7 photo storage drives before all your data is moved to the new 8TB drive AND your new drive is completely backed up to multiple locations. * 

If it were me I would hold them somewhere safe, ideally at a different location until your offsite (Backblaze) is compete. 

One poor sole was working on basically the same consolidation process that you are doing was erasing their source drives as they were moved to the new drive. But before they made any backup of the new drive it crashed and they lost their entire image collection. 

-louie


----------



## domsueads (Apr 19, 2019)

i just read an article a few weeks ago about why NOT to get an ssd drive...tbh, i didn't read it because my computer is 4 years old and i'm sure a new computer would solve most of my issues. although, i'm not ready to pull the plug yet ($$$$) but i'm considering it!  we travel in a motor home and as much as my husband would love his own imac (i'd give him this one!!!), we probably don't have the room to travel with both....ahhhh-decisions decisions. i think i'll stick with your original plan of action and when that's done, i may ask you guys about creating a new catalog (in case this one is corrupt) and migrating into that one....i am STILL backing up the catalog ( been about 36 hours now)....as soon as that's done, i'll start moving my files to my 8tb....this is a true lesson in patience


----------



## clee01l (Apr 19, 2019)

domsueads said:


> i'm wondering if my catalogue is damaged somehow ?


What is the size of your catalog file? How many image are cataloged?  Some times the catalog file will get lots of orphan records that can't be resolved in the relational database.  Optimising the catalog will rebuild indexes and compress any empty space between records.  But these orphan records will only get copied to the new file when the old file is backed up.  Sometimes the only recourse is to export all of the cataloged images into a new catalog using the "Export As a Catalog". NOTE:  I did not say Import. "Exporting as a new catalog" will copy only those records that relate to images in the catalog and leave orphans behind and producing a clean catalog file.


----------



## PhilBurton (Apr 19, 2019)

domsueads said:


> i just read an article a few weeks ago about why NOT to get an ssd drive...tbh, i didn't read it because my computer is 4 years old and i'm sure a new computer would solve most of my issues.


When SSDs were new, there were a lot of issues with them.  But they are much more reliable and long-lasting, and much more affordable.         Putting an SSD into your four year old system would be like a real performance enhancement, and not all that much money, ceertainly less than a new system.  Most laptops these days come with an SSD, by the way,


----------



## philromford (Apr 20, 2019)

If I understand your first post, you have one catalogue for all your folders. if this is the case, it can mean that loading the catalogue and accessing photos will be very slow. it would be better to do as I have done: this is to have multiple catalogues for different subject matter, or by date such as year and month.  I prefer the catalogues to be for subject matter types, this makes it very easy to find what I'm looking for and means fast access.

as was pointed out by Hal; never save backups to the same drive, I have a drive dedicated to backups only. I learned this the hard way soon after starting to use Lightroom a few years ago.

good luck.


----------



## domsueads (Apr 27, 2019)

ok, i think i'm in  a position to actually proceed. While copying files to my 8TB drive I realized several things .  My early photos from 2005-2010 are not all in LR, so although there are files on the hard drive, I will need to import them later on into LR. but, i don't think that has anything to do with this, right now..
I'd like to reiterate what I think I should be doing, before I do it...
1. copy files to 8TB (Done)
2. disconnect  the drive that was copied (so that LR thinks the files are missing)
3. with the 8tb drive connected, i will point LR to that drive to find missing photos. i am a bit confused as to what victora is saying here Lightroom thinks my photos are missing—how do I fix it? | The Lightroom Queen.   but i think i do have parent/child relations set up ok, so i will just search for the parent, right?
 4. when done, i expect my catalogue will look to my 8TB drive for those photos. 
5. but, here's my question.....
i want all my photos from a given year (for example 2015) to be on a single drive. right now, i have photos from 2015 spread over 3 external drives.
for instance, i have a parent label PHOTOS 2015 on drive 1. i have folders beneath that labeled 20150101, 20150102, etc....I have  another parent on drive 2 labeled  PICTURES 2015 with folders beneath labeled 20150101, 20150102, etc. 
how do i combine the two folders labeled 20150101 into one under one parent ????

ouch, my head hurts~~~


----------



## Rob_Cullen (Apr 27, 2019)

I have not been following your thread for a few days (C: SSD crashed in my desktop  ) I will try to help-

ok, i think i'm in  a position to actually proceed. While copying files to my 8TB drive I realized several things .  My early photos from 2005-2010 are not all in LR, so although there are files on the hard drive, I will need to import them later on into LR. but, i don't think that has anything to do with this, right now..
I'd like to reiterate what I think I should be doing, before I do it...

1. copy files to 8TB (Done)
In the same folder structure I hope!

2. disconnect  the drive that was copied (so that LR thinks the files are missing)
3. with the 8tb drive connected, i will point LR to that drive to find missing photos. i am a bit confused as to what victora is saying here Lightroom thinks my photos are missing—how do I fix it? | The Lightroom Queen.   but i think i do have parent/child relations set up ok, so i will just search for the parent, right?
You [Right-Click] the top-most folder that is missing [?] and select [Find Missing Folder]
Another link with instructions to Find Folders in Part#1-  Adobe Lightroom - Find moved or missing files and folders
Lightroom does not look to your drive automatically to find folders- you have to search the Finder dialog for the Folder you are trying to re-link.

4. when done, i expect my catalogue will look to my 8TB drive for those photos.
 Yes, 

5. but, here's my question.....
i want all my photos from a given year (for example 2015) to be on a single drive. right now, i have photos from 2015 spread over 3 external drives.
for instance, i have a parent label PHOTOS 2015 on drive 1. i have folders beneath that labeled 20150101, 20150102, etc....I have  another parent on drive 2 labeled  PICTURES 2015 with folders beneath labeled 20150101, 20150102, etc.
how do i combine the two folders labeled 20150101 into one under one parent ????
So copy both folders PHOTOS 2015 and PICTURES 2015 as separate folders on the new drive keeping the folder heirarchy then-
You have options-
You can keep the PHOTOS 2015 and PICTURES 2015 as separate folders on the new drive keeping the sub-folder heirarchy -  Or You could then follow this up after re-linking by using the Folder panel in Lightroom to 'move' sub-folders (with photos)  into one existing 'parent' folder if you desire (eg. PHOTOS 2015) by Drag&Drop. (Later after a successful re-link, delete the now empty PICTURES 2015 folder). How you do this will depend on any sub-folders contained in these parent folders. There is no reason to combine other than OCD to be tidy.


----------



## Califdan (Apr 27, 2019)

#1)  Yes - assuming you keep the same folder structure.

#2)  Yes, 

#3)  Once you disconnect the old drive, LR won't be able to find the images which it still thinks are on the old drive.  right click on the highest level folder on the old (disconnected) drive and select "Find Missing Folder".  Then in the navigation window that pops up, selecet the same folder that is now on the new drive and LR will change it's pointers to point to the new drive for all that folder and all it's subfolders and all the images it knows about in any of those folders.  If you don't have a common "parent" folder that contains ALL the folder images, you may have to repeat this for several high level folders.

#4)  Correct

#5)  Correct


----------



## domsueads (Apr 27, 2019)

thanks, everyone....keep your fingers crossed, i'm going for it!!!!!
i'll let everyone know what happens .....


----------

