# What a mess!!



## Kendig (Oct 20, 2012)

*I have really gotten myself into a bind with my photo and LR catalogue filing system. I shoot a lot and since beginning with digital in 2003, I have have just over 191,000 images. About 70 % of these are shot in RAW. 

Due to poor attention to filing details, these images are located in folders (mostly by year) on my PC hard drive and on a separate RAID system. The LR catalogues are all over the place in a number of different files some are lost (to me).

I have since read that all photos should be stored in one folder. I understand that I need to move/rearrange the folders while ‘in’ lightroom but I do not know where some of the catalogues are located. It is a convoluted mega mess. 

Any advice on where to begin fixing this mess? Is my ultimate goal to just have 1 catalogue?*


----------



## Tony Jay (Oct 20, 2012)

Wow!
I have to confess to a few palpitations reading this post.
Nonetheless welcome to the forum Kendig.

This issue is fixable but will take some preparation and effort to sort out.

Although the preferable end point will be to consolidate all your images in one catalog there is some work to be done before even attempting the process of consolidation.

The very first issue is to ensure that each of your images across all your catalogs have unique file names.
It is very likely that you have shot with multiple cameras sinced 2003.
Did you have a specific renaming system from the start?
If not my suggestion is to tackle each catalog in turn and rename images in batches appropriately.
I use a template incorporating date, custom text, and sequence(#0001). I use my template for illustrative purposes only the point being that each image ends up with a completely unique filename with no chance of duplication - ever.

The second point to raise is to ask how discerning your original selection process was.
Actually you have an opportunity here to review all your selects to determine whether all of them really deserve to be kept.

The next point is to rate your remaining selects - 1 to 5 stars. (more on this shortly)

You need to keyword you images if you have not done so already.
My suggestion is that you put together an hierarchical keyword collection that appropriately encompasses the subject matter of your photography.
So, if you are a fashion photographer don't bother including complex taxonomic keyword hierarchies of plants and animals.
Make sure that you use the same keyword collection in each catalog.

Now, with the original metadata, star ratings, and now keywords it will be possible for you to use smart collections to find images using very complex critieria if needed very simply within a single catalog.

Once you have done all these things then only start moving images in their folders from within lightroom to whichever hard drive will be appropriate.
You are right in thinking that it is preferable to ultimately place every folder in a single apex folder named to make it instantly identifiable to you.

Once you have done this then start to combine each catalog into a master catalog using export as catalog and then import as catalog.

Review a strategy for backing up and archiving your new master catalog. This is a big topic in itself so a detailed response here is inappropriate.

Review issues:

I strongly suggest that you do a bit-for-bit backup of each hard drive before starting on the process outlined above.

Thought and preparation is required for each step - take the time to identify any potential issues before initiation of any steps.
If you cannot resolve any of the detail issues then it is very likely that posting the question on this forum will lead to an answer in any detail appropriate to its solution.

This forum is very a VERY friendly and helpful site so no question should be regarded as to simple (stupid) to ask.

The suggestion that I have outlined will work but do require a fair bit of work.
Others, probably much more expert than I, will also no doubt offer certain alternative approaches or clarifications.

While the task you have set out to do is a big one, and a right royal pain in the arse, it is possible to achieve with a divide and conquer approach.

I look forward to hearing how you go - keep us informed.
Good luck.

Tony Jay


----------



## Replytoken (Oct 20, 2012)

Kendig said:


> *The LR catalogues are all over the place in a number of different files some are lost (to me).
> *



How many catalogues are we talking about?

--Ken


----------



## Kendig (Oct 20, 2012)

*[h=2]I feel like I just attended my first 12-step support group meeting, “Hello my name is Kendig and I have really screwed up my Lightroom digital workflow and filing system!”  I feel much better just putting it out there!  Thanks Tony for not making me feel even worse than I already do![/h][h=2]As I understand the recommended sequence of sorting this out: [/h]
1.  Backup all photos (or the complete drive?)
2.  Make sure that all file names are unique. I have some work to do here.
3. Star rating (already done, I did something right, yay!)
4. Keywords (I do very little key wording, even though I know it is important and will pay back big in the future  I am usually in a big hurry and skip it)
5. Rearrange folders
6. consolidate catalogues

They say you should never lie to your doctor,  lawyer or your accountant, I think it would follow that if you are going to get the best advice, adding the Lightroom Forum to this list would be appropriate!  

Here goes... even after years of using LR I had NO idea what a catalogue was or what it did. My workflow consisted of importing photos from an SD card through LR to a folder named for  the month (‘October 2012’, that lived in a folder called ‘2012 Images’).  Then I would review each image, assign a 1 star rating to the images I wanted to look closer at and edit. As editing progressed, if I had a really good image I would change star ratings to 2 or 3 stars (I’m too critical, I have yet to shoot a 5 star image!) and finally export images with the appropriate settings to folders on my desktop for use on the web or to be printed. Then, one day a few years ago something got accidently changed on the import settings and some photos were sent to the wrong folder; when I attempted to move them back (maybe outside LR?) I created a bigger mess. At some point, in an attempt to improve things, I think I clicked on something to do with the catalogue and the a bunch more of my images disappeared...... bla bla bla

So right now, I can visit my hard drives (internal and external raid) and know I still have my photos, but can only see images from 2012 on LR!  The catalogues for the earlier years 2003-2011 are someplace on my drive(s) but I do not know where. I assume I can search the drives for the .lrcat files and open them? Ken to answer your question, “How many catalogues?” I’m not sure. Because I did not know what a catalogue was, I have not been deliberately creating lots of them. I would guess that there are about a dozen catalogues in total? But that is just a guess. 

I have not messed with that yet as I wanted to get a sequence of attack before beginning to tackle this mess. 

Any other thought on the sequence of this process before I begin?

Thanks again guys, I really appreciate your time!

-K*


----------



## Brad Snyder (Oct 20, 2012)

"*So right now, I can visit my hard drives (internal and external raid) and know I still have my photos, but can only see images from 2012 on LR! The catalogues for the earlier years 2003-2011 are someplace on my drive(s) but I do not know where. I assume I can search the drives for the .lrcat files and open them?"

Yes. Keep in mind that's LRCAT. Folks commonly read that as an upper case 'i' instead of a lower case "L".  LightRoomCATalog. 

If you can find any of those that are MIA it will save you a lot of work going forward. There are ways to merge multiple catalog files that folks here can help with.*


----------



## Replytoken (Oct 20, 2012)

Kendig said:


> * “How many catalogues?” I’m not sure. Because I did not know what a catalogue was, I have not been deliberately creating lots of them. I would guess that there are about a dozen catalogues in total? But that is just a guess.
> 
> I have not messed with that yet as I wanted to get a sequence of attack before beginning to tackle this mess.
> *



While Tony Jay has given you a very detailed process, it is important to map this out on paper before proceeding ahead because the devil is in the details.  First, and foremost, you have the right attitude and approach so that is probably half of the battle.  I am going to suggest a few first steps, and I am sure that others will offer additional advice.  I am not a "my way or the highway" kind of poster, so do consider any additional suggestions.  If I were you, I would begin as follows:



Are your images backed up off-line?  If not, do you want to and/or can you back them up before moving ahead?
Search all drives for LRCAT files.  These are your catalogs, and you should see how many you can find.  Do back them up manually (by copying the files) before proceeding ahead, and know that you may have previously asked LR to automatically create backups of them as well.  That's OK.  Back all of them up.
Determine if you have synced your ratings and changes to the images that you have worked on.  If not, know that these "instructions" and metadata only reside within the LR catalog.  Data that has not been synced with the file does not follow the file as it is moved or copied.
Combine your catalogs into one catalog.  Life will be much easier for you if you are working with one catalog while you sort out this situation, and if you combine your catalogs successfully, all of your metadata that is associated with your files will follow into the main catalog.  I am going to let somebody else describe how to successfully and safely merge catalogs as it is not a part of my work routine.
Once you have one catalog, complete with all of your ratings, metadata, and image edits, back up this catalog and pat yourself on the back.  Hopefully, the second half of your clean-up will be somewhat easier, albeit somewhat longer.

Any questions?

--Ken


----------



## Tony Jay (Oct 21, 2012)

As an addendum:

If it is true that some of your catalogs really are lost then the only option will to be reimport those images from scratch using the 'copy and move' option in the import dialog.

Tony Jay


----------



## Kendig (Feb 17, 2013)

*Thanks folks for the great list of possible solutions. 

I have a little time to devote to fixing this problem today, so far I have:



made sure everything is backed up 
searched the 2 drives where all .lrcat files should be. 

The search found 104 files with .lrcat in them here are 3 screenshots with the results.
I apologize for the quality of the screenshots, I hope you can read them.

How would you recommend that I proceed with merging them all into 1 catalogue?

Thanks again for your thoughts.

-K

*


----------



## Replytoken (Feb 18, 2013)

And I thought I was behind in my work load? :hm:  While I tried to read your screenshots, the images were a bit small for me to read the fonts, and it looks like some of the columns were not fully expanded to show the entire path for each file.  Nonetheless, as it appears that you have a number of catalogs with the same name, You probably do not have to merge all 104 catalogs.  If a catalog is in a folder called Backups, it probably does not need to be merged if you can determine if the catalog it is backing up is functioning and has not been deleted.  If possible you may want to print out a list of these files (with their full path) so you can have a master list to work with as you move ahead.  There are a number of free utility programs that can also help you identify duplicate files.  Most do not move or alter the files unless you specifically ask it to do so.  This may help you get a handle on how many catalogs need to be merged.  Personally, I do not work with many catalogs, but I am sure others will be joining this thread and contributing good suggestions who have had to deal with similar situations.  You may want to see if one of our gurus has some good advice.

Hang in there,

--Ken


----------



## RachelRJ (Mar 9, 2013)

*What a mess - also.*


Determine if you have synced your ratings and changes to the images that you have worked on.  If not, know that these "instructions" and metadata only reside within the LR catalog.  Data that has not been synced with the file does not follow the file as it is moved or copied. 
Combine your catalogs into one catalog.  Life will be much easier for you if you are working with one catalog while you sort out this situation, and if you combine your catalogs successfully, all of your metadata that is associated with your files will follow into the main catalog.  I am going to let somebody else describe how to successfully and safely merge catalogs as it is not a part of my work routine. 
Once you have one catalog, complete with all of your ratings, metadata, and image edits, back up this catalog and pat yourself on the back.  Hopefully, the second half of your clean-up will be somewhat easier, albeit somewhat longer. 

Any questions?

--Ken[/QUOTE]

Hi Ken,

So I'm in a similar situation. My question is how do I go about synching my ratings and changes? Is this done within each catalog? before I move the catalog?


----------



## Replytoken (Mar 10, 2013)

RachelRJ said:


> Determine if you have synced your ratings and changes to the images that you have worked on.  If not, know that these "instructions" and metadata only reside within the LR catalog.  Data that has not been synced with the file does not follow the file as it is moved or copied.





Hi Ken,

So I'm in a similar situation. My question is how do I go about synching my ratings and changes? Is this done within each catalog? before I move the catalog?[/QUOTE]

Hi RachelRJ,

Welcome to the forum!  It was a beautiful day here in the PNW, and I hope you were able to enjoy the warm and sunny weather.

When you work on an image or add metadata to it, LR keeps that information in its catalog associated with that image.  Some folks have LR set up to automatically sync that data with the actual image file, if it is a jpeg, or with an associated XMP  sidecar file, if it is a raw file.  Most of us leave the auto-sync setting off because it can be quite a resource hog, depending on your workflow.  But, LR does give you the option to manually sync that information to the file or sidecar file.  When in the Library module, there is a command in the Metadata menu called "Save Metadata to File" (Ctrl+S).  If you invoke this command, LR will sync the data for the files selected.  This is a reasonably quick command for a single file, but can be quite involved if you are trying to sync the data for every file in your catalog in one command.  Also know that while most data gets sync'ed, there is some data that does not, and it only resides in the LR catalog.  This is why it is important to back up your catalogs.  If you are merging catalogs, the data stored in the catalog should carry forward when merged.  Report back with any additional questions.

Good luck,

--Ken


----------



## RachelRJ (Mar 10, 2013)

Ken,

Thanks for the welcome. Yes it was nice here in Portland. We even tried to see the comet PanSTARR, but clouds were on the horizon. 

Auto-synch? I cannot seem to find that setting in the Preferences area. I am still confused about this step. What is the purpose for this? (I mainly process RAW.)

Rachel


----------



## Replytoken (Mar 11, 2013)

RachelRJ said:


> Ken,
> 
> Thanks for the welcome. Yes it was nice here in Portland. We even tried to see the comet PanSTARR, but clouds were on the horizon.
> 
> ...



Hi Rachel,

As you are working with raw files, LR would write the metadata instruction to an XMP (sidecar) file for each image when you use the Save Metadata to File command in the Metadata menu (Ctrl+S).  This can be done manually, as I described above, so as to keep processor calls to a minimum, or it can be done automatically.  To set LR up to automatically write out metadata to XMP files, you need to go to the Edit menu in the Library module.  Choose Catalog Settings... and LR will bring up a three tab dialog box.  Click on the Metadata Tab (the last tab in LR 3.X).  The third setting down is "Automatically write changes into XMP".  If you check this setting, LR will write the metadata changes to the associated XMP file for each image file as you work on an image.  You may notice a decrease in performance because LR is now writing this information in real time to the XMP files, so do not be alarmed if your processor seems to be working a bit more than usual.  As I mentioned previously, some folks like to write their metadata to the XMP files in addition to having it reside in the LR catalog.  The XMP file makes that metadata portable so other programs that are capable of reading it can make use of it.  There are a number of other reasons to write out metadata to XMP files, but that is a little bit beyond the scope of this discussion.

--Ken

P.S.  I will probably be away from my computer the next few days, so hopefully somebody else will be able to assist you if you have additional questions.  The forum is filled with very knowledgeable folks, so you are in good hands.


----------



## RachelRJ (Mar 19, 2013)

Hi Ken,

I'm baaaack. 

I've been doing some work moving files around and keywording. I discovered, however, that I've got multiple references to the same image. That is to say I've got images which are not associated with the main catalog showing up in the catalog as "orphans" (they have the ?). I tried to have LR "find" the image to fix the problem, but when I tried to connect it, LR said the file was already associated with another image in the catalog. I did have a copy of the folder on another drive for a while, but then copied it over to the new drive. I also had a separate LR catalog for this particular folder. 

Should I skip the connecting part for now, and just make sure everything is where it should be before I set up the NEW catalog? I will be including the catalog that this folder is associated with when I make the new catalog. Thanks of your help!


----------



## Victoria Bampton (Mar 21, 2013)

Hi Rachel

Sorry for the delay replying.  Ken's dropped me a PM to ask me to look at this thread, as he's not around at the minute.  I just wanted to say hello and acknowledge your response... I just need a little time without a puppy biting my ankles to be able to read through the thread so far.  I will be back!


----------



## RachelRJ (Mar 22, 2013)

LOL - thanks!!!


----------



## Victoria Bampton (Mar 22, 2013)

Hi Rachel

Ok, puppy's in bed!  Since you've got multiple copies of the photos in the same catalog, we probably need to sort out which ones (i.e. which set of settings) to keep.  You'll probably find that one copy of the image has no settings and the other does.  Once we've got it down to one version of each image, then you can link up the missing files and you'll be straight from there.

Before we go any further, let's see a screenshot of your Folders panel so we can see whether whole drives are duplicated.


----------



## RachelRJ (Mar 23, 2013)

You might be terrified by what you see. not for the faint of heart! The folder in question is Erica & Isaac


Here is where the other copy of the images lives on the R: drive.


----------



## Victoria Bampton (Mar 24, 2013)

Great, now check the same photo on the D drive and the R drive in LR, and see which has the right settings.  If they both have exactly the same settings, go to Develop and check the history panel for both photos and see if one has a longer history than the other.

It's not a huge disaster... or at least, I've seen worse!  At least you still have the original images!


----------



## RachelRJ (Mar 25, 2013)

Victoria,

Thank you. I'll do it and get back to you.


----------



## Allan Olesen (Mar 25, 2013)

There is another possible variation of your multiple copies problem:
Is there a risk that you have the same photo registered in several catalogs?

I don't really know what will happen if you combine catalogs which have the same photo registered - but I fear it can create a lot of trouble if you are not ready to answer which version you want to keep when you start combining the catalogs.

If you don't know if you have more catalogs referring to the same photo, I would suggest these steps to investigate it:
1. Open each catalog and check that you have no "?" in the folder tree.
2. If you do have "?", find the files.
3. Make one new photo folder for each catalog.
4. Open each catalog and inside Lightroom move all the folders to the new folder belonging to that catalog.
5. Repeat step 1.

The whole purpose of this rather tedious exercise is to deliberately break references to photos stored in multiple catalogs since it is easier to find photos with broken references than it is to find photos belonging to multiple catalogs.

Of course, it is possible that Lightroom has built-in tools to handle this situation better than the way I suggest. But I am pretty sure that someone will chime in if that is the case.


----------



## RachelRJ (Mar 26, 2013)

Allan,

That is definitely a possibility. I integrated several older catalogs into this one. Which is why I want to start anew! How do I break those references?


----------



## Allan Olesen (Mar 26, 2013)

Step 4 will break the references.

But if you already combined the catalogs without seeing the problem I anticipated, there is hardly any reason to go back and do what I recommended.


----------



## RachelRJ (Mar 27, 2013)

I do have at least one folder with images referenced more than once. Did you see the jpgs I posted?


----------



## Allan Olesen (Mar 27, 2013)

No, I missed those last screenshots with tho photos referencing the same file. Just saw them now. I assume they are taken after merging two catalogs?

If yes, what happened? Did you have to choose immediately and blindly which one to keep? Or were you offered an option of looking at both of them to see which edits had been made? Or were you offered an option of duplicating the files and keep two photos in Lightroom?

I don't think my method would help you at this stage. The steps I described was only intended to investigate whether you have a problem with duplicate references. They will not help you in solving the problem.


----------



## Kendig (Oct 20, 2012)

*I have really gotten myself into a bind with my photo and LR catalogue filing system. I shoot a lot and since beginning with digital in 2003, I have have just over 191,000 images. About 70 % of these are shot in RAW. 

Due to poor attention to filing details, these images are located in folders (mostly by year) on my PC hard drive and on a separate RAID system. The LR catalogues are all over the place in a number of different files some are lost (to me).

I have since read that all photos should be stored in one folder. I understand that I need to move/rearrange the folders while ‘in’ lightroom but I do not know where some of the catalogues are located. It is a convoluted mega mess. 

Any advice on where to begin fixing this mess? Is my ultimate goal to just have 1 catalogue?*


----------



## RachelRJ (Mar 28, 2013)

Victoria,

Here's another question. I did this:



I've got several more. I loathe this system, but hey, when I did this, I didn't know! Two questions:

Would it be a good idea to gather these all together and then merge them with the new catalog? 
Could I access the images in them now, just using them as i need them, then merge them with the new catalog?

Thank you for your time. Lots of deep breaths here.


----------



## RachelRJ (Mar 28, 2013)

Thanks Allan. I think i merged one catalog from that standalone catalog, then merged the BIGGER catalog with a blended later catalog. 

So the moral of the story is...? 

What is the best way going forward? I do have a number of catalogs that are stand alone.


----------



## Allan Olesen (Mar 28, 2013)

RachelRJ said:


> Thanks Allan. I think i merged one catalog from that standalone catalog, then merged the BIGGER catalog with a blended later catalog.
> 
> So the moral of the story is...?
> 
> What is the best way going forward? I do have a number of catalogs that are stand alone.


I think that you can give the best answer to those two questions. It all depends on what happened when Lightroom discovered the double reference. Did Lightroom give you good options of sorting the problem, or did you have to take decisions blindly, without knowing if one or both of the two versions had already been edited.

Losing edits is the problem which I am afraid of. If two catalogs points to the same file, but you haven't done any edits to that file in any of the catalogs, there is no problem at all. You can can just let Lightroom do what it wants to do. But if you have made different edits to the same file in two catalogs, you have to decide which one you want to keep, or if you want to keep both of them. If Lightroom can't handle that - and I don't know if it can - your best option is to sort out the duplicate references before you merge the catalogs. But if Lightroom handles it nicely, I would just merge the catalogs.


----------



## Victoria Bampton (Mar 28, 2013)

Allan Olesen said:


> If Lightroom can't handle that - and I don't know if it can - your best option is to sort out the duplicate references before you merge the catalogs. But if Lightroom handles it nicely, I would just merge the catalogs.



When you import from catalogs, it gives an option to create virtual copies when there are different edits, so yes, that's certainly a possibility.

Since it sounds like there's some cleaning up to do anyway, you might as well merge it all in together and clean up everything at once.

Did you get any further with my earlier question?  



> check the same photo on the D drive and the R drive in LR, and see which  has the right settings.  If they both have exactly the same settings,  go to Develop and check the history panel for both photos and see if one  has a longer history than the other.


----------



## RachelRJ (Mar 28, 2013)

Victoria,

I looked at the two drives, and the D drive has the images with longer history. How should I proceed with this conflict?


----------



## Victoria Bampton (Mar 29, 2013)

But they both have different Develop settings?  If you're not sure, I think we'll export them out into 2 separate catalogs, relink the D drive with the current location on R, and then merge them back together using import from catalog so it merges them and creates VC's of the separate settings.


----------

