# Lightroom importing duplicates



## Dredog (Sep 26, 2010)

Anyone else have a problem importing duplicates even though I have "Don't Import Suspected Duplicates" check???

I usually have to them go to my Finder and delete all the "-2" files 

[img width=6'' height=424]http://img839.imageshack.us/img839/4116/screenshot2'1''926at123z.png[/img]


----------



## b_gossweiler (Sep 26, 2010)

Welcome to the forums, Dredog  

What your screenshot shows is a symptom that occurs when you import with the "Copy" action from a folder, with the very same folder set as the destination.

Which images do your Lightroom photos refer to? Do they carry the name with or without the "-2" suffix?

If you want to import from a folder without copying/moving the files, then you should select "Add" as the import action. This will just register the photos in LR, without moving or copying them.

Beat


----------



## Dredog (Sep 26, 2010)

You are correct, I am COPYING the pictures. I can't ADD the pictures as a reference because I'm importing from my memory card. 

The weird thing is that it doesn't happen with all my pictures, just some of them. I don't delete the pictures after they are imported. So after I take some more pics I only want to import the new pictures, but Lightroom gets confused whether some pictures are new or not. Not sure why. 

The "-2" are duplicates.


----------



## b_gossweiler (Sep 26, 2010)

OK, so it's not what I suspected. Since you're copying from a memory card (and I don't assume your screen shot is the directory of your memory card), it can't be a problem of copying within the same folder.

Another possible cause could be using "Make a Second Copy To" during import, pointing to the same directory where your originals go. Check your import dialog for that.

Beat


----------



## Dredog (Sep 27, 2010)

Nope. I don't that selected. I'm bit surprised that I'm the only one seeing this. 

It's weird because Lightroom is smart enough to rename the files with the "-2" suffix but fails to prevent these from imports.


----------



## b_gossweiler (Sep 27, 2010)

After the import, are you seeing both photos (the one without the "-2" suffix and the one with the "-2" suffix) in LR), and not just in the folder?

Beat


----------



## Jim Wilde (Sep 27, 2010)

Given that it's none of the above problems, there is another possibility. I just ran a quick test as follows:

1. Setup to use LR2.7....not sure what release the OP is using, profile says 1.' which I don't have access to. 
2. Imported 13 pictures from a CF memory card, using Copy To location on my hard drive, no second copy option, and no file renaming on import.
3. After import, I selected and deleted 5 of them (as if I was doing a post-import cull), but *crucially I did not choose to Remove from Disk, instead used the Remove option.* 
4. So at this point the hard drive folder still has all 13 files, though only 8 of them are still in the catalog.
5. Then repeated the import from the card (as if it was after some more pictures had been taken), by default all 13 pictures are selected though when I choose Import I get the warning message that 8 of them were not imported as they already exist. *However, the 5 that I had previously culled WERE imported, as expected.*
6. Because these 5 had NOT been deleted from the hard drive folder, and the OS won't allow duplicate file names in the same folder, these 5 are given the "-2" suffix as per the attached screenshot.

So, is any of this possible? Do you routinely not "Remove from Disk" when you cull within Lightroom? What is the situation in your catalog regarding these "pairs"....are both files in the LR catalog, or only one? If the latter, are they the "-2" versions?

Of course if you have both versions still in the catalog, this theory goes bust and we have to think again! 

BTW, this theory also holds for LR3, except that the 8 pictures already in the catalog would be 'greyed out' and not selectable in the Import dialog....the 5 previously imported and Removed WOULD be selected for import again.


----------



## b_gossweiler (Sep 27, 2010)

[quote author=TNG link=topic=11161.msg75'87#msg75'87 date=1285578'19]
...
3. After import, I selected and deleted 5 of them (as if I was doing a post-import cull), but *crucially I did not choose to Remove from Disk, instead used the Remove option.* 
...
[/quote]

In which case they really are no duplicates, because the definition of a duplicate is a photo already being in your catalog.

I guess we need to know which photos are contained within the catalog after the suspected duplicate import, the ones with the original filenamen, the ones with the "-2" suffixed filename or both.

Beat


----------



## Dredog (Oct 1, 2010)

Thanks for your thorough response. Think u may be on to something. I've been removing a lot of pictures from both the folder or catalog since my machine crashed. My catalog is bit screwed up and I have a lot of duplicate files. To be honest I'm not sure if the dupes show up in the catalog but the -2 suffix caught my attention in the folders. I think the best thing might be to remove evrything and start over. 

Will I lose all my edits and adjustments to my pics????


Thanks for ur help!!! 

[quote author=TNG link=topic=11161.msg75'87#msg75'87 date=1285578'19]
Given that it's none of the above problems, there is another possibility. I just ran a quick test as follows:

1. Setup to use LR2.7....not sure what release the OP is using, profile says 1.' which I don't have access to. 
2. Imported 13 pictures from a CF memory card, using Copy To location on my hard drive, no second copy option, and no file renaming on import.
3. After import, I selected and deleted 5 of them (as if I was doing a post-import cull), but *crucially I did not choose to Remove from Disk, instead used the Remove option.* 
4. So at this point the hard drive folder still has all 13 files, though only 8 of them are still in the catalog.
5. Then repeated the import from the card (as if it was after some more pictures had been taken), by default all 13 pictures are selected though when I choose Import I get the warning message that 8 of them were not imported as they already exist. *However, the 5 that I had previously culled WERE imported, as expected.*
6. Because these 5 had NOT been deleted from the hard drive folder, and the OS won't allow duplicate file names in the same folder, these 5 are given the "-2" suffix as per the attached screenshot.

So, is any of this possible? Do you routinely not "Remove from Disk" when you cull within Lightroom? What is the situation in your catalog regarding these "pairs"....are both files in the LR catalog, or only one? If the latter, are they the "-2" versions?

Of course if you have both versions still in the catalog, this theory goes bust and we have to think again! 

BTW, this theory also holds for LR3, except that the 8 pictures already in the catalog would be 'greyed out' and not selectable in the Import dialog....the 5 previously imported and Removed WOULD be selected for import again.
[/quote]


----------



## Jim Wilde (Oct 1, 2010)

Dredog,
I'm a little confused, one post you said both originals and the -2 versions are in the catalog, but in the next post you said you're not sure. It's going to be difficult to give you advice unless we know exactly what is in your catalog, so if you could clarify that point it may help.

As for starting over, if you really mean with a new clean and empty catalog, yes you would lose your edits and adjustments. So may be this should be a last resort, if we could get a better feel for the extent of the problem we may be able to help you recover. For a start, how many pictures you do have in the catalog? And roughly how many of these have this 'duplicate' issue?


----------



## Dredog (Oct 1, 2010)

Sorry for the confusion. It was late when I was responding and I couldn't think straight. 

I confirmed that dupes show up in both the folder and catalog. A lot of them are from my iPhone since I regularly sync it w lightroom. I even have triplicates of some photos (-3). 

Just to confirm once more, I do see duplicates or even triplicates of the same photo in lightroom 3. 

Thanks for ur help


----------



## Jim Wilde (Oct 2, 2010)

OK, thanks for the clarification.....though that means that my theory of how the duplicates are being created and getting into your catalog is a bust (in my theory only ONE copy of the pictures can exist in the catalog).

So we have to look elsewhere, and maybe your iPhone is a good place to start. When you say "a lot of them are from my iPhone" to do acually mean ALL of them, or are there still duplicates from other camera sources?

I also don't understand what you mean by "regularly sync with Lightroom"....could you expand on this a little? I have an iPhone, which only syncs with iTunes....if I was to use the iPhone camera I would treat it as any other picture source and import directly into Lightroom using the Copy To function. But that is NOT syncing in my world, so what do you mean exactly?


----------



## Dredog (Oct 3, 2010)

By regular sync, I mean that I regularly copy pictures from my iPhone to lightroom. I don't use any other software to import my pictures from iPhone. So every week or so when I connect my iPhone I use the import command (copy to) to copy pictures to my library. 

Since I take much more pictures with my iPhone and regularly use the copy to function, it's hard to say if the iPhone is the only one w duplicates since its used much more than any other camera source. However, I have in fact seen duplicates from other sources although not as frequent due to lower activity.


----------



## Jim Wilde (Oct 4, 2010)

Thanks for the info, though I have to say I am a little bit stumped by this.....if you definitely have 'Do Not Import Suspected Duplicates' checked when you import, I can't see how you are getting second (and third) copies of the same pictures in your catalog. Can you post a screenshot of your import screen the next time you do an import so that I can see the various settings? Might give me a clue....


----------



## B64Photo (Mar 27, 2011)

Experienced the same problem when computer crashed and I had to reinstall all bases (new system altogether, really), an did an upgrade at the same time. My problem is due to that the time on the original version and the file with "-2" markings are one hour different, for some reason - which makes sense as Lightroom checks filename and date/time stamps when importing.


----------



## CatchAFire (Apr 19, 2011)

I'm getting the same issue. Photos get imported twice and LR adds -2 to the file name. So frustrated. I've been using LR since version 1. This is something that started happening very recently.

BN64Photo, did you ever figure out the problem for yourself?

I'm using "Copy as DNG" to import the photos from the card to my harddrive. My card reader recently failed, so I'm plugging my camera directly in to my Mac with a USB cable.

"Don't Import Suspected Duplicates" is checked. "Make a Second Copy To:" is not checked.

I'm renaming files with a template as I've always done. After the duplicate is imported, I've compared the files and all the metadata is the same.

I just Optimized the catalog and that didn't help.

I'm just starting to troubleshoot. I'll report any findings I make.

Thanks for any thoughts.


----------



## Victoria Bampton (Apr 23, 2011)

Hi CatchAFire and B64Photo, welcome to the forum!  Sorry we missed the thread before.

Are you both updated to 3.3 or 3.4RC?  And does it only happen if you convert to DNG or on all imports?

B64 - which timezone are you in?  Your date stamps observation interests me.


----------



## CatchAFire (Apr 25, 2011)

Hi Victoria,

Yes, I'm using version 3.3. I'm trying to replicate the issue right now, but it seems to be working. My duplicates are grayed out. It's nice to see that!

It was happening on Copy as DNG. Didn't try it with just "Copy."

At the moment, I can't try importing the photos that weren't working, but as soon as I can, I will. If I have trouble again, I'll come back and mention it. And if I figure out why it was happening, I'll post here as well.

Thanks for your help.


----------



## artsupreme (Apr 26, 2011)

I have the same problem in 3.3  I've been consolidating several external HD's worth of photos into one large catalog the last couple weeks and my LR is importing duplicates for some reason.  My box is checked "do not import suspected duplicates" and I'm using the "move photos to a new location and add to catalog"...basically, I'm trying to get all my photos moved from several externals onto 1 external and I know for sure it's been importing duplicates.

*Is there a program out there that does a good job at removing duplicates from LR??*  The file names might not be the same now considering LR adds a -2 at the end of the file.


----------



## clee01l (Apr 26, 2011)

artsupreme said:


> I have the same problem in 3.3  I've been consolidating several external HD's worth of photos into one large catalog ....
> 
> *Is there a program out there that does a good job at removing duplicates from LR??*  The file names might not be the same now considering LR adds a -2 at the end of the file.


There is a Plugin that you can purchase for LR.  http://www.lightroom-plugins.com/DupesIndex.php it is not perfect, but  it will discover suspects and let you make the final cull


----------



## artsupreme (Apr 26, 2011)

clee01l said:


> There is a Plugin that you can purchase for LR.  http://www.lightroom-plugins.com/DupesIndex.php it is not perfect, but  it will discover suspects and let you make the final cull


 

When you say it is not perfect, what do you mean by that?  Does that mean it will not necessarily identify all my duplicates or does that mean it's hard to use??


----------



## clee01l (Apr 26, 2011)

artsupreme said:


> When you say it is not perfect, what do you mean by that?  Does that mean it will not necessarily identify all my duplicates or does that mean it's hard to use??


It is very easy to use.  Sometimes it will find two (or more) photos taken is a close time interval and call these duplicate when in face they are visually clearly different subjects. 

My Suggestion: D/L the plugin and run it for free.  It will stop after the first 10 duplicates found, but should give you enough of an idea that you will purchase a license if you really want to clean up the whole catalog.


----------



## artsupreme (Apr 26, 2011)

okay, is that my only option for removing my duplicates or perhaps there's another program that runs on MACs that could identify/remove my duplicates.  I have approx 200K images in my catalog now and it's growing.   I would bet there are 15-20K duplicate images in there.

Thanks for your advice.


----------



## b_gossweiler (Apr 26, 2011)

I just noticed recently that if you change the capture date/time in LR on a photo, this photo is no longer detected as duplicate during import (since the date/time now differs between LR and external).

Could this be the source of your duplicates?

Beat


----------



## clee01l (Apr 26, 2011)

There are many shareware/third party applications that will find duplicates on your HD. Most work with Windows since that is where the product demand lies. None of these work inside of LR.  You really need to work inside of LR or you will be creating havoc by deleting these from outside of LR and having LR display the {?} Missing message and then having to go into LR clean up mode.

By running the Dup Finder plug-in you will ID all of the potential Dups with a "Duplicat ID field Created by the plug-in.  If you have 200K images, then it will take a bit of time but probably no longer than if you use some other program to search your HD outside of LR.


----------



## Dredog (Sep 26, 2010)

Anyone else have a problem importing duplicates even though I have "Don't Import Suspected Duplicates" check???

I usually have to them go to my Finder and delete all the "-2" files 

[img width=6'' height=424]http://img839.imageshack.us/img839/4116/screenshot2'1''926at123z.png[/img]


----------



## Victoria Bampton (Apr 26, 2011)

I'd second the vote for the Dup Finder plug-in - I've used it on my own catalog, and it did a nice job of marking them up, and then I was able to flick through at my leisure and sort them in attainable chunks.


----------



## clee01l (Apr 26, 2011)

b_gossweiler said:


> I just noticed recently that if you change the capture date/time in LR on a photo, this photo is no longer detected as duplicate during import (since the date/time now differs between LR and external).
> 
> Could this be the source of your duplicates?
> 
> Beat


More often it is when the user chooses to store images in a theme named folder option instead of using one of the default date named folder options.  The OS will not permit two files to have the same name and reside in the same folder. I am presently working with someone that has (by the plugin's count) about 1/3 of her LR catalog as duplication.  Most of this could have been prevented if a Data named folder scheme were consistently used


----------



## b_gossweiler (Apr 26, 2011)

I don't understand this statement, Cletus 

As far as I understand, LR will not allow an import of a "real" duplicate, no matter where it comes from nor where it's going to be stored in. If it's in the catalog, the import detects duplicates. And if there is an file name conflict during import of a non-duplicate with an existing name, LR will append "-2" to the file name upon import automatically.

Beat


----------



## artsupreme (Apr 26, 2011)

clee01l said:


> More often it is when the user chooses to store images in a theme named folder option instead of using one of the default date named folder options.  The OS will not permit two files to have the same name and reside in the same folder. I am presently working with someone that has (by the plugin's count) about 1/3 of her LR catalog as duplication.  Most of this could have been prevented if a Data named folder scheme were consistently used


 

Well, this could be my issue too I guess.  My folder structure has several parent folders and then subfolders named 2009, 2010, etc and then the dated folders inside those...so there are several 2009, 2010 folders in there all under different parent folders and probably multiple duplicate folders (i.e "05-03-09") inside.  

Once I get this duplicate thing worked out would there be a way to start a new catalog and "import photos from catalog" (current catalog) and have LR move all my photos to the new location and create a new folder structure so I don't have multiple duplicate folders inside the parent folders?


----------



## b_gossweiler (Apr 26, 2011)

artsupreme said:


> ... Once I get this duplicate thing worked out would there be a way to start a new catalog and "import photos from catalog" (current catalog) and have LR move all my photos to the new location and create a new folder structure so I don't have multiple duplicate folders inside the parent folders?



I'm affraid this is not possible, as the target folder options during "Import from Catalog" are very limited, if not non-existing.

One option (which I would _*not*_ recommend) could be writing metadata to files and then re-import, but there will be loss of information (collecitons, stacks, VC's, develop-history, flags, ...) by doing so.

Beat


----------



## clee01l (Apr 26, 2011)

artsupreme said:


> Once I get this duplicate thing worked out would there be a way to start a new catalog and "import photos from catalog" (current catalog) and have LR move all my photos to the new location and create a new folder structure so I don't have multiple duplicate folders inside the parent folders?


  When I started LR I had an existing folder structure in place that did not precisely follow a date named theme. I lived with that for a while and imported new images using one of the OEM suggested folder date named themes.   Being an 'anal retentive' kind of guy, it bugged me that everything was not ordered in the same date named order. In my spare time I used the filter bar to filter by 'Date' and would drag  images into their proper date order.  In helping my friend untangle her LR catalog, I took her images in theme named folders and filtered by date captured.  Sometimes LR would complain and leave images in the theme named folder because there were already images with that file name  in the date named folder. A lightbulb moment! These are duplicates.  With only Duplicates remaining in the theme named folders, I only need to determine which have LR development adjustments and which do not.  If I find two "master Original" copies with adjustments, then an executive decision needs to be made to determine which one gets kept and which gets deleted.


----------



## artsupreme (Apr 26, 2011)

clee01l said:


> When I started LR I had an existing folder structure in place that did not precisely follow a date named theme. I lived with that for a while and imported new images using one of the OEM suggested folder date named themes.   Being an 'anal retentive' kind of guy, it bugged me that everything was not ordered in the same date named order. In my spare time I used the filter bar to filter by 'Date' and would drag  images into their proper date order.  In helping my friend untangle her LR catalog, I took her images in theme named folders and filtered by date captured.  Sometimes LR would complain and leave images in the theme named folder because there were already images with that file name  in the date named folder. A lightbulb moment! These are duplicates.  With only Duplicates remaining in the theme named folders, I only need to determine which have LR development adjustments and which do not.  If I find two "master Original" copies with adjustments, then an executive decision needs to be made to determine which one gets kept and which gets deleted.


 
Cletus, thank you!  This is a very good suggestion and I will probably hit you up again when I'm ready to tackle this if I can't figure it out.  In a nutshell, is this what you are suggesting:  I figure out how to filter by date, let's say it's 5/3/09.  When I filter it turns out I might have 2 or 3 folders titled "05-03-09" within different parent folders.  I pick one folder I'm going to use and drag my images from the two other folders into the one I decide to use as a master folder for "05-03-09".  If they don't move, then they are likely duplicates and at that point I just decide which duplicates I want to delete....am I on the right track here?

Thanks again.


----------



## clee01l (Apr 26, 2011)

You are on the right track.  The idea is to move all the folders under a single branching parent. Folders that have the same name will not move, but the image contained within one can be moved into the other.

The idea is that you have one one folder for each unique date.  I prefer to segregate the folders by year under a common parent:
CommonParent
2009
2009-05-03
2009-05-04​2010
2010-06-30
...​
2011...​
By selecting the "All Photographs"  Collection and using a Metadata filter, every photo in your catalog can be sorted and filtered by date.  When filtered by a unique data, you can then drag all the images from that unique date into the child folder for that unique date.  Only one of te images that share a common file name will move to the designated folder.


----------



## artsupreme (Apr 26, 2011)

Sounds good Cletus I really appreciate your help.  I'll play with it after I've imported all my photos and have everything backed up.


----------



## tscheinok (May 14, 2011)

*Timezone causing import suspected duplicates to fail*

After a day of trying to figure out why LR was willing to import duplicates (same filename, exact size and exim capture date) I finally realized that the original import into LR was done overseas while I was traveling. Now that I back home I changed my timezone.  By setting my timezone back to what it was while traveling LR is finding the duplicates for me.

This seems like a bug because the exim date is absolute, not adjusted +- GMT.


----------



## Mark Sirota (May 14, 2011)

Good catch, and I agree that it sounds like a bug.  You should report that on the official bug report forum -- there's a link in the grey bar at the top of the page.


----------

