# Lightroom Not recognising duplicates on import



## happygun (Aug 30, 2016)

Evening all

I exported my OS X photos library to a folder yesterday, and imported to lightroom. On import i renamed the files ( (YYYYMMDD) - Hour - Minute - Second - Make - Model - original filename).

Today i needed to check which folders i imported yesterday, and pointed lightroom to the above folder, expecting lightroom to confirm that there were no new images, but lightroom recognised some as imported, but not all. I have confirmed that some of the images that Lightroom does not recognise as duplicate are imported.

Why is Lightroom not recognising these files as duplicates?

Screenshots

Currently in library
https://www.evernote.com/l/AAIppnOyCatB4L_O-4wXNMo5FtkXUOcOF7I

Import dialogue showing duplicates not recognised
https://www.evernote.com/l/AAInwWlzh2ZAmaxlZ0nTTAyKO2QPxsmOs0g


----------



## Johan Elzenga (Aug 30, 2016)

Your screenshots aren't showing. I suspect to posted a link to your local disk.


----------



## happygun (Aug 30, 2016)

Edited original post, hopefully the links should now work.


----------



## Johan Elzenga (Aug 31, 2016)

I understand that some of the images are indeed imported as duplicates now? If so, show them side by side in the Grid (put both in the Quick Collection for example) and then check the file names. Look for differences in the part that was generated based on date and time, to see if you can find any differences. That would give you a clue. Otherwise, I don't see anything out of the ordinary.


----------



## happygun (Aug 31, 2016)

Thanks for getting back. I have now imported one image that i recognise as duplicate, and Lightroom didn't.

Lightroom imported the image as the same filename, but with -2 appended. The metadata is identical (apart from an update to the metadata copyright information that i have added as a preset).

In my understanding, this shouldn’t happen, at least from what i wish to see happen...

Screenshots of original image, and the duplicate i imported:-


----------



## Johan Elzenga (Aug 31, 2016)

I'm out of ideas. :(


----------



## Victoria Bampton (Sep 2, 2016)

There are a few things that can cause the duplicate detection to fail. Lightroom's looking for a match on the filename (as it was at the time of import) plus the capture time plus the file size. The filename is where it usually falls down, either because the duplicate photo's name doesn't match the existing photo's import filename, or when the original filename has been lost through Import from Another Catalog or suchlike. Those are the most likely suspects.


----------



## happygun (Sep 2, 2016)

Victoria Bampton said:


> There are a few things that can cause the duplicate detection to fail. Lightroom's looking for a match on the filename (as it was at the time of import) plus the capture time plus the file size. The filename is where it usually falls down, either because the duplicate photo's name doesn't match the existing photo's import filename, or when the original filename has been lost through Import from Another Catalog or suchlike. Those are the most likely suspects.



Hi Victoria, the files are renamed by Lightroom on import, but this has never previously caused any issues. The original files have not been changed in any way.

I created a test catalog, and tested importing a handful of files. I renamed the files on import, renamed the original files in Finder, and lightroom correctly recognised these as duplicates despite the name changes.


----------



## rob211 (Sep 2, 2016)

Lr is VERY conservative in this regard in my experience. If I add metadata and then do a duplicate in the Finder on a Mac and then try to import that duplicate, for example, Lr often won't realize it's a dupe. I simply don't rely on Lr's algorithms to do this; I use either a plugin (Lightroom Plugins - Duplicate Finder for Lightroom) or another application like PhotoSweeper or Snapselect to do it. Or I simply look at the destination list and uncheck the ones that I know would be duplicates (as when I am certain I've imported all of the 2016-09-02 images). If I'm unsure I let Lr import and do a visual, but even that isn't reliable, since the image content might be a duplicate, but perhaps the metadata (esp a time adjustment, or GPS data) might NOT be the same. Or maybe I adjusted it on the road and forgot.

It just isn't a process that can be left to something like a checkbox I'm afraid.


----------



## Gnits (Sep 2, 2016)

rob211 said:


> I simply don't rely on Lr's algorithms to do this;



I may have asked before .... have Adobe published the criteria which determines a duplicate ?


----------



## happygun (Sep 2, 2016)

I was on the phone with Adobe for >90mins last night, and the documentation they referred to stated that any file renaming (either on import, or renaming the original file) would not allow identification of duplicates. They ran through a couple of scenarios and found that that was not the case, so the ticket has been escalated....


----------



## rob211 (Sep 2, 2016)

happygun said:


> I was on the phone with Adobe for >90mins last night, and the documentation they referred to stated that any file renaming (either on import, or renaming the original file) would not allow identification of duplicates. They ran through a couple of scenarios and found that that was not the case, so the ticket has been escalated....


Meaning that it did NOT catch an image as a dupe even if it had NOT been renamed?


----------



## happygun (Sep 3, 2016)

rob211 said:


> Meaning that it did NOT catch an image as a dupe even if it had NOT been renamed?



No, it was that files renamed (with both original and imported files renamed) _were_ identified as duplicates in our tests, contrary to their documentation.

However, the issue that caused the call was some files that were renamed on import (all my files always have) was not identified as duplicates when i pointed the import dialogue to the original folder with files not changed at all.


----------



## Stephanie Booth (Sep 19, 2018)

I know this is an old thread but I'm having a similar problem and was wondering if any news on the issue had turned up since 2016. 

I have photos that were in Lightroom, and where then imported into Apple Photos, and exported out of Apple Photos. No changes have been made except keywords. Now when I try to import them into Lightroom they are not recognised as duplicates.


----------



## Johan Elzenga (Sep 19, 2018)

Stephanie Booth said:


> I know this is an old thread but I'm having a similar problem and was wondering if any news on the issue had turned up since 2016.
> 
> I have photos that were in Lightroom, and where then imported into Apple Photos, and exported out of Apple Photos. No changes have been made except keywords. Now when I try to import them into Lightroom they are not recognised as duplicates.


So why do you try to import them in the first place if you know they are duplicates that should not be imported?


----------



## Stephanie Booth (Sep 19, 2018)

JohanElzenga said:


> So why do you try to import them in the first place if you know they are duplicates that should not be imported?



Well, as I mention in the excerpt you quoted, they have different keywords, that I want to preserve.


----------



## Johan Elzenga (Sep 19, 2018)

Stephanie Booth said:


> Well, as I mention in the excerpt you quoted, they have different keywords, that I want to preserve.


So how would that work? When you try to import duplicate images, you only have two options. Either you *do* import them, but that will be as duplicates. Or you *don’t* import them, but then you won’t import the keywords either. You can’t *not* import the images, but still merge the keywords somehow.


----------



## Stephanie Booth (Sep 19, 2018)

You can update metadata for duplicate images.


----------



## Johan Elzenga (Sep 19, 2018)

Stephanie Booth said:


> You can update metadata for duplicate images.


Yes, but that means you first have to import them, so you *do* want duplicates in that case. I’m beginning to think that you don’t understand how this feature works. Lightroom can detect duplicate images when you try to import them, but that means they will *not* be imported. There is no option where they are imported and then flagged as duplicates or something like that.


----------



## Stephanie Booth (Sep 19, 2018)

JohanElzenga said:


> Yes, but that means you first have to import them.


Yes... import from catalog.  The idea is to use the import from catalog feature to update metadata on existing photos in the catalog, taking advantage of how lightroom allows you to update metadata for duplicate photos when you try to import them from a catalog.

Which in theory works fine, and I've tested it out with trial catalogues, only with my "proper" catalogs lightroom is not recognizing my duplicate photos as duplicates, and is therefore wanting to import them into the catalog as new photos, not giving me the option to update the metadata for them.


----------



## Johan Elzenga (Sep 19, 2018)

OK, now you finally make some sense. You don’t want to import the _images_, you want to import another catalog that contains the images. If that does work with trial catalogs and not with your main catalog, then you could try the following. Make a copy of your main catalog. Start Lightroom with that copy and remove all images from it. Then import those duplicate images into this copy. Now switch to your main catalog again and import this copy. Maybe this time the duplicates are detected.


----------



## Stephanie Booth (Sep 19, 2018)

JohanElzenga said:


> If that does work with trial catalogs and not with your main catalog, then you could try the following. Make a copy of your main catalog. Start Lightroom with that copy and remove all images from it. Then import those duplicate images into this copy. Now switch to your main catalog again and import this copy. Maybe this time the duplicates are detected.



Thank you for the idea, I will try this.



JohanElzenga said:


> OK, now you finally make some sense.



Have I done anything to annoy or offend you? From the beginning of your answers to me, including on my other thread, they come across as condescending and unpleasant. I am not stupid. I do not know lightroom as well as you, certainly, but you seem to assume that I have not taken the trouble to think. I have. My language may not be as precise as it should have been because this is not my specialty area, but that does not mean that the reasoning behind it "makes no sense". 

So, I appreciate the helpful information, but do want to draw your attention to the way your interactions with me are coming across, in case it was not intentional or you were not aware of it.


----------



## Johan Elzenga (Sep 19, 2018)

Sorry, if my reaction came across as unpleasant. That was not the intention.


----------



## Stephanie Booth (Sep 19, 2018)

Apology appreciated, glad it wasn't intentional!

So, I tried what you suggested, and unfortunately I still get the same result, meaning that when I look to import the secondary catalog into the master catalog, only a tiny portion of the photos are recognized as duplicates. 

To try and understand a bit better what is going on, I took a subset of those photos (2 out of the 10 or so years) and put them through photosweeper to see how many duplicates it recognized. 

And this is where I think I figured out the problem: for some reason, many (most!) of my photos in my master catalog have filenames that start with a space or two. Those that have taken the trip through Apple Photos and out again have lost their leading spaces. So although they seemed to me to have identical filenames, they don't.

Now, I ran another test: I exported some photos out of a catalog "as catalog", "with negative files". In the secondary catalog, I renamed the files. I then reimported that secondary catalog into the primary one, and it recognized the renamed files as duplicates, despite them not having the same name anymore.

So I'm suspecting that this non recognition of duplicates has to do with the leading spaces in the filenames, but on the other hand the test I can seems to indicate that different filenames do not prevent LR from identifying photos as duplicates when importing a catalog back into another. So I'm not too sure what to think.


----------



## Johan Elzenga (Sep 19, 2018)

Different file names alone should not cause Lightroom to ignore duplicates. Lightroom looks at a number of things, like filename, capture date, file type, and I assume other metadata like camera model and lens. What seems more likely is that Apple Photos changed some of those metadata, especially some dates.


----------



## happygun (Aug 30, 2016)

Evening all

I exported my OS X photos library to a folder yesterday, and imported to lightroom. On import i renamed the files ( (YYYYMMDD) - Hour - Minute - Second - Make - Model - original filename).

Today i needed to check which folders i imported yesterday, and pointed lightroom to the above folder, expecting lightroom to confirm that there were no new images, but lightroom recognised some as imported, but not all. I have confirmed that some of the images that Lightroom does not recognise as duplicate are imported.

Why is Lightroom not recognising these files as duplicates?

Screenshots

Currently in library
https://www.evernote.com/l/AAIppnOyCatB4L_O-4wXNMo5FtkXUOcOF7I

Import dialogue showing duplicates not recognised
https://www.evernote.com/l/AAInwWlzh2ZAmaxlZ0nTTAyKO2QPxsmOs0g


----------



## Stephanie Booth (Sep 20, 2018)

JohanElzenga said:


> What seems more likely is that Apple Photos changed some of those metadata, especially some dates.



I examined a subset of photos where some seemed recognized by LR as duplicates, and some not, from the same shoot. Taking them through Photosweeper I thought I had identified the problem: all the photos that were recognised as duplicates when importing from catalog had no keywords set in the destination catalog. That is the only visible difference I was able to find, aside from the file name. And indeed, some of the photos recognized as duplicates did have different filenames, so that does not seem to be the issue.

I therefore tried removing the keywords from those photos in the destination album to test my hypothesis, but their "duplicates" are still not recognized as such. I have looked at all the metadata I can see in LR and Photosweeper and can see no difference. I have photos side-by-side in the folder, from the same shoot, which seem identical in every way, except for that keyword issue, and is recognized as duplicate when importing catalog, and not the other.

Is there a tool that might help me see more metadata than what lightroom and photosweeper shows me, maybe, to surface an as-yet-invisible difference between these photos?

EDIT: actually, that preexisting keyword indicated that the photo in question was in LR before a *previous* import I seem to have done from Apple Photos. So, that might explain things: the photos that "resist" being identified as already in the destination catalog would be those that were in there longer, before that initial import. Remains that clearly there is something different between the Apple Photos versions and the ones preexisting in LR and that I can't manage to pinpoint what it is (or how to fix it).


----------



## Jim Wilde (Sep 20, 2018)

Just for the record, Lightroom's duplicates detection is based solely on three things: the filename, the capture date, and the file length (i.e. its size in bytes). When an image is first imported, an "ImportHash" record containing this data is stored in the catalog database, against which subsequent imports are checked. If there's a match, it's a duplicate. So looking for differences in metadata isn't going to find the cause of the issue. 

What's unclear to me is the workflow that you have followed here. I understand that you have images in a catalog (call it Cat A) that have been passed through Apple Photos where keywords were added, and that you are attempting to import these images (using Import from another catalog) into another catalog (call it Cat B) which already contains (a duplicate of) the same images. What I don't understand is how did the images get into Cat A in the first place....did you create a new catalog solely to import the images that were in Apple Photos? And how did the keywords get included in the images by Apple Photos? And are they physically separate copies of the image files (which I assume they are), and what file type are they: Raw or Jpeg? And one final question: how did the copies of the images originally get into Cat B?

A couple of possibilities: 

1. If the keywords that were added in Apple Photos have been embedded in the image files either by Apple Photos of by Cat A, and if the image files are NOT proprietary raw, then it's possible that the file length will be different to the set of images in Cat B.

2.  If the images in Cat B were themselves imported from another catalog, then you may be tripping over a bug that has been documented elsewhere. Specifically, when importing images from another catalog the original ImportHash record is NOT imported, thus it remains blank. So when attempting to import a real duplicate of an image, the duplicate detection fails if the original was itself imported from another catalog. Unlikely in this situation, as either the images in Cat A, or the copies in Cat B, would have to have been imported into the catalog via the import from another catalog route.


----------



## Stephanie Booth (Sep 20, 2018)

Jim Wilde said:


> What I don't understand is how did the images get into Cat A in the first place....did you create a new catalog solely to import the images that were in Apple Photos?



So, until end 2015 I used only Lightroom. Then I started using Apple Photos as my main photo management tool. Most of my new photos from then on didn't make it to Lightroom. I also imported images from Lightroom to Apple Photos for 2014, 2013, etc. 

During my "Apple Photos time" I made albums, edited photos, selected favorites.

In Apple Photos, I then gave keywords to my photos to indicate if they were favorites or in this or that album, with the intention that I would then be able to rely on those keywords to recreate my albums in LR.



Jim Wilde said:


> And how did the keywords get included in the images by Apple Photos? And are they physically separate copies of the image files (which I assume they are), and what file type are they: Raw or Jpeg?



I exported the images from Apple Photos, which included the keywords in the sidecar. Apple Photos creates sidecars for JPGs, so I used exiftool to write that metadata to the JPG so it could be read. 

Some of these images are RW2, some jpg. Some ended up in Apple Photos after having lived in Lightroom for a while, and others were imported directly into Apple Photos and never saw Lightroom.

They are all physically separate image files.



Jim Wilde said:


> And one final question: how did the copies of the images originally get into Cat B?



I exported a catalog from my master catalog (A), emptied it of photos, then imported "from disk" all the "apple exported" photos to have them in Cat B.

Here is a write-up of how I approached things: Moving From Apple Photos to Adobe Lightroom Classic CC

I guess I am tripping on the hash problem. One thing I noticed (with my test subset, which we'll call catalogs A' and B') is that if I use the import dialog to try to import the B' photos from disk, or if I use "import from catalog", different files get recognized (or not) as duplicates: if I use the "from catalog" import, about 15 of the 25 files are recognized as duplicates. If I use the "import from disk" function, then the 25 are.

If you have suggestions for getting myself out of this mess... (roughly 20k photos I'm trying to bring over from Apple Photos; main library has 80k)


----------

