# Don't import suspected duplicates - not working in Lightroom 6



## GrahamP (May 25, 2015)

For a long time I have relied on checking "Don't import selected duplicates" to do just that,  I can be lazy (or, as I prefer to put it, prudent) in not deleting files from my card in my camera before taking more pictures if I do not have to, or if I forget to clear the card first, and then I just rely on using this option only to import the new files when I hook up my camera to my computer.

That is, until Lightroom 6.  This worked faultlessly under Lightroom 5 and earlier, with never a problem, literally hundreds of times.  Today, I used exactly the same workflow as previously, but Lightroom 6 (standalone, not CC version) refused to recognise the old previously imported files on my card which were already on my hard drive and catalogued as duplicates.

I did not have time to sort out the new from the old in the import dialogue box, so imported all the files on my camera card into the location required for the genuinely new files, then sorted by date and moved the obvious duplicates into another folder, which I called "suspected duplicates" (imagination and creativity in my folder naming were never my strong points...).  As an experiment, and having double checked that the files in the suspected duplicates folder really were duplicates of files already on my hard drive and in the catalogue from a previous import, I removed all the files in the suspected duplicates folder from the catalogue (I did not delete the files though).  I then synchronised the folder tree which contained the suspected duplicates folder, and again all of the files in the suspected duplicates folder showed up as new imports, and were not caught by the "Don't import suspected duplicates" filter.  I then removed them from the catalogue a second time, and then used the import dialogue to point to the suspected duplicates folder - same result, not recognised as duplicates.

Looks like this simply does not work properly in Lightroom 6.  Anyone else encountered this?  From now on, I will try to remember to clear my cards once the photos have been downloaded from them, before I shoot anything else.

Graham


----------



## happycranker (May 25, 2015)

Graham. I just tried with a non formatted card using LRCC and Windows 7 64 Bit and I get the normal functionality all the pictures greyed out in the import module, as you would normally expect. But I do not have the standalone version so cannot try that I am afraid. Are you using the correct catalogue, do you have more than one?


----------



## GrahamP (May 25, 2015)

Peter, thanks for the reply. I only have one (huge) catalogue.  It contains over 200k pictures (mainly RAW files) and is about 10gb in file size.  All I can think of is that there are issues with very large catalogues.  It does not, for example, backup to a zip file, but only to a copy of itself.  Like you, I am using Win 7 64 bit OS. 

I have never had the problem elsewhere. Funnily enough, only a day or so ago I posted to another forum, trying to help someone with a haphazard filing system with a load of duplicates, suggesting using LR import to identify the duplicates, moving the non-duplicated files to a safe location and then deleting those indentified by LR as duplicates, but now I am not confident this will work as well as I had assumed.  

Graham


----------



## Jim Wilde (May 25, 2015)

Graham,

I've recently been looking at a couple of issues that I found with the "Don't import suspected duplicates" function, but they were limited to importing via directly-connected camera, and don't seem to be directly related to your issue (unless you sometimes import from card reader as well as from camera?). It might be worth trying the import from a card reader to see if the same problem occurs.

The problem of catalog backups not zipping correctly is related to larger catalogs (>4gb), I believe. I expect it will be fixed soon.


----------



## GrahamP (May 25, 2015)

Jim, the initial problem occurred on an import from a directly connected camera, but was in practice replicated when I deleted the files from the catalogue and then imported them again from their new location on the hard drive.

Graham


----------



## Jim Wilde (May 25, 2015)

OK, then you may have found another issue which I'll try to reproduce.

To be really honest, I think there are so many problems with importing via direct camera attachment (which never seem to get fixed) that I'd always people to use a card reader instead.


----------



## Lasseman (May 29, 2015)

GrahamP said:


> Looks like this simply does not work properly in Lightroom 6.  Anyone else encountered this?  From now on, I will try to remember to clear my cards once the photos have been downloaded from them, before I shoot anything else.
> 
> Graham



I have exactly the same issue. I import from a memory card reader and lightroom does not recognise that photos are duplicates even though they have the same name. It simply renames the files to filename-2.
This needs to be fixed ASAP since I use Lightroom in the same way as GrahamP.


----------



## LRList001 (May 31, 2015)

Lasseman said:


> I have exactly the same issue. I import from a memory card reader and lightroom does not recognise that photos are duplicates even though they have the same name. It simply renames the files to filename-2.
> This needs to be fixed ASAP since I use Lightroom in the same way as GrahamP.



I know this is not a fix, but a slight change to your workflow could work round it.  If you re-name the folder with the images in it, you keep the originals on your card until you are ready to clear the card down.  You lose the ability to review those images with the camera.  By using the date you copy the images in the new directory name (eg lr20150601), you can have multiple folders with each previous transfer of files from the card and a reminder as to when you did it.


----------



## coreypine (Jun 9, 2015)

Following this thread. Came to the Forum today for exactly this - importing from my SD card onto a MacBook Pro (Yosemite OS) and unless I re-format my card each and every time, photos get imported twice. I have triple checked that "do not import duplicates" is checked. File names are even the same!! (so that it has to import them as 1120-2 because there already is an 1120). Help!


----------



## clee01l (Jun 9, 2015)

coreypine said:


> Following this thread. Came to the Forum today for exactly this - importing from my SD card onto a MacBook Pro (Yosemite OS) and unless I re-format my card each and every time, photos get imported twice. I have triple checked that "do not import duplicates" is checked. File names are even the same!! (so that it has to import them as 1120-2 because there already is an 1120). Help!


This is not my experience at all.  The import dialog shows the images on the card that have already been imported but these are grayed out and non selected, Only new images (or images that have been imported and then deleted from LR)  are selected. 

Can you post a screen shot of the import dialog showing the images that have been already imported as being selected and the "Do not import duplicates" is also checked?


----------



## Jim Wilde (Jun 9, 2015)

coreypine said:


> Following this thread. Came to the Forum today for exactly this - importing from my SD card onto a MacBook Pro (Yosemite OS) and unless I re-format my card each and every time, photos get imported twice. I have triple checked that "do not import duplicates" is checked. File names are even the same!! (so that it has to import them as 1120-2 because there already is an 1120). Help!



And how are you importing: direct via camera connection, or using the SD slot on the MBP, or a USB card reader?


----------



## coreypine (Jun 10, 2015)

Importing via the SD card slot directly on the computer. Another odd thing - LR did not recognize newer photos on the card, even though I could view them in the Finder and see that they were in the same folder as the older photos. I solved that problem by ejecting the card and putting it back in. But it puzzles me that with the same import preset, sometimes I get duplicate photos and sometimes I don't.


----------



## Jim Wilde (Jun 10, 2015)

Can you track that back to maybe one specific card that causes the problem?


----------



## coreypine (Jun 10, 2015)

Good question, Jim. I just bought a new SD card, so will try it with that and see how it goes. Right now I'm thinking it's some kind of quirk, maybe with the card. I'll check back in if there's more problems. Hope I haven't hijacked Graham's thread.


----------



## GrahamP (Jun 14, 2015)

> Hope I haven't hijacked Graham's thread.



Not at all! I have just returned from a short trip.  On most days, because I had plenty of cards, I used a separate card per day, to avoid the problem of Lightroom not recognising duplicates. I created a new catalog on my laptop for the trip, to merge into my master catalog on my return.  On the last day, however, I did add images to an already used card, then imported them directly from my camera on to my laptop.  There was no problem with the duplicates being recognised. 

The new catalog on the laptop was a copy of my master catalog from my desktop computer, to preserve all keywords etc., but with the images removed from the catalog, to shrink it in size.  

So I have the same catalog on desktop and laptop. One is huge and has the import problem, the other is small and does not (admittedly only judging from a single use - my priority was to get my images into the catalog correctly, not to test for problems). This makes me wonder whether the problem stems from Lightoom's apparently difficulty in coping with very large catalogs. The version on my desktop contains over 200k images and is about 11GB in size. 

Graham


----------



## Branwell (Jul 28, 2015)

I have the same problem.  I have even created a new catalogue and deleted the preferences file (someone elsewhere suggested the .prefs file may be corrupt in this situation) but the problem still prevails.  I am trying to sort out a huge number of photos that were mostly on lightroom previously (before I lost my catalogue) and I decided to put all photos from all external drives on the same drive and take the opportunity to get fully organised using Lightroom - As the photos were collected over many years, there are a lot of duplicates to be removed from the composite drive.  Unfortunately Lightroom is not helpful.
Sometimes it does the right thing and greys out a batch of photos as duplicates - excellent!
Sometimes it greys out some of them only - once again, it's working correctly.
But, most times it imports the lot - sometimes just creating duplicates and sometimes adding -1 or -2 to the filename.
I'm going stir crazy trying to crack what's happening here.  Does anyone have any helpful comments? I would very 
much appreciate it if you do.
Branwell


----------



## Jim Wilde (Jul 28, 2015)

You need to know that Lightroom will only be able to determine that a file is a duplicate of one already imported if:

1. The file name is the same.

2. The file length is the same. This means that multiple copies of the same file that were saved/exported using different settings, or were cropped differently, or had different metadata applied, or were developed differently, would all fail the duplicates check because they'll be different file sizes.

3. The capture date/time is the same. 

But if you're confident that all three items of the checking criteria are really identical, then you probably need to report it as a bug using the link at the top of the page.


----------



## GrahamP (Jul 28, 2015)

I have tried to monitor the circumstances in which this problem occurs, ever since I initially reported it.  I now have a theory as to the cause, which is that Lightroom cannot cope with looking at more than one source of images.

My camera is a Canon EOS 5D Mark III, which has two card slots,  I tend not to clear the SD card, since on the average day out I will not fill even the CF card, and even if I do, I will use a second CF card rather than use the SD card in the second slot: SD cards are so small and fiddly that I am always scared of dropping and losing them!  I only actually use the SD for extended trips, as additional storage, and do not clear it until I need it again, but the SD card with old images nevertheless remains in the camera - better to have it with me than not, just in case I were to need it.  The old files presently on the SD card have long been in my LR catalog.

When I return home, I connect my camera and import directly from it into LR.  I have found that if I have both cards in the camera, with new images on the CF card and the old duplicates on the SD card, LR does not recognise the SD images as duplicates.  Nor does it recognise, in these circumstances, duplicate images that are on the CF card: it tries to import the whole lot, from both cards without recognising any of them as dulpicates.

The solution I have found to my particular problem is to remove the SD card from the camera before importing from the CF card,  If there is only this single source, LR correctly identifies duplicates on the CF card.  The problem definitely exists and is replicable, but for my particular usage I have found this way to overcome it.  

That does not help Branwell, of course, but comparing his situation to mine may help to identify the nature of the bug.  Branwell must have multiple folders as his import source, and is having the same issue,  It seems reasonable to conclude that the LR detection of duplicates falls over if it has more than one potential source which it has to examine for new files.  Maybe, at a guess, the solution for Branwell will be to import from one folder at a time, ignoring subfolders so that only one is examined, then deleting duplicates found from that, then moving on to the next one - a similar approach to mine of letting LR examine only one card in the camera at a time.

Hope this works, and helps in identifying the nature of the bug if anyone is reporting this to Adobe.

Graham


----------



## Jim Wilde (Jul 28, 2015)

Graham,

I've submitted several bug reports in this area of "duplicates checking", one of which covers the scenario you mention here, i.e. importing directly from camera which has two cards inserted. Sadly, I doubt these will ever be fixed as they'll never get a sufficiently high priority on the basis that the workaround is so simple.....in this case either remove one card or, perhaps better still, use a card reader rather than a direct camera attachment. There are bugs, related only to imports from a direct camera attachment, which go back a very long way, and the workaround is always the same....use a card reader.

In the testing that I did, I was only ever able to reproduce the problem on Windows, it all seemed to work correctly under OSX.


----------



## Branwell (Jul 28, 2015)

Thanks Jim and Graham.
Jim, The only difference between the files is the keywords that were applied to them when they were previously in Lightroom, vs the keywords applied to files that are already in Lightroom.  Not realising that keywords could determine if a file was a duplicate or not, I added some new ones as I imported.  Everything else is identical.
This does not seem to be the problem though, as after importing folder A, applying some new keywords, and then (as a test) attempting to re-import the files in Folder A, Lightroom correctly identifies them as duplicates, and does not import (even though the keywords are different).
The tedious process I am using until I get a better offer, is something like Graham suggests: importing one folder or group of files at a time, searching the imported files for -1.jpg, -2.jpg, and 'copy of' to eliminate duplicates formed through these renaming activities Lightroom has been getting up to, and then searching all files in the Catalogue for duplicate files using the plug in 'Duplicate Finder' and removing them all before importing the next folder.
There are other equally tedious processes I could use, but I have selected this one because it is done through Lightroom and not the operating system, and seems less risky.
However, it is a pain. Lightroom 6 does seem to be faulty in this respect as I didn't have the problem with the previous version.


----------



## Jim Wilde (Jul 28, 2015)

I agree that LR6 does seem to have some issues with its duplicates checking....perhaps if you have a problem that you can reproduce consistently, it would be worth filing a bug report at the official feedback site (link at the top of the page).


----------



## Roelof Moorlag (Jul 28, 2015)

> sometimes I get duplicate photos and sometimes I don't.


I did also and i will summit a bug report for this.

The first time i noticed it was on my main computer (windows) on a relative big catalog. I never attach the camera, instead i'm using a cardreader and sometimes LR recognizes the new images correct but other times it went importing the whole card (renaming the images the same with -2 as addition). Because i was used to reformat the card after processing (and back-ups) this was not a very big problem until now. 

I'm just back from a trip and used a macbook and a smaller (project) catalog. I used two cards (in two different camera's) and i imported the photo's every evening. Because I want to use the space on my cards as efficiently as possible I used them the next day(s) untill they are 'full'.

The first import went good (the first set images). The second time (on both cards) went good also. LR recognized the new images and imported only those. The next time it looks like LR recognized them again (some images were grayed out) but after import a lot of earlier images were imported (again) too...
I used the same import preset and the option "Don't import selected duplicates" was activated, i checked this several times. It is not clear why this happens. 

Now I went to uncheck all images in the import menu and only selecting te images from that day... Not very efficient.


----------



## robson (Dec 10, 2015)

Same problem in 5.7, *except* that my duplicates end up with different time stamps--the *minutes* remain the same, but the hours are different. I suspect something bizarre happening with time zones, because I travel a lot. The differences reflect eastern, central, mountain and Pacific time zones, depending--and I often end up with four copies of a single image. The Duplicate Finder plug-in can't locate them because the time stamps are different. I'm manually removing a lot, and hoping that something (as yet undefined) changes before I get another proliferation of files as in the attached. I've got tons of these. . . .

I'm doing further checking. This happens with all image sources: iPhone, Sony Alpha NEX-6, little Canon point-and-shoot. For the phone, I use a cable to connect (of course). With the cameras, I pull the SD card out and plug it directly into the computer.


----------



## Victoria Bampton (Dec 10, 2015)

You could be right about the time zones getting complicated.

Lightroom checks the capture time (plus file size and original file name) rather than the file creation date shown there.  Have you edited the capture time in LR at all?


----------



## robson (Dec 10, 2015)

I haven't done any editing of the capture time in Lightroom. 

The folks from the Duplicate Finder developer suggested altering the times by an hour on some images, but the process of doing that--and of sorting out +1, -2. -3 for varying files to get everything aligned has me sitting here manually deleting instead of making up a spreadsheet of what files need to be altered in which directions to get the time stamps all synchronized! They also have a beta that will allow a user to deselect capture time as an identifier, but (a) I don't have access to that yet and (b) I'm not sure it would work.

As I look at the times I was in the U.K. and Iceland, well, yeah, it gets even more complicated to try to make a coherent plan of adjustment.

Love your resources, by the way. Have been darting in an out of both your quick guide and your LR 5 Missing FAQ. I haven't upgraded to 6 yet because I barely have command of 5 so far--due to travel! (Shifted last year from Photos to LR when Photos misplaced *all* my keywords.)


----------



## GrahamP (May 25, 2015)

For a long time I have relied on checking "Don't import selected duplicates" to do just that,  I can be lazy (or, as I prefer to put it, prudent) in not deleting files from my card in my camera before taking more pictures if I do not have to, or if I forget to clear the card first, and then I just rely on using this option only to import the new files when I hook up my camera to my computer.

That is, until Lightroom 6.  This worked faultlessly under Lightroom 5 and earlier, with never a problem, literally hundreds of times.  Today, I used exactly the same workflow as previously, but Lightroom 6 (standalone, not CC version) refused to recognise the old previously imported files on my card which were already on my hard drive and catalogued as duplicates.

I did not have time to sort out the new from the old in the import dialogue box, so imported all the files on my camera card into the location required for the genuinely new files, then sorted by date and moved the obvious duplicates into another folder, which I called "suspected duplicates" (imagination and creativity in my folder naming were never my strong points...).  As an experiment, and having double checked that the files in the suspected duplicates folder really were duplicates of files already on my hard drive and in the catalogue from a previous import, I removed all the files in the suspected duplicates folder from the catalogue (I did not delete the files though).  I then synchronised the folder tree which contained the suspected duplicates folder, and again all of the files in the suspected duplicates folder showed up as new imports, and were not caught by the "Don't import suspected duplicates" filter.  I then removed them from the catalogue a second time, and then used the import dialogue to point to the suspected duplicates folder - same result, not recognised as duplicates.

Looks like this simply does not work properly in Lightroom 6.  Anyone else encountered this?  From now on, I will try to remember to clear my cards once the photos have been downloaded from them, before I shoot anything else.

Graham


----------



## robson (Dec 10, 2015)

Just came across a batch where time/date stamps are the same and still there are duplicates. I'm back in 2014, having cleaned 2015 (many photos).


----------



## robson (Dec 10, 2015)

I'll settle for a routine that will keep the duplicates from showing up again once I get them all out. . . . It sounds like making folders on the SD cards will help there. The phone is another question.


----------



## Neil Hunt (Oct 21, 2016)

This has gotten worse and worse over the past 18 months.
Used to be sometimes it missed a batch of dups.
Now Lightroom fails to spot duplicates at all.
Mac ElCap-Sierra, Lightroom 6.0-6.5, loading from a card in the Mac reader.


----------



## Victoria Bampton (Oct 22, 2016)

Before you do anything else Neil, upgrade to 6.7, as earlier versions aren't supported on Sierra.


----------



## [email protected] Shooter (Nov 1, 2016)

I have had the same problem for a while... 5D Mark IV ... Lightroom CC... now this worked for me several times now ..so I present it as a work around... When I go to Import ..My SD card I just loaded is at the top left of importable files under Devices... All of the pictures I just imported are not greyed out... I just went down to the other folders under FILES below the SD card which included my hard drive I import to and just get it to recognize any RAW file in the lower files... I don't import ..then...I just click back on the top SD card under Devices which didn't grey out old imports before and now it works... been good for a week or so now...???????? I'm just Sayn'


----------



## Gnits (Nov 1, 2016)

robson said:


> As I look at the times I was in the U.K. and Iceland, well, yeah, it gets even more complicated to try to make a coherent plan of adjustment.



*I know it will not suit everyone*.... but I set my cameras to UTC time .... no matter where I am in the world.  I never have to worry about Time Zones or Winter / Summer time. Particularly useful if doing a lot of travel.

Regarding checking for Duplicates... since the first beta of Lr I have never trusted this function in Lr.  I format my cards immediately after import to specifically avoid a duplicate image scenario. 


Also, so sad that  .....


Jim Wilde said:


> Sadly, I doubt these will ever be fixed as they'll never get a sufficiently high priority



that known fundamental flaws are not resolved.


----------



## clee01l (Nov 1, 2016)

Gnits said:


> *I know it will not suit everyone*.... but I set my cameras to UTC time .... no matter where I am in the world. I never have to worry about Time Zones or Winter / Summer time. Particularly useful if doing a lot of travel.


Nikon seems to have solved that problem.  My camera has two times.  UTC and a local time zone.  UTC gets updated automatically every time I connect my GPS.  I need to remember to change my local time zone and if I forget, then I can adjust after the fact when I remember or in LR.


----------



## Gnits (Nov 1, 2016)

clee01l said:


> UTC gets updated automatically every time I connect my GPS.



A side benefit of the inclusion of GPS chips in cameras.


----------



## Jim W (Feb 27, 2017)

[email protected] Shooter said:


> I have had the same problem for a while... 5D Mark IV ... Lightroom CC... now this worked for me several times now ..so I present it as a work around... When I go to Import ..My SD card I just loaded is at the top left of importable files under Devices... All of the pictures I just imported are not greyed out... I just went down to the other folders under FILES below the SD card which included my hard drive I import to and just get it to recognize any RAW file in the lower files... I don't import ..then...I just click back on the top SD card under Devices which didn't grey out old imports before and now it works... been good for a week or so now...???????? I'm just Sayn'



I've been having this same issue, and this suggestion solved it for me. ...for now, anyway!


----------



## Stephanie Booth (Sep 19, 2018)

Similar issue here, but importing from catalog or hard drive (SD card seems to work) so following this thread (just in case)


----------

