# wanted: plugin to find duplicates



## schwerd (Sep 4, 2008)

as a result of years of mismanagement, I've ended up with multiple copies of the same photo with different filenames....I'm going thru the library manually, sorted by capture date/time, which in most cases puts the files together, then deleting the dupes....

it would be nice to have a plugin that would scan exif data, find dupes based on lots of the data EXCEPT filename...and display those files so that we could prune out the duplicates.

anyone have any thoughts????? wish I had the skillz to write it for the whole community


----------



## golf1025 (Sep 5, 2008)

*Add my vote!*



			
				schwerd;216'7 said:
			
		

> as a result of years of mismanagement, I've ended up with multiple copies of the same photo with different filenames....I'm going thru the library manually, sorted by capture date/time, which in most cases puts the files together, then deleting the dupes....
> 
> it would be nice to have a plugin that would scan exif data, find dupes based on lots of the data EXCEPT filename...and display those files so that we could prune out the duplicates.
> 
> anyone have any thoughts????? wish I had the skillz to write it for the whole community



I would add my vote for this plugin request. I have many duplicates of my edited RAW files in JPG, PSD, TIFF versions and would love a plugin that would filter them out based on a query like:
filename + extension = .nef AND .jpg/.psd/.tiff - etc.
Jan


----------



## Victoria Bampton (Sep 6, 2008)

Hi to both of you, welcome to the forum!

Definitely put in a feature request( Official Feature Request/Bug Report Form) - I think that's outside the realms of the plugin SDK at the moment, but it's something I'd love to see added too.


----------



## Carlo Balistrieri (Oct 12, 2009)

I take it no solution has presented itself yet.... I'm now in the same boat. After a mass conversion to DNG, I've got duplicate files all over and need a way to get rid of the NEF's (LR is recognizing the DNG's so I'm good there...).
Anyone?


----------



## johnbeardy (Oct 12, 2009)

I suspect it would be possible to code such a plug-in. It might involve writing to a custom field, maybe entering a code built up from the camera, exif time etc. You'd then be able to filter the catalogue by each code, grouping the DNG and the NEF. 

However, it would be hit and miss with some EXIF data, and it would still leave you having to do a load of clean up work afterwards. Anything to make it easier would entail the sort of logic that will differ from case to case - you want to loop through DNGs and see if there's a NEF, others will want to do the opposite, or the duplicate might be a JPEG, or a duplicate DNG etc. I can see so many permutations that it's not a plug-in that tempts me. Best not to get in such a mess in the first place!

John


----------



## clee01l (Oct 12, 2009)

An application outside of LR might resolve most of your duplication issues. One I've used for images is NoClone

http://www.reasonablesw.com/


----------



## Carlo Balistrieri (Oct 12, 2009)

I'm looking for a way to line up the NEF and DNG files in something like Finder (I'm on a MAC), verify the duplicate file names (minus extensions) and eliminate the NEF's. Doesn't have to be fully automated...

John, the new files are all coming in as DNG's and being saved that way so there is no longer an issue. It's just the initial few thousand.


----------



## jimkeir (Dec 7, 2009)

Hi,

Just noticed this thread. Perhaps you might like to try this plugin?

http://www.jimkeir.co.uk/Lightroom/DupesIndex.html

Cheers,
Jim


----------



## Victoria Bampton (Dec 7, 2009)

Thanks Jim, I'll have a look at that! And welcome to the forum!


----------



## Nomad (Jan 20, 2010)

[quote author=jimkeir link=topic=3221.msg57981#msg57981 date=126'185275]
Hi,

Just noticed this thread. Perhaps you might like to try this plugin?

http://www.jimkeir.co.uk/Lightroom/DupesIndex.html

Cheers,
Jim

[/quote]

I just downloaded and installed this plugin. When I relaunched LR & tried to run it I got an error message "Lightroom 2 performance warning. A memory leak in Lightroom 2 means you'll probably have to restart after running this". 

Does anyone know what's causing this? I'm using LR 2.6. I thought I read elsewhere that the memory leak problem was fixed back in 2.3.


----------



## jimkeir (Jan 20, 2010)

Hi Nomad,

Well, *a* memory leak problem was fixed at 2.3 

Each time you do a search from a plugin, it looks like some memory is set aside for the records that are accessed. The plugin frees its own resources but Lightroom doesn't. You can check this by running the same search twice - memory usage will go up dramatically the first time but hardly at all the second.

I added a new search mode a couple of weeks ago which allows you to search a section of the database at a time to try and help people who did hit problems.

Cheers,
Jim


----------



## Nomad (Jan 20, 2010)

Hi Jim,

Thanks for the follow-up. I also just noticed the memory leak caveat at the end of the plugin documentation "Known Problems". 

One thing that I'm confused by: After running the plugin on a small subset, a few duplicates were reported. When I selected "Show in Finder" for two duplicates both images were pointing to the same image file. It ended up that the duplicate was actually a virtual copy within LR. Is there any way to avoid including virtual copies in the results so that only genuine duplicates show up on the list?


----------



## jimkeir (Jan 20, 2010)

Hi,

It should be aware of virtual copies - I have several in my database, which I obviously test against - but I'll double-check and get back to you.

Cheers,
Jim


----------



## jimkeir (Jan 20, 2010)

Hi,

You're right, it does include virtual copies. Duh. :-[

I've added an option which will allow you to exclude them from the results, and uploaded to http://www.lightroom-plugins.com . Give it a go and see if it works for you - let me know how it goes.

Thanks for the heads-up - if you contact me via PM with your email address I'll send you a license for it.

Cheers,
Jim


----------



## Scotty (Jan 30, 2010)

Hi,

Just happened upon this thread and forum this morning since I'm working on a similar situation with too many digital photos that have been mismanaged. Fault is mine since I really didn't have a good, er, any, sort of work flow to go from camera to print and everywhere in between. Random backups, copies, proofs, and what not are now scattered across my system and its time to clean it up. But I rant.

I can say that NoClone from Reasonable isn't too bad. My opinion it works better on music files and jpg's than raw files. I was hoping a recent upgrade to 2'1' would address some of the issues they have but it doesn't appear to be the case. The beauty of noclone is that it uses a true byte by byte comparison on the file. Its not fast by any means but what it appears to me is that it uses a ton of memory if you have files scattered everywhere such as I. 2',''' or so images in raw format and each over 13MB in size, well, the memory requirements appear to be a lot. It never finishes when I search for raw but has worked fine with jpg's and mp3's. Thus my comment. I don't work for them but just a end user who has some luck with it but wanted to warn about how it works with raw files and the number of them. I'm run a q955' intel with 4gb of ram it its not enough on windows 7. I also tried to get it to work within ubuntu and wine but no luck as of yet thinking memory may behave different since all I have is windows 7 32-bit version.

I will try the plug-in here that was posted and see how that goes. I think at the end of the day though the issue that needs to be addressed for me is to get my work flow down so I don't run into this mess again. I use both LR 2.5 and Phase One - and they both seem light on the end to end work flow to me anyway. But I'm not a professional so I probably would like more at the app to guide rookies like me on a process.

Anyway - just wanted to add $'.'2 about noclone.


----------



## Brad Snyder (Jan 31, 2010)

Scotty, thanks for the info, and welcome to the forums.


----------



## ceroni (Feb 19, 2010)

I've been using *Duplicate File Finder* with success for quite some time now. It works on any kind of file.


----------

