# Lightroom Validation - Suggestions and feedback solicited



## Linwood Ferguson (Dec 4, 2013)

I've been using ImageVerifier for some time, and remain annoyed by it, and finally decided to write something myself. 

I'm working on a program that does the following: 

- Periodically (on request) reviews newly added images in the catalog, checksums them, and saves data about them.

- Periodically (on request), uses previously captured information to see if the files are still there, still readable, and identical to what they were the last check.

- Ancillary routines to accept/revalidate, show the files found from disk or from the preview cache so you can decide on a problem.

The main difference from lightroom, which has the facility to notice missing files, and (through synchronize) find files lost from the catalog, is to verify that through disk failure, bad copies, etc. that your raw files are not changing over time.   If you use DNG I guess that's somewhat built in, but if you keep regular raw files it is not, and I've been burned.

I am doing this as a windows program.  I was going to do it as a plugin, but (perhaps through not being familiar enough) I wasn't sure that I could do everything well, especially fast enough (in particular I wasn't sure I could reasonably keep track of attributes of images that Lightroom may have lost from it's catalog).

This posting is for a few purposes, to find out if you have suggestions for such a program and/or interest in it.  I have no idea what I will do with the program - put it up as open software, give it away, sell it, or throw it away; this is mostly a learning project at the moment.

But it is also to see if it seems of interest, is there a better way to do this (perhaps a plugin already available), etc.

Summary so far: 

Does: 

- Checksum new images added, and tells you if they change later
- Finds files not present but expected (pretty much same as find-missing) 
- Verifies new and existing files can be read on disk, fully
- Find files that were present, are still on disk, but are not now in the catalog (like synchronize, but gives you background info on when last seen)
- Verifies all file types
- Gracefully handle properly done moves on disk (i.e. if new location is right in lightroom), and helps you figure out what happened in bad moves (as it tracks the last valid location separately from lightroom)

Does not: 

- Verify XMP's
- Display video (though it will checksum them)
- Verify that an image is well-formed (e.g. if it is corrupt when this program first sees it, it never knows; however the hooks are there for doing an initial verification if I could figure out a reliable way) 
- Not sure if it handles all the variations of off-line media; right now it just flags them as not validating, and you can mass-revalidate when you put it online
- Haven't even looked at smart previews, when it displays an image to provide context it is from the conventional preview cache.
- Have tried it only on lightroom 5.3


----------



## johnbeardy (Dec 4, 2013)

I'm pretty sure no plugin covers this area.

You may want to consider launching your app from a small plugin, if only to gain experience. You can gather each file's name and path data and send it to your app via a command line or a data file.

John


----------



## Linwood Ferguson (Dec 17, 2013)

Thank's John. 

Lacking other response there's probably not much point in distributing it, but maybe someone will be interested enough to test.  I've decided to release it publicly as free software including source code.   That way if you are (rightly so) reluctant to use binaries from a stranger, you can pull the source code down, review it, and build it yourself.

I did it as a Windows WPF application instead of a plugin for speed -- both so I could write it quicker (as I know C#) and so that it would run faster; the result is heavily multi-threaded and pretty quick (on my system it does 2-5 raw files per second initial scan, and about 10-15 revalidation; it will saturate your image storage disk or CPU when scanning, whichever is slowest).

It stores data in the catalog database; it does not write to the catalog tables, so it should have no interaction with Lightroom (though lightroom will not share the database, thus you should not run both at once).

It detects: 

- Files imported into lightroom which cannot be read initially for validation (missing, bad directory, file lost); this is basically the same as Lightroom's find missing.

- Files previously validated but: 
* That are now not physically present (disk file lost, catalog present -- could be offline, or just list) - like "find missing"
* That are present, but cannot be read
* That are present, but have been changed (compared by checksums before/after)
* That are present on disk, but NOT in the catalog (lost? removed but file not deleted?)

These last three are the real purpose, to detect file changes due to disk failure, inadvertent user action, cosmic rays, etc.

It has been lightly tested, and I have both a 32 and 64 bit versions (though the 32 will run on 64, or should, if for some reason you prefer it).  I've installed on Windows 7 (32 & 64), 8 (64) and 8.1 (64).

I've tried it against Lightroom 5.2 and 5.3.  It depends on a few lightroom tables being present and in the current format; I do not know if older versions had the same format but I suspect they did (it doesn't depend on anything related to newer features), so I suspect it will run on quite old versions, but cannot say.

I've run it on a catalog with about 27,000 images.  It takes a while to run, and slows down as it does the initial scan (you should optimize soon afterwards).   I don't know of any specific capacity issues to prevent it from running on as large of catalog as Lightroom can handle.

In places where it displays the raw photo, it depends on the  O/S codec to do so, so if you want to see the images from disk install the Nikon, Canon, etc. codec first.  The preview cache is JPG so it can always show that (if found).  This is on the display of error information.

If someone wants to test it, and give some feedback, please do.  I created a Codeplex project so all the source code is there to build your own copy, as well as binaries (if you are the trusting sort). 

Sorry, there will be no apple version, I don't speak Cupitino.

If you try it, please do back up your catalog first.  Or better yet, set up a special one just for testing.  I THINK it is safe, but you can't be too careful.

Some screen shots attached.

Project, binaries, etc are here: http://lrvalidate.codeplex.com

I've not used Codeplex before, so let me know if it doesn't make sense.


----------



## Linwood Ferguson (Dec 17, 2013)

Well, that's interesting.  Was checking on Google to make sure the project showed up, and stumbled across this:

http://bayimages.net/blog/lightroom/validator/

Someone has already done this as a plugin.  

I haven't tried this one, but it looks like it does almost the same thing (the one thing I think being in a plugin prevents is noticing unexpected loss of an image from the catalog but not disk, something I've never known to happen-- but could).

When I get some time I plan to run a performance test, am really curious whether integrated as a plugin it slows things down.   But regardless -- here's another validation alternative from a different author, if you haven't found it before.


----------



## Linwood Ferguson (Dec 17, 2013)

I tried the plugin.  Very cool, easy to use.  It doesn't find cases if a file was lost from the catalog but is still on disk.  It finds cases where the file is not readable (or missing) but it doesn't report them well, as best I can tell you have to look in the log (not difficult but out of the paradigm for the rest which deals with collections.  It was very fast on small test catalogs, but I tried it on a 28k image catalog and it slowed down dramatically, 100 images took 66 seconds.  But nights are free time to do validation, so that's not a terrible problem.  Revalidation took only 28 seconds.  Compared to a standalone program that's pretty good (takes about 5-6 seconds for either), considering all the overhead to deal with inside of Lightroom as a plugin.

But it's darn close to exactly what I was looking for and didn't find until I wrote my own.  Not sure if to be happy for the education or annoyed my google skills are so poor.


----------

