# Photo Set Storage Database



## matsmithphotog (Sep 26, 2011)

Hi all,

Warning - this post is a bit geeky.

When I'm shooting on location, I typically shoot to memory card until it gets full, at which point I switch it for another card and insert the full card into an on-the-go backup disc. At the end of the day, I therefore have two sets of photos: the ones on memory cards and the ingested image files on the backup disc, residing in a folder for each memory card. This allows me to travel back to the studio / home with two sets of images in different places in case of problems. (I'm anal, I literally put the data at opposite ends of the car.)

When I return back to base, I take a copy of the RAWs (from my backup disc) and dump them straight onto a server (with RAID mirrored discs), and import them into LR on my laptop (this is where I keep them and work on them until the editing is complete).

At this point I can then re-use memory cards safe in the knowledge my photos are stored in three locations:
- my laptop (with me all the time)
- RAWs on my backup disc
- RAWs on my server 

Depending on the job type or amount of editing, I may want to take interim backups of the edits, in which case I'll use LR "export to catalog", and copy the whole catalog to the server as well as the RAWs that already reside there. Pointless duplication, yes - but I don't mind as I cull all backups and working copies at the point of archiving.

As I work through the set on my laptop, I usually delete dud images (reject then delete) and ones I know I won't use.

Okay - imagine you are working on photos from 10 clients at any given point. You have the following for each shoot:

- RAWs on the mobile backup disc (primary ingested)
- a copy of these RAWs on the server (secondary ingested)
- a working copy on the laptop (working copy)
- and possibly an interim copy of a part-editing LR catalog (catalog backup)
- if the job is super important I may also put RAWs on another disc and store them offsite (tertiary ingested)

Oh - and when the job is complete and images handed-over, there's also the final location: the archive location.

I've never lost a client's images, but I once came too close for comfort, hence the regime. At this point, I didn't just want to know what was stored where, but I also needed to have seen a *log* of a backup I had previously stored and consciously decided to remove.

How on EARTH do you keep track of what photo sets and backups are where? Let alone a history of which photo sets *were* in various locations but are no longer?

More to the point, how do you know for *sure* that you remembered to complete all stages of the workflow? (After all, a workflow / backup process is useless unless you monitor it.)

So I started to keep tabs on my photo data sets in an Excel spreadsheet, so I could quickly see - at a glance - what I put where. The spreadsheet got big quickly, what with multiple clients for whom I may have done one or more shoots, each shoot having a variable number of data sets associated with it, which may or may not have been removed as my workflow progressed.

Naturally I thought I should really have a database for this. I have spent most of my Sunday getting my head around this database. I've almost succeeded in doing so.

But I'm really interested to hear others' thoughts on the subject. Do others take data set management so seriously? Does anyone have any systems - or even little mechanisms - in place to prevent accidents?

Has anyone ever come across data set asset management software? I wouldn't even know what category of software this comes under!

Hope I've not bored you all to tears / slitting wrists...

Mat


----------



## Victoria Bampton (Sep 26, 2011)

It's great to hear of someone who has actually THOUGHT about their backup systems.

One bizarre question - how well's that server protected?  Assuming you're sat at home one day (server's always plugged in, maybe this time your laptop's plugged in too) - what happens if there's a power surge which wipes out all of your connected gear?  Your super import job's protected as it's offsite, but the others?

Personally I'm fond of good old fashioned checklists for things like that, organized by shoot date.  I haven't run into any software set up for that specific purpose, and as everyone's setup is likely to be slightly different, your own database is probably your best bet.


----------



## Brad Snyder (Sep 26, 2011)

Two semi-random thoughts.

1) [Memory cards]>>> [field backup disk].   [Field backup disk] >>>>> to [RAID backup server] and [Laptop]    Are the copies from the memory cards ever verified? As you describe it, sounds like the [field backup disk copies] could be a weak point.  Do the memory cards get dumped separately/parallel into the system somewhere?

2) I've been doing a lot of playing with Evernote, with its 'omnipresent' auto-sharing. It has a reasonably functional catalog(notebook) organization with keyword tagging. From desktop, to laptop, to mobile phone, this puts a lot of stuff at my fingertips.  (Which I find I need as more of my gray cells turn into gray hairs.)  It's nowhere near an Excel or a 'database' but it's always on.  Some faffing (am I using that correctly?) about with page templates might get you somewhere productive.


----------



## clee01l (Sep 26, 2011)

And just in case you need another opinion.  I would do everything in Lightroom with smart Collections.
First, I like the 'Ingested image Backup Disk"  And I like the Master images stored on a Raid Server.  But there is one more piece of hardware/software that I would add. I would get another (matching if possible) Raid Server to back up the first AND to backup the catalog.

The In-the-field- backup disk is as temporary and as volatile as the memory cards. It will fail.  And the RAID controller will fail leaving you with several perfectly good mirrored HDs and no way to read them. With a matching Raid Controller, you can read the HDs from the RAID server gone bad with the duplicate controller.  Now there is software that will read the data off of many flavors of RAID if you have no controller, but this is tedious and not for the faint of heart. It is also expensive if you relegate this data recovery to a third party. 

So, what you end up with is two temporary field data copies (memory card and on-the-go backup disk.)  Two master copies of data and catalog (Original and backup)
Now manage it all with LR.  John Beardsworth has a Workflow Collection to manage your processing, I would extend that to include data tracking. 
You can add internal (not for export) keywords for each image file. And a Smart Collection or Collections to track the data flow as well as the work flow.  These keywords would mirror the data actions that have taken place (i.e. Server backed up, data removed from field sources, moved to archive, etc.) in your case, I would envision a series of collections for each client.  If you feel the need to Export Client data as a catalog. All you need to do is choose the appropriate collection as your export and then add a keyword noting the event. 

Paraphrasing Tolkien: 
_One Catalog to rule them all,
One Catalog to find them,
One Catalog to bring them all
and in the darkness bind them._


----------

