# Compressing RAW files within the catalog



## matsmithphotog (Nov 21, 2010)

Hi all

Like many here, I sometimes capture many hundreds of images in one photo shoot (be it a wedding, portrait session, my personal holiday snaps...).

I have terabytes of storage; my laptop has two internal drives, my desktop has 4, I have a server with 4 internal drives, plus various external discs, and a mobile backup device. For my clients' files, I back up images on-the-go as well as backing up when I return back to the studio. At many points in my workflow, my images are stored in three places although I like to reduce this to two by the end of the work on a given set.

I shoot RAW. At 5616 x 3744. That makes for 16 bit files.

Instead of "the cull", which I do regularly on clients' images as well as my own, I would sometimes rather hit a button that keeps my images in LR but reduces their size by compressing to JPEG, reducing the JPEG quality, and reducing the resolution.

I know I can export and re-import, but this is

a) prone to error (re-importing back to another folder, accidentally forgetting to change "COPY" to "MOVE", etc.)
b) takes too long. I cull sets of files all day, every day, I want to do it in a single click!

*Does anyone know of any feature in LR, plugin, or module, that can achieve this?*

It is useful for my personal photos more than those of my clients. I like to shoot loads of personal ones and come back in a month time, and I really know I don't need to keep them in CR2 format unless I decide I want to do this at the time of reviewing them in the first place.

Thanks!


----------



## dj_paige (Nov 21, 2010)

When you export, click the box that tells Lightroom to include the export in your catalog.


----------



## sizzlingbadger (Nov 21, 2010)

I don't know of any plug-ins that will do that. The only built-in option is to convert to DNG, maybe that will save some space ?

What Paige suggested will take out the error prone part. You would still have to delete the originals afterwards but that should be easy enough.


----------



## matsmithphotog (Nov 21, 2010)

[quote author=dj_paige link=topic=11661.msg78316#msg78316 date=129'371234]
When you export, click the box that tells Lightroom to include the export in your catalog.
[/quote]

Thanks! Is there any way to automate the subsequent removal of the original files? Maybe using post-processing actions? I guess I could just save "reduce files" as a preset this way...


----------



## sizzlingbadger (Nov 21, 2010)

You could run a post action script of some sort to remove the original, but it would need some careful testing.


----------



## johnbeardy (Nov 21, 2010)

I think it would be possible to write a plug-in to automate that workflow, but no-one has seen fit to do so. Read into that what you will! 

John


----------



## matsmithphotog (Nov 23, 2010)

[quote author=johnbeardy link=topic=11661.msg78321#msg78321 date=129'373662]
I think it would be possible to write a plug-in to automate that workflow, but no-one has seen fit to do so. Read into that what you will! 

John
[/quote]

Indeed, although I can think of a number of one-click functions in LR3 that are *surely* more universally useless than the ability to reduce file sizes of multiple images (thus claiming back potentially hundreds of gigabytes of disk space!). For example, "convert to DNG"... haha.

On second thoughts, I think an export action is a pointless waste of time. The idea would be to keep images in the cat without losing their flag statuses and edits and the like. Re the latter, an edit on a RAW file would look very different to that of a JPG, so maybe this is a lost cause anyway.

Still - some option to help me tame my disk space issues would be nice, and I can't imagine I'm the only one with such issues wanting LR to help me out. Maybe I'll submit a feature request.

ETA: i7-93'? 12GB RAM? Is that really enough RAM?


----------



## Graeme Brown (Nov 23, 2010)

Paint a keyword called Reduce across all then files you want to reduce.
Filter them by the keyword and file type so that only the raw files are showing.
Export as jpg with 'add to catalog' option selected
Once it's done and you've checked them then filter the raws again as above and delete.

It's not one click, but it's still pretty quick. Personally I'd rather keep the raw files so I'd just buy more hard discs - a 1tb external is not expensive and it will hold an awful lot of raw files.


----------



## matsmithphotog (Nov 23, 2010)

Right, the lack of in-app file conversion is hacking me off.

I just spent the last hour going through tens of Photoshop edited files, some of which were 16 bit multilayered, some were just under half a GIG each. Had to open each one in PS, flatten, convert to 8 bit, save.

Thank god my disc is an SSD, but seriously, it would have been SO much easier had I been able to do this from within LR at the click of a button.

Thanks Graeme for the point you mentioned - I have been doing something similar already for a while, but this doesn't help me when I need to flatten images and keep their LR metadata (flag status, edit datetime, etc.), and it also doesn't help the fact that when you go through exporting, re-importing, and deleting originals, this is SO prone to error. Therefore the extra time it takes for me to check everything over and over again before pressing the delete button - this is taking me too long.

Maybe it's my workflow. If anyone can suggest something useful that helps me do what I need to do given what I've written above, that would be most appreciated. I know I could save the PS files as flattened 8 bit in the first place, but I only want to do this about a month after editing and reviewing, so I can go back through the image layers and make changes if I need to - whilst the job is still in progress.


----------



## Victoria Bampton (Nov 23, 2010)

Are all of the ones you want to convert all in the same folder? If so, I'd switch to Bridge/PS and batch them. It would all happen behind LR, so when LR next looked for the file, it would just see the reduced version.


----------



## matsmithphotog (Nov 24, 2010)

Hi Victoria

I will usually have say 2-8 Photoshopped images from within one folder (my folders are by date e.g. 2'1'/25-1'-2'1'/) but typically yes I will get to a point where I need to flatten from across e.g. 1' or more folders.

You are right, I should really have done that last bit of work by batching them in Photoshop. But that wouldn't solve my other problem of wanting to compress the images down from RAW to JPG.

I know a lot of people say "get another disk" but I'm getting through a terabyte every six months if you include the multiple copies I store for clients. I'm always buying disks.

Every professional photographer I know IRL, at least all who shoot of a tonne of images in one shoot, needs to do 'the cull', it reduces the footprint taken up by a shoot by 5'% or more, removing otherwise useless images, it would just be nice not to have to be quite so brutal about it.

Anyway I am not complaining - just wanted to figure out if there was *any* way whatsoever to do what I am trying to do.

Thanks for your comments, all.


----------



## Graeme Brown (Nov 24, 2010)

I keep all of the raw files from the images the client saw, plus tiff versions of any images edited in Ps, so around 3''-35' files in total. Add in a copy of the lr catalog for that job and the whole thing comes to maybe 5gb or so. Even if I was shooting 21mp files it would still be less than 1'gb, so I can get 1''+ jobs on a 1tb disc which costs less than £7'. That works out at 7'p per job and I've still got the original files ..... can't think of a cheaper way of archiving, and it's certainly not worth my time compressing files to jpegs and checking them all just to save 7'p


----------



## Victoria Bampton (Nov 24, 2010)

[quote author=matsmithphotog link=topic=11661.msg78514#msg78514 date=129'58796']
But that wouldn't solve my other problem of wanting to compress the images down from RAW to JPG.
[/quote]

Ok, I'd make it simple (and I haven't read back through the whole thread again so this might have already been mentioned).

I'd stop using flags that aren't stored in XMP and use stars/labels instead. Develop history and settings would be no use to JPEGs so no need to worry about those. The keywords etc should automatically end up with the exported files, so they're not a problem. 

As I'm sorting through, I'd drop photos I want to swap into a collection, maybe QC. Once there's a batch, I'd export JPEGs to original folder, with the export dialog set to add to catalog, which will import those replacement JPEGs back in. Once the export finishes, the number of photos in the 'last import' collection should match that quick collection (there's your double check that you got them all), and then you can delete all of the photos out of the QC using Ctrl-Alt-Shift-Delete which will delete from the hard drive as well as the catalog even when you're viewing a collection. Any better?


----------



## johnbeardy (Nov 24, 2010)

The other point I'd make is that if it's not worth keeping as raw, is it really worth keeping in any form? 

That would, I think, be the view of most of the pros I know - in fact I can't think of any who apply a JPEG system like you describe. Some ditch a high proportion of their captures (95% is atypical but deleting 66-75% is common) while others will keep every raw file but won't necessarily keep them online. In the last case, the guy burns reject images to DVD before deleting them from his hard drive. He does this by using flags to evaluate the shoot, filters so he shows only the rejects, adds them to a subfolder called "on DVD", and then goes into Explorer and deletes those files. So in his catalogue he has the rejected files' previews.

John


----------



## matsmithphotog (Nov 24, 2010)

[quote author=Graeme Brown link=topic=11661.msg78516#msg78516 date=129'591725]
Even if I was shooting 21mp files it would still be less than 1'gb, so I can get 1''+ jobs on a 1tb disc which costs less than £7'. That works out at 7'p per job and I've still got the original files ..... can't think of a cheaper way of archiving, and it's certainly not worth my time compressing files to jpegs and checking them all just to save 7'p
[/quote]

Ha! Ok so the shoots I do routinely yield 2'GB of RAW files (21mp captures) from 12 hours of shooting, double or triple for perhaps a two or three day job on the road. Add to this 1' to 3' multilayered files in PS... Much of the time half the images will be either complete duds or lighting tests, and depending on the job I often don't get the chance to delete in-camera before putting straight onto mobile backup. Everything from mobile backup then gets imported straight to LR. Typically, like you, I would show around 3''-4'' images to clients (this varies massively though) but it's the other 7'', which you presumably get rid of (?), many of mine may be duds (lighting tests etc.), but I don't like to cull. For example I recently needed to look at a test image I shot at a wedding where I set up a studio, the image was of the studio setup itself, in the room, I needed it so that I could make an estimation of space required for a similar setup - I had got rid of that test shot because at the time I knew it was of no use to the client therefore assumed it was of none to me.

Anyway, the focus of this chat has clearly become 'workflow' rather than 'Lightroom'; it has been most useful comparing workflows. I still maintain it's possible to adjust my workflow to LR but wish LR could accomodate my ideal workflow. "Pipe down", you might say, "unless you want to come up with something better" ...

[quote author=Victoria Bampton link=topic=11661.msg78517#msg78517 date=129'592526]
I'd stop using flags that aren't stored in XMP and use stars/labels instead [...] The keywords etc should automatically end up with the exported files, so they're not a problem. 
[...]
Once the export finishes, the number of photos in the 'last import' collection should match that quick collection (there's your double check that you got them all), and then you can delete all of the photos out of the QC using Ctrl-Alt-Shift-Delete which will delete from the hard drive as well as the catalog even when you're viewing a collection. Any better?
[/quote]

Much better. Thank you. I didn't use flags at first, I always used star ratings. Then I moved over to using flags just for the purpose of culling the images. I'll go back to what I did before!

[quote author=johnbeardy link=topic=11661.msg78519#msg78519 date=129'593988]
The other point I'd make is that if it's not worth keeping as raw, is it really worth keeping in any form? 

That would, I think, be the view of most of the pros I know - in fact I can't think of any who apply a JPEG system like you describe. Some ditch a high proportion of their captures (9'% is atypical but deleting 66-75% is common) while others will keep every raw file but won't necessarily keep them online. In the last case, the guy burns reject images to DVD before deleting them from his hard drive. He does this by using flags to evaluate the shoot, filters so he shows only the rejects, adds them to a subfolder called "on DVD", and then goes into Explorer and deletes those files. So in his catalogue he has the rejected files' previews.

John
[/quote]

Useful insights. See previous comment about if not worth keeping as RAW, then is it worth keeping in any form? I doubt it is worth it in most cases, but for the very reason that I know storage is so cheap, I can't help thinking that if the compression ratio is 7:1 then it's worth keeping some kind of record, just for me.

3'' RAW
+7'' JPG
= same space as 4'' RAW.

Anyway I'll stop harping on because clearly I'm in the minority in thinking like this!

Thanks again for the comments.


----------



## MarkNicholas (Nov 25, 2010)

In my opinion such a feature is far too obscure for Lightroom.


----------



## Brad Snyder (Nov 25, 2010)

Rather than obscure, I'd say it's a valid request, but likely to be at the low end of the demand curve.


----------



## MarkNicholas (Nov 25, 2010)

I'm happy with "obscure".


----------



## b_gossweiler (Nov 25, 2010)

[quote author=matsmithphotog link=topic=11661.msg78312#msg78312 date=129'37'221]
I shoot RAW. At 5616 x 3744. 16 bit.
[/quote]

Mat,

May I ask what camera you're shooting these Raws with? (16 bit?)

Beat


----------



## matsmithphotog (Nov 28, 2010)

[quote author=MarkNicholas link=topic=11661.msg7856'#msg7856' date=129'654957]
I'm happy with "obscure".
[/quote]

Call it what you will, it was just a question.

[quote author=b_gossweiler link=topic=11661.msg78565#msg78565 date=129'668156]
May I ask what camera you're shooting these Raws with? (16 bit?)
[/quote]

I meant I'm shooting in RAW and dithered up to 16 bit in PS, I think most people knew what I meant!

(Actually I have a camera that records the entire visible spectrum ...  )


----------



## simondebrun (Jul 6, 2011)

For me this request is far from obscure. What Matt is looking for is exactly what I want. I'm surprised it's not an option.

All this talk about hard-disk space is cheap is fine if you're a pro. I'm an amatuer -- I take pictures because I love photography -- 90% of my images are of family -- yet last month I took 16gb of stills (Nikon d7000 raw). Once I've processed my files I'd love to be able to create jpegs of them in a single click. I use my macbook pro for all my photo editing, and display. I don't want to carry multiple hard-drives with me.

In an ideal world there would also be an option to have the converted jpeg link to a raw file on a separate disk when plugged in. Of course, now I'm just dreaming. 

For now the export workflow described seems like a workable but cludgy hack. 

If any LR developers are listening please add 'convert to jpeg' in your to-do list. It would be super simple to implement and please a lot of people.


----------



## Replytoken (Jul 6, 2011)

simondebrun said:


> All this talk about hard-disk space is cheap is fine if you're a pro.
> 
> ===============
> 
> In an ideal world there would also be an option to have the converted jpeg link to a raw file on a separate disk when plugged in. Of course, now I'm just dreaming.



Your request is a bit confusing to me.  You are initially concerned about the amount of storage that you need, but then you mention a link from a JPEG file to the raw image.  This arrangement that you are dreaming about does not reduce your need for storage.  Are you concerned about your total storage, or the size of the files that LR needs for a catalog that you wish to show others on your laptop?  If it is the latter, why not have a master catalog of all of your raw files, and also have a "portfolio" catalog that is a subset of your master catalog, and that uses JPEG copies of the processed images.

--Ken


----------



## Victoria Bampton (Jul 7, 2011)

Hi Simon, welcome to the forum!

So just to confirm, are you actually asking to SWAP a raw file for a converted JPEG, deleting the raw file in the process?  I'll be honest, I don't think the devs are ever likely to do that, because it would be far too easy for people to delete their raw files without backups, and then complain that Adobe deleted them without them realising.  That said, you could put in a request direct to Adobe on the Official Feature Request/Bug Report Forum where users can vote for it.  They are listening.


----------

