# Sidecar Files



## Wasabi Bob (Sep 22, 2010)

I am a relatively new LR user who just upgraded to 3.2 from 2.'
In 2.' RAW and JG files appeared separately, while in 3.x they are grouped together.
My understanding is that LR creates a "sidecar" file, which I assume is the JPG file?

My questions is, if I select 3 RAW files and apply the development settings and export them is LR exporting the RAW files or the JPG?

How about when I want to use the JPG files, how do I select which file type is exported?

Is there any way that I can have 3.x show both file types separately?


----------



## Jim Wilde (Sep 22, 2010)

Hi Bob, welcome to the Forum.

Go to Edit&gt;Preferences&gt;General Tab.....in the Import Options section there is an option to "Treat Jpeg files next to Raw files as separate photos". To have Lightroom treat the RAW and associated JPEG as two indepedent photos that you can develop and export separately, then you need this option to be checked.

Note, however, this is NOT a retrospective action, so any RAW+JPEGs that you have imported already will NOT suddently appear as two photos.

BTW, for existing RAW photos with the JPEG 'sidecar', you will only ever work on the RAW....to all intents and purposes the associated JPEG cannot be referenced in Lightroom.


----------



## MarkNicholas (Sep 23, 2010)

Sidecar files are the xmp files created by LR (if to choose to do so in the catalog preferences) to record the edits you make to your RAW files. They are not the Jpeg files.


----------



## Jim Wilde (Sep 23, 2010)

[quote author=MarkNicholas link=topic=111'9.msg74831#msg74831 date=12852514'1]
Sidecar files are the xmp files created by LR (if to choose to do so in the catalog preferences) to record the edits you make to your RAW files. They are not the Jpeg files.
[/quote]

There are many different types of "sidecar" files, XMP being just one of them. Another type is RAW+JPEG, and when the option to "Treat Jpeg files next to Raw files as separate photos" is unchecked, Lightroom processes the JPEG as a 'sidecar' file, and records it as such in the RAW metadata.

See this article: http://en.wikipedia.org/wiki/Sidecar_file


----------



## MarkNicholas (Sep 24, 2010)

In LR there is only "one" type of side car file and that is the XMP file.


----------



## clee01l (Sep 24, 2010)

[quote author=MarkNicholas link=topic=111'9.msg74881#msg74881 date=1285298132]
In LR there is only "one" type of side car file and that is the XMP file.
[/quote]This is very much not true. Jim Wilde is correct. If you shoot RAW+JPEG as did the OP, and import them with "Treat Jpeg files next to Raw files as separate photos" unchecked, LR will list the JPEGS as Sidecar files in the Metadata panel. Somewhere in my catalog I have files listed just like that when I made that mistake.


----------



## Mark Sirota (Sep 24, 2010)

There are also audio files for some cameras, which are handled as sidecars.

And didn't some really old cameras use sidecar JPEGs rather than embedded JPEGs for raw files? Though I guess that's basically just like today's cameras when you shoot raw+JPEG.


----------



## MarkNicholas (Sep 24, 2010)

I stand corrected ! I do not shoot Jpeg anymore and so missed this feature.


----------



## Victoria Bampton (Sep 24, 2010)

[quote author=Mark Sirota link=topic=111'9.msg74888#msg74888 date=12853'5272]
And didn't some really old cameras use sidecar JPEGs rather than embedded JPEGs for raw files?
[/quote]

Yes, the original Canon cameras, i.e. 1'd, had .thm thumbnail files too.


----------



## carbonwolf (Oct 27, 2010)

Hi, I'm also fairly new to Lightroom and have a few related questions:

Is it generally accepted as best practice to import only RAW files and export JPGs as required? If so, is there any point shooting RAW+JPEG in camera?

When using the LR Flickr publisher with RAW files only I assume the RAW is automatically converted to a JPG before uploading?


----------



## Jim Wilde (Oct 27, 2010)

Hi, welcome to the forum.

My opinion only, but there's no such thing as 'best practice' in this particular area.....everyone develops their own workflow according to their individual needs. Sure, many will only shoot RAW (I'm in that camp), but equally there will be many who shoot RAW+JPEG, and there are also many who only shoot JPEG (in many cases because that's all their camera might allow).

I do understand that shooting RAW+JPEG - *and importing them as separate photos, i.e. not 'paired'* - allows the busy photographer to take a quick look at his images and send off the JPEGS for client review if these are 'good enough', and only spend valuable time post-processing on the subsequently selected RAW files. 

But I absolutely do not "get" shooting RAW+JPEG and them importing using the default setting of treating the JPEG as a sidecar file that you can neither see or otherwise process in Lightroom. I personally don't see the point, but I have little doubt that others here probably do.

Your last assumption is correct, I believe.


----------



## tzalman (Oct 27, 2010)

> If so, is there any point shooting RAW+JPEG in camera?


I have never seen the point of RAW+jpg except for somebody on a tight deadline who needs immediate jpgs. If you are using the camera maker's software a jpg identical to a camera-made one can be had in seconds. Using LR, the default rendering will be different, although with a little work a preset that is a close approximation can be created. But if someone thinks the camera rendering is so great, why would he bother with RAW and LR?



> When using the LR Flickr publisher with RAW files only I assume the RAW is automatically converted to a JPG before uploading?


Yes. A RAW file is not a color image file. Only after conversion can there be an image for uploading.


----------



## carbonwolf (Oct 27, 2010)

@TNG:

Many thanks for that. I see what you mean; 'horses for courses' as it were.

I've been importing CR2 and JPEGs in 'paired' mode for a short while without realising, until very recently, the implications of this. Specifically, I wasn't aware that the JPEG's only function was to act as a sidecar file when imported this way.

I think that, rather than making an immediate jump to RAW only shooting, I'll still import the JPEGs but as separate images (not paired). That way I can have a play with the LR developer settings and see how my processing compares with the camera-generated JPEG. If I can easily replicate (or improve upon) the original JPEG I'll probably stop shooting JPEG altogether.

@tzalman:

Thanks also.

I guess my reluctance to go 'RAW only' has partly stemmed from a lack of confidence in my ability to produce a processed RAW as good as or better than the camera's JPEG. I realise now that (1) if I don't try, I won't know and (2) using that mentality means I might as well shoot JPEG only and forget about the RAW, which would also make using LR a little pointless. As I want to improve as a photographer, that's certainly no way to go about things.

So, I'll try what I mentioned above and see how I go.

Thanks again for your advice.


----------



## carbonwolf (Oct 27, 2010)

One more question/clarification, if I may? Am I right in that setting a color space on my camera only applies to the JPEGs it produces (i.e. RAW/CR2 files have no set color space)?


----------



## Jim Wilde (Oct 27, 2010)

Yes, and this is also true for the majority of in-camera settings.....which is why a lot of people may initially think that their RAW files look absolutely horrible in relation to the SOOC JPEG. Have a quick look through these threads, which are mainly talking about a SONY camera but the underlying issues/thoughts are equally applicable to your Canon camera:

http://www.lightroomqueen.com/community/index.php?topic=11315.'
http://www.lightroomqueen.com/community/index.php?topic=11353.'

The further you go into these threads the more useful information you will glean about the RAW vs JPEG issue.


----------



## carbonwolf (Oct 27, 2010)

Thanks again, Jim, much obliged for your help. I'll have a read through the other posts now.


----------



## Jim Wilde (Oct 27, 2010)

No worries. Come back if you have more questions, this is a great place for having them answered.....though you will notice that quite often there is no "right way", merely different opinions about what is best. But even when opinions differ, its almost always done in a very civil manner, which is great.


----------



## carbonwolf (Oct 27, 2010)

I certainly will do. It makes a change to use a forum where people take the time to give polite, considered replies; even when the questions are somewhat basic!

I know what you mean about there being no "right way". Photography is all about personal taste and the reasons for taking photos range from holiday snaps to pro photography so different people will all have their own "right way".

By the way, and going slightly off topic, I notice you have a 5D MkII and wondered if you could recommend it? I'm saving up to upgrade my 45'D and am torn between the 7D and 5D. I'm not particularly interested in video so that isn't really an issue. I must say that the 5D's full-frame sensor and 21MP are very appealing but, aside from the extra £5'' cost over the 7D, I have to consider that I would probably have to upgrade my lenses to 'L' series ones in order to fully benefit, which obviously adds considerably to the price.

I've also heard that the 5D uses an older AF system than the 7D which, according to some, struggles in certain situations. Others say that this is either not the case at all or only manifests itself in situations involving things moving at extreme speed in relation to the camera. Could I ask what your take is on this?

I suppose that, no matter which camera I decide to get, there's always going to be a more expensive version with a higher megapixel count (not always an indication of quality, I believe), a better sensor, etc. But, in the end, none of these things is a replacement for ability so the choice will ultimately boil down to what I can afford (and justify). It would still be interesting to hear your opinion though, if you care to give it?


----------



## Jim Wilde (Oct 27, 2010)

Ahh....good question! 

Would I rcommend the 5D2....absolutely! But I'm sure the 7D owners would recommend their model as well. I think they are both great cameras, albeit meeting different needs. I drove myself mad for a few weeks while trying to decide between the 7D and the 5D2, every comparison review I read tended to cause me to switch back and forth between the two. Each have their own strong points, and I guess the key is not to be blinded by one or two particular features (which is probably what I was initially doing), instead focus on what's important to you and your photography. For me it eventually came down to the extra detail that a full-frame sensor should be providing, even though knowing that I would really need L glass to make best use of it, which I'm slowly adding to the camera bag. Would love to have been able to buy both though! Maybe when the 7D's replacement comes out....

Regarding the AF 'issue', can honestly say I've never had a problem yet....though I've never trying shooting a formula 1 car! Plenty of other 5D2 users on this forum, no doubt they may have something to add.


----------



## LouieSherwin (Oct 28, 2010)

Hi,

OOps... off topic but here goes.... 

I think your choice really depends on what kind of pictures you mostly take. If your primarily action sports than the 7D is your best bet. On the other hand if you are mostly taking landscape, portrait and especially low light the the 5DmkII is likely your choice. as far as image resolution they are pretty much identical. 

A big difference is the sensor noise. This is a simple fact of physics. The smaller pixels simply can't collect the same quantity of electrons so their signal to noise ratio is higher. LR3 handles this as well or better than the in camera jpg processing. For normal iso 1''-4'' this is probably not a big deal but 8'' and above you will begin to see noticeable differences.

I have been using the following lenses on my 5dmkII. 5'mm f1.4, 1''mm f2 and 28mm f1.8. This is a killer set for a fraction of the cost of big heavy L zooms. You just have to keep in mind for the 5'mm and 28mm to stop down if edge to edge sharpness is important. All of them are up to the task of meeting the resolution of the sensor. 

-louie


----------



## MarkNicholas (Oct 28, 2010)

[quote author=LouieSherwin link=topic=111'9.msg76795#msg76795 date=1288228437]
I have been using the following lenses on my 5dmkII. 5'mm f1.4, 1''mm f2 and 28mm f1.8. This is a killer set for a fraction of the cost of big heavy L zooms. You just have to keep in mind for the 5'mm and 28mm to stop down if edge to edge sharpness is important. All of them are up to the task of meeting the resolution of the sensor. 
[/quote]

Very nice lens set... but what you have are relatively cheap fast lenses for which there are no comparable Canon zooms. The nearest would perhaps be the 24-1'5mm/f4 which is only about 75% the cost of your 3 lenses and has IS and is an "L". Of course the f/4 does not compare with your f/1.4, F1.8 and f2.


----------



## carbonwolf (Oct 28, 2010)

Can open, worms everywhere!

I think I'm most likely to land on the 5D2 side of the fence, mainly due to the larger sensor. Although I don't currently have L lenses, I can always replace what I have gradually as finances allow. Thanks for all your input.

Bringing the conversation back to LR for a moment, am I correct in thinking that developer adjustments made in LR are stored in the catalogue unless manually exported to sidecar files? And, if that is the case, how concerned should I be about catalogue corruption? I back my catalogue up to an external drive and to an online backup source so I'm not too worried about recovering it quickly but if partial corruption occurred my backups could potentially be overwritten before I realised that corruption had occurred. Should I export my processed RAW images as TIFFs or would you consider this to be overkill?


----------



## b_gossweiler (Oct 28, 2010)

Yes, your development adjustments are stored within the catalog, and, if you do "Write Metadata to File" (which can be automatically done by saying so in Preferences), most of it is also written into XMP sidecar files or XMP sections in DNGs, JPEGS, PSDs, TIFFs.

Writing metadata to file is generally looked at as a last resort "_backup_", because the XMP data does not contain all the information available in the catalog for a photo (no stacks, virtual copies, develop history, collection memberships, flags).

So your primary issue should be backing up the catalog, and not overwriting older backups by newer ones as you go along (LR provides the date based backup folders for that reason).

I personally would consider exporting developed TIFFs an overkill. Make sure your catalog backup system is waterproof.

Beat


----------



## carbonwolf (Oct 28, 2010)

That's good to know. I'll enable date based backup as soon as I get home tonight as that definitely sounds like the best solution.

My apologies if some of these questions are quite basic. I'm not getting much time to experiment with LR at the minute so the more I can learn about it while I'm at work the less of what little spare time I have will be 'wasted' working out settings and changing workflow and the more I can spend actually using LR to produce better photos.


----------



## Jim Wilde (Oct 28, 2010)

Answering your last question first, I would agree that exporting all developed RAWs as TIFFs - *purely for backup/contingency purposes* - would be overkill.

However, first things first.....there is no "unless" regarding your develop edits. These are ALWAYS stored in the catalog, though you can ADDITIONALLY write most, but not all, metadata changes to XMP sidecar files. I'm not going to get into the pros and cons of that particular debate, that's had plenty of recent air time on this forum and I suspect it will rage on and on (have a look at this thread for some of the arguments). FWIW I myself do not write to XMP sidecars.

While I've been typing I see that Beat has already replied, so I'll not add any more other than to support what he says about your catalog backup routine. I use my own backup system using SyncToy, which I run after every Lightroom session to backup BOTH the catalog and any new imports. But with this system there is a danger that initially undetected catalog corruption (which I've never had) could be propogated into my backups, making the catalog backup worthless. To protect against this I also use Lightroom's own backup system (which ADDS, not replaces, dated backups to the "Backups" folder alongside the catalog)....and I include this folder in my SyncToy backups - so I am backing up the many dated backups as well. My backup strategy also includes an offsite element (nothing fancy, stored at the Mother-in-Laws). Over time, if not attended to, the number of Lightroom-created backups can get VERY large (especially if done daily) so you need to remember to occasionally go in and cull some of them, but making sure you keep sufficient to do the job if needed.


----------



## Wasabi Bob (Sep 22, 2010)

I am a relatively new LR user who just upgraded to 3.2 from 2.'
In 2.' RAW and JG files appeared separately, while in 3.x they are grouped together.
My understanding is that LR creates a "sidecar" file, which I assume is the JPG file?

My questions is, if I select 3 RAW files and apply the development settings and export them is LR exporting the RAW files or the JPG?

How about when I want to use the JPG files, how do I select which file type is exported?

Is there any way that I can have 3.x show both file types separately?


----------



## carbonwolf (Oct 28, 2010)

My apologies, I meant to put "...developer adjustments made in LR are only stored in the catalogue unless manually exported to sidecar files".

I use a similar system to yours for backing up my catalogue (and other files) using SyncBack for local backups and Mozy for offsite backup but I'm pretty sure that I don't have the 'dated backups' option enabled in LR so it looks like a quick tick in that box when I get home should cover me as far as backups are concerned. As you say, I just need to remember to have a clear out of the older files every so often, I could probably use an automated script/batch file to do that based on the file dates.


----------



## Jim Wilde (Oct 28, 2010)

Just a point of clarification about Lightroom's dated backup system.....it's not actually the catalog backup file that is dated: when creating a backup Lightroom puts a copy of the catalog, *with the exact same name as the original*, in a *dated sub-folder *in the Backups folder. See attached screenshot.


----------



## carbonwolf (Oct 28, 2010)

Thanks for that, Jim. I think I'll knock up a quick Autohotkey[sup]1[/sup] script to automate deletion (probably based on the year and month in the folder name).

[sup]1[/sup]http://www.autohotkey.com/


----------



## b_gossweiler (Oct 28, 2010)

[quote author=carbonwolf link=topic=111'9.msg76816#msg76816 date=1288259'58]
Thanks for that, Jim. I think I'll knock up a quick Autohotkey[sup]1[/sup] script to automate deletion (probably based on the year and month in the folder name).

[sup]1[/sup]http://www.autohotkey.com/
[/quote]

Also, LR catalogs, and therefore also their backups, are very well compressible with WinZip, as they almost only contain text data. If you zip your backups, you'll end up with a fraction of the space used by the catalogs themselves.

Beat


----------



## Brad Snyder (Oct 28, 2010)

[quote author=b_gossweiler link=topic=111'9.msg768'8#msg768'8 date=1288254739]

I personally would consider exporting developed TIFFs an overkill. Make sure your catalog backup system is waterproof.

[/quote]
I know one pro who maintains a complete additional library backup of his commercial work in 16-bit TIFF, just so he can always get back to *exactly* the image he sold. He points to evolution in the Lr rendering engine as a potential problem in *exact* backwards compatibility with previously processed images. I would call his use case a statistical outlier, but he has given this a lot of thought, makes consistent arguments, and is aware of the workflow costs in storage and productivity.


----------



## carbonwolf (Oct 29, 2010)

@Beat

That's good to know, I think Syncback has an option to compress on backup so I'll have a look at that, thanks!

@Brad

I can see why, for a pro who needs to keep a back catalogue of his commercial work, keeping exact TIFF copies of sold work could be attractive, especially as storage is relatively cheap these days. I think for me personally it would definitely be overkill but it's good to know that the option exists to fix an image and it's adjustments in stone, as it were, without compromising on quality.


----------



## carbonwolf (Oct 29, 2010)

Continuing on with the RAW-only theme. I've uploaded a screenshot to http://drop.io/akgnmd7 showing a zoomed-in section of a RAW file (left) and JPEG (right). On my monitor at home, which I've calibrated using an X-Rite Eye One Display 2, the RAW shows noticeable grain, even though it was only taken at ISO 2''. The JPEG shows no grain but is far softer and much less defined than the RAW.

However, looking at the two images here at work, and on a cheaper, uncalibrated monitor, both images look almost pure black unless I zoom right in, at which point I can see the small, sharp points of colour on the RAW and the 'blockyness' of the compression of the JPEG.

My question is, can anyone else see any difference between the images on their monitors at normal zoom levels?


----------



## carbonwolf (Oct 31, 2010)

Anyone?


----------



## Jim Wilde (Oct 31, 2010)

At 1''% zoom level, I see two black boxes....though it's possible that to my untutored eye the one on the right (the JPEG?) is not quite as dark as the one on the left. Looking at the RGB values in Photoshop I can see that the left box is more variable, some points being pure black (average values &lt; 1'), and the one on the right is indeed consistently a bit lighter (average values &gt; 1').


----------



## carbonwolf (Nov 1, 2010)

Thanks for having a look at this, Jim. I've dropped the brightness on my home monitor and that seems to have brought it more into line with what the photos look like on other monitors.

I got to spend a good few hours experimenting with Lightroom over the weekend and I must admit I'll never go back to SOOC JPEGs now. I managed to get hold of a copy of Scott Kelby's LR3 book which has helped a great deal but without the advice I got here on the forum I think I'd still be battling with some of the concepts so many thanks once again for your invaluable advice


----------



## Jim Wilde (Nov 1, 2010)

Back in my early daya with LR2 I bought both the Kelby and Martin Evening's books, and I found both to be helpful (although I had a definite preference for the Martin Evening book). I have to say, though, that since I came across this forum at the beginning of this year both books have been virtually untouched. I know that any questions I might have have already been asked and answered somewhere here (out of habit I read every new post, so it's a constant learning experience). The only book I now use is Victoria's "Missing FAQ", which I would definitely recommend if you are planning on using Lightroom as your DAM (for the link and discount code, check Victoria's sig in the post just before your first one in this thread).


----------



## carbonwolf (Nov 1, 2010)

I suspect that might be the case for me too once I've got my teeth stuck into things properly. I'll have a look at the Martin Evening and Missing FAQ book though; the more sources of information the better at the moment!


----------

