# Is the new DNG "fast load" actually faster?



## turnstyle (Jan 14, 2012)

Hi all,

So far, I've been keeping my camera RAW files in the camera-specific RAW format.

But I'm interested in some of the new DNG features, and so I'm considering converting my camera-specific RAW files into DNG.

Is the new DNG "fast load" actually noticeably faster? What exactly does it speed up, and by how much? It is "noticeable"?

I'm also tempted to use the lossy compression for all the not-so-great pictures that I can't quite throw away.

(It would also be nice to be able to update the embedded preview, which I can't do with my camera-specific RAW file.)

Last time I considered switching to DNG, it didn't seem necessary -- but it's getting more tempting...

Thanks for any thoughts!


----------



## Hal P Anderson (Jan 14, 2012)

The Fast Load data helps speed up the initial load of the image in the Develop Module. It doesn't help with anything else. And I haven't noticed any speed up with it. For me, it's snappy, anyway. 

DNGs keep adding advantages. A nice one is the newish internal checksum that gives a warning when the data has become corrupted. Eric Chan on the Adobe forum describes DNG as a wrapper that can enclose any of the filetypes that LR handles, including JPEGs. JPEGs have their data simply copied into the DNG without a decompression-recompression step that would lose data.

Personally, I'm thinking strongly of converting my collection to DNG, many of which would be lossy-compressed.

Regarding lossy DNGs, produce a few, get a friend to re-name both them and their parents randomly and see how well you do telling them a part in a "blind" test. 

Hal


----------



## turnstyle (Jan 14, 2012)

For the bulk of my photos, I don't mind a bit of lossy compression -- the big plus (for me) of RAW is the ability to fix white balance, recover highlights, etc. If I can still do all that with lossy DNG files, then that's good enough for most of my pics -- seems to combine the advantages of RAW and JPG.

So Fast Load does make it snappier if you're flipping though your catalog? For example, for non-cached files? And for cached files too? ie, will I see less "Loading..."?


----------



## Hal P Anderson (Jan 14, 2012)

No, I said I didn't notice a difference. I've got a snappy computer, so the loading bezel flashes on and off very quickly, anyway. I'd be interested to hear how it goes for you in a test.

Hal


----------



## turnstyle (Jan 14, 2012)

I don't actually have the Beta -- but if anybody else here has indeed tried it, please do post your observations!


----------



## Victoria Bampton (Jan 14, 2012)

Imagine that fast load data as removing the need for the ACR cache - which holds partially processed data.  The first time you look at a DNG without fast load data, it has to read the full data, do initial processing, and then store that in the ACR cache. Future times it reads the ACR cache first so it's quicker.  If that rolls out of the ACR cache before you next come to it though, it has to start again.  Fast load data, on the other hand, is stored in the file, so it only has to do it that the first time round and in future it only reads it.  The bigger or more complicated the file (certain sensors, certain opcodes like forced lens corrections), the greater the advantage.


----------



## turnstyle (Jan 14, 2012)

Since you mentioned lens corrections -- does that mean the fast load data within the DNG file itself will change if something like a lens correction is enabled/disabled?

btw, do you happen to know of any videos/web pages that specifically demonstrate these new features of DNG?


----------



## Victoria Bampton (Jan 14, 2012)

Nope, the opcodes are for things like some Panasonic cameras which insist on their lens corrections being applied - you can't turn them off and they're performed much earlier in the pipeline.  It doesn't apply to LR's lens corrections.

I haven't seen much on the new DNG stuff yet.  It's on my to do list but the last week's been insanely busy, so it might not get done until the final release is out, although I'll try to get it done in the meantime.


----------



## Hal P Anderson (Jan 14, 2012)

I'm pretty sure that Victoria was referring to some of the more recent point-an-shoot cameras that do lens corrections in software for their JPEGs. Raw converters (including LR) automatically do that lens correction when they render the raw file. You don't have to ask for it. That's what would be sped up by the fast-load data.

There are a couple of videos, one with Julianne Kost on Adobe TV and one on Lynda.com that talk about the new DNG features:
http://www.lynda.com/Lightroom-4-tutorials/Photoshop-Beta-Preview/96215-2.html
http://tv.adobe.com/show/whats-new-in-lightroom-4-beta/

Actually, all those videos in those series are worth watching.

Hal


----------



## turnstyle (Jan 17, 2012)

Thanks, I hadn't known about those Lynda videos. It didn't actually seem to show Fast Load Data making anything faster -- and another detail that struck me: it appeared that using Lossy DNG might slow things down. In any case, I'm glad to see DNG progressing.


----------



## Hal P Anderson (Jan 17, 2012)

If lossy DNG slows things down, it won't be by much. Essentially, lossy DNGs are JPEG-compressed, and decompression of a JPEG seems blindingly fast on any modern hardware.

Hal


----------



## turnstyle (Jan 17, 2012)

Hal P Anderson said:


> If lossy DNG slows things down, it won't be by much. Essentially, lossy DNGs are JPEG-compressed, and decompression of a JPEG seems blindingly fast on any modern hardware.
> 
> Hal



I'm referring to a bit after 6 minutes into this video:
http://www.lynda.com/home/Player.aspx?lpk4=96419&playChapter=False

When he zooms into the lossy DNG, it seems to sit on "Loading" for about twice as long, no?


----------



## Victoria Bampton (Jan 17, 2012)

That may be a beta bug.


----------



## turnstyle (Jan 17, 2012)

Victoria Bampton said:


> That may be a beta bug.



It'll be interesting to find out -- I'll be tempted to use the lossy option for some of my photos -- but if it takes noticeably longer to open them, I'm not so sure I'd bother.

And if anybody does find a video (or any other info) that demonstrates the real-world benefit of Quick Load Data, please do post!


----------



## Hal P Anderson (Jan 18, 2012)

I converted a folder's worth of images to lossy DNGs with the fast-load option. The size on disk went from 615 MB to 241 MB, which is a sizable reduction, indeed. Going from one to the next in the Develop Module seemed to be neither faster nor slower than in a folder with unconverted NEFs. In all cases, the "Loading" message appeared for somewhere around half a second.

Hal


----------



## turnstyle (Jan 18, 2012)

Great info, thanks! fwiw, that came out to a bit less compression than I had expected (I thought it would be closer to 1/4 the original).

If you're in a testing mood, I'd be interested to know if fast-load, non-lossy DNGs actually seem faster than traditional DNGs -- in this case, Adobe describes them as 8 times faster, so it seems that should be noticeable.

Thanks again. :hail:


----------



## Hal P Anderson (Jan 18, 2012)

Yes, they are faster. Maybe twice as fast. If I had a slower computer, I'd have a chance to do better timings, but having quick-load definitely makes a difference, at least when the images aren't in the Camera Raw cache. 

Hal


----------



## turnstyle (Jan 18, 2012)

Thank you Hal, you've shown patience in helping me.


----------



## watsonm (Jan 23, 2012)

*Lossy DNG*

Anyone tried converting existing DNGs to the lossy format.  
If I try the lossy option is greyed out so I assume I cannot convert which seems a bit of a pain.   
Maybe I'm not setting things up right or it can only be done on import


----------



## Hal P Anderson (Jan 23, 2012)

watsonm,

That's a bug that has been reported at the Adobe site. Sometimes it's greyed out, and sometimes not. I've managed to get it to work, although earlier in the same session, it wouldn't. 

Hal


----------



## watsonm (Jan 23, 2012)

Well I tried on my laptop and it worked.  So went back to Main machine and when I tried again I noticed the pop up panel  had the previous  Camera Raw settings of 5.4   .   I flipped it to 6.6 and the "use lossy Compression" check box "Ungreyed" itself.  I selected OK and it worked.

Next experiment was to 
Select several images  in the film strip and do the same but it would only convert one image.  
Selected them in the grid view and it converted them all.


----------



## turnstyle (Jan 23, 2012)

How do you feel about the compressed DNGs? Notice anything about the quality? Or performance flipping through in LR?


----------



## watsonm (Jan 23, 2012)

I'll cut and paste a reply I made in flickr:

"I tried it tonight.   I reduced three images I use to make an HDR snap  to lossy DNG.  I then selected them and exported them to Photomatix.
On re import I could not tell the difference bewteen the two shots  either at initial re import or if I synched the post processing.  That  was at 4:1.  Not a conclusive test I know but seeing I do a lot of  landscape HDR it was relevant to me.   

Better test would probably be a couple of A3 prints from a complete file before and after conversion"

As a subjective test I didn't notice any major performance difference in selecting  a lossy or lossless DNG in lightroom and from the screen could not see any difference between images either.  So for non critical stuff I suspect I may move over when the final release comes along.  But then I am just an  amateur  that only prints up to A3 for personal use.


----------



## Victoria Bampton (Jan 24, 2012)

If you've got Photoshop Mike, try opening the original and lossy DNG in PS and putting them as layers on top of each other.  Then change the blend mode at the top of the layers panel to Difference mode, and you can see exactly what the difference is.  On the ones I tried, I was surprised to see how little difference there was, even at 100% view.


----------



## turnstyle (Jan 24, 2012)

Victoria Bampton said:


> If you've got Photoshop Mike, try opening the original and lossy DNG in PS and putting them as layers on top of each other.  Then change the blend mode at the top of the layers panel to Difference mode, and you can see exactly what the difference is.  On the ones I tried, I was surprised to see how little difference there was, even at 100% view.



hah, I used to do that -- fwiw, I'd also use the magic wand with tolerance set to zero -- that can give a nifty visual showing how the compression is applied.


----------



## turnstyle (Jan 14, 2012)

Hi all,

So far, I've been keeping my camera RAW files in the camera-specific RAW format.

But I'm interested in some of the new DNG features, and so I'm considering converting my camera-specific RAW files into DNG.

Is the new DNG "fast load" actually noticeably faster? What exactly does it speed up, and by how much? It is "noticeable"?

I'm also tempted to use the lossy compression for all the not-so-great pictures that I can't quite throw away.

(It would also be nice to be able to update the embedded preview, which I can't do with my camera-specific RAW file.)

Last time I considered switching to DNG, it didn't seem necessary -- but it's getting more tempting...

Thanks for any thoughts!


----------



## watsonm (Jan 27, 2012)

Victoria Bampton said:


> If you've got Photoshop Mike, try opening the original and lossy DNG in PS and putting them as layers on top of each other.  Then change the blend mode at the top of the layers panel to Difference mode, and you can see exactly what the difference is.  On the ones I tried, I was surprised to see how little difference there was, even at 100% view.




Tried that Victoria... very impressive for a DNG file 1/3 the size of the original.


----------



## turnstyle (Jan 27, 2012)

turnstyle said:


> hah, I used to do that -- fwiw, I'd also use the magic wand with tolerance set to zero -- that can give a nifty visual showing how the compression is applied.



A minor correction -- I don't think the wand would work if you combine the images via separate layers -- I used to use "difference" to generate a 3rd image, and then used the wand on that.


----------



## sizzlingbadger (Jan 27, 2012)

This is interesting...  http://chromasoft.blogspot.com/2012/01/lightrooms-new-lossy-dng-compression.html

... and reinforces my sentiments about lossy DNG, its a great idea but many people assume that DNG means its a raw file and this may lead to some problems. Its an education issue not an issue with the technology itself. DNG is used predominately to encapsulate raw data and it is often mistaken as another raw format rather than a "container" for many file types.

The experiments I have done so far with lossy DNG (or clever JPG as I like to think of it) have given very good results.


----------



## JimHess43 (Feb 5, 2012)

I watched the Julienne Kost video, and decided to give the compressed DNG a try. Now, admittedly, I don't have the best computer. But I didn't see any increase in performance. And the video didn't mention anything about the compressed DNG not containing raw data anymore. It seems to me that Adobe is pushing DNG so hard (at least by their trainers) that they will neglect to provide those kinds of important details. I tried the compressed DNG, reduced the size of my files by about 30%, no increase in performance. Sticking with my original thoughts of ignoring DNG completely. Thankfully, I was working on copies of the images.


----------



## turnstyle (Feb 5, 2012)

I don't think compressed DNG is necessarily expected to increase performance -- unless, perhaps, you access your RAW files over a local network.

The performance improvement is supposed to come via selecting the "Quick Load" option, which is independent of compression -- but both are traits of the new DNG.

Also, when you say you saved roughly 30%, you mean that a roughly 10MB only wound up as 7MB when compressed?


----------



## JimHess43 (Feb 6, 2012)

Yes, as far as size reduction is concerned, that is exactly what I meant. I watched the video, and I guess I misinterpreted what she said. The compressed DNG files didn't seem to load any faster, and that's what I thought they were supposed to do. I still haven't found any compelling reason to convert to DNG on a regular basis. I thought this would be a reason, but after experimenting with it a little bit I'm still not going to convert.


Disclaimer: I don't take a lot of pictures so worrying about disk space isn't a big issue for me. I think I only took something like 5000 images last year. I'm just interested in Lightroom, and enjoy using it.


----------



## Victoria Bampton (Feb 6, 2012)

That's an excellent point Jim, that definitely needs making clear.  Thanks for bringing it up.

Personally, I see DNG as a workflow tool.  My originals will stay as proprietary raw or DNG full res, but lossy DNG's will be handy for taking out on my laptop or keeping in the cloud.


----------



## JimHess43 (Feb 6, 2012)

I did notice something, however.  Even though the image data are not raw in the compressed DNG, I still have access to all of the camera profiles and other controls that are unique to raw images.  This is interesting and calls for a little rethinking.  If I do the majority of my post processing on the original NEF file, then I suppose minor adjustments in the DNG file wouldn't be so bad.  Am I thinking correctly?


----------



## Hal P Anderson (Feb 7, 2012)

Jim,

I don't think it works that way. The squished DNG won't be the result of baking your adjustments into your original raw image, as if you were exporting a JPEG. The conversion will act on the raw file without any of the adjustments applied. The final image will be the same whether you do a little or a lot of work before converting.

Hal


----------



## JimHess43 (Feb 7, 2012)

Perhaps I misspoke in my previous comment, or else I am misunderstanding you, Hal. I just converted a single NEF to DNG using lossy compression. This is after I had made a number of adjustments in sharpness, color, contrast, etc. Oddly, this time the file size was reduced by 50%. The only difference is that last time I converted a number of images. Not sure what is going on there. In the resulting compressed DNG file, all of the adjustments I made before converting were still in place. So, with my limited knowledge, it seems to me that if I take the time to post process my images before converting to DNG then (in most cases for me) any minor adjustments that would be required could effectively be applied to that DNG file.


However, I'm still uncomfortable doing any DNG conversion with or without compression. I just feel as though I'm giving up some options, and I don't like that when it really isn't necessary. The conversion might be a reasonable choice for questionable images that I might want to hold onto anyway. But for my important "five-star" images I don't think conversion is a good idea.


----------



## Victoria Bampton (Feb 7, 2012)

The difference is, a standard DNG file takes the sensor data in its original format and wraps it in a DNG wrapper.  No processing applied, just a different container with some lossless compression.

A lossy DNG takes the same mosaic sensor data, but does the very early stage processing (demosaic) and then wraps that in a DNG wrapper.  Any slider adjustments made in LR make no difference at all to the stored data - they're just stored as xmp text metadata instructions (like xmp sidecars, but in the file header) and the embedded preview can be updated to the new settings.  That early-stage-processed-data then has JPEG compression applied, which shrinks it down with surprisingly minimal loss of quality because of the way it's stored.  It's a cross between a JPEG and a raw file - still behaves like a raw file, but it's smaller.  So yes, handy for questionable images and workflow.

For standard DNG, there's no quality loss - the only thing you're giving up is compatibility with some other programs, and the gains are things like embedding updated previews, no sidecars but metadata embedded (you can update that at any time, not just conversion), slightly smaller files in some cases, and most importantly, the DNG Hash which can let you know if there's even minor corruption.

For lossy DNG, there is quality loss, although minimal.  I wouldn't use it for archiving my 5 stars, or actually, considering the amount of disc space I have available, for archiving anything.  But it'll be handy for having access to files on my laptop, or for people who shoot vast quantities (i.e. timelapse) but still want files that are more editable than JPEGs.


----------



## JimHess43 (Feb 7, 2012)

Thank you for that explanation, Victoria.  I know a lot of the Lightroom presenters on Adobe TV and on Lynda.com really push the DNG format.  Personally, I haven't seen anything to really compel me to use it as part of my standard workflow.  It is, however, nice to have it as an option.


----------



## Victoria Bampton (Feb 7, 2012)

It's good timing, I've been rewriting the DNG section of my book today.  You know the reason I use DNG?  The data validation.  That's it.  I could take or leave the archival format, sidecars, file sizes, built-in previews, lossy compression, etc. but knowing that my data is safe is worth the conversion to me.  I've been bitten by corruption in the past, and once is enough for me.


----------

