# How can lightroom possibly corrupt files during import?



## RzzB (Jul 3, 2011)

Using...

Lightroom 3.4.1
Windows 7 (64bit) - fully up to date with patches
System Rating 5.9
Processor i7 CPU - [email protected] - no overclocking
Memory - 6 GB
Display Adaptor - Nvidia GeForce GTX 460
No yellow bits in device manager 

Canon 5D Mk II and 7D
Sandisk Extreme 60MB/s 32GB & 8GB
Sandisk Card reader - directly plugged

This happens all the time - but for example - yesterday I took 350 photos, when importing them directly from the CF card with Lightroom, seven of them were corrupt. 

However, if I manually copy the files to the HD and then import into lightroom - no corruption.

I have used a hex comparison tool to compare the good and bad files and sure enough there are a couple of picked bits here and there.

I'm a bit confused. I have some basic questions...

How can Lightroom introduce corruption at this stage? Seems implausible?

If there was a corruption - why on earth wasn't it detected at transfer time? Surely there is some sort of basic checking, CRC at a minimum - yes/no?

Thanks,
RzzB


----------



## DonRicklin (Jul 4, 2011)

Likely bad CF card reader, cable or memory. Try different reader of direct from camera connection. 

Don


----------



## RzzB (Jul 4, 2011)

Don,

Thanks for your response.

This occurs with different card readers, connected direct and through a hub. The host machine is very stable so I doubt that I have a memory problem of this magnitude.

Let me be very clear ...

When I transfer the files using Windows Explorer there are NO corrupted files.

When I transfer the files using Lightroom I get corrupted files.

It's as if Lightroom is using a different transfer mechanism that is causing corruption.

What is worrying me more is - why aren't the corruptions detected and an error raised? I believe USB transfers have a CRC checking / retry mechanism built in for most transfer modes. Is Lightroom perhaps using a modes that has this turned off? 

Thanks,
RzzB


----------



## DonRicklin (Jul 4, 2011)

Okay, time to post screen shots of some of these files  in LR so we can see what corruption you are talking about.

Don


----------



## Victoria Bampton (Jul 4, 2011)

Hi Rzzb, welcome to the forum!

I can certainly understand your concern, and that is a very odd problem.  It's not one that I've heard any other reports of, so I would still look at hardware amongst the other possibilities.

I assume you've ruled out the other possibilities, such as a hard drive, and you've run a low level memory test such as Memtest?  Perhaps LR is accessing memory that has a minor glitch, whereas Explorer isn't hitting it?  LR's very good at finding the most hidden minor memory glitches, and it could cause corruption, so it would be worth ruling out.

And how are you discovering that the transferred files are corrupted?  Is LR telling you, or are you spotting it in the previews?  Or just in the hex comparison that you're seeing it?


----------



## RzzB (Jul 5, 2011)

Many thanks for your responses guys.

I'm detecting the corruption because some of the images in the preview window are a mess - see the attached. When I inspect these files (hex comparison) with the versions that I transferred manually I can see that there are some picked/dropped bits. I can see no pattern sensitivity - they look pretty random. 

I have done nothing to rule out hard drive problem, but I can't see how this could be associated with a hard drive problem. Not sure what I should do? Thoughts?

I haven't run memtest recently. I did run all sorts of test when I originally built the machine (6 months ago) - Memtest, Burnin, Prime95 - just to make sure I had a stable system. But I agree - it would be sensible to run Memtest again to be sure. I will do this in the near future and report back - might be in a couple of days.

I do agree that this feels like a hardware problem, but it seems to be exacerbated by Lightroom. 

Possibly memory - I'll do some testing as mentioned above. 

The other thing I need to do it double check all the failing scenarios - just to make sure. ie Cables, readers, hubs etc. I'll probably do this over the weekend.

What concerns me is that none of the software involved is detecting the corruption and raising an error.

Many thanks for your help,
RzzB


----------



## Kiwigeoff (Jul 5, 2011)

What is the build number of Lightroom? Check in system info in the Help drop down menu.
This looks similar to the issue with 3.4.


----------



## Hal P Anderson (Jul 5, 2011)

Geoff,

Wasn't that problem with jpegs from a non-Canon?

Hal


----------



## Kiwigeoff (Jul 5, 2011)

I have no idea Hal, but it looked like something similar I think.


----------



## b_gossweiler (Jul 5, 2011)

The issue was noticed with images from a HP PhotoSmart R607, but you never know ....

Beat


----------



## RzzB (Jul 5, 2011)

Thanks for your interest gents.

Lightroom version is 3.4.1 - see first post for other info.

One thing I haven't mentioned is that these are all raw CR2 files. I only shoot in raw. 

Thanks,
RzzB


----------



## Kiwigeoff (Jul 5, 2011)

I have another example here from a forum that looks very similar to yours and funnily enough from a Canon 40D.
I seem to recall it being something to do with the drive or connections.
Did you move them from one drive to another when importing? Where do you store your files - on an eternal drive???


----------



## RzzB (Jul 5, 2011)

Kiwigeoff said:


> I have another example here from a forum that looks very similar to yours and funnily enough from a Canon 40D.
> I seem to recall it being something to do with the drive or connections.
> Did you move them from one drive to another when importing? Where do you store your files - on an eternal drive???



Geoff,

Many thanks for your response - could you post a link to the other example please. My cameras are a 7D and 5D but similar in respect of being Canon.

The importing is done from a CF card using a card reader directly to my D: drive which is a 1T mirrored raid drive. I do have external backup storage in the form of Synology DS210 boxes but I never import to them.

I have a non Raid drive in my machine - perhaps I'll create a new catalogue and import to that drive just to rule out any Lightroom/Raid problems. 

Thanks for the thought...

Regards,
RzzB


----------



## Kiwigeoff (Jul 5, 2011)

I have the file on my computer and it looks like this:

Interesting it was on a D drive as well. I'm trying to recall what the solution was but am too busy at the moment to search. I'd think your idea to test on your C drive would be a good one!!


----------



## RzzB (Jul 7, 2011)

RzzB said:


> Geoff,
> ...... snip.....
> I have a non Raid drive in my machine - perhaps I'll create a new catalogue and import to that drive just to rule out any Lightroom/Raid problems.



Hmmm... I'm now beginning to regret starting this thread 


I tried using the non-raid drive - and hey presto no corruption! 


However, I retried to the raid drive also and guess what - it worked ok there too.


In the last couple of days I have transferred the same 350 images using all sorts of hardware and I have seen just one corrupt file.


So - something has changed - and I have been racking my brain trying to work out what.


Oh well - I think I now have to wait till I see the corruptions again and look at what is happening on my machine at the time.


Thanks,
RzzB


----------



## Victoria Bampton (Jul 7, 2011)

That sounds like an excellent plan RzzB.  Funny how things fix themselves when you say them out loud!


----------



## Brad Snyder (Jul 7, 2011)

Good news. 

 I'd say we don't like to come across as Lightroom fanboys, but I don't think I've ever seen convincing evidence that Lr can cause raw file corruption at the import or editing stages. Lr never touches the raw files at all (with one arcane exception) and uses the OS services exclusively for file operations. 

But if you just flat out say that, it sounds like defending an Adobe who can do no wrong, which is patently false.


----------



## Roomservice (Jul 17, 2011)

RzzB said:


> ... I tried using the non-raid drive - and hey presto no corruption!
> However, I retried to the raid drive also and guess what - it worked ok there too.
> In the last couple of days I have transferred the same 350 images using all sorts of hardware and I have seen just one corrupt file.
> ...


 
It seems this is the same problem I have: 
when importing 400 cr2 files directly from the camera into LR, one (1) file gets corrupted. It does not happen every time.
And: when I transfer the files using Windows Explorer there are NO corrupted files.

Lightroom 3.4.1
Windows 7 (64bit)
Canon 5D Mk II
Sandisk Extreme 60MB/s 16GB

Checked memory and hard drive.
The corrupt file seams to appear after 6-8 GB of the 10 GB has been transfered.
I have also used a hex comparison tool to compare the good and bad files, and there is a certain pattern: a multiple of the 6th and 14th byte are replaced... somewhere in the last 20-40% of the file content.


----------



## Roomservice (Jul 17, 2011)

Maybe the file corrupt problem is related to this problem:

Today I took some test shots with camera setting RAW+jpg, to another empty memory card.
When importing them to LR it hanged. I made that import repeatedly: sometimes LR freezes, sometimes not. When it freezes you can see in windows explorer that it has lost the connection to the card. Does Lightroom cause the card to disconnect?

But there are no problems when copying with Windows explorer.
So RAM, hard disk, memory card, camera and usb cable seems ok (and RAM + hard disk are tested).
But when Lightroom is involved...

Any suggestions would be appreciated


----------



## clee01l (Jul 18, 2011)

Roomservice, welcome to the forum.  Unless you have actually used a second USB cable, you can not eliminate 'Badd Cable" from the list of possible causes.  By using two different cards, you probably have eliminated the "Bad Card" cause (Or validated that you have two bad cards).  The camera needs a fresh battery to begin the transfer,  Using a real card reader instead of the camear can help isolate the camera/battery as a cause. 

If you can successfully copy images to the computer and Import them into LR from the computer location, you can then remove the camera/card/cable as a source of the problem. 

I would also like to note that with flash memory, it very important to give the camera time to write its buffered image data to the card before shutting the camera off or ejecting the card from the camera. Either operation while the file is still being written can result is a corrupt image.


----------



## Roomservice (Jul 18, 2011)

Hi clee01, and thank you for the feedback.

The files are getting corrupt during/after the transfer (copy) to Lightroom. They are OK on the CF card.

When copying with Windows explorer - with the same cable/card/camera - there is no problem. Neither with lost usb connections, hanging, or corrupt files. I have also tested another cable (from a canon 60D).

There are 2 problems during file transfer: LR hangs and files gets corrupt when using LR. I believe LR hangs because the connection to the card is lost during the transfer. Maybe Lightroom has some problem with the camera windows driver? The camera driver from camera CD is installed (it's actually a MS driver from 2006).

I will test with a card reader, even though it works fine with the camera when using Windows explorer and LR is not involved.


----------



## Roomservice (Jul 18, 2011)

A clarification about the hanging:

LR hangs during the initial phase of import (sometimes, not always):
after clicking the left "Import..." button, and before clicking the right "Import" button.

Note:
Historically Lightroom has had problems with freezing, check out this almost identical problem from en early version:
http://kb2.adobe.com/cps/401/kb401357.html
That problem seem had something to do with the previews.

Some threads about hanging issues in current versions of LR:
Lightroom 3 freezing when using import:
http://forums.adobe.com/thread/664110
LR 3 freeze when trying to import (with a workaround):
http://forums.adobe.com/message/2981433
Horrendous import problems in Lightroom 3 64bit:
http://forums.adobe.com/message/3022353


----------



## clee01l (Jul 18, 2011)

> I'd say we don't like to come across as Lightroom fanboys, but I don't think I've ever seen convincing evidence that Lr can cause raw file corruption at the import or editing stages. Lr never touches the raw files at all (with one arcane exception) and _*uses the OS services exclusively for file operations*_.


quoted from Brad Snyder above.  If you are an exception to this with your Canon, you will be the first.  When you copied these CR2 files with windows, what viewer did you use to view the files?  Most viewers will only show you the embedded JPEG thumbnail in the RAW. The JPEG thumbnail in in the header section of the file and not in the compressed RAW data section that follows. The Compressed raw data could still be corrupt.  What happened when you tried to import these files that you successfully copied via windows?.  LR only calls standard OS File-handling APIs to do the file operations like read, copy, and write.  These are the same APIs that the built in file import Windows applet uses when it imports images from a card to a HD.

And as for USB cables, I'll leave you with this comforting thought. Remember, Canon is a camera manufacturer.  All of their cables distributed with their camera likely came from a supplier that was lowest bidder. 

You mentioned the camera device driver. There is a possibility that it is corrupt. Have you checked Canon for updates? Or perhaps just re install the driver that came with your camera.


----------



## clee01l (Jul 18, 2011)

Roomservice said:


> A clarification about the hanging:
> 
> LR hangs during the initial phase of import (sometimes, not always):...


 Hanging is not the same as corrupted image files, Which is your problem? I have been giving you remedies for images that get corrupted between the time the camera writes to the camera card and viewing the file that was transferred to the HD with LR.


----------



## Roomservice (Jul 18, 2011)

clee01l said:


> quoted from Brad Snyder above.  If you are an exception to this with your Canon, you will be the first.  When you copied these CR2 files with windows, what viewer did you use to view the files?  Most viewers will only show you the embedded JPEG thumbnail in the RAW. The JPEG thumbnail in in the header section of the file and not in the compressed RAW data section that follows. The Compressed raw data could still be corrupt.  What happened when you tried to import these files that you successfully copied via windows?.  LR only calls standard OS File-handling APIs to do the file operations like read, copy, and write.  These are the same APIs that the built in file import Windows applet uses when it imports images from a card to a HD.
> 
> And as for USB cables, I'll leave you with this comforting thought. Remember, Canon is a camera manufacturer.  All of their cables distributed with their camera likely came from a supplier that was lowest bidder.
> 
> You mentioned the camera device driver. There is a possibility that it is corrupt. Have you checked Canon for updates? Or perhaps just re install the driver that came with your camera.



Yes today I reinstalled the driver from the CD. I've also checked canons site but there are no upgrades regarding that driver.
About the cables: I don't think any of the two cables are bad since they work fine when LR is not involved.

I checked the raw files with two methods: added them to LR to be able to see the raw files (and not just the embedded jpg's as you pointed out). And binary compare with UltraCompare, which verified the previous result.

Next thing to do is to test a card reader. And also test importing from hard disk to LR.

Maybe there is no bug in LR. But if there is, it probably makes the software do something quite unexpected (as bugs often do).


----------



## RzzB (Jul 3, 2011)

Using...

Lightroom 3.4.1
Windows 7 (64bit) - fully up to date with patches
System Rating 5.9
Processor i7 CPU - [email protected] - no overclocking
Memory - 6 GB
Display Adaptor - Nvidia GeForce GTX 460
No yellow bits in device manager 

Canon 5D Mk II and 7D
Sandisk Extreme 60MB/s 32GB & 8GB
Sandisk Card reader - directly plugged

This happens all the time - but for example - yesterday I took 350 photos, when importing them directly from the CF card with Lightroom, seven of them were corrupt. 

However, if I manually copy the files to the HD and then import into lightroom - no corruption.

I have used a hex comparison tool to compare the good and bad files and sure enough there are a couple of picked bits here and there.

I'm a bit confused. I have some basic questions...

How can Lightroom introduce corruption at this stage? Seems implausible?

If there was a corruption - why on earth wasn't it detected at transfer time? Surely there is some sort of basic checking, CRC at a minimum - yes/no?

Thanks,
RzzB


----------



## Roomservice (Jul 18, 2011)

clee01l said:


> Hanging is not the same as corrupted image files,  Which is your problem? I have been giving you remedies for images that  get corrupted between the time the camera writes to the camera card and  viewing the file that was transferred to the HD with LR.



I mentioned the hanging because maybe it is related to the corrupt problem


----------



## clee01l (Jul 18, 2011)

> I mentioned the hanging because maybe it is related to the corrupt problem


In that case, I would even be more suspect of a bad cable or card reader (or in your case, camera)  Is it correct that your Camera uses SD/SDHC/SDXC cards rather than the CF type?  SD card readers are more durable and less likely to get damaged. 
Just to review the steps you've taken to eliminate the source of the problem:

Tried two different cables to see if bad cables might be the cause
Tried two different cards to eliminate bad card as a cause
Successfully imported the card contents using the Windows import apple
To be Tested:

Import the Windows imported image into LR
Use a card reader capable of reading SDXC or SDHC cards (if you have that format) (Note: SD card readers are not forward compatible. SD card readers can only read <4GB cards. SDHC card readers won't read SDXC cards above 32 GB


----------



## Roomservice (Jul 19, 2011)

After some more testing it seems the cause is:
- some Canon camera driver or software (from version 23.0 of the EOS Digital solution disk, or EOS utility 2.9.0.0).
- or maybe something in Lightroom 3.4.1 - the Windows7 64bit version - that doesn't handle things as it should.

Anyway, my solution is:
- don't import directly from the canon camera.
Use a card reader, or copy with windows explorer to hard disk first.

Note: it may work to import to LR directly from camera if you haven't installed any canon drivers on that computer. Or if you are using the mac version of LR.


----------



## clee01l (Jul 19, 2011)

It is possible that the drivers that you used are for 32 bit OS and not compatible with your 64bit OS. According to the Canon website no drivers are required for the 60D on Win7-64
http://www.usa.canon.com/cusa/support/consumer/eos_slr_camera_systems/eos_digital_slr_cameras/eos_d60#DriversAndSoftware

The EOS Solutions disk is now up to about V30, so the one that you have may be very much out of date.


----------



## Roomservice (Jul 19, 2011)

camera is 5D Mk II
Software on CD is for 64bit. It's the EOS utility that handles communication with camera when using canons file transfer, and as I said before there is an update from 2.9 to 2.10 but it doesn't seem to contain anything related to the problem. And I'm not running that utility. But canon software must still be involved somehow when handling the card directly in camera, because cr2 previews is viewed in windows explorer (but only when viewing the camera card directly).
But the usb driver used for the camera is a MS driver.


----------



## Roomservice (Jul 21, 2011)

Completion to post #30:

More info in this thread (from post #6):
http://www.lightroomqueen.com/commu...reviews-not-available-when-import-from-camera


----------



## papaya74 (Jul 24, 2011)

Same problem here: LR first shows the preview, than this:




i am shure its not the way you copy the files to your computer (i tried 3 different ways.)

SOLUTION:

- turn the image in the library, usig the small arrow at the bottom
- zoom in the corrupt file - and there is the proper version!
- enjoy the magic and dont ask questions 

Worked for me now.


----------

