# CPU usage at 100% during Publish (or export)



## camner (Jan 8, 2020)

I understand that LR now uses multithreading for certain tasks, including exporting and publishing.  For the sake of speeding up the process, that's wonderful.  But, when I do a large publish or export, my 2015 27" iMac (quad core) becomes virtually unusable because all for cores are running at 100%.  Is there any way to throttle this (or tell LR not to use all cores?), or is just time for me to take a break and grab a cup of coffee and relax?


----------



## Replytoken (Jan 8, 2020)

I would opt for the cup of coffee and relaxing.  Unless you need to be running another process, what gain is there in artificially throttling the CPU?  It would just take longer to complete the process.  Are you concerned about heat build-up?

--Ken


----------



## camner (Jan 8, 2020)

I am not too concerned about heat build up, because I use TG Pro to control my iMac's fans (for reasons I don't understand, out of the box the iMac never seems to speed up its fans, even when the CPU temps get really high), which leads to fans running on max during an extended publish or export process.

My reason for wanting to be able to keep a little CPU power for some (light) work is that I am typically not in a hurry to have the publish/export finish up, and I would rather not have to stop my other work while its going on.  But, this is definitely a "1st world problem."  I was more curious than anything else; it's really only a minor annoyance.


----------



## LouieSherwin (Jan 8, 2020)

How much RAM do you have? 

Export and Publish will use a lot. Once all available RAM is in use it will take more time for the computer to make space to run other applications. 

-louie


----------



## Linwood Ferguson (Jan 8, 2020)

What kind of publish?  Smugmug's publishing plugin has a feature to upload in parallel (though interestingly I just looked and can't seem to disable it). Also, if publishing over the internet and you have a sufficiently sophisticated router you can put bandwidth limits to where you publish, which will inherently slow down LR.

But the cup-of-coffee answer is probably the best one. Perhaps not the desirable one, but the least frustrating as most other techniques are going to have side effects of some sort or be complicated.


----------



## camner (Jan 8, 2020)

LouieSherwin said:


> How much RAM do you have?
> 
> Export and Publish will use a lot. Once all available RAM is in use it will take more time for the computer to make space to run other applications.
> 
> -louie


32GB.  Although Activity Monitor never reports less than 8GB or so free, it also typically reports that some swap files have been used (right now about 10GB).  I presume (but don't know for sure) that swap files work like this: When open apps collectively want more than 32GB, the iMac writes out open but not currently in use RAM to the hard drive; then, that written-to-swap-files RAM stays out on disk until it is needed, so even if the RAM actually needed drops a lot, the swap files will remain in use until their contents get swapped back into RAM.


----------



## camner (Jan 8, 2020)

Ferguson said:


> What kind of publish?  Smugmug's publishing plugin has a feature to upload in parallel (though interestingly I just looked and can't seem to disable it). Also, if publishing over the internet and you have a sufficiently sophisticated router you can put bandwidth limits to where you publish, which will inherently slow down LR.
> 
> But the cup-of-coffee answer is probably the best one. Perhaps not the desirable one, but the least frustrating as most other techniques are going to have side effects of some sort or be complicated.


I use SmugMug's publishing plugin.  I didn't realize that it uploads in parallel; that probably explains the CPU usage pinging 100% for longer publish events.


----------



## Linwood Ferguson (Jan 9, 2020)

Well, I'm really not sure what it does now. If you look in plugin manager, you'll see the option under settings.  But on mine it's greyed out and not exactly checked.  The others check with a check mark, not the black blob.  

When they first came out with this, long ago, there were issues, and they disabled it temporarily.  I never really followed up, and just assumed they fixed it and it worked.  But given this, I'm not sure what it's doing.

Parallel uploads will better saturate a very fast internet connection; a single upload (with any tool) has trouble keeping the link busy.  And more throughput on the upload allows LR to generate faster, which consumes lots of CPU.  So parallel uploads will slow down your computer.

But given what I see below, I'm not sure if it's parallel or not; and at least on mine I can't turn it on or off.

If curious and a problem you might contact [email protected] if you can't control it on yours either.


----------



## Ed Anderson (Jan 9, 2020)

Interesting Linwood, I'm using a slightly older version of the plugin and have the checkbox.  When I restored my Mac last month the Adobe add on site couldn't push the push the plugin to LR even though it would show installed.  I eventually just downloaded and did a manual install.

I just published a few photos to see what my system was doing in the Activity monitor.  The CPU usage bumped up to over 2% leaving 97% idle and Ram usage increased by 3GB to just under 17.5 used. No swap file usage. All this held fairly steady throughout the process.  I'm using a late 2015 iMac, i7, 32GB ram and running the latest Catalina with LR 9.1 Classic.  

Sounds like there is a process or other underlying issue running thats using up resources.  As Camner reported in a post above "Activity Monitor never reports less than 8GB or so free, it also typically reports that some swap files have been used (right now about 10GB)."


----------



## clee01l (Jan 9, 2020)

Lightroom's Export/Publish process is multi threaded  And depending on the number of CPU cores available, will spawn several threads and multiple file handles as it goes about completing your task as soon as possible.  The process creates many temporary file in working storage and swaps data in an out of the swap file .     For this reason,  I recommend keeping at least 15% or 100GB of freespace available for working storage and the Swapfile.  
The Export/Publish process is I/O intensive so you also need to consider the I/O speeds available.  The process should improve if you have multiple discs (not partitions on the same physical disc) and faster discs.


----------



## Linwood Ferguson (Jan 9, 2020)

You're saying that CPU during publish says in the single digits?   CPU used?    That's total CPU utilization for all processes?

Again, don't speak Mac, but memory usage as a percentage is generally not a very telling number.  Well... if there is a lot free, it's not a limiting factor, but if there is little free that might not mean anything.  The only way you can tell for sure is look at page fault hard I/O rate, which is usually more difficult to tell (sometimes called Hard Faults).  Essentially if those are near zero even if you have no apparent free  memory, memory is not constrained. (Note below) 

As Cletus above mentioned, disk I/O could be an issue if you have a really busy disk during export.  If that's the system disk, or one used for temp and scratch (a good bet it may be) that can make the overall system seem sluggish as well.

Finally lightroom itself, in order to do publishing and also do other work inside lightroom, has to be "smart".  Every once in a while people stumble on cases where some programmer was not quite so clever, and it might be getting tangled up in itself for some reason.  If the overall system is responsive but only lightroom unresponsive during publish, I would suspect a bug in lightroom (or if Adobe, a feature). 

Footnote: My comment on memory applies mostly to programs that use memory as needed from virtual space.  There are some programs, and I do not know if Lightroom is one, that try to outsmart the operating system, and look to see how much physical memory is available either free or in total, and then adjust their own algorithms to compensate.  This is a little like walking down a sidewalk and approaching someone and you both move to avoid each other, often resulting in a very unproductive little dance.  The best defense against such dumb "smart" programs is to have lots and lots of memory so you do not have to worry if they are in a tug of war with the O/S/.   And programmers should learn to trust the operating system, not try to outsmart it!


----------



## camner (Jan 9, 2020)

This is getting even more interesting.  I'm running the same (current) version of the SmugMug plugin, but my settings look the same as Ed's, not Linwood's (i.e., I can check/uncheck the simultaneous uploads box but the Use URL Handler for Authorization is greyed out.


----------



## Linwood Ferguson (Jan 9, 2020)

Are you on Mac also.  I'm windows.  I wonder if that's relevant.


----------



## camner (Jan 9, 2020)

Ed Anderson said:


> I just published a few photos to see what my system was doing in the Activity monitor.  The CPU usage bumped up to over 2% leaving 97% idle and Ram usage increased by 3GB to just under 17.5 used. No swap file usage. All this held fairly steady throughout the process.  I'm using a late 2015 iMac, i7, 32GB ram and running the latest Catalina with LR 9.1 Classic.


I'm flabbergasted that your CPU usage is only 2% during a publish process (or for that matter, even if you are NOT publishing...).  I have the same machine, but I'm running Mojave, not Catalina.  Whoops, I just noticed you said yours has an i7, whereas mine has an i5, so that's  a difference.  At "idle" I run at about 20% CPU utilization.  Admittedly I've got a bunch of stuff running in the background, ranging from Dropbox and BackBlaze to other addons such as MacBreakZ (which reminds me every 45 min to take a break, for the sake of my back) and DriveDx (which monitors drive SMART status), and a bunch of other smaller stuff.  Also, I have a tendency to keep lots of tabs open in Chrome, which is notorious for both RAM and CPU usage.  

Still, when I publish, CPU usage ramps up and pretty much hangs between 95%-100% until the publish process has finished, making my iMac rather unresponsive.


----------



## camner (Jan 9, 2020)

clee01l said:


> Lightroom's Export/Publish process is multi threaded  And depending on the number of CPU cores available, will spawn several threads and multiple file handles as it goes about completing your task as soon as possible.  The process creates many temporary file in working storage and swaps data in an out of the swap file .     For this reason,  I recommend keeping at least 15% or 100GB of freespace available for working storage and the Swapfile.
> The Export/Publish process is I/O intensive so you also need to consider the I/O speeds available.  The process should improve if you have multiple discs (not partitions on the same physical disc) and faster discs.


Does LR use its own swap file?  If so, can one move it to another location besides the startup disk (which, of course, is thrashing around doing whatever else I've ordered it to be handling...)


----------



## Ed Anderson (Jan 9, 2020)

So I went back and did some different tests.  When publishing starts, CPU usage ramps up to 95-97%, drops off to around 15 then settles down and during my capture was 3.3% (see capture 2) while publishing continued.  You can see the spike in the graph with the drop off before settling.  I then opened and closed a couple of programs and CPU ramped back up to the 15% range, but at no time am I restricted from using the machine.  Capture 1 shows the CPU usage while here reading this post on LRQ.  As an after thought, I went back a did republish of the same photos and did a capture of the memory usage during the process (capture 3).

As an aside, I don't have a lot of programs running in the background. I think it goes back to the days whenI was doing 3D graphics and need all the resources I could get to render.  My main back up is CCC which runs nightly at 1AM so its not interfering while I'm on the system.  I can understand the break for the back too as mine isn't in the best of shape anymore either. I'm doing it low tech though, an old alarm clock on the desk.


----------



## camner (Jan 10, 2020)

I tried the same test as you, Ed.  I, too, noticed the relatively quick ramp up of CPU usage. However, as you can see from the screenshots, my 2015 iMac with 32GB is showing Lightroom CPU usage of 353% (whatever THAT means) which make the entire system CPU usage over 99%, which in turn, unsurprisingly, makes my iMac rather unresponsive.

I tried unchecking the "simultaneous uploads" in the SmugMug publish service, but saw no difference in CPU usage.


----------



## Linwood Ferguson (Jan 10, 2020)

camner said:


> Does LR use its own swap file?  If so, can one move it to another location besides the startup disk (which, of course, is thrashing around doing whatever else I've ordered it to be handling...)


I don't quite know what "own swap file means, but let me see if I can answer more generally.

The O/S uses the page and/or swap file as needed, programs cannot create their own at that level.

Lightroom has lots and lots of temporary storage it uses, though, among them (and likely not a complete list):  the ACR cache, the preview cache, the smart preview cache (not sure if this is properly called a cache), temp storage for files being exported/published, temporary storage for files being imported from cards, (different from) temporary storage for items being imported from non-card disks, write-ahead and similar logs for sqlite that it automatically creates, catalog backup folders (also not sure if this is "temporary"). 

Some of these can be moved from the GUI.  Some can be moved if you understand the operating system (e.g. previews can be moved with a symbolic link in place of the folder pointing to a different place), some can be changed with operating system environment variables.   There's just no simple answer to how you control all the pieces.


----------



## Ed Anderson (Jan 10, 2020)

camner said:


> Lightroom CPU usage of 353% (whatever THAT means)


If each core in the CPU was maxed out, the activity monitor could theoretically show 400%, so during your publish, LR is nearly maxing out all the cores in your CPU.  This is why the Mac is unresponsive.


----------



## Conrad Chavez (Jan 10, 2020)

Ferguson said:


> Again, don't speak Mac, but memory usage as a percentage is generally not a very telling number.  Well... if there is a lot free, it's not a limiting factor, but if there is little free that might not mean anything.


All of that is completely true. Mac (and I guess Windows) RAM management has gotten complex enough that most of the individual numbers don't show you  enough of the whole picture. It’s true that the amount of "free" RAM no longer means much. I think it is why the Free readout for RAM has disappeared from macOS Activity Monitor.

If free RAM starts to get low, macOS has more strategies than just the OS swap file. It can "raid" the RAM that's marked Inactive (in Unix), since other applications have marked it as free for higher priority processes. It can free up RAM by compressing it; this is kind of a newer thing that macOS (and Windows?) started to do once CPUs got fast enough that it could take less time to compress/decompress data in RAM than to write/read it in swap files. If there isn't enough RAM to do that, then it will go to a swap file (which, if on a fast SSD, is less of a performance penalty than it once was). Because of all this, the easiest way on a Mac to see if RAM is a problem is to look at Memory Pressure. You can see in Ed's third screen shot that Memory Pressure is green. This means all RAM requests are being handled easily and immediately by the system, with no cause for concern even if "a lot" of RAM appears to be "in use." And if you are troubleshooting a problem with a feature like Export, if Memory Pressure is green, look somewhere else because the problem isn't about RAM.

If it’s really important to be able to use other applications in the middle of an export that’s maxing out all the CPU cores, and you’re comfortable working in the Terminal application for command-line Unix in macOS, you might look into using the renice command. That command changes the CPU priority of applications, so in theory you could lower the CPU priority of Lightroom so that other applications could get more CPU time. I have not tried this myself.

If you’re not comfortable using the Unix command line, there is  a macOS application called AppTamer that is supposed to do the same thing (it’s probably just a convenient GUI for renice). I haven’t used AppTamer either, but it comes from a developer who also creates the excellent Default Folder utility that I’ve been using on my Macs for maybe 20 years.


----------



## johnrellis (Jan 10, 2020)

camner said:


> when I do a large publish or export, my 2015 27" iMac (quad core) becomes virtually unusable because all for cores are running at 100%.  Is there any way to throttle this (or tell LR not to use all cores?)



On most systems, LR _exports_ uses about 85 - 95% of CPU, leaving enough for interactive use by LR or other applications. According to Adobe employee Rikk Flohr, this is by deliberate design. But a fair number of people have complained over the years that LR can use 100% of CPU, making the computer, and LR in particular, unusable for interactive use until the export completes.  

It would be simple enough to have an advanced option that throttles exports, but Adobe has not chosen to implement it. I recommend that you add details of your issue to this bug report in the official Adobe feedback forum:
Lightroom Classic: High CPU usage makes it unusable during export | Photoshop Family Customer Community 

Be sure to click Me Too and Follow in the upper-right corner. This will make it a tiny bit more likely Adobe might pay attention.


----------



## johnrellis (Jan 10, 2020)

camner said:


> when I do a large publish or export, my 2015 27" iMac (quad core) becomes virtually unusable because all for cores are running at 100%.  Is there any way to throttle this (or tell LR not to use all cores?)



On most systems, LR _exports_ uses about 85 - 95% of CPU, leaving enough for interactive use by LR or other applications. According to Adobe employee Rikk Flohr, this is by deliberate design. But a fair number of people have complained over the years that LR can use 100% of CPU, making the computer, and LR in particular, unusable for interactive use until the export completes.  

It would be simple enough to have an advanced option that throttles exports, but Adobe has not chosen to implement it. I recommend that you add details of your issue to this bug report in the official Adobe feedback forum:
Lightroom Classic: High CPU usage makes it unusable during export | Photoshop Family Customer Community 

Be sure to click Me Too and Follow in the upper-right corner. This will make it a tiny bit more likely Adobe might pay attention.


----------



## johnrellis (Jan 10, 2020)

Regarding CPU usage of _publishing_ to services like SmugMug, observed CPU usage can vary widely depending on which images are being exported and the bandwidth of the network connection.

A publish operation consists of two sets of operations, often performed concurrently: exporting the photos to temporary disk copies and uploading those exported photos to the service.  If you're exporting large raws, the constraining factor is often the CPU, since rendering raws is CPU-intensive and a fast network connection can keep up with the rate at which rendered JPEGs are produced. But if you're exporting JPEGs, the rendering of the temporary copies is very fast in comparison, and the network can then become the bottleneck.  

So when reporting experience with publishing, say, with the Smugmug pluin, be sure to report the type of files being exported, the CPU usage, and the network bandwidth usage (both the actual usage and the nominal maximum bandwidth of the connection).


----------



## SaraLH (Jan 11, 2020)

FYI, I just noticed this behavior myself with an "internal"publish. I use Jeffrey Friedl's "Folder Publisher" to publish edited files that meet a certain criteria to the network harddrive for family viewing. I changed some keywords which triggered a republish on a couple of thousand files and now my system (i5) is running as follows: CPU 85-100%, Memory 70% (16GB), Disk 5%, Network 2%, GPU 3-18%. DNG files are being exported to JPGs.


----------



## camner (Jan 14, 2020)

johnrellis said:


> On most systems, LR _exports_ uses about 85 - 95% of CPU, leaving enough for interactive use by LR or other applications. According to Adobe employee Rikk Flohr, this is by deliberate design. But a fair number of people have complained over the years that LR can use 100% of CPU, making the computer, and LR in particular, unusable for interactive use until the export completes.
> 
> It would be simple enough to have an advanced option that throttles exports, but Adobe has not chosen to implement it. I recommend that you add details of your issue to this bug report in the official Adobe feedback forum:
> Lightroom Classic: High CPU usage makes it unusable during export | Photoshop Family Customer Community
> ...


Thanks, John.  I will report to Adobe.


----------

