# Internal Catalog Changes (was: Edit history and catalog size)



## Hoggy (Nov 2, 2017)

Exact Lightroom Version: Classic 7.01

It's likely that the new catalog version hasn't been used by many extensively as of yet, so this may still be an unknown.  Of course, I also realize the limited usefulness of keeping edit histories - but this is a little more towards hypothetical/academic.  (Although it _could_ mean that I might be more likely to keep a few more 'useful' edit histories on some images than in pre-7.0 versions.)

As many may have noticed, their LR catalog pre-7.0 has been reduced quite a bit, post-7.0.  Mine is over half reduced - from ~1.2GB to ~512MB unzipped.

I also have been recompressing the zip backups to 7zip-Zstandard codec, level 17.  Pre-7.0, they would recompress from a ~280MB (IIRC?) zip file to an ~86MB 7zip file.  Post-7.0, they recompress from a ~260MB zip file to a ~154MB 7zip file.  So it seems as if 7.0 might be writing compressed data directly into the catalog file, leading to poorer recompression to 7zip-Zstandard-level17.  Of course, that's just a guess as to what might be happening in order to explain those findings (therefore I'm open to other possible reasons to explain that, btw).

Sooooo..  Has anyone done any testing to see if keeping long edit histories bloats the catalog as much as pre-7.0 edit histories did?  Or otherwise have any 'direct' knowledge if they should bloat it less now, post-7.0?


----------



## johnbeardy (Nov 2, 2017)

As part of the work optimising performance, Adobe now compress the History log data and I would not expect the LR7 catalogue to grow as quickly as the LR6 equivalent. History compression during catalogue upgrading is the main reason why LR7 catalogues are smaller than LR6, and half does not surprise me (it is consistent with when I cleared a big LR6 catalogue's History). Your observations about zipping backups are also what I would expect, since the History is already compressed.

John


----------



## Gnits (Nov 2, 2017)

Good to be aware of these subtle changes under the hood.


----------



## Hoggy (Nov 2, 2017)

johnbeardy said:


> History compression during catalogue upgrading is the main reason why LR7 catalogues are smaller than LR6, and half does not surprise me (it is consistent with when I cleared a big LR6 catalogue's History).



Thanks for confirming my observations/suspicions.  But as far as the quoted statement here, there must something else quite big that it's compressing.  As a rule, I always clear history data after making a snapshot except maybe for current in-progress editing - so there would be practically no history there unless I forgot on a couple/few images.

However, each non-tiff/psd image has at least one 'v1.00' snapshot, and there may be an average of 2 or 3 on most images (~70%) - with some rare ones that might have about 10 snapshots (maybe ~6 images in total there).  I always make it a point to try and delete redundant/now-pointless snapshots as I make my way though my entire 'artsy' catalog a 4th/5th or 60th time  in order to keep the size low.  .....  But even solely considering snapshot compression, I'm quite shocked that it could have accounted for that much of a decrease.  I think there was less of a decrease in size when I first started deleting all my [long] edit histories. - but it's been quite a while since I got done with that, so I could be remembering that wrong.

(ETA: oh.. This 'artsy' catalog is only ~3000 images - so probably quite small compared to others'.)


----------



## johnbeardy (Nov 2, 2017)

My guess is that the difference is because upgrading also includes the standard optimisation which executes a SQL vacuum statement, cleaning up any temporary or orphaned data.


----------



## clee01l (Nov 2, 2017)

johnbeardy said:


> My guess is that the difference is because upgrading also includes the standard optimisation which executes a SQL vacuum statement, cleaning up any temporary or orphaned data.


This is good to read. In the past, I have executed my own SQL vacuum statement on recalcitrant catalog files to achieve this same end,


----------



## johnbeardy (Nov 2, 2017)

Optimise definitely includes vacuum, Cletus. I am not as sure about what else it may do.


----------



## Wernfried (Nov 2, 2017)

I compared old and new DB version with SQLite browser. Development history and XMP content are compressed now. In LR6 they were stored as plain text string, now they are compressed binary data.


----------



## Hoggy (Nov 3, 2017)

Wernfried said:


> I compared old and new DB version with SQLite browser. Development history and XMP content are compressed now. In LR6 they were stored as plain text string, now they are compressed binary data.



That leads the techno-geek in me to wonder now, about which codec they're using.  It would seem to me that Zstandard would be ripe for LR to use internally, due to it's ultra-crazy-fast compression & decompression (yet superb ratio).  I suppose I might have to finally check out one of the free SQL browsers I've downloaded recently to see if it can be figured out...  I've long been back-burner curious about SQL, anyways.

Although, I must say, I have mixed feelings about writing compressed data directly into the database.  On the one hand, it obviously makes the 'uncompressed' catalog size much smaller.  But on the other hand, I've been enjoying recompressing them (via unzip then solid-7zip) with various codecs so as to make the best use out of my free space with Google Drive - while at the same time trying hard not to abuse that free space, because this 'po-boy' depends heavily on that free space .  By playing around with the command-line options of 7-zip[-Zstandard], I've been able to deep-freeze older catalogs down to 48MB (of 1.2GB original) by using optimized options with LZMA(1)-level-9ultra codec.

Ugh!  I suppose I'll have to go through some more rounds of researching and testing various options of various codecs to see if some might do better with the new post-7.0 compressed-uncompressed catalog format.  Not looking forward to that, though, as I just got done with that a few months ago - and have already set up several menu-button scripts in Directory Opus for it.

Just when you think you have something figured out..  BOOM - technology goes changing on ya.  Will it ever end?


----------



## Wernfried (Nov 3, 2017)

I had a look at HEX data. It seems to be something "special", the signature "00 00 0" does not match any of these: List of file signatures - Wikipedia


----------



## Linwood Ferguson (Nov 3, 2017)

Hoggy said:


> Although, I must say, I have mixed feelings about writing compressed data directly into the database.  On the one hand, it obviously makes the 'uncompressed' catalog size much smaller.  But on the other hand


Personally I think this was a solution in search of a problem.  It was handy (for those who like to query in SQL) to have the history there, often I have looked for a specific event (e.g. preset applied, etc.) in the history, now that's not happening.

My uncompressed LR6 catalog was 3.1GB, my LR7 catalogs about 1.7GB.  Evening ignoring regular optimization the compression saved at must 1.4GB.

My preview cache is 75GB.

My images are 1700GB.

Seriously -- what's 1.4GB?  (for those not interested in doing arithmetic it's 0.079% of my space; it's not even a rounding error).

Now they have to uncompress and recompress it all the time, plus it's useless to outside queries.  Databases do NOT get significantly slower because they get bigger if they are properly designed, and I would be amazed if the CPU tradeoff for a bit smaller file was a performance improvement in fact rather than theory.

Wish they would have spent that time on other performance issues, personally. 

-- Signed grumpy old programmer.


----------



## johnbeardy (Nov 3, 2017)

I don't think Adobe owe anything to those who want to dig around the SQL - they've never promoted it as a way to get information - and it's certainly not fun to parse the Lua statements that are often returned, is it?

There was a problem with history data corruption too, though I didn't believe it until I had a case where 10 years of history data appeared to cause a slowdown. I had tried everything I could think of, and clearing history was the last resort. Performance was restored, and the halving of the file size was a handy by-product.

John


----------



## Linwood Ferguson (Nov 4, 2017)

johnbeardy said:


> I don't think Adobe owe anything to those who want to dig around the SQL - they've never promoted it as a way to get information


Well, I'll agree they are not interested in making it an open system all that much.


----------



## PhilBurton (Nov 4, 2017)

Ferguson said:


> Well, I'll agree they are not interested in making it an open system all that much.


Makes it that much harder for a future product that claims to import and process LR edits.


----------



## Hoggy (Nov 4, 2017)

PhilBurton said:


> Makes it that much harder for a future product that claims to import and process LR edits.



OH, that's a good point to consider...  Hopefully they don't also write compressed XMP to the files themselves when doing a 'write metadata to files'.  I would hope..


----------



## PhilBurton (Nov 4, 2017)

Hoggy said:


> OH, that's a good point to consider...  Hopefully they don't also write compressed XMP to the files themselves when doing a 'write metadata to files'.  I would hope..


I don't think that compressed data is allowed in the XMP standard.


----------



## johnbeardy (Nov 4, 2017)

Ferguson said:


> Well, I'll agree they are not interested in making it an open system all that much.



They are, or at least were. That's why they have the SDK.



Hoggy said:


> Hopefully they don't also write compressed XMP to the files themselves when doing a 'write metadata to files'.



The history log has never been written to the xmp, and it wouldn't be much value in another app. On the other hand, unlike competitors, Adobe has always written adjustment instructions to xmp in a form that could be interpreted by another app. So rather than trying to make Adobe the bad guys on this.....

John


----------



## Wernfried (Nov 4, 2017)

Hoggy said:


> OH, that's a good point to consider...  Hopefully they don't also write compressed XMP to the files themselves when doing a 'write metadata to files'.  I would hope..



Come on, it takes 10 seconds to verify this. XMP files are still clear XML text.


----------



## Hoggy (Nov 4, 2017)

johnbeardy said:


> The history log has never been written to the xmp, and it wouldn't be much value in another app. On the other hand, unlike competitors, Adobe has always written adjustment instructions to xmp in a form that could be interpreted by another app. So rather than trying to make Adobe the bad guys on this.....



That's good to hear..  I'm not concerned about history - just current develop settings and snapshots.



Wernfried said:


> Come on, it takes 10 seconds to verify this. XMP files are still clear XML text.



I have a disability that causes some 'issues' with me easily checking such things..  And it's been a very long time since I've done that, so I can't remember which program I used anymore (PhotoME, perhaps?).  Also bear in mind that I don't use separate XMP files - just embeds in DNG.

Do you know of an easily understood, still-being-developed/recent GUI type program to check that myself?  I mean, I know there's ExifTool - but that might take some good time for me to figure out how to use, and I haven't had a pressing need for it yet. (Although I still have my original intelligence..  It just doesn't work properly or quickly anymore.  )


----------



## Hal P Anderson (Nov 4, 2017)

Hoggy said:


> Do you know of an easily understood, still-being-developed/recent GUI type program to check that myself?


If you have a .XMP file, Notepad works just fine.


----------



## Hoggy (Nov 4, 2017)

Hal P Anderson said:


> If you have a .XMP file, Notepad works just fine.



I only use DNG here, so wouldn't mind knowing what's current out there for that.


----------



## PhilBurton (Nov 4, 2017)

Hal P Anderson said:


> If you have a .XMP file, Notepad works just fine.


Search for "microsoft XML notepad."  It's from 2007, but it works fine on my Win 10 64 system.

Phoil


----------

