lemp stack: support upgrading from older versions of mysql when the grain shut down uncleanly.

[zenhack]

After the TTRSS upgrade, yesterday James Cook reported that his ttrss grain wasn't booting:

https://groups.google.com/forum/#!topic/sandstorm-dev/JggZ_kwFPEw

Inspecting the grain backup he sent me, I found this in the mysql error log:

2020-08-12T17:12:25.941031Z 0 [ERROR] InnoDB: Upgrade after a crash is not supported. This redo log was created before MySQL 5.7.9. Please follow the instructions at http://dev.mysql.com/doc/refman/5.7/en/upgrading.html
2020-08-12T17:12:25.941062Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-08-12T17:12:26.243949Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-08-12T17:12:26.244242Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-08-12T17:12:26.244397Z 0 [ERROR] Failed to initialize builtin plugins.
2020-08-12T17:12:26.244417Z 0 [ERROR] Aborting

So, my understanding of what happend here is that the grain had last shut down uncleanly before the upgrade. Unclean shutdowns are normally fine, as mysql (and any other database worth its salt) can recover automatically. But it seems that mysql cannot recover from an unclean shutdown if the crash happened when running an older version. So if mysql is shut down uncleanly and then the grain is updated to a newer version, recovery will fail on grain boot.

One way we could solve this is to embed older versions of the database for the purposes of recovering from crashes during upgrade.

It looks like the previous version used in ttrss was mysql 5.5?

[ocdtrekkie]

That's ugly and horrifying. :/

Is there any halfway decent way to tell it to abandon whatever changes it was trying to make prior to the crash or something like that instead? I loathe the idea of having to permanently include previous releases of MySQL. (Particularly since we don't have a working implementation of minUpgradeableVersion.)

[zenhack]

Is there any halfway decent way to tell it to abandon whatever changes it was trying to make prior to the crash or something like that instead?

This is what the recovery process does, so no apparently -- not without the version that was trying to make the changes.

But yes, we should get minUpgradableVersion working and then we could keep it to just one old version.

[ocdtrekkie]

It might be plausible to lift this from the TTRSS port as well... Our previous stacks would've been that version of MySQL as well, and it won't do anything to a grain where it's not applicable. And then if we upgrade the stacks' MySQL versions in the future we should be able to similarly mirror this behavior for additional versions of MySQL?

[zenhack]

Yeah, that should work.

(Side note, it might be nice to have a nix stack).