udb: add upgradeToVersion12 #1126
Conversation
|
Few issues: A long time ago we had several different components (address manager, tx store, stake store) all being saved in the database, but they were versioned separately and one component could not use data from another because it could not assume the data format. This was fixed by moving everything under the udb package and merging all of the components under a single database version. The old upgrade code for each individual component was still necessary to bring it up to the state where the merge could be performed, but following that, no more upgrades are made to the components individually. See upgrades.go, upgrades_test.go, and the testdata directory for the new upgrade mechanism and testing infrastructure. Second, you can not delete or put values when using a cursor or within a ForEach function (which also wraps a cursor). This invalidates the cursor and further cursor usage can silently cause database corruption. Read all values which need changes first and then modify them at the end. |
|
Also, be aware of golang/go#18007 when generating the testdata files. I have needed to rename testdata to something else just to |
|
I didn't get why to generate new testdata since it is only data inconsistency fix. It should be fully compatible with v7 version, isn't it? |
wallet/udb/upgrades.go
Outdated
| // removeMultisigOutsWithoutTransactions is the tenth version of the database. | ||
| // It fixes issue with MultisigOuts without related transactions. Simply find all | ||
| // detached MultisigOutputs and remove | ||
| removeMultisigOutsWithoutTransactionsVersion = 10 |
There was a problem hiding this comment.
This name is a bit long. removeExtraMultisigOutsVersion reads better to me. The comment should also describe why they were an issue (failure to insert transactions due to multisig output double spend checks, because they were not always correctly removed in previous database code). And please don't use "simply" when describing the fix, just describe how the fix is performed.
wallet/udb/upgrades.go
Outdated
| metadataBucket := tx.ReadWriteBucket(unifiedDBMetadata{}.rootBucketKey()) | ||
| txmgrBucket := tx.ReadWriteBucket(wtxmgrBucketKey) | ||
|
|
||
| // Assert this function is only called on version 8 databases. |
wallet/udb/upgrades.go
Outdated
|
|
||
| var multisigOutsMarkedForDelete [][]byte | ||
| multisigOutBucket := txmgrBucket.NestedReadWriteBucket([]byte("ms")) | ||
| multisigOutBucket.ForEach(func(k, v []byte) error { |
There was a problem hiding this comment.
I'd prefer a cursor to be used here for consistency, instead of a ForEach. See the other upgrade functions, which all use cursors.
wallet/udb/upgrades_test.go
Outdated
| @@ -15,6 +15,7 @@ import ( | |||
| "path/filepath" | |||
| "testing" | |||
|
|
|||
| "errors" | |||
There was a problem hiding this comment.
move this to the above group with all of the stdlib packages
wallet/udb/upgrades_test.go
Outdated
| @@ -34,6 +35,7 @@ var dbUpgradeTests = [...]struct { | |||
| // No upgrade test for V7, it is a backwards-compatible upgrade | |||
| {verifyV8Upgrade, "v7.db.gz"}, | |||
| // No upgrade test for V9, it is a fix for V8 and the previous test still applies | |||
| {verifyV10Upgrade, "v7.db.gz"}, | |||
There was a problem hiding this comment.
If we are testing a 9 -> 10 upgrade, we should have a version 9 database with the error.
|
After upgrading, the database will be compatible with the older db test artifacts, however those won't expose the problem we are trying to fix here. |
|
Hi @jrick . I was exploring methods to create mock objects in multisigOut bucket and didn't find any public functions for it. Could you clarify that thing and maybe suggest some approach you would like to see? |
|
@jrick Travis CI error log doesn't seem like related to new code |
|
The error is fixed on master now. As for creating the test db, you can just write keys/values directly to the DB. This is fine for this case since we are intentionally trying to create an invalid DB that will be fixed during the upgrade. |
|
Is it ok If I will leave more complex solution for invalid DB creation? I didn't manage to create valid value of multisig record without using multisig methods |
|
The invalid DB for the test can be created in a more simpler way by using keys and values created from the |
|
Changed approach to use |
kLkA
force-pushed
the
force_db_upgrade_to_remove_mso_without_tx_exist
branch
from
eba6b21
to
3e900aa
Compare
wallet/udb/upgrades.go
Outdated
| Height: int32(mso.BlockHeight), | ||
| }) | ||
| if txVal == nil { | ||
| multisigOutsMarkedForDelete = append(multisigOutsMarkedForDelete, k) |
There was a problem hiding this comment.
to be on the safe side, make a copy of the key. bolt is unfortunately ridden with bugs related to cursor invalidation
There was a problem hiding this comment.
@jrick could you check this change in fresh version. Did I make it right?
kLkA
force-pushed
the
force_db_upgrade_to_remove_mso_without_tx_exist
branch
3 times, most recently
from
e57e14c
to
4072ab6
Compare
kLkA
force-pushed
the
force_db_upgrade_to_remove_mso_without_tx_exist
branch
4 times, most recently
from
6bde0c1
to
17d79ba
Compare
In this upgrade function we do check all MSOs for related tx existence. If TX doesn't exist we drop MSO
kLkA
force-pushed
the
force_db_upgrade_to_remove_mso_without_tx_exist
branch
from
17d79ba
to
0ce1ec7
Compare
|
Done |
#1036
In this upgrade function we do check all MSOs for related
tx existence. If TX doesn't exist then we drop MSO