Boltdb panic during initial chain sync

[bajohns]

Expected behavior:

Start a fresh lnd node, point at suitable neutrino forked btcd node and sync chain to tip.

Actual behavior:

Lnd crashed after roughly 500k blocks due to panic from boltdb. From issues linked below this seems to be a race condition with respect to boltdb access.

lnd_1       | 2018-02-21 02:43:35.649 [DBG] BTCN: Sending getcfilter to 192.168.0.104:8333 (outbound)
lnd_1       | 2018-02-21 02:43:35.654 [INF] LTND: LightningWallet opened
lnd_1       | 2018-02-21 02:43:35.666 [INF] HSWC: Starting HTLC Switch
lnd_1       | 2018-02-21 02:43:35.671 [INF] RPCS: RPC server listening on 127.0.0.1:10009
lnd_1       | panic: page 25606 already freed
lnd_1       |
lnd_1       | goroutine 57 [running]:
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*freelist).free(0xc4204e8540, 0x1fa4a, 0x7fbd46406000)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/freelist.go:121 +0x29a
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*node).spill(0xc4204dbdc0, 0xc42014b300, 0x14ebcc0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/node.go:363 +0x1e0
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*node).spill(0xc4204dbd50, 0xc42014b2e0, 0x14ebcc0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/node.go:350 +0xb4
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*node).spill(0xc4204dbc70, 0xc4202710e0, 0xc4203f2d40)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/node.go:350 +0xb4
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4204f2700, 0xc420271000, 0xc4203f2fb0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:570 +0x17b
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4204f26c0, 0xc420270f00, 0xc4203f3220)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:537 +0x8f8
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4201b9518, 0x284007a3, 0x1538920)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:537 +0x8f8
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Tx).Commit(0xc4201b9500, 0xa, 0x14f3620)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/tx.go:163 +0x125
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb.(*transaction).Commit(0xc4204f0088, 0xc4204f0088, 0x0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb/db.go:91 +0x32
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb.Update(0x14eddc0, 0xc4201421e0, 0xc4203f34a8, 0x87caee, 0x180001)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/interface.go:225 +0xd1
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/filterdb.(*FilterStore).PutFilter(0xc420086000, 0xc42014ab80, 0xc420270ea0, 0xc4203f3600, 0x0, 0x0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/filterdb/db.go:170 +0x77
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).GetCFilter(0xc42042a780, 0x485af27c8018b071, 0x97193b0ae97a4b61, 0x57f8057bd2a24, 0x0, 0x57f8057bd2a00, 0x0, 0x0, 0x0, 0x0, ...)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/query.go:450 +0x527
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).blockFilterMatches(0xc42042a780, 0xc4204fe780, 0xc420270664, 0x0, 0x7cf25a4871b01880, 0xa3b1997614b7ae9)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:530 +0x138
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).notifyBlock(0xc42042a780, 0xc4204fe780, 0xc4204db730, 0xc420270660, 0x1, 0x2, 0x0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:452 +0x145
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).Rescan(0xc42042a780, 0xc4204f50e0, 0x7, 0xc, 0x0, 0x0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:432 +0xeff
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*Rescan).Start.func1(0xc4204f4fc0, 0xc4204f5080)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:741 +0xb7
lnd_1       | created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*Rescan).Start
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:751 +0x8d

Affects:
LND commit 2a61ccec
BTCD commit 0dcaa7f9
Running on Ubuntu 16.04 (in Docker)

More detailed log excepts can be found here

Related boltdb issues
boltdb/bolt#731
etcd-io/bbolt#72

Note: this occurs only on mainnet and not on testnet

[halseth]

Referring to discussion in lightningnetwork/lnd#1107, looks like one of the byte slices passed to the db transaction gets changed during the transaction?

The filter is fetched asynchronous from the peers, so it is a potential candidate:

neutrino/query.go

Line 429 in 36d9a50

filter = gotFilter

[halseth]

@bajohns You still seeing this? If you could compile with the -race flag that would be helpful :)

[Roasbeef]

Closing due to inactivity.