Migrate from boltdb to sql #585

sputn1ck · 2023-05-17T20:10:51Z

This PR adds the migration from boltdb to a local sqlite db or a postgres db. This will make it easier for future improvements such as taproots assets and other protocol changes.

Currently the migration is only done once from boltdb to the specified sql store. We should add a migrate command to loop which will allow the migration from sqlite to postgres and back.

TODO

loopdb/sqlite/queries/swaps.sql

GeorgeTsagk

cACK, not super expert in db migrations

will do another review run soon so I can also provide a tACK

GeorgeTsagk · 2023-05-23T12:26:15Z

client.go

@@ -192,13 +192,13 @@ func NewClient(dbDir string, cfg *ClientConfig) (*Client, func(), error) {
 }

 // FetchSwaps returns all loop in and out swaps currently in the database.
-func (s *Client) FetchSwaps() ([]*SwapInfo, error) {
-	loopOutSwaps, err := s.Store.FetchLoopOutSwaps()
+func (s *Client) FetchSwaps(ctx context.Context) ([]*SwapInfo, error) {


weird that we didn't have a ctx before. Isn't it like a rule-of-thumb to have a context with timeout when dispatching db calls?

+1 for adding timeout, perhaps something large like 30s would be a good start.

@bhandras do you mean on every call to add a timeout to the context?

What I mean is that we could perhaps wrap the ctx to a timeout context (per call to the store), to avoid hanging if the DB is for any reason unresponsive.

I'm not sure, but think the sql driver handles that? There are some timeout settings we pass in sqlite for example

GeorgeTsagk · 2023-05-23T17:20:06Z

loopd/migration.go

+
+	// If the migration was successfull we'll rename the bolt db to
+	// loop.db.bk.
+	err = os.Rename(


is keeping the backup what usually happens with migrations?

what if db was super big and keeping a duplicate causes problems?
(e.g a user with many swaps)
is the following a possible scenario:

user upgrades & triggers migration

migration completes but now they suddenly reached 99% storage capacity

service outage

or what if migration fails without completing, possibly exhausting all of storage space?

Good point. My rationale is to have a backup in case something goes wrong. IMO that's important for client side migration. Users can then delete their database on their own.

Should the user run out of space the migration would fail on the migration step before and loopd wouldn't start and the db wouldn't rename, so a user could expand their db. There would be no fallback just an exit from loopd.

I agree with @sputn1ck, it's important to keep the old DB around. It's also likely not large enough to cause any issues regarding storage space and if it is, then the migration above will already fail since this is just a rename.

loopd/migration.go

loopdb/sqlite/queries/swaps.sql

sqlc.yaml

loopdb/sqlite/queries/liquidity_params.sql

Roasbeef · 2023-05-27T00:09:20Z

loopdb/migrate.go

+
+	// For each loop out, create a new loop out in the toStore.
+	for _, loopOut := range loopOuts {
+		err := m.toStore.CreateLoopOut(ctx, loopOut.Hash, loopOut.Contract)


In order to speed this up, we want to be able to batch all the insertion here. Otherwise, you're doing a single db transaction for each swap. Instead a series of swaps can be created, within a single DB transaction.

bhandras

Awesome work! First pass looks really good already! Very close 🚀

bhandras · 2023-05-19T13:30:20Z

go.mod

@@ -28,10 +31,11 @@ require (
 	github.com/stretchr/testify v1.8.1
 	github.com/urfave/cli v1.22.9
 	golang.org/x/net v0.7.0
-	google.golang.org/grpc v1.41.0
+	google.golang.org/grpc v1.45.0


The bump is because of golang-migrate?

I'm actually not sure where the bump comes from.

loopdb/macaroons.go

client.go

bhandras · 2023-06-01T16:21:29Z

client.go

@@ -192,13 +192,13 @@ func NewClient(dbDir string, cfg *ClientConfig) (*Client, func(), error) {
 }

 // FetchSwaps returns all loop in and out swaps currently in the database.
-func (s *Client) FetchSwaps() ([]*SwapInfo, error) {
-	loopOutSwaps, err := s.Store.FetchLoopOutSwaps()
+func (s *Client) FetchSwaps(ctx context.Context) ([]*SwapInfo, error) {


+1 for adding timeout, perhaps something large like 30s would be a good start.

loopdb/interface.go

loopdb/sqlite_store.go

loopdb/postgres.go

lightninglabs-deploy · 2023-06-08T21:29:18Z

@sputn1ck, remember to re-request review from reviewers when ready

This commit adds a context to our loopdb interface, which we should use in the sqlite migration.

This commit adds a batch insert to the interface. This greatly reduces the time the migration will take. It is not implemented for the boltdb, as we will not be supporting migrating to the boltdb.

loopdb/sqlc/migrations/000001_swap.up.sql

bhandras

tACK, LGTM! Excellent work @sputn1ck 🎉 🎉 🎉

Please don't forget to update the release notes as well.

loopdb/sql_store.go

loopdb/sql_test.go

loopd/migration.go

bhandras · 2023-06-20T16:32:37Z

loopd/migration.go

+)
+
+// migrateBoltdb migrates the boltdb to sqlite.
+func migrateBoltdb(ctx context.Context, cfg *Config) error {


Perhaps as a separate PR we can also add the option to migrate sqlite to postgres. Not needed now since there're no users yet on sqlite.

yes I thought about that as well, shouldn't be too much extra work

This commit adds a nil check to the boltdb liquidity test. There was a difference between the sqlite and boltdb implementation. This test covers that.

This commit adds a postgres store to the loopdb package. Ths postgres migrator uses a replacer filesystem to replace the sqlite types to postgres types in the migration.

This commit adds a migrator to the loopdb package that manages migrating between 2 databases

This commit adds the ability to choose between different sql stores for loopd. The default is sqlite, but it can be changed to postgres.

addressed the requested changes

sputn1ck force-pushed the sqlite branch 5 times, most recently from 0322c84 to 0c26796 Compare May 19, 2023 13:11

sputn1ck marked this pull request as ready for review May 19, 2023 13:12

bhandras requested review from bhandras and GeorgeTsagk May 19, 2023 13:13

sputn1ck force-pushed the sqlite branch 7 times, most recently from 79160c1 to 1bb62b7 Compare May 19, 2023 13:42

sputn1ck commented May 23, 2023

View reviewed changes

loopdb/sqlite/queries/swaps.sql Outdated Show resolved Hide resolved

sputn1ck force-pushed the sqlite branch 2 times, most recently from 0247b6e to cfc3c0c Compare May 23, 2023 16:53

GeorgeTsagk reviewed May 23, 2023

View reviewed changes

Roasbeef previously requested changes May 27, 2023

View reviewed changes

sputn1ck force-pushed the sqlite branch from cfc3c0c to 40884be Compare May 30, 2023 07:43

sputn1ck changed the title ~~Migrate from boltdb to sqlite~~ Migrate from boltdb to sql May 30, 2023

sputn1ck force-pushed the sqlite branch 6 times, most recently from fe55f1c to 148e9e4 Compare May 30, 2023 09:19

bhandras reviewed Jun 1, 2023

View reviewed changes

sputn1ck added 2 commits June 19, 2023 12:28

multi: use context in loopdb call

becc8a3

This commit adds a context to our loopdb interface, which we should use in the sqlite migration.

loopdb: add batch insert

5d9d7c5

This commit adds a batch insert to the interface. This greatly reduces the time the migration will take. It is not implemented for the boltdb, as we will not be supporting migrating to the boltdb.

sputn1ck force-pushed the sqlite branch from 12104e4 to de9b3b9 Compare June 20, 2023 12:11

sputn1ck requested a review from bhandras June 20, 2023 12:11

sputn1ck force-pushed the sqlite branch from de9b3b9 to 1d2b9b5 Compare June 20, 2023 12:13

sputn1ck commented Jun 20, 2023

View reviewed changes

loopdb/sqlc/migrations/000001_swap.up.sql Show resolved Hide resolved

sputn1ck commented Jun 20, 2023

View reviewed changes

loopdb/sqlc/migrations/000001_swap.up.sql Outdated Show resolved Hide resolved

sputn1ck force-pushed the sqlite branch from 1d2b9b5 to 9822139 Compare June 20, 2023 12:17

tools: add sqlc generation tools

5c07c20

sputn1ck force-pushed the sqlite branch from 9822139 to a60de09 Compare June 20, 2023 12:19

sputn1ck added 4 commits June 20, 2023 17:31

loopdb: add loopdb sqlite migrations

3c58e98

loopdb: implement sqlite queries

85a757f

loopdb: generate sqlc files

206e463

loopdb: create base sqlite store

f8c65bb

sputn1ck force-pushed the sqlite branch from a60de09 to b592bc6 Compare June 20, 2023 15:31

bhandras approved these changes Jun 20, 2023

View reviewed changes

sputn1ck added 8 commits June 20, 2023 18:59

loopdb: add sqlite store implementation

0826b74

loopdb: add nil check to boltdb liquidity test

9278c18

This commit adds a nil check to the boltdb liquidity test. There was a difference between the sqlite and boltdb implementation. This test covers that.

loopdb: add sqlite test files

ab8923f

loopdb: add postgres store

3ee5bb6

This commit adds a postgres store to the loopdb package. Ths postgres migrator uses a replacer filesystem to replace the sqlite types to postgres types in the migration.

build: add postgres unit tests to makefile and CI

9d7f3bf

loopdb: add migrator

69b4df0

This commit adds a migrator to the loopdb package that manages migrating between 2 databases

loopd: Run migration if boltdb exists

a2abcd0

multi: add sql stores to loopd

270bc31

This commit adds the ability to choose between different sql stores for loopd. The default is sqlite, but it can be changed to postgres.

sputn1ck force-pushed the sqlite branch from b592bc6 to 270bc31 Compare June 20, 2023 16:59

build: add sqlite/postgres releasenotes

b41752f

sputn1ck force-pushed the sqlite branch from 3cc3fa3 to b41752f Compare June 20, 2023 17:11

sputn1ck merged commit 0781caf into lightninglabs:master Jun 20, 2023

sputn1ck deleted the sqlite branch June 20, 2023 17:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate from boltdb to sql #585

Migrate from boltdb to sql #585

sputn1ck commented May 17, 2023 •

edited

Loading

GeorgeTsagk left a comment

GeorgeTsagk May 23, 2023 •

edited

Loading

bhandras Jun 1, 2023

sputn1ck Jun 9, 2023

bhandras Jun 9, 2023

sputn1ck Jun 20, 2023

GeorgeTsagk May 23, 2023 •

edited

Loading

sputn1ck May 23, 2023

bhandras Jun 13, 2023

Roasbeef May 27, 2023

bhandras left a comment

bhandras May 19, 2023

sputn1ck Jun 9, 2023

bhandras Jun 1, 2023

lightninglabs-deploy commented Jun 8, 2023

bhandras left a comment

bhandras Jun 20, 2023

sputn1ck Jun 20, 2023

Migrate from boltdb to sql #585

Migrate from boltdb to sql #585

Conversation

sputn1ck commented May 17, 2023 • edited Loading

TODO

GeorgeTsagk left a comment

Choose a reason for hiding this comment

GeorgeTsagk May 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GeorgeTsagk May 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhandras left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightninglabs-deploy commented Jun 8, 2023

bhandras left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sputn1ck commented May 17, 2023 •

edited

Loading

GeorgeTsagk May 23, 2023 •

edited

Loading

GeorgeTsagk May 23, 2023 •

edited

Loading