Lots of trouble migrating to new server.

[damanic]

I have been failing to migrate a node with LND for a over a week now. Seriously stuck.

LND on a BTCPAY install. Data from all the docker volumes are copied across to new server, so all of the LND folder data is there. All the configurations are the same. Everything boots up as it should but LND show 0 channels.

    "num_pending_channels": 0,
    "num_active_channels": 0,
    "num_inactive_channels": 0,
    "num_peers": 11,
    "block_height": 860383,
    "block_hash": "000000000000000000005f3afe18f1cb004bd0114a09891f930badd7dfddc462",
    "best_header_timestamp": "1725758856",
    "synced_to_chain": true,
    "synced_to_graph": true,
    "testnet": false,
    "chains": [
        {
            "chain": "bitcoin",
            "network": "mainnet"
        }
    ],

I have run this past BTCPAY support and have confirmed the migration process completed as expected. I have run this multiple times and always same result.

For reference here are the backup and restore scripts used by BTCPAY . As you can see its fairly straight forward.

• https://github.com/btcpayserver/btcpayserver-docker/blob/master/btcpay-backup.sh
• https://github.com/btcpayserver/btcpayserver-docker/blob/master/btcpay-restore.sh

The new BTCPAY instance on new server has all the same versions as old. New server is aarch_64 and old is x86_64 in case that has any bearing on the situation.

So the question is. What could be preventing my channels from being picked up by the new install, and what can I do to bring those channels in?

Help hugely appreciated as I have been struggling with this for some time now.

[johnheenan]

Easy solution might be to

1. Disable lightning before backup
2. Do migration, checking lightning is still left disabled when migrated btcpay is backup
3. Check static channel backup states are the same each instance (previous and migrated)
4. Enable lightning on migrated server only

There are two backup scripts, the other one is btcpay-backup.sh

Both broadly do the same: they dump the database, shutdown btcpay, backup selected docker volumes and restart btcpay, so LND needs to be disabled before starting backup.

If both LND instances are up at the same time this is going to confuse the network or if the channel state gets out of sync with the backup then funds can be lost.

[damanic]

Yes sorry I linked the wrong backup script, edited comment to update link.

The backup script brings down all containers during backup. I then kept the 'old' server DOWN so LND not running whilst attempting to restore on the 'new' server.

All done as you described.

[johnheenan]

The backup script restarts what has not been disabled, so even if you stop btcpay before backup it will restart the LND container if it is not disabled.

The proposed solution is to not to bring down the LND container but to disable it before starting backup

[damanic]

How to disable LND before backup if not simply bringing the container down?

[johnheenan]

First record the result of

env | grep BTCPAYGEN_LIGHTNING

For LND it will be lnd, but confirm anyway.

Then to disable LND try:

cd btcpayserver-docker
export BTCPAYGEN_LIGHTNING=""
. ./btcpay-setup.sh -i

Confirm lightning is disabled.

To enable LND in migrated server:

cd btcpayserver-docker
export BTCPAYGEN_LIGHTNING="lnd"
. ./btcpay-setup.sh -i

[johnheenan]

Some more points. Disabling LND should leave LND docker volume in place.

You need to be careful a 'clean' operation with restore does not delete the LND docker volume of the currently disabled LND container.

[damanic]

The purpose of disabling LND ahead of the backup is to prevent if from being started at the end of the backup process right - if so could I not just copy the restore script and remove the line for btcpayup?

Or is there more to it than that?

[johnheenan]

As an alternative to disabling lightning, you would need to remove the btcpay_up command from the backup script in particular. I would also remove it from the restore script as I would want to crosscheck lightning static backup channel data between servers first.

Personally I would prefer to disable lightning as it gives more flexibility. I would not want to enable lightning again unless everything else was functioning as expected.

[johnheenan]

If I was going ahead with an actual migration and using the same web site DNS address, rather than testing the migration process:

1. I would redirect the old server website to a maintence page.
2. I would redirect the new server website to a maintence page.
3. Make DNS alterations for the new website IP or CNAME
4. Disable lightning (if I use it on the same server)
5. Remove 'btcpay_up' commands from backup script
6. Run backup script on old server
7. Copy backup files over
8. Remove 'btcpay_up' command from restore script
9. Remove any clean command from restore script
10. Run restore script on new sever
11. Run btcpay_up if restore runs without problems and everrything else is OK
12. Reenable lightning (if used) and the channel state backup is OK
13. Remove the redirection on the new server website to the maintenace page.

[damanic]

Apart from the removal of btcpay_up in the backup/restore scripts , that's pretty much what we are doing. I will try to backup and restore again with the btcpay_up removed.

In regards to :

crosscheck lightning static backup channel data between servers first
and
12. Reenable lightning (if used) and the channel state backup is OK

How would you determine if the channel state backup is OK?

[johnheenan]

crosscheck lightning static backup channel data between servers first
and
12. Reenable lightning (if used) and the channel state backup is OK

How would you determine if the channel state backup is OK?

Static Channel Backup (SCB) is a core feature of LND.

Check your channel.backup files are identical with checksums such as sha256sum. They exist in $LND_HOME/data/chain/$LND_CHAIN/$LND_NETWORK/channel.backup

Use find -name channel.backup to find.

There is a set of perhaps overcomplicated scripts to automate backup compatible with S3, with rsync (so ssh) and with custom backup at https://github.com/darwin/lnd-auto-backup. The link has further links to opinionated Medium aricles.

The core of the backup strategy is based around using inotifywait, which I illustrated in https://chat.btcpayserver.org/btcpayserver/pl/nk7fuhawjpfrpdrwgriixbjbgy

while inotifywait -r /directory/*; do
    rsync -avz /directory host:/target
done

In practice I would add in something like sleep 1 as the second line and would log behaviour with timestamps to see if the script needed adjustment, until I was satisified. I would also record previous checksum and compare, to avoid uncessary network activity.

An addiitonal subtle point: with regard to migration and DNS with lightning, most servers will use a different IP address, so you must keep your previous lightning server offline until DNS changeover is complete. This can take up to 48 hours. It does not mean you cannot be up within an hour say for some lightning migration. Just make sure the previous server does not enable lightning under any circumstances at the previous IP address for 48 hours after DNS changes, even if you are going to set up another lightning server at the old IP address. It also explains why a lightning migration might be sluggish for a few hours.

[johnheenan]

To speed up DNS change, a common technique is to reduce DNS TTL 48 hours before changover.

Also so some sort of file locking is advised. Since it may not be practical to signal a file lock to LND, the next best thing is compare a checksum of same files before and after backup. If they differ, try again. If they keep differing, anotrher strategy is required.

A file lock to LND might work by opening a channel.backup file for writing but not actually write to it. It would be necessary to open an issue or examine source code to see if this would cause problems. Source code changes might be required.

An improvement would be open channel.backup file for writing, copy contents to another file locally, close channel.backup for writing, copy copied file to remote backup. The improvement minimises the time a file lock would be held by opening channel.backup for writing (without actual writing).

[johnheenan]

With regard to redirecting a web site during maintenance:

1. It is easier to do if the reverse proxy web server is run outside BTCPay docker containers, such as with
   BTCPAYGEN_EXCLUDE_FRAGMENTS=nginx-https
2. If BTCPay is up but not ready for customers use, there is a useful changedomain.sh to change the domain to an unpublished name while redirection is in place.

As far as I can make out so far, there is only one channel.backup file for LND for lightning on BTC, no matter how many lightning channels are open. The file is called emergency.recover for clightning.

From https://docs.corelightning.org/docs/backup#backup

Core Lightning has an internal bitcoin wallet and you can backup three main components from the wallet:

• HSM Secret: On-chain funds are backed up via the HD wallet seed, stored in byte-form as hsm_secret
• Static Channel: Lightning channel states to recover are stored in a file named emergency.recover
• Database: Detailed information for funds locked in channels are stored in the database

Further down it is stated Static Channel Backup (SCB) is only updated when a new channel is opened. I assume it is also updated when a channel is closed. The purpose is to recover funds stuck in a channel.

Also further down:

This database is required for on-going channel updates as well as channel closure.

A way to do a sucessful migration without funds getting stuck in a channel (requiring closure ro recover funds) is to:

1. Completely disable LND and do not allow to start up again on old server
2. Start DNS changeover
3. Do a clean of unused docker containers before migration, if required, without BTCPay running on new server.
4. Do a migration without a clean of unused docker containers/volumes
5. Renable lightning on new container
6. Do a clean, if desired

If a lightning server crashes then with Static Channel Backup funds can be recovered (if file backed up externally). But don't expect business to just carry on as after some downtime, as if nothing happened.

[damanic]

@johnheenan
I have migrated many web services before so there is no need for you to post general server/dns migration tips here, although I sincerely appreciate the effort.

I will do another run keeping LND down during backup and restore, and add a checksum verification step. Thanks.

[johnheenan]

I have migrated many web services before so there is no need for you to post general server/dns migration tips here, although I sincerely appreciate the effort.

As you can see, I have moved to more to an approach that can be adapted into general guidelines.

It is not good that any skilled person has to put up a message that are having lots of trouble migrating and have been trying for a week.

[johnheenan]

I have migrated many web services before so there is no need for you to post general server/dns migration tips

This is not just web migration. DNS with lightning migration are not general tips and should be included in any comments about migration

You will potentially lose funds if another server thinks the old server is still active

So I will repeat in capitals:

KEEP LIGNTING DOWN ON OLD SERVER UNTIL DNS CHANGEOVER IS COMPLETE (48 HOURS). DOES NOT APPLY TO NEW SERVER.

For lightning to success we need to emphaise and encourage sound practices

[damanic]

It is not good that any skilled person has to put up a message that are having lots of trouble migrating and have been trying for a week.

touché

[damanic]

OK another run on backup and restore with modified sh scripts to remove btcpayup.

After backup and restore with all services down on both servers I get a checksum match on channel.backup:

old server

96aca72f279912b23dfed0a112eaf29a709ef0c35cf5f200913a8a5ff3f907a4  /var/lib/docker/volumes/generated_lnd_bitcoin_datadir/_data/data/chain/bitcoin/mainnet/channel.backup

new server

96aca72f279912b23dfed0a112eaf29a709ef0c35cf5f200913a8a5ff3f907a4  /mnt/nvme/docker/volumes/generated_lnd_bitcoin_datadir/_data/data/chain/bitcoin/mainnet/channel.backup

btcpay-up and still it is not picking up the lightning channels.

   "num_pending_channels": 0,
    "num_active_channels": 0,
    "num_inactive_channels": 0,
    "num_peers": 12,
    "block_height": 860542,
    "block_hash": "0000000000000000000196780fc440882cc5ba785ae8e78beff1d82d48f7311b",
    "best_header_timestamp": "1725841897",
    "synced_to_chain": true,
    "synced_to_graph": true,

Because the new server has not once managed to load in channels and therefore has been unable to perform any channel activity I have been able to restart the old server after failed migration attempts and continue running my node with ALL channels present.

I think at this point I would like to know how it is possible for LND on the new server instance to not be able to pick up the channels when it is running on the exact same data as the old server that can pick up the channels.

Here are some LND logs on boot that might point to an issue:

2024-09-09 00:54:11.946 [INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
2024-09-09 00:54:11.947 [INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
2024-09-09 00:54:11.948 [INF] CMGR: Server listening on [::]:9735
2024-09-09 00:54:11.949 [INF] SRVR: Initializing peer network bootstrappers!
2024-09-09 00:54:11.949 [INF] SRVR: Creating DNS peer bootstrapper with seeds: [[nodes.lightning.directory soa.nodes.lightning.directory] [lseed.bitcoinstats.com ]]
2024-09-09 00:54:11.949 [INF] DISC: Attempting to bootstrap with: Authenticated Channel Graph
2024-09-09 00:54:12.024 [INF] DISC: Obtained 3 addrs to bootstrap network with

Quite a few of these:

Pruned peer *redacted* from persistent connections, peer has no open channels

Pruned unconnected node

resend failed: unable to fetch channel sync messages for peer 

But no obvious errors.

[johnheenan]

I see this message in the logs

2024-09-09 00:54:12.024 [INF] DISC: Obtained 3 addrs to bootstrap network with

Can you confirm you are getting lightning gossip as a result of the bootstrap?

[damanic]

OK so new failed migration server is down, my old server is back up and running. All channels are up within 5 mins, the node is routing payments.

Now if both servers run the same version LND and docker, and both are using identical data volumes, we would expect the new server to behave as the old and boot up all channels fairly quickly.

So what is the difference? Could it be network connectivity?

lncli on the new server shows chain and graph sync, and as many peers as the old server.

old (finds channels)

    "num_pending_channels":  0,
    "num_active_channels":  11,
    "num_inactive_channels":  0,
    "num_peers":  12,
    "block_height":  860549,
    "block_hash":  "00000000000000000002f05d048e4988f60a5b06015f347c4bc0aff591aa1d2f",
    "best_header_timestamp":  "1725848042",
    "synced_to_chain":  true,
    "synced_to_graph":  true,
    "testnet":  false,

new (does not find channels)

   "num_pending_channels": 0,
    "num_active_channels": 0,
    "num_inactive_channels": 0,
    "num_peers": 12,
    "block_height": 860542,
    "block_hash": "0000000000000000000196780fc440882cc5ba785ae8e78beff1d82d48f7311b",
    "best_header_timestamp": "1725841897",
    "synced_to_chain": true,
    "synced_to_graph": true,

Is there not a way to force LND to load in the channels, or a tool to confirm it is aware of the channels, or a debug tool to find reason channels not loading in ?

[damanic]

Can you confirm you are getting lightning gossip as a result of the bootstrap?

Not right this minute because I new server is down, old is up. But looking at the logs on old I can see a lot more activity in boot that was not there on new. I have saved the log. When I attempt another migration I will save the log on new to compare.

Lets assume no gossip, would only cause be firewall?

I have allowed the following ports:
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 8333/tcp
ufw allow 9735/tcp

Gossip requires 9735 right?

[johnheenan]

So what is the difference? Could it be network connectivity?

It is not IP network connectivity.

This looks odd, 12 peers with no apparent chance of channel recovery.

    "num_pending_channels": 0,
    "num_active_channels": 0,
    "num_inactive_channels": 0,
    "num_peers": 12,

Gossip requires 9735 right?

Yes

The conventional wisdom is to close channels on old after sweeeping and reopen on new. CLN has a feature to auto-reblance channels that is regarded as relevant, LND does not.

I have not been able to find a single example of the migration approach working.

It does not mean it cannot work.

I would like to have a go myself.

Thanks for trying and reporting.

[damanic]

I did consider close and reopen but doc/articles point to copy of LND folder working in migration scenarios. I want to keep my nodes identity and avoid cost of reopen if not necessary, so I will hold off my migration until I get some kind of explanation as to what might be causing this issue.

I am very interested to hear how you go with a btcpay+lnd migration including channel(s).

[johnheenan]

First it is good the previous server came up with all channels open. Hence the new server was incorrect about closed channels.

These suggest the lightning network has a security feature to refuse to route to a new IP address between peers once a channel between peers is opened.

Does it make sense? It does not seem to. After all, keys can be re-exchanged to establlsh authenticity.

[johnheenan]

We have got a specific answer from LND maintainer guggero at #9075 (reply in thread)

The migration is faulty because the database was backed up before LND was shutdown

[johnheenan]

Sorry, the LND database is not part of the postgres backup. Will try my own migration tests to pinpoint the problem.

[johnheenan]

I have just ordered two addiitonal VPS servers becuse I am not going to interfere with existing servers. Since LND now supports pruned nodes I will hopefully be able to shorten the time to get my own tests going

[damanic]

@johnheenan Some good digging there.

In my lnd.conf btcpay sets two entries for externalip. One is set to mydomain.com:9735 and the other to mytoraddress.onion:9735 . The clearnet value is set via env var BTCPAY_ANNOUNCEABLE_HOST

I had another go at running migration steps and I tried setting the env var BTCPAY_ANNOUNCEABLE_HOST to the servers IP address but btcpay will always overwrite it with the BTPAY_HOST value unless it is a .lan or .local domain.

Its interesting that while there is a externalhosts config option, btcpay chose to place the servers domain in the externalip config option.

And also interesting that when BTCPAY_ADDITIONAL_HOSTS is set, they are not included in lnd config.

The other issue this presents is that when the domain for BTCPAY is intended to present a public HTTP/HTTPS webservice, the service could be placed behind a proxy like cloudflare which points DNS for the domain to cloudflare servers. In this scenario LND would be announcing the cloudflare IP address through domain resolution. So it seems necessary for BTCPAY to allow for LND to be set up with an external IP address instead of domain. I have asked about this on btcpay chat/support.

[damanic]

@johnheenan

Since LND now supports pruned nodes

Yes, the server I am trying to migrate away from is a pruned node opt-save-storage-s on VPS and has been operating without issue for a few years. One of the reasons for this migration is to put it onto bare-metal, full node.

[damanic]

Issue posted to BTCPAY in regards to LND ip address issues -> btcpayserver/btcpayserver-docker#927

Not sure if this is the cause of channels not being picked up, but having that issue resolved would help debug this issue as IP address could be set explicitly instead of having to hope for correct dns resolution/propagation.

[johnheenan]

Might as well start the ball rolling on how the tests will be conducted.

The two VPS servers are each 4 vCPU, 6GB RAM, 600GB SSD in Sydney with Contabo. They are currently both at 91% synced to BTC. LND is the lightning software. Their LN alias names are set to be au8b and au8c. The pruning is set to opt-save-storage (1 year or 100GB). The fast sync file used to accelerate catch up is date stamped last December.

The intention is to make as few changes as practical. I probably won't even enable BTC on BTCPay, only lightning on one strore on each server. That is all that is necessary. One difference from the norm is that I will use my own reverse proxy with nginx, because that is what I am used to:

export BTCPAY_HOST="......................."
export NBITCOIN_NETWORK="mainnet"
export BTCPAYGEN_CRYPTO1="btc"
export BTCPAYGEN_ADDITIONAL_FRAGMENTS="opt-save-storage"
export BTCPAYGEN_EXCLUDE_FRAGMENTS="nginx-https"
export BTCPAYGEN_LIGHTNING="lnd"
export LIGHTNING_ALIAS="au8b"
export BTCPAY_ENABLE_SSH=true
export REVERSEPROXY_HTTP_PORT=10080

The sequence of tests will be:

1. Connect outgoing channels each between au8b and au8c
2. Connect at least one outgoing channel from each to outside
3. Test btcpay backup and restore on each server to themselves individually
4. Backup btcpay on au8b and au8c. Copy au8b backup to au8c. Shut down au8b. Restore au8b backup to au8c.
5. If the above works, copy au8c backup to au8b. Restore au8c backup on au8b.
6. Deal with problems as they arise
7. Report results
8. Ask if anyone wants any other tests tried.

[johnheenan]

Pre testing channel state below

au8b and au8c are listed in 1ml.com

Their channel peer connections to random public nodes Caffeine (for au8b) and Cappero (for au8c) are currently not showing in 1ml.com.

Anyone who wants to open a channel for passive observation is welcome but not advised. I will be taking down the nodes in a few weeks.

Of particular interest might be observing what happens when nodes are migrated, as discussed above. Remember ip addresses will hopefully be sucessfully swapped!

Keys from 1ml.com or

For au8b
03946a15098e68b483035447e0e14eec3dbf7b19831a1b2b0f0c3d3e9f322a7868@46.250.240.39:9735
03946a15098e68b483035447e0e14eec3dbf7b19831a1b2b0f0c3d3e9f322a7868@tmjlmhybsxxddx7vexw3uieowzv4puebiieeck3io75vhakik57a34qd.onion:9735

For au8c
034d8d3b13f68c2109827b113636aa1144ed52ecf42c5ad8108bd22026d66a4ca6@46.250.241.55:9735
034d8d3b13f68c2109827b113636aa1144ed52ecf42c5ad8108bd22026d66a4ca6@nzi32vyubgqiqcdiigis6vtbuayhvke34th67mxdugdd3pqwcbusicid.onion:9735

Current au8b channel state:

Current au8c channel state:

[johnheenan]

The first phase of the tests are successful, which is backing up each server and restoring to self.

The LND server was disabled before backup on each server and remained disabled after restore. The LND server on each server was enabled after restore. All LN channels came back up again.

No alterations were made to backup and restore scripts.

I might leave the next phase until tomorrow.

[johnheenan]

The source of the problem has been determined. It is an apparent bug in btcpayserver.

The quickest and easiest way to get this dealt with is to submit an issue to btcpayserver tomorrow, as it close to midnight where I am, with the evidence and let them take over. It is puzzling. On the face of it, btcpayserver has done nothing wrong.

When the issue is resolved I will submit an answer here.

[damanic]

Close to midnight here too. Sounds like we are close to wrapping this up. Thanks for helping out, I might have given up otherwise!

[johnheenan]

Delete line 113 of file btcpay-backup.sh

  --exclude="volumes/generated_lnd_bitcoin_datadir/_data/data/graph" \

Pull Request created at btcpayserver/btcpayserver-docker#928

[damanic]

I did wonder about that line but did not have the balls to tinker at that level considering potential for loss of all funds.

Setting up a test environment was the way - really appreciate your time getting that all set up.

Will leave a bit of time to see how the posted issues are received then give the migration another crack towards end of week - and report back.

[damanic]

Delete line 113 of file btcpay-backup.sh

  --exclude="volumes/generated_lnd_bitcoin_datadir/_data/data/graph" \

Pull Request created at btcpayserver/btcpayserver-docker#928

You nailed it. Migration complete , all channels are up and active, nothing unusual in the logs.

This is x86_64 to aarch_64 which is untested and not recommended according to LND docs -> https://github.com/lightningnetwork/lnd/blob/master/docs/safety.md#migrating-a-node-to-a-new-device

I will hold off celebrations until I see first channel payment routed. But so far it is looking good.