SPX Boost Testing

[brendalee]

Thanks to the Boost team (@dirkmc, @nonsense, @aarshkshah1992, @jacobheun) for the great work in building a new and improved markets experience, driven by Storage Provider feedback around dealmaking.

Overview

The purpose of Boost is to replace the current version of Lotus Markets. It is currently being developed as a standalone repository to allow the team to iterate quickly on a redesign of the system, prove the merit of the new design early and often through internal and community facing demos, and to be releasable as a standalone binary that can run alongside the existing Lotus process.

Motivation

Current state & pain points

• Today, online deals trigger data transfer with a SP as soon as the SP accepts the deal. Often, the SP is in the middle of something else when this happens — e.g. sealing a CC sector, sealing a sector with deals, unsealing, etc. In other words, their sealing pipeline is full (may even have a queue of tasks) and is not able to accept new data. As a result, sometimes the data transfer fails (which fails the deal for the user), or the miner stops accepting online deals altogether for some time until their sealing pipeline is open again. The former leads to terrible client dealmaking UX. The latter is manual, inefficient, crude, and not very trustworthy for SPs.
• Clients have very limited information about SPs on the open storage market before they attempt to make storage deals — mostly pricing information through the ask protocol. In practice, there is much more info that a client needs to know in order to make a successful deal. For example, if an SP has set a minimum piece size, a client needs to comply with that otherwise their deal will be outright rejected. Clients operate on limited information when choosing SPs, and often encounter lots of surprises about availability and other hidden SP configuration parameters when it is time to make a deal. This leads to significant deal failure rates. Clients may not be monitoring the outcomes of their deal attempts closely, so may not know the deal has failed until much after the fact, at which point it can be exhausting for clients to re-attempt a deal.
• Standard online deals require clients/SPs to transfer data via Graphsync. While Graphsync is a solid data transfer protocol, clients sometimes desire to use other non-Graphsync data transfer methods/protocols with their online deals. E.g. wanting to use http or streaming protocols.
  The Filecoin deal protocol today does not allow this level of flexibility, so many clients today work around this restriction by circumventing online deals altogether and using the "offline deal" flow. This creates significant pain for clients b/c "offline deals" require clients to do upfront CAR/commP processing directly (for online deals today, lotus takes care of this upfront processing for the client) or remove their own deal control (e.g. Textile bidbot makes deals on behalf of its users, so they don't "own" the deals).

Expectations

• Improve the storage deal flow
  • Make replay-ability of deal stages much easier when the process experiences a recoverable error, such as "out of funds".
  • Reduce state complexity and debuggability through reduction based state transitions
  • Provide more flexibility of each deal stage to enable future extensibility/customization
• Improve visibility for Storage Providers into their systems
  • Add a simple webui to leverage the new Markets APIs
  • Better visibility of fund allocation, deal states including data transfer, and sealing pipelines
• Extensible data transfer
  • Online deals support fetching a CAR file over HTTP
• Improve team velocity
  • Reduce complexity in deal stage management

Instructions

• Please upgrade your test lotus-miner to the latest v1.15.1 release.
• All instructions on using boost can be found on the boost website under migrating from lotus to boost section.
• For any issues with the documentation, please also include that directly in your feedback.
• Please fill out this report template with your results, as well as any qualitative feedback that you may have.

Title	Metric
Changing deal ask config from UI is working	[x]yes []no
Adding fund to market actor is working through UI	[x]yes []no
Deal transfer is working in UI for all deals	[x]yes []no
Publishing deals from UI is working	[x]yes []no
All the deal parameters and logs are available in deal detail page	[x]yes []no
All the legacy deals are available in the UI	[x]yes []no
Sealing Pipeline displays the correct stage for deal sealing	[x]yes []no
Success rate and number of deals made via the Boost API (w/ the Boost client)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)
Hardware spec of Boost node
Average speed of transfers experienced previously with legacy markets (GraphSync)
Average speed of transfers on Boost
Network bandwidth
Number of deals initially migrated

Reminders

• Risk Mitigation
  • Please backup everything in the repositories before making any changes.
  • Roll back to legacy markets
• Reporting Issues
  • Please fill out this template
• Questions
  • Start here
  • @LexLuthr will be monitoring the relevant slack channels and Github
• User tutorials
  • We would appreciate help with user tutorials.
  • Example tutorial on how to store files with boost

[SBudo]

Testing results on Calibnet

Title	Metric
Changing deal ask config from UI is working	[x]yes []no
Adding fund to market actor is working through UI	[x]yes []no
Deal transfer is working in UI for all deals	[x]yes []no
Publishing deals from UI is working	[x]yes []no
All the deal parameters and logs are available in deal detail page	[x]yes []no
All the legacy deals are available in the UI	[x]yes []no
Sealing Pipeline displays the correct stage for deal sealing	[]yes [x]no (not in the case of snap-up sectors)
Success rate and number of deals made via the Boost API (w/ the Boost client)	92% (23 deals) - errors due to open file limit on OS
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	TBD

Testing results on Mainnet

Title	Metric
Changing deal ask config from UI is working	[x]yes []no
Adding fund to market actor is working through UI	[x]yes []no
Deal transfer is working in UI for all deals	[x]yes []no
Publishing deals from UI is working	[x]yes []no
All the deal parameters and logs are available in deal detail page	[x]yes []no
All the legacy deals are available in the UI	[x]yes []no
Sealing Pipeline displays the correct stage for deal sealing	[x]yes []no
Success rate and number of deals made via the Boost API (w/ the Boost client)	86/90
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	92/114
Hardware spec of Boost node	64GB RAM, 8-vCores AMD EPYC 7262 , 5TiB HDD SAS
Average speed of transfers experienced previously with legacy markets (GraphSync)	10-40 Mbps
Average speed of transfers on Boost	150 Mbps
Network bandwidth	200Mbps
Number of deals initially migrated	13461

[SBudo]

On the storage deals UI page, I sometimes gets two errors while deals are transferring:

• "Error: subscription connect timeout"
• "Error: context deadline exceeded"
  (see screenshot below)

Hard to reproduce (no real idea on how to), but I've noticed it only happens when multiple deals are transferring simultaneously.
Note that a refresh of the page fixes the issue

(running boostd v0.0.1+git.f62df00)

[LexLuthr]

#411

[tmyuu]

Miner ID: f0104671

Title	Metric
Changing deal ask config from UI is working	[x]yes []no
Adding fund to market actor is working through UI	[x]yes []no
Deal transfer is working in UI for all deals	[x]yes []no
Publishing deals from UI is working	[x]yes []no
All the deal parameters and logs are available in deal detail page	[x]yes []no
All the legacy deals are available in the UI	[x]yes []no
Sealing Pipeline displays the correct stage for deal sealing	[x]yes []no
Success rate and number of deals made via the Boost API (w/ the Boost client)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)

Tippy's index

Title	Metric
Hardware spec of Boost node	LXD Container (Xeon W1270/128GB RAM /1TB SSD)
Average speed of transfers experienced previously with legacy markets (GraphSync)	27MiB/s
Average speed of transfers on Boost	400MiB/s 🚀🚀🚀
Network bandwidth	10Gbps
Number of deals initially migrated	1633

[tmyuu]

Migration has been completed on mainnet.

Lotus version:

Daemon:  1.15.1-rc5+mainnet+git.913478b21+api1.5.0
Local: lotus-miner version 1.15.1-rc5+mainnet+git.913478b21

make note for something went wrong for me:

After running the migrate command, I found unable to access market (boostd) API from the storage miner. Due to missing the token file, I copied it manually then work fine. I also copied the dagstore directory in the same situation; however, it doesn't work fine. Therefore I'm currently running the initialize-all command.
The Web UI cannot be accessed directly from the Global IP. It looks like I'll need to use SSH tunnel or reverse proxy and so on. Error message is Error: Failed to fetch - check connection to Boost server.

[LexLuthr]

@tmyuu Did you keep the error you faced when migrating the Dagstore? Boost migrate should have done it automatically. I am going to create a new discussion to keep this thread clean and we can debug there.

[tmyuu]

I ran the migrate command again in another directory, but I don't see anything like an error.
There were no errors in the previous migration, so I had thought it was completed without any problems at the moment.

Getting markets repo '/usr/local/filecoin/.market'
Initializing boost repo
Trying to connect to full node RPC
Checking full node sync status
Worker: 6039; Base: 1701096; Target: 1701096 (diff: 0)
State: complete; Current Epoch: 1701096; Todo: 0

Done!
Checking if repo exists at /usr/local/filecoin/.boost_
Checking full node version
Creating boost repo
Migrating datastore keys
Importing all legacy markets datastore keys under /deals/provider
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 428 legacy markets datastore keys
Imported 30124 legacy markets datastore keys under /deals/provider
Importing all legacy markets datastore keys under /retrievals/provider
Importing 616 legacy markets datastore keys
Imported 616 legacy markets datastore keys under /retrievals/provider
Importing all legacy markets datastore keys under /storagemarket
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 858 legacy markets datastore keys
Imported 4954 legacy markets datastore keys under /storagemarket
Migrating markets keystore
Migrating markets config
Adding miner address f0104671 to datastore
Migrating storage.json file
dagstore directory size: 404 kB
dagstore directory size is 404 kB. Copy [c] / Move [m] / Ignore [i]:
m
Moving /usr/local/filecoin/.market/dagstore to /usr/local/filecoin/.boost_/dagstore
Boost repo successfully created, you can now start boost with 'boostd -vv run'

[TippyFlitsUK]

Miner ID: f08403

Title	Metric
Changing deal ask config from UI is working	✓
Adding fund to market actor is working through UI	✓
Deal transfer is working in UI for all deals	✓
Publishing deals from UI is working	-
All the deal parameters and logs are available in deal detail page	✓
All the legacy deals are available in the UI	✓
Sealing Pipeline displays the correct stage for deal sealing	✓
Success rate and number of deals made via the Boost API (w/ the Boost client)	9/10 (1 failure client side)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	1/1
Hardware spec of Boost node	EPYC 7282 - 500GiB RAM - 5TiB NVMe
Average speed of transfers experienced previously with legacy markets (GraphSync)	10MiB/s
Average speed of transfers on Boost	200MiB/s 🚀
Network bandwidth	1GiB
Number of deals initially migrated	4736

[neondragon]

Miner ID: f0694396

Title	Metric
Changing deal ask config from UI is working	[x]yes []no
Adding fund to market actor is working through UI	[x]yes []no
Deal transfer is working in UI for all deals	[]yes []no
Publishing deals from UI is working	[]yes []no
All the deal parameters and logs are available in deal detail page	[x]yes []no
All the legacy deals are available in the UI	[x]yes []no
Sealing Pipeline displays the correct stage for deal sealing	[x]yes []no
Success rate and number of deals made via the Boost API (w/ the Boost client)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)
Hardware spec of Boost node	Dual E5-2697 v2 (48c), 256GB RAM, 2x3.8TB NVME, 2x4TB HDD
Average speed of transfers experienced previously with legacy markets (GraphSync)	1-10MB/s
Average speed of transfers on Boost
Network bandwidth	1000Mb/s
Number of deals initially migrated	11284

[neondragon]

Web UI doesn't work when accessed remotely -- Error: Failed to fetch - check connection to Boost server

[LexLuthr]

#427

[dirkmc]

@neondragon note that you can now access the web UI over a local network since this PR landed: #420

Note also that you can run an ssh tunnel to access the web UI remotely:

ssh -L 8080:localhost:8080 myserver

Then open http://localhost:8080

[jamerduhgamer]

Miner ID: f0678914

Title	Metric
Changing deal ask config from UI is working	[X]yes []no
Adding fund to market actor is working through UI	[X]yes []no
Deal transfer is working in UI for all deals	[X]yes []no
Publishing deals from UI is working	[X]yes []no
All the deal parameters and logs are available in deal detail page	[X]yes []no
All the legacy deals are available in the UI	[X]yes []no
Sealing Pipeline displays the correct stage for deal sealing	[X]yes []no
Success rate and number of deals made via the Boost API (w/ the Boost client)	12 / 12 (100%)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	2209 / 47433 legacy deals (5%)
Hardware spec of Boost node	256 GB RAM, 16C AMD EPYC CPU, 7 TB SDD
Average speed of transfers experienced previously with legacy markets (GraphSync)	Varied too much
Average speed of transfers on Boost	500 Mbps
Network bandwidth	1 G
Number of deals initially migrated	2209

[cryptowhizzard]

Testing results on Mainnet f01608291

Title	Metric
Changing deal ask config from UI is working	[x]yes []no
Adding fund to market actor is working through UI	[X]yes []no
Deal transfer is working in UI for all deals	[X]yes []no
Publishing deals from UI is working	[x]yes []no
All the deal parameters and logs are available in deal detail page	[X]yes []no
All the legacy deals are available in the UI	[X]yes []no
Sealing Pipeline displays the correct stage for deal sealing	[X]yes []no (not in the case of snap-up sectors)
Success rate and number of deals made via the Boost API (w/ the Boost client)	100%
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	close to 100%
Network bandwitdh	30 Gbps
Average speed transfer Boost	around 600 Mbps
Average speed of transfers experienced previously with legacy markets (GraphSync)	depends . With 0 to 10 m/s latency > 300 Mbps.

[Angelo-gh3990]

Miner ID: f010446

Title	Metric
Changing deal ask config from UI is working	[X]yes []no
Adding fund to market actor is working through UI	[X]yes []no
Deal transfer is working in UI for all deals	[X]yes []no
Publishing deals from UI is working	[X]yes []no
All the deal parameters and logs are available in deal detail page	[X]yes []no
All the legacy deals are available in the UI	[X]yes []no
Sealing Pipeline displays the correct stage for deal sealing	[X]yes []no
Success rate and number of deals made via the Boost API (w/ the Boost client)	100%
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	80-100%
Hardware spec of Boost node	64GB RAM, 12C Intel CPU, 3 TB SDD on vmware/vSPhere
Average speed of transfers experienced previously with legacy markets (GraphSync)	50-100 Mbps
Average speed of transfers on Boost	400 Mbps
Network bandwidth	1 G
Number of deals initially migrated	11056