Proposal: Add minerID param to retrieval deal

[Fatman13]

Proposal

Add minerID as a params for retrieval deal.

return Params{
	Selector:                &cbg.Deferred{Raw: buffer.Bytes()},
	PieceCID:                pieceCid,
	PricePerByte:            pricePerByte,
	PaymentInterval:         paymentInterval,
	PaymentIntervalIncrease: paymentIntervalIncrease,
	UnsealPrice:             unsealPrice,
}, nil

Use case

From experiences working with enterprise SPs who has on-boarded PiBs of verified deal, venus-market team has developed features that focuses on multi SP support.

The idea is that multiple SPs sharing the network infrastructure within a IDC could all share one market service to cut down costs and exposing just one multi-address on-chain.

Key merits includes...

• CapEx saving on infrastructure (IP, boxes, rack space)
• Easier deployment and management
• Potential new line of business for "market service provider"

Motivation

In order to realize the above use case, we propose adding minerID as a params for retrieval deal.

Given the above architecture, imagine the following scenario...

• Both SP f01001 and SP f01002 store pieceCID001
• Retrieval client queries SP f01001 for pieceCID001
• Retrieval client then initiate retrieval deal on pieceCID001 to market service
• At this point, market service would have no idea if SP f01001 or SP f01002 is responsible for unsealing the data

Discussion

Please feel free to let us know if we should expand more on anything we have presented. Thank you!

Resources

• Previous discussion on Slack

[dirkmc]

Thank you for putting together this comprehensive explanation with diagrams 🙏

Currently the procedure for making a retrieval deal is typically:

1. Client looks up miner ID (eg f0123445) on chain to get the miner's multi-address (eg /ip4/1.2.3.4/tcp/24001/p2p/Qm1234567890)
2. Client creates a query ask object with the root data cid (to get the price of the retrieval)
3. Client opens a libp2p connection to the multi-address and sends the query-ask
4. Storage Provider sends a query ask response with the price of the retrieval
5. Client opens a libp2p connection to the multi-address and makes a retrieval deal proposal with the root data cid
6. Storage Provider sends the data

From your explanation we understand that:

• for simplicity you would like to have a single markets process that handles incoming storage and retrieval deals from the network, and routes them to the appropriate Storage Provider
• each Storage Provider may want to charge a different price for retrieval
• if there are multiple Storage Providers who provide the same piece, it may not be possible to distinguish which Storage Provider to send the retrieval request to

In practice the retrieval price is associated with a multi-address (eg /ip4/1.2.3.4/tcp/24001/p2p/Qm1234567890) and not with a miner ID (eg f0123445). For this reason we're not sure if it would make sense to include a miner ID in the query-ask and in the retrieval deal protocol.

Our suggestion is that you use a similar approach to how a reverse proxy works:

• run a single markets process
• the process listens on many ports: one port per Storage Provider
• the Storage Provider advertises their multi-address on chain, including the port, eg
  • SP A advertises /ip4/1.2.3.4/tcp/24001/p2p/Qm1234567890
  • SP B advertises /ip4/1.2.3.4/tcp/24002/p2p/Qm0987654321
  • etc
• the client sends
  • a query-ask to /ip4/1.2.3.4/tcp/24002/p2p/Qm0987654321 and receives the pricing information
  • a retrieval proposal to /ip4/1.2.3.4/tcp/24002/p2p/Qm0987654321 and receives the deal data

[Fatman13]

Hello, @dirkmc! Thank you so much for the detailed reply.

However we still believe that this proposal would serve substantial utility for the use case presented above.

1

In practice the retrieval price is associated with a multi-address (eg /ip4/1.2.3.4/tcp/24001/p2p/Qm1234567890) and not with a miner ID (eg f0123445)

According to Filecoin Spec, we believe that retrieval (not pricing) is associated with a miner ID on the provider (miner) side.

Filecoin Spec:

The Retrieval Provider depends on the following dependencies
Host: A libp2p host (set setup the libp2p protocols)
Filecoin Node: A node implementation to query the chain for pieces and to setup and manage payment channels
StorageMining Subsystem: For unsealing sectors
BlockStore: Same as one used by data transfer module
Data Transfer: Module used for transferring payload. Reads from the blockstore

There is no clear definition on StorageMining Subsystem, but in order for Retrieval Provider to unseal sectors a retrieval request will have to be associated with a miner ID.

2

Our suggestion is that you use a similar approach to how a reverse proxy works:

We are aware of a workaround but it will complex management of different SPs using the market service and open up security risks.

Imagine market service serving up to 30 SPs. Normal add, remove and update of on-chain multiaddress would greatly increase management overhead. There are also operation overhead where the operation team has to rearrange firewall to open and close ports every time to support the aforementioned CRUD operation, which we don't think are practical.

Conclusion

Looking forward on your feedbacks and possibly give this proposal a revaluation. Thank you!

[dirkmc]

@Fatman13 could you please put together a proposal to add the miner address to

• the query ask request and response
• the deal proposal request and response

Note that

• in the requests the miner address should be optional
• the response messages should include a signature, signed by the miner address

[Fatman13]

Hello, @dirkmc happy new year! Hope you had a great holiday!

• the query ask request and response
• the deal proposal request and response

Both of the above would remain the same. What we propose to change is to add minerID to retrieval proposal.

Before:

// DealProposal is a proposal for a new retrieval deal
type DealProposal struct {
  PayloadCID cid.Cid
  ID         DealID  
  Params
}

Proposed:

// DealProposal is a proposal for a new retrieval deal
type DealProposal struct {
  PayloadCID cid.Cid
  ID         DealID   
  MinerId    *abi.ActorId
  Params
}

Responses would remain the same. Looking forward to your feedbacks and review! Thank you!

[dirkmc]

@Fatman13 I have some questions:

1. When the retrieval client sends a query-ask request to the markets endpoint, how does the markets endpoint know which miner to send the query ask to?
2. What if there are multiple miners who are storing the same deal data (with the same payload cid)?
3. Can they each send a response with their pricing information?

[Fatman13]

When the retrieval client sends a query-ask request to the markets endpoint, how does the markets endpoint know which miner to send the query ask to?

Thank you for pointing that out! Venus engineering missed it on the initial run. Yes, Query struct would need minerID too. And you are correct that right now market service doesn't know which miner to send query ask to.

Before:

// Query is a query to a given provider to determine information about a piece
// they may have available for retrieval
type Query struct {
	PayloadCID  cid.Cid // V0
	QueryParams         // V1
}

Proposed (in addition):

// Query is a query to a given provider to determine information about a piece
// they may have available for retrieval
type Query struct {
	PayloadCID  cid.Cid // V0
        MinerId    *abi.ActorId
	QueryParams         // V1
}

What if there are multiple miners who are storing the same deal data (with the same payload cid)?

Right now the pricing information of the first minerID in the list will be returned, which is not the ideal way of doing this.

Can they each send a response with their pricing information?

Only the first one on the list for now.

[dirkmc]

Right now the pricing information of the first minerID in the list will be returned, which is not the ideal way of doing this.

If we're going to make a protocol change, I think it needs to be comprehensive and thought through so that it works for all implementations (which may have their architecture set up in different ways). Could you please make a comprehensive proposal for this change. Note that if there is one endpoint that can return responses for multiple miners, each of those responses should be signed by the miner (so that the client can verify that the response indeed comes from that miner).

[Fatman13]

Simple Summary

Add a new param minerID *abi.ActorId to query-ask request and retrieval proposal.

Abstract

Current deal protocol as it is structured is not equipped to support multi miner use case. By adding minerID *abi.ActorId to both query-ask request and retrieval proposal, an endpoint that is responsible for managing multiple miners will be able to distinguish and tell which miner is asked to price information and which miner is asked to retrieve the CID.

Use case

From experiences working with enterprise SPs who has on-boarded PiBs of verified deal, venus-market team has developed features that focuses on multi SP support.

The idea is that multiple SPs sharing the network infrastructure within a IDC could all share one market service to cut down costs and exposing just one multi-address on-chain.

Key merits includes...

• CapEx saving on infrastructure (IP, boxes, rack space)
• Easier deployment and management
• Potential new line of business for "market service provider"

Motivation

Given the above architecture, imagine the following scenario...

• Both SP f01001 and SP f01002 store pieceCID001
• Retrieval client queries SP f01001 for pieceCID001
• Retrieval client then initiate retrieval deal on pieceCID001 to market service
• At this point, market service would have no idea if SP f01001 or SP f01002 is responsible for unsealing the data

Specification

For query-ask request

Before:

// Query is a query to a given provider to determine information about a piece
// they may have available for retrieval
type Query struct {
	PayloadCID  cid.Cid // V0
	QueryParams         // V1
}

Proposed:

// Query is a query to a given provider to determine information about a piece
// they may have available for retrieval
type Query struct {
	PayloadCID  cid.Cid // V0
        MinerId    *abi.ActorId
	QueryParams         // V1
}

Responses will stay the same.

For retrieval proposal

Before:

// DealProposal is a proposal for a new retrieval deal
type DealProposal struct {
  PayloadCID cid.Cid
  ID         DealID  
  Params
}

Proposed:

// DealProposal is a proposal for a new retrieval deal
type DealProposal struct {
  PayloadCID cid.Cid
  ID         DealID   
  MinerId    *abi.ActorId
  Params
}

Responses will stay the same.

Product consideration

Note that if there is one endpoint that can return responses for multiple miners, each of those responses should be signed by the miner (so that the client can verify that the response indeed comes from that miner).

The above product concern is addressed by current Venus architecture where the endpoint (venus-market) will initiate a sign request to remote venus-wallet (under that particular miner's custody) by routing the request to venus chain service and thus getting the signature from the remote wallet. Request to different miner will be routed to respective wallet.

[dirkmc]

@Fatman13 thanks for putting this proposal together.

I would imagine that it may be useful for a client to send a query for a particular payload CID to the endpoint, and to get back responses from several miners, each one specifying a price.

For example

1. The client sends a request for payload cid bafy123 to endpoint 1.2.3.4
2. Two miners have the piece corresponding to payload bafy123. Each miner includes their pricing information in the response:

• for miner A the unseal price is 1 FIL
• for miner B the unseal price is 2 FIL

3. The client chooses the miner with the cheapest price (or best reputation, etc).
   For example imagine that the client chooses miner A. The client sends a deal proposal to endpoint 1.2.3.4 with miner A's miner ID

[Fatman13]

Will com back to this soon!

[Fatman13]

Hello, @dirkmc, sorry for the late late reply. The team was caught up by holidays and nv18 upgrade.

I would imagine that it may be useful for a client to send a query for a particular payload CID to the endpoint, and to get back responses from several miners, each one specifying a price.

This would be greatly useful, but a change to existing protocol to accommodate this need would be drastic for a protocol. The proposal here would be simpler for small upgrade to the protocol.

If bigger changes are allowed as suggested, maybe we could do the following...

before

type QueryOffer struct {
  Err string
  Root  cid.Cid
  Offers MinerOffer
}

After

type QueryOffer struct {
  Err string
  Root  cid.Cid
  Offers []MinerOffer
}

Appendix

type MinerOffer struct {
  MinerId                 abi.ActorId
  PaymentAddress          address.Address
  MinerPeer               retrievalmarket.RetrievalPeer

  Piece                   *cid.Cid
  Size                    uint64
  MinPrice                types.BigInt
  UnsealPrice             types.BigInt
  PricePerByte            abi.TokenAmount
  PaymentInterval         uint64
  PaymentIntervalIncrease uint64
}