[EPIC] Celestia-Node API

[renaynay]

Context

This issue supersedes and encompasses #169, serving as the canonical tracking issue for work related to celestia-node API.

Current design

Simple HTTP server with endpoints accessing certain methods mounted on the servemux. See here.

Desired design

An RPC client that provides modules based on the services provided by the node.

Goals

• eliminate need for go-cnc and DALC
• automated documentation of methods

Issue breakdown

• 1. Finish ADR
• 2. Node pkg refactoring #958
  • node: consider usage of new FX feature #923
  • refactor(nodebuilder): Move parse functions + flag defs to the respective modules #1126
  • node: Refactor how services are stored on the node #304
• 3. Per-module refactoring #959
• 4. Implement additional modules that don't yet exist #977
• 5. Scaffolding for new RPC server/client #962
  • Choose the RPC framework. #256
  • [Feature Request]: OpenRPC integration into Celestia Node #873
• 6. Reintroduce REST gateway with most frequent endpoints #1174
  • Enable gateway with flag #1183
  • add JSON-RPC autogenerated docs #1188
• 7. Complete RPC
  • Add client wrapper #1185
    • fix(rpc): unskip test once marshaling is finished #1303
    • fix(fraud): fraud.Subscribe should return a channel of Proofs #1207
    • feat(rpc): JSON (Un)marshaling for fraud.Proof and ExtendedHeaders #1208
  • Add fraud service to rpc #1186
  • add authorization #1187
• 8. Auxiliary
  • Rename nodebuilder/<pkg>/service.go files to respective package name #1278
  • fix(gateway): Ensure r.Context is getting canceled once the processing of the request ends with error or success or anything else. #726
  • feat(rpc/gateway): move to ListenAndServe #1286
  • fix(nodebuilder): clean up Config usage pattern #1287
  • chore(nodebuilder): change log loggers to $MODULENAME/constructor instead of $MODULENAME-module #1288
  • Provide DASer stub on bridge node #1299

Related issues

• fix(cmd/params): Repair encapsulation for network params #1076

[Bidon15]

Added #709 after grooming

[Wondertan]

Ok, so the main reason I would like to use JSON/Open RPC is the ability to do the following. Imagine we have a simplified DA interface:

type DataAvailability interface {
  GetByNamespace(context.Context, namespace.ID, Root) ([][]byte, error)
  SubmitData(context.Context, namespace.ID, [][]byte) Root 
}

and its implementation:

// implements actual logic to read and write from node network
type availabilityService struct {}

// performs network ops to get the data
func (availability) GetByNamespace(context.Context, namespace.ID, Root) ([][]byte, error) 
// sends PFD
func (availability) SubmitData(context.Context, namespace.ID, [][]byte) (Root, error) 

With Open/JSONRPC tooling we are aware of, we can do this:

server := jsonrpc.NewServer(...)
server.Register(new(DataAvailability), &availabilityService{})
server.Serve()

This is almost all you need to do to set up your own RPC server. The tooling parses the given interface and automagically setups the endpoint for each existing method.

Similarly on the client side:

client := jsonrpc.NewClient(serverAddr, new(DataAvailability))

client.GetByNamespace() // does request to the remote server instance of DataAvailabiliy interface and returns the results

Simple, powerful, and composable.

With regular Rest APIs there is a lot of boilerplate code that you cannot avoid. GRPC is somewhat similar to RPC, but it generates interfaces for you, which I personally hate. Interfaces have to be handwritten with care and succinctness and cohesionin mind

[renaynay]

🚀