Make cln-grpc a first-class citizen

[kilrau]

Tl;dr: We at Boltz are building a new major client on CLN communicating with CLN solely via gRPC and we dearly miss decent gRPC support in CLN.

For us this means:

• Support all calls, add new calls to cln-grpc immediately - no more adding "on demand"
• Fix cln-grpc bugs with the same priority than lightning-rpc
• Enable cln-grpc in CLN by default

The current mode of adding gRPC calls on demand and not knowing if a node has gRPC available or not clearly doesn't work for anyone trying to build something serious:

Open Issues:
#7163
#6996
#6986
#6731
#6193
#5634
#5354
#5345

Closed Issues:
#6892
#6792
#6635
#6409
#6318
#6196
#6194
#6192
#6186
#6112
#6064
#5988
#5926
#5763
#5248

[cdecker]

Thanks @kilrau for taking the lead here. cln-grpc is a core component of CLN and Greenlight alike, and so I am interested in getting coverage and stability to the same level as the other client bindings too.

The JSON-RPC schema, from which the GRPC bindings are generated via msggen, are sometimes rather complex, meaning that the msggen converter can't handle them all as of now. We can either simplify the schema (my preference, but also slow due to deprecation cycles) or we can teach msggen how to handle these more complex structures.

So we'd be attacking this on 3 fronts:

• JSON-RPC Schema must be simplified to match the capabilities of other RPC systems we intend to use. GRPC for now.
• msggen needs to learn how to deal with the structures we have already
• Testing needs to be expanded to allow testing through the grpc interface too.

The last point already partially exists, and is what I used when building msggen for my own needs. If you set the envvar CLN_TEST_GRPC=1 you'll switch to using the cln-grpc interface instead of the direct JSON-RPC:

lightning/contrib/pyln-testing/pyln/testing/utils.py

Lines 831 to 835 in e331532

	if os.environ.get('CLN_TEST_GRPC') == '1':
	logging.info("Switching to GRPC based RPC for tests")
	self._create_grpc_rpc()
	else:
	self._create_jsonrpc_rpc(jsonschemas)

Technically, we have grpc2py.py which translates requests on the python side into the corresponding grpc messages, then sends them to the cln-grpc interface, and then it translates back into the JSON-RPC format so the tests don't have to change. This roundtripping could be a good way to ensure both interfaces remain stable.

What do you think?

[kilrau]

So first step is to fix msggen. @michael1011 gave it a look - it honestly looks like someone from your team who's already familiar with it will need to do that, too steep of a learning curve for an external contributor. I can't judge if simplifying the schema is currently possible or teaching msgen to handle the existing structures, but who from your team could work on it?

[rustyrussell]

My preference is to simplify APIs where necessary: the GRPC can simply use the new APIs so the only deprecation cycle is on the JSON-RPC side, and I can handle those.

Can we make a checklist of what APIs are missing? GH supports those natively, so we can use this issue to coordinate and aim to complete that by next release? (Sure, that means we need it done in three weeks, but pressure makes diamonds, right?)

[ohenrik]

I seconds the importance of this. While Building Torq and using CLN for Eden we notice that the biggest weakness that CLN has is the APIs and the gRPC. I think that improving this will have a massive impact on CLN adoption especially for enterprise.

Another thing. Not having subscription for certain data forces the need to build plugins. For example for payment status and channel interception.

[kilrau]

New issues since I opened this one:
#7168
#7169
#7176

[daywalker90]

I think these belong here aswell:

#7185
#7186

[NicolasDorier]

Just noting that in my case, the fact it is a plugin let me keep supporting arm32v7 due to this annoying issue. #6596

The fact I can remove it easily from the build in our fork fixed my issue. (I spent 1 day trying to fix it)

[Impa10r]

Don't want to start a new discussion, but hope for experts here to answer my question. Can gRPC methods be added dynamically at runtime for a CLN plugin? Looking for ways to make my PeerSwap Web compatible with Core Lightning.

[kilrau]

Can we make a checklist of what APIs are missing?

Here you go #7209 @rustyrussell - now even tests can catch missing calls

[daywalker90]

Can we make a checklist of what APIs are missing?

Here you go #7209 @rustyrussell - now even tests can catch missing calls

I have commits ready for all of them except sql/sql-template

[kilrau]

What's missing here?

[daywalker90]

Everything is in now except the sql/commando methods and methods that dont make sense via grpc e.g. batching

[kilrau]

Nice! Are there tests for the coverage via #7209 ?

[daywalker90]

Not that i know of

[kilrau]

I think we need tests. Then we can close here and be sure for missing grpc calls to at least cause a test to fail. @rustyrussell @cdecker

[cdecker]

I had started a grpc testing mode some time ago. It was swapping out the direct JSON-RPC interactions with one that was converting to grpc, call via cln-grpc and then concerts the protobuf result back to python dicts. That works nicely for some tests, but the conversion back to python is hacky (especially enum values) and so it never made it very far. I am hoping that we can pick that up again and fix it up, and add a test run to the CI.

[cdecker]

For future reference, this is the place we're we switch from json-rpc to grpc based on an envvar:

lightning/contrib/pyln-testing/pyln/testing/utils.py

Line 831 in a064b1a

if os.environ.get('CLN_TEST_GRPC') == '1':