Ultimate UI goal: Scenarios that do everything without a graph file

[pinheadmz]

Most experiments with Warnet require at least one graphml file and one scenario. Graphml is a fairly annoying format and is likely to be overkill:

• it requires pre-defining all data tag "keys" at the top of the file
• we need to use tools to generate large graphs
• lightning channels and layer 1 p2p connections both use graphml edge's and the difference is subtle

SO WHAT IF...

• We restructure warnet more like a library
• Break out all the functions in scenarios like ln_init.py and eventually onion_init.py so they can be imported into new scenarios as library functions
• Add methods to generate graphs with a more robust format:
  • "i need 100 nodes, each with lnd and this circuitbreaker image" could be one single command, etc

ALSO...

There are tons of functions in the bitcoin test framework we cant support without stubbing (i.e. assert_debug_log()) and we do not need proxies-on-proxies for, like, code coverage. That's why I think we should plan long term on dropping the bitcoin test framework and writing our own.

HELL...

These single-point-of-entry scenario files could even launch kubernetes locally or ensure its already running or whatever. Then we can also remove all the justfile commands and integrate everything into one place.

Imagine:
"How do I use Warnet?"
"You write one file, and execute one command"

[josibake]

Agree with this direction longterm. I think we are outgrowing the initial idea of "everyone is going to use the bitcoin test framework" and have learned instead that usually there is a lot of setup required to get something working, e.g. opening lightning channels, mining a chain so wallets have UTXOs, etc.

The library approach makes a lot of sense to me in that you can pull stuff in as needed and then flexibility to write code for what you want to happen.

HELL...

These single-point-of-entry scenario files could even launch kubernetes locally or ensure its already running or whatever. Then we > can also remove all the justfile commands and integrate everything into one place.

This is the north star, imo. We can also dogfood any of the code we are exposing in a "library" by building the CLI directly on top of it, i.e. the CLI is a client of the library code we are writing.

[willcl-ark]

I mean, also concept ACK on this.

I'd just not that this was the intention of the graph generate command originally. It had capability to do n nodes, connected using any networkx-supported graph type, and could use random bitcoin core versions. It could in theory be reverted to behave like this, in addition to suporting LND and circuitbreaker stuff?

IMO I think one mistake we have arguably made is going too genric over everything. Perhaps we should define warnet as:

• Makes a network of n nodes
  • where m of those can optionally have associated LND/Core Lightning nodes
  • where p of those can optionally have associated circuitbreaker nodes
• Connected in this specific topology by default
• etc.

Perhaps the easiest way to do this using some "kind of" DSL as was previously discussed, would be with a config yaml/toml which warnet parses?

# Network Configuration Schema

[network]
# Number of total nodes in the network
n = 10

# Number of nodes with associated LND/Core Lightning nodes
m = 5

# Number of nodes with associated Circuitbreaker nodes
p = 3

# Specify the topology
topology = "erdos-renyi"

# Docker images for each node type
[docker_images]
# Docker image for regular nodes
regular = "node:latest"

# Docker image for nodes with LND/Core Lightning
lightning = "lnd:latest"

# Docker image for nodes with Circuitbreaker
circuitbreaker = "circuitbreaker:latest"

# Bitcoind node options
[bitcoind_options]
debug = 1
blocksonly = 1

At this point, it may not even be particularly worth bothering to parse this all into graphml. Just use networkx to generate a topology-type graph with the required nodes and edges, then use that mapping to make connections...

[pinheadmz]

Quick example i started writing that works but doesn't do anything yet. This file doesn't even have to be in the warnet directory as long as warnet is installed:

from pathlib import Path
from warnet.warnet import Warnet
from warnet.tank import Tank
from warnet.lnnode import LNNode

config_dir = Path("/tmp/wn_test")
network_name = "wn_test"

wn = Warnet(config_dir, "compose", network_name)
tank0 = Tank(0, wn)
tank0.lnnode = LNNode(wn, tank0, wn.container_interface, {"impl": "lnd", "ln_image": "lightninglabs/lnd:v0.17.0-beta", "cb_image": None, "ln_config": None})
tank1 = Tank(1, wn)
tank1.lnnode = LNNode(wn, tank1, wn.container_interface, {"impl": "lnd", "ln_image": "lightninglabs/lnd:v0.17.0-beta", "cb_image": None, "ln_config": None})

wn.generate_deployment()
wn.warnet_build()
wn.warnet_up()

[josibake]

but this example you've written here would work perfectly for what I am doing

[pinheadmz]

(w/e we wanna call it).

BATTLE ?

[josibake]

Thinking further: if we are to do what you're doing here with the example script, is there any reason to not just make warnet a python library you can pip install from anywhere? And then warcli is included as part of the python package?

The flow would then be:

1. setup docker / minikube / hosted k8s
2. pip install warnet
3. nvim my_cool_warnet_thing.py *python python python*
4. $ warcli scenario run-file my_cool_warnet_thing.py

This way, if someone has a graphml file they want to use, they can easily read it using networkx and then create the warnet objects they need, like in your example above, from their own graphml file. Similarly, if someone has a json file exported from LND, they can turn that into something warnet native, etc. We can also provide helper methods for the most common ones, like parsing a lightning network graph, or even a graphml utility?

[willcl-ark]

@josibake I hope you mean uv pip install warnet 😛

Packaging for pypi is no problem. The only disadvantage is that, unless we have an action to automagically push to pypi on merge (or daily or something), then it's a bit more maintenance work vs clone and uv pip install -e . Much nicer for end users though 💯

I do like this direction; I don't think we can have a "pure library" as we will need something (a server) to recieve commands? And possibly some state?

I suppose this could in theory be stateless though if we imagine the state to be the config file, in the same way bitcoin-cli will read bitcoin.conf before sending every command... So run-file would still spawn a warnet server (can be done with subprocess.Popen(spawn_new_session=True): link)

Subsequent warcli commands would know how to contact the spawned server using details found in the same config used to spawn it?

[josibake]

In a k8s / minikube world, we don't need to run a server. We need to create a client config in the script (e.g. wn = Warnet(config_dir, "compose", network_name) and then operate everything else from there.

Even if using docker, I think we should move the RPC server to running inside compose. I do think we need to get away from this idea that the script / user ever needs to spawn a warnet server on their machine, outside of warnet backend.

For warcli this could be something like warcli init and then it asks you to choose your backend and then deploys everything needed to get that backend working and ready to accept commands (e.g. starting the rpc server inside the backend, starting the port forwarding, etc). Once init is done, you can continue per usual with writing scripts and sending them to the RPC server with run file.

[mplsgrant]

I like the idea of using a library approach rather than a graphml + scenario file approach.

The support for moving away from using the test framework surprises me a little. I imagined us making certain incisions in the test framework code to transplant TestNode out of its local popen process and into a pod. Some functions need to be modified such as connect_nodes, but it seems like assert_debug_log just works. Unless the test framework changes a lot in Core, it may not add meaningful technical debt to support it as an option.

Then again, if I came to this project and it did not support the test framework, but it did have the ability to act as a warnet in its own way, then I actually may not be surprised that it didn't support the test framework. And perhaps the costs of supporting a framework that is out of our control may not be in our interest.

[willcl-ark]

I also agree that it would likely be a shame to drop the test framework. It has tonnes of useful utilities which would be great to have work out of the box. Of course, people could copy and paste what they need from that...

I also don't think we've scratched the surface of e.g. using a p2p node to perform lightweight actions on the network for us...

[josibake]

I'm ambivalent to using the test framework or moving away from it. I do think there is a ton of stuff in there we don't need and some of how the test framework is designed doesn't fit well with warnet, but there's also tons of utilities and ways that it does work really well.

My rule of thumb would be if we find that we are making design decisions in warnet around how the test framework works, that's a red flag. We should use whatever parts of the test framework make sense for warnet and re-write the parts that don't.