Events

[gedw99]

Is your feature request related to a problem? Please describe.

SSE is the browsers way to send an event.

So how can we design an SSE abstraction for Server side workers.

also in a browser web workers use message passing so it’s not SSE after all.

It would be great if a worker can publish an event and any other worker can get it over a broadcast or pub sub mechanism.

Describe the solution you'd like

I don’t know how to map the pub sub pattern onto this project architecture.

broadcasting to all other workers would flood the system so that’s why we need a pub sub mechanism.

Describe alternatives you've considered

Event systems are meant to be late bound in that who do dunes the event is unknown at design time.

i guess a namespace with topics and subjects would be a decent structure into which an event can be published.

Then each workers that wants to listen for events can describe it in their config.

Additional context

To me there is sone aspect of this that relates to kv which was recently introduced.

I was thinking that we Puls reuse the kv system in order to hold and forward the event . It’s really like a kv with a watch flag in that you want to listen for any changes on that kv namespace.

So if you follow my gist then if the kv had a namespace then we can store data in kv as well as route events with it.

Most event systems have durability for the serverless pattern because that other service may not be running and so it can get the event later when it is up. So if I wanted this I would run a durable kv / pub System elsewhere and have my wasm code call that Server cluster. I imagine the endpoint would be an environment variable. So I wanted to ask if the team has any thoughts on how to describe external servers and how best to lay out config for workers to know this config

[Angelmmiguel]

Hey @gedw99,

Thank you for sharing this idea! I agree having different ways to invoke a worker is valuable. Currently, we can split Wasm Workers Server into 3 layers:

flowchart TD
    A(HTTP server) --> B(Wasm runtime)
    B --> C(Workers)

Loading

The only way you can invoke workers is by calling the related API endpoint externally. Those are the only events it supports, as it cannot listen to other kind of events. Supporting other approaches will open new use cases and applications.

Many systems work based on events. Reading your idea, I would go even a step further. Instead of embedding an event queue or reusing the current KV implementation, I believe it's more valuable to invoke workers based on external queues. In that way, you can run workers based on external events, allowing you to connect wws with your existing services.

The web workers you mentioned is an interesting example and I believe it's related to #206. The usage of SSE / Web Workers would depend on the JS / Browser implementation of wws. In any case, I see EventSource as another external queue we can connect workers.

[Angelmmiguel]

Linking this here as I think it's relevant to this discussion: #194 (comment).

Multiple config files could allow you to add other parameters in the future (like a scheduled job)

Cron-jobs could be another way to invoke workers that it's not related to external API calls.

[gedw99]

Curious what you think about this ??

a local pub sub that you can maximise perf by using sockets would be much better. So it can run isolated inside a browser or a server .

• low latency
• Network partitioning tolerant.
• fast

Let me call it a “Leaf”:

nats jetstream has what they call a Leaf server. A leaf is a pub sub server that can run even if the network is dead / partitioned and then catches up when the network is back up. https://docs.nats.io/running-a-nats-service/configuration/leafnodes/jetstream_leafnodes

Funnily bough others are compiling nats,go to wasm today and running it in a browser as a service worker to connect to a NAT Leaf Server somewhere.

So a a “ Embedded leaf” for wasm in the browser or native ( desktop / server ) would use message passaging and sockets. Not tcp or web sockets.

The hard bit is the sockets working on windows and Unix. Rust has a few libraries for this I believe.
The pub sub aspect is pretty easy as it just needs a fs and a way to take a message and store it and send it on to all subscribers.

about durability. It’s possible a red herring because in a service worker the OS browser can remove storage when’re ever it wants. On native it’s worth having possibly. Not sure.

Queue is also like cron tasks in that a message is sent and the first worker that takes the message runs. It’s a mini election.

Rpc is also possible which is messaging between two workers in both directions.

nats also has KV and Buckets that can be subscribed to from any workers. The KV in this repo can then leverage nats kv when the network is up, just like the pub sub.

Then you want to link outwards to a network based pub sub. Say nats server running on Leaf node.

i am basically talking about a Russian doll just like NATS but a smaller doll designed to run inside a service worker or a binary using sockets.

Nats has rust,golang, js, python drivers. So we could use the rust driver to link. Just like how others use nats.go ( as wasm ) to link to NATS server ( daemon service ).

Obviously we can’t restrict developers to just nats so we need a binding layer.

[Angelmmiguel]

@gedw99 sorry for the delay on my response. I’m currently attending WasmCon and the week has been a bit busy. I will reply you as soon as I come back :)

[ereslibre]

Thanks for starting the discussion @gedw99!

I also think having some form of pub/sub mechanism is interesting. I would start with a very barebones solution.

Having an implementation akin to the KV store would be a good first approach, but I would keep it separate from the KV store itself, so that we don't have to overcomplicate the KV implementation itself, and keep responsibilities separate.

I'm not fully sure how SSE can help in this case, I might be misunderstanding. As I understand it, Server-Side-Events are flowing in one direction (server -> client), they don't allow a client to send an event to the server, but the other way around.

In general, I think this is an idea worth exploring. Also, we'll find use cases that will allow us to refine the feature as we go, but I would keep the initial solution as simple as possible at the beginning: events stored only in memory, single instance, no orchestration of any kind, no permissions...