Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database change notifications with Server-sent Events (SSE) #1807

Draft
wants to merge 44 commits into
base: master
Choose a base branch
from

Conversation

kodeFant
Copy link
Contributor

@kodeFant kodeFant commented Sep 1, 2023

Ref #1800

Usage example: https://github.com/kodeFant/ihp-sse

image

This is a proposal for enabling EventSouce endpoints that subscribe to database changes through Server-sent Events (SSE), letting you execute custom behaviour on the client triggered by change notifications from the database.

What's the win here, really?

The simplicity of autoRefresh, but allows fine-grained control on what and how updates reflect in the view.

No need to make stateful SPA component islands if you need to only auto-refresh certain parts of the page. Just have the database be the source of truth and sync the data, for example with htmx.

One example that is currently not easy with other techniques in IHP: An always in-sync inbox indicator in the navbar layout showing amount of unread messages.

image

With autoRefresh, you would need to use autoRefresh everywhere and explicitly track that table in every endpoint since it doesn't take in arguments from the controller, definitely not the right tool for this job.

With DataSync, it's a bit of overkill to set up just for this little feature.

Long-polling could be a viable option, but it means higher latency, more bandwidth-hungry and a higher toll on server resources.

Syncronizing this with SSE is simple, non-hacky, and cheap.

Why SSE

  • Simpler that websockets
  • Very cheap for both the server and client
  • It's nothing fancy, just a part of the web platform and supported by all modern browsers

Browser support: https://caniuse.com/eventsource

Description

Like autoRefresh it tracks tables, but instead of forcing a full page reload, you create an EventSouce endpoint that will subscribe to PostgreSQL event triggers.

Here is the controller code that is needed to make an event listener endpoint. You can track more than only one table if needed:

    action StreamPostsEvents =  withTableReadTracker do
        -- withTableReadTracker will automatically pick up database tables to subscribe to
        trackTableRead "posts"
        streamPgEvent "posts_updated"

Note that even though this is an HTTP endpoint, it does not render html or JSON, but streams an EventSouce subscription to the client.

Practically, this can give you DataSync with equal barrier to entry as autoRefresh.

Some advantages over DataSync is less need for JavaScript, and also very little Haskell code as seen above.

This for example hydrates updated HTML from PostAction into the view each time the posts_updated event is triggered from the StreamPostsEvents endpoint we defined above.

        <div hx-ext="sse" {...[("sse-connect", pathTo StreamPostsEvents)]}>
            <div hx-get={PostsAction} hx-trigger="sse:posts_updated">
                {printPosts posts}
            </div>
        </div>

Instead of HTMX, you could also recieve these events from JavaScript directly by instantiating an EventSource. In other words, this can be equally useful for those who use vanilla JS or jquery.

The example below could be an example solution on how to solve an unread messages indicator as described above.

function initializeEventSource() {
    const eventSource = new EventSource("/StreamPostsEvents");

    eventSource.onopen = function () {
        console.log("Connection opened.");
    };

    eventSource.onerror = function (err) {
        console.error("EventSource failed:", err);
    };

    eventSource.addEventListener('posts_updated', function (e) {
       // Fetch the new count of messages with a dedicated endpoint that will simply respond with a number
        fetch("/PostsCount")
        .then(response => {
            if(!response.ok) {
                throw new Error('Network response was not ok');
            }
            return response.text();
        })
        .then(data => {
            // Update the element with the new message count
            const newMessageCountElement = document.querySelector('#new-message-count');
            if (newMessageCountElement) {
                newMessageCountElement.textContent = data;
            }
        })
        .catch(error => {
            console.error("Failed to fetch the new message count:", error);
        });
    }, false);

    return eventSource;
}

TODO

  • Gracefully close heartbeat loop and postgres listener
  • If green light for the implementation, document in IHP guide
  • Look at possibility for making SSE endpoints that responds with HTML data directly (I think better to look at later)

@kodeFant kodeFant changed the title Database change notifications with Server Sent Events Database triggered Server Sent Events (SSE) Sep 1, 2023
@kodeFant kodeFant changed the title Database triggered Server Sent Events (SSE) Database change notifications with Server-Sent Events (SSE) Sep 1, 2023
@kodeFant kodeFant changed the title Database change notifications with Server-Sent Events (SSE) Database change notifications with Server-sent Events (SSE) Sep 1, 2023
@kodeFant kodeFant marked this pull request as ready for review September 2, 2023 01:27
@kodeFant
Copy link
Contributor Author

kodeFant commented Sep 2, 2023

Open to suggestions about the naming. Not quite sure DBEvent is the coolest, but SSE or EventSource would be too generic, like naming AutoRefresh to Websockets

Edit: Changed to PGEventSource and streamPgEvent, but still open to other suggestions

@s0kil
Copy link
Collaborator

s0kil commented Sep 8, 2023

At this point, should we consider integrating HTMX as part of IHP?

@kodeFant
Copy link
Contributor Author

kodeFant commented Sep 12, 2023

At this point, should we consider integrating HTMX as part of IHP?

Although (just to be clear) this implementation is purely frontend agnostic, I would most certainly use an HTMX boilerplate :)

@kodeFant kodeFant marked this pull request as draft September 15, 2023 10:41
@kodeFant
Copy link
Contributor Author

Converting it back to draft for now and letting it sit until I try it in real-life production first. I think the API could be improived

@mpscholten
Copy link
Member

Sounds good, let me know when things are for further review 👍

@kodeFant
Copy link
Contributor Author

kodeFant commented Oct 2, 2023

@mpscholten do you know if there already exists a good way to use row-level security (or something else) to prevent that the postgres notification triggers each time any column in the tracked tables changes?

@mpscholten
Copy link
Member

To make it work with RLS you need to refetch the data on every change. This is how it's done in DataSync as well. You can likely copy the code from here: https://github.com/digitallyinduced/ihp/blob/master/IHP/DataSync/ControllerImpl.hs#L138

It uses sqlQueryWithRLS which you likely want to use here as well (that function just wraps the query so that Row level security policies are applied, so that the refetching is done with RLS enabled)

@kodeFant
Copy link
Contributor Author

kodeFant commented Oct 2, 2023

thanks 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants