MWS: Sync Thread

[hoelzro]

Per @Jermolene's suggestion, let's start a discussion around the sync mechanism within multi wiki server.

Existing comments:

• Introduce "Multi Wiki Server" Plugin #7915 (comment)

[hoelzro]

In his comment about this, @Jermolene mentioned Server-Sent Events, which I am also a big fan of 🙂

There are other options such as WebSockets and WebRTC, but in my opinion they'd probably be overkill for TiddlyWiki's needs, and they can sometimes result in fun network shenanigans.

There are a lot of libraries designed to help with syncing; is the goal to avoid libraries if possible, or are we open to using one if it brings enough value?

[joshuafontany]

I have been expermeating with websockets over the last few years. At first with Jed's "Bob server" app, most recently with a Yjs + websockets live multiplayer sync experiment with tiddlywiki.

I once got a TiddlyWiki hosted on a node.js server in Google Cloud live-syncing (with bugs) from Oregon to Italy. That was fun(!!!!!), and has lodged the idea in my head ever since.

I’ve come around to the idea that there should be a “fast-sync” layer that gets active wikis in memory in sync with each other (and could optionally also sync user details like cursor positions), and a “slow sync” layer that is actually writing to a shared data store that new wikis loaded into memory get “initialized” from.

Hard problem, especially around “source of truth” and possibly CRDT live multiplayer code.

[Jermolene]

I’ve come around to the idea that there should be a “fast-sync” layer that gets active wikis in memory in sync with each other (and could optionally also sync user details like cursor positions), and a “slow sync” layer that is actually writing to a shared data store that new wikis loaded into memory get “initialized” from.

I wasn't anticipating keeping hosted wikis in memory (we do need to support attaching a wiki object for static renderings, but I am hoping we can avoid keeping hosted wikis in memory).

[Jermolene]

Hi @hoelzro @joshuafontany I think the threshold for integrating a 3rd party library should be fairly high. The ones we already have in the core are:

• sjcl.js
• diff-match-patch
• bas64-utf8
• css color parser
• cssescape

The thing that they all have in common is that that they are implementations of non-trivial algorithms, and that none of them have any other dependencies. I would contrast that with libraries that are focussed on introducing new abstractions. Generally we have our own abstractions, and so I don't find much utility for them.

We need to discuss it over in the schema thread, but I am keen to update the schema to retain old versions of tiddlers. At that point, the database can be treated as a write only append store, which makes syncing significantly simpler.

I am not a fan of CRDTs. I think they are solving the wrong problem; they are mostly concerned with giving users the illusion of seamless simultaneous editing a la Google Docs. I am not a fan. If I'm concentrating on writing then the last thing I want to see is the distraction of what somebody else is doing. I am much more interested in providing users with tools for interactive conflict resolution.

[joshuafontany]

Agree. I don't mean to propose CRDTs for this sync case, but researching them has taught me a lot about data-syncing problems. We should also think about cases where one user has a wiki open in multiple browsers or devices. Bugs like this one (#7938) can pop up, and the TW UI is not really setup to expose the conflict to users. Still researching around this problem myself.

An append-only data store with a "tombstone" table could contain the type of action or a "reason" for the action that deleted the tiddler. This would allow a lookup, or history of the multi-step refactors (like renaming or relinking). This is a common pattern is distributed databases.

[Jermolene]

Thanks @joshuafontany a "reason" field on the delete marker does seem like a very useful idea.

Did you mean #7938? I see that one as being an issue with the draft mechanism.