Introduce "Multi Wiki Server" Plugin #7915

Jermolene · 2024-01-02T19:12:58Z

Introduction

This PR introduces the "Multi Wiki Server" (MWS) plugin. It adds support for hosting multiple wikis at the same time. For performance and scalability, it uses SQLite to store tiddlers.

Hosted wikis are independent of the main wiki, which is still used for administration functions:

storing configuration for managing the hosted wikis
generating templates for static HTML routes

Recipe/bag Model

Hosted wikis can share content using the recipe/bag model from @cdent's TiddlyWeb. This is a simple but powerful way to organise content:

Tiddlers are stored in named "bags"
Bags have access controls that determines which users can read or write to them
Recipes are named lists of bags, ordered from lowest priority to highest
The tiddlers within a recipe are accumulated in turn from each bag in the recipe in order of increasing priority. Thus, if there are multiple tiddlers with the same title in different bags then the one from the highest priority bag will be used as the recipe tiddler
Hosted wikis are composed by splicing the tiddlers from the corresponding recipe into the standard TW5 HTML template

A very simple example of the recipe/bag model might be for a single user who maintains the following bags:

recipes - tiddlers related to cooking recipes
work - tiddlers related to work
app - common tiddlers for customising TiddlyWiki

Those bags would be used with the following recipes:

recipes --> recipes, app - wiki for working with recipes, with common custom components
work --> work, app - wiki for working with work, with common custom components
app --> app - wiki for maintaining custom components

All of this will work dynamically, so changes to the app bag will instantly ripple into the affected hosted wikis.

A more complex example might be for a teacher working with a group of students:

student-{name} bag for each students work
teacher-course bag for the coursework, editable by the teacher
teacher-tools bag for custom tools used by the teacher

Those bags would be exposed through the following hosted wikis:

student-{name} hosted wiki for each students work, including the coursework material
teacher-course hosted wiki for the coursework, editable by the teacher
teacher hosted wiki for the teacher, bringing together all the bags, giving them an overview of all the students work

Usage

# Clone the repo into ./TiddlyWiki5
git clone https://github.com/Jermolene/TiddlyWiki5.git --branch multi-wiki-support
cd TiddlyWiki5

# Install dependencies
npm install

# Start the server
npm start

Then visit the administration interface in a browser: http://127.0.0.1:8080/

The npm start command is a shortcut for the following command:

node ./tiddlywiki.js ./editions/multiwikiserver --mws-listen

Note that changes are written to the topmost bag in a recipe.

Note that until syncing is improved it is necessary to use "Get latest changes from the server" to speed up propogation of changes between users.

To run the tests:

./bin/test.sh

Related Work

SQLite Browser Wasm Experiments

Last year, I worked on #7329 which involved using the Wasm version of SQLite in the browser to make an alternate implementation of the wiki store object. The current status is that everything works but the overall performance is worse than the current JavaScript store implementation. I believe that it is possible to significantly improve on the JS performance by compiling filters directly into SQL. I hope to return to that work, but in the meantime I think we can get a lot of value from SQLite on the server without requiring the ability to execute filters against tiddlers in the database.

Bob.Exe

MWS has similar goals to @inmysocks's Bob.Exe at https://github.com/OokTech/TW5-Bob with the key differences being:

MWS uses SQLite for storing the content of the wikis
MWS uses the recipe/bag model for sharing content

Progress

vercel · 2024-01-02T19:13:04Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Jul 16, 2024 9:38am

pmario · 2024-01-03T11:48:38Z

@Jermolene -- Is this PR intended to collect early feedback, or just to track your own progress?

Jermolene · 2024-01-03T12:34:16Z

@Jermolene -- Is this PR intended to collect early feedback, or just to track your own progress?

It's no different from any other draft PR, which is to say that it is the product of working in the open, and is published so that others can review and provide feedback.

linonetwo · 2024-01-15T10:29:10Z

Cool, can different bags belong to different folders?

(So they can be synced to different Github repos, some of them are private bags and in private Github repo, and some public will be used to build public gh pages blog).

Jermolene · 2024-01-15T11:27:52Z

Hi @linonetwo

Cool, can different bags belong to different folders?

This is not a hierarchical structure; bags are flat and do not contain other bags.

Bag can appear in multiple recipes at the same time.

(So they can be synced to different Github repos, some of them are private bags and in private Github repo, and some public will be used to build public gh pages blog).

That is correct, there are several motivations that might apply for placing a particular set of tiddlers in a bag of their own:

To be able to independently include those tiddlers in any other recipe (ie to reuse them in other wikis)
To make it easier to sync those tiddlers independently
To be able to apply access controls to those tiddlers (bags represent access control boundaries)

linonetwo · 2024-01-15T13:58:17Z

Sorry for the confusion,

can different bags belong to different folders?

I want to ask about the folder structure on the file system, will it still be one tiddlers/ folder?

I know bag is a field on the .tid file or .meta file, so maybe all of these "bags" are still on the same tiddlers/ folder on the file system?

Jermolene · 2024-01-15T23:57:43Z

I want to ask about the folder structure on the file system, will it still be one tiddlers/ folder?

The tiddlers for the hosted wikis are stored in a SQLite database, they are not stored as files in the file system.

May not actually be needed

* fix breaking bug in image tiddler attachment * fix comments * fix code format * refactor processIncomingTiddler flow * remove whitespaces after if statements * refactor attachment_blob persistence flow * refactor process tiddler to support different attachments * add tests for attachment * add more attachement test cases * working on adding instanbul for test coverage report * code coverage report generation * remove unnecessary packages * fix comments

* fix breaking bug in image tiddler attachment * fix comments * fix code format * refactor processIncomingTiddler flow * remove whitespaces after if statements * refactor attachment_blob persistence flow * refactor process tiddler to support different attachments * add tests for attachment * add more attachement test cases * working on adding instanbul for test coverage report * code coverage report generation * remove unnecessary packages * fix comments * handle directory creation if doesn't exist for test store * resolve issue with CI tests failure

So that we can run on older Node.js versions

@pmario

Function definitions within a condition do not function as expected thanks to JS hoisting, so it is better to avoid it, and keep ESLint happy as well. See discussion at #8622 with @pmario

pmario · 2024-10-02T12:06:16Z

@Jermolene -- Which tools do you use to inspect the SQLite database. So I can see recipes, bags and tiddlers -- for dev testing

Jermolene · 2024-10-02T12:09:40Z

@Jermolene -- Which tools do you use to inspect the SQLite database. So I can see recipes, bags and tiddlers -- for dev testing

@pmario I use this: https://sqliteonline.com/

But I would love to spend a little time at some point making it possible to import arbitrary SQLite database files into TW as tiddlers, and then run SQL queries to extract/insert data as tiddlers. That would make a very nice extensible version of https://sqliteonline.com/ that users could customise with UI that directly reflects the schemas they are working with.

* mws authentication * add more tests and permission checkers * add logic to ensure that only authenticated users' requests are handled * add custom login page * Implement user authentication as well as session handling * work on user operations authorization * add middleware to route handlers for bags & tiddlers routes * add feature that only returns the tiddlers and bags which the user has permission to access on index page * refactor auth routes & added user management page * fix Ci Test failure issue * fix users list page, add manage roles page * add commands and scripts to create new user & assign roles and permissions * resolved ci-test failure * add ACL permissions to bags & tiddlers on creation * fix comments and access control list bug * fix indentation issues * working on user profile edit * remove list users command & added support for database in server options * implement user profile update and password change feature * update plugin readme * implement command which triggers protected mode on the server * revert server-wide auth flag. Implement selective authorization * ACL management feature * Complete Access control list implementation * Added support to manage users' assigned role by admin * fix comments * fix comment

* mws authentication * add more tests and permission checkers * add logic to ensure that only authenticated users' requests are handled * add custom login page * Implement user authentication as well as session handling * work on user operations authorization * add middleware to route handlers for bags & tiddlers routes * add feature that only returns the tiddlers and bags which the user has permission to access on index page * refactor auth routes & added user management page * fix Ci Test failure issue * fix users list page, add manage roles page * add commands and scripts to create new user & assign roles and permissions * resolved ci-test failure * add ACL permissions to bags & tiddlers on creation * fix comments and access control list bug * fix indentation issues * working on user profile edit * remove list users command & added support for database in server options * implement user profile update and password change feature * update plugin readme * implement command which triggers protected mode on the server * revert server-wide auth flag. Implement selective authorization * ACL management feature * Complete Access control list implementation * Added support to manage users' assigned role by admin * fix comments * fix comment * Add user profile management and account deletion functionality

pmario · 2024-10-31T22:07:33Z

package.json

+    "@playwright/test": "^1.47.2",
+    "better-sqlite3": "^9.4.3",
+    "node-sqlite3-wasm": "^0.8.10",
+    "playwright": "^1.47.2"


Do we really need @playwright/test and playwright as project dependencies. I think they should be moved to devDependencies

pmario · 2024-10-31T22:10:01Z

package.json

@@ -27,15 +27,23 @@
    "eslint": "^9.12.0",
    "@eslint/js": "^9.12.0"
  },
-  "bundleDependencies": [],
  "license": "BSD",
  "engines": {
    "node": ">=0.8.2"


I think we should update minimum Node version. This one is "stone age"

That's very true, but needs to be addressed in "master", not here

pmario · 2024-10-31T22:18:17Z

bin/ci-test.sh

@@ -2,6 +2,8 @@

 # test TiddlyWiki5 for tiddlywiki.com

+npm install


In line 15 there is npm install playwright ... which is also added to package.json, which is installed here. -- I think line 15 is redundant

pmario · 2024-10-31T22:19:33Z

bin/readme-bld.sh

@@ -10,6 +10,7 @@ fi

 # tw5.com readmes
 node $TW5_BUILD_TIDDLYWIKI \
+	+plugins/tiddlywiki/multiwikiserver \


Why does readme-bld.sh need the multiwikiserver plugin?

It's a temporary measure while the PR is in development. It allows us to include content from the MWS plugin in the readme.

pmario · 2024-10-31T22:34:58Z

package.json

+    "@playwright/test": "^1.47.2",
+    "better-sqlite3": "^9.4.3",
+    "node-sqlite3-wasm": "^0.8.10",
+    "playwright": "^1.47.2"
  }


better sqlite3 version does not "npm install" on my Windows machine using Node.js 22.10.0
Version 11.5.0 which is latest does install.

better-sqlite3 v11.0.0 -- dropped support for Node.js v21 -- see: https://github.com/WiseLibs/better-sqlite3/releases/tag/v11.0.0

pmario · 2024-11-01T01:35:48Z

plugins/tiddlywiki/multiwikiserver/modules/routes/handlers/manage-user.js

+				"user-initials": user.username.split(" ").map(name => name[0]).join(""),
+				"user-role": JSON.stringify(userRole),
+				"all-roles": JSON.stringify(allRoles),
+				"is-current-user-profile": state.authenticatedUser && state.authenticatedUser.user_id === $tw.utils.parseInt(user_id, 10) ? "yes" : "no",


@webplusai - If you use $tw.utils.parseInt(str) there is no need to add the radix param. Since the utility function takes care of it. See:

TiddlyWiki5/core/modules/utils/utils.js

Lines 980 to 982 in c7531e5

exports.parseInt = function(str) {

return parseInt(str,10) || 0;

};

JFTR, you don't need it with the language's parseInt either, if you're calling it as a global function, where the radix defaults to 10. Of course if you're passing a reference to it into a function that calls it with additional parameters, you can be in for a surprise or two. The classic example is:

['1.1', '2.2', '3.3'].map(parseFloat) //=> [1.1, 2.2, 3.3] // but ['1', '2', '3'].map(parseInt) //=> [1, NaN, NaN] // hmm

That is not an issue, though with the current usage, either of the native version or the TW utility.

netlify · 2024-11-06T15:21:10Z

✅ Deploy Preview for symphonious-seahorse-84f7f9 ready!

Name	Link
🔨 Latest commit	`3a5f67d`
🔍 Latest deploy log	https://app.netlify.com/sites/symphonious-seahorse-84f7f9/deploys/672b8be30df05d0008a09c0e
😎 Deploy Preview	https://deploy-preview-7915--symphonious-seahorse-84f7f9.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Tests were breaking in GitHub Actions but work OK on my machine. Problem was better-sqlite3 not being prebuilt, so I am testing to see if an update helps.

vercel bot deployed to Preview January 2, 2024 21:45 View deployment

vercel bot deployed to Preview January 3, 2024 16:31 View deployment

Jermolene added 2 commits January 3, 2024 16:47

Add new multiwikiserver edition

e9d3f67

Use a persistent disk-based database

68a89b6

vercel bot had a problem deploying to Preview January 5, 2024 10:59 Failure

Fix broken test

8543dda

vercel bot deployed to Preview January 5, 2024 11:05 View deployment

Remove debugging code

3f1f7c7

vercel bot deployed to Preview January 5, 2024 11:12 View deployment

Jermolene added 2 commits January 5, 2024 15:37

Fix create recipe SQL bug

1eed613

Add another recipe, improve docs

0799177

vercel bot had a problem deploying to Preview January 5, 2024 15:41 Failure

Fix broken test

1fb8b2e

vercel bot deployed to Preview January 5, 2024 15:49 View deployment

Jermolene added 2 commits January 17, 2024 22:41

First pass at admin user interface

615dc0c

Server extension framework

8941bd1

May not actually be needed

vercel bot deployed to Preview January 17, 2024 22:46 View deployment

Fix CI tests

50d0b14

vercel bot deployed to Preview January 18, 2024 09:49 View deployment

Merge branch 'master' into multi-wiki-support

a16338c

vercel bot deployed to Preview January 18, 2024 16:47 View deployment

Jermolene added 2 commits January 18, 2024 21:47

Fix getTiddler query

2f09c32

Admin styling

82fae45

Jermolene and others added 14 commits August 27, 2024 09:19

Merge branch 'master' into multi-wiki-support

eac8a2c

Clean ups from #8456

8a5a684

Merge branch 'master' into multi-wiki-support

c733116

Change for the sake of CI

90848a1

revert playwright test (#8573)

b698b3e

Avoid optional chaining syntax

4050ba5

So that we can run on older Node.js versions

Whitespace clean up

a9c124c

Fix load wiki folder command

0f4f60f

Merge branch 'master' into multi-wiki-support

6e7efeb

MWS icon remove redundant paths (#8621)

d0b5f41

Merge branch 'master' into multi-wiki-support

517dae8

Refactor conditional function definitions

6dd6b7b

Function definitions within a condition do not function as expected thanks to JS hoisting, so it is better to avoid it, and keep ESLint happy as well. See discussion at #8622 with @pmario

Jermolene mentioned this pull request Oct 2, 2024

i18n(ES) update to latest code base #8653

Merged

Jermolene and others added 4 commits October 17, 2024 13:19

Merge branch 'master' into multi-wiki-support

d0575d6

Fix plugin bag names

5d6ddae

pmario reviewed Oct 31, 2024

View reviewed changes

pmario reviewed Nov 1, 2024

View reviewed changes

Merge branch 'master' into multi-wiki-support

ad901a9

Update dependencies

3a5f67d

Tests were breaking in GitHub Actions but work OK on my machine. Problem was better-sqlite3 not being prebuilt, so I am testing to see if an update helps.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce "Multi Wiki Server" Plugin #7915

Introduce "Multi Wiki Server" Plugin #7915

Jermolene commented Jan 2, 2024 •

edited

Loading

vercel bot commented Jan 2, 2024 •

edited

Loading

pmario commented Jan 3, 2024

Jermolene commented Jan 3, 2024

linonetwo commented Jan 15, 2024

Jermolene commented Jan 15, 2024

linonetwo commented Jan 15, 2024

Jermolene commented Jan 15, 2024

pmario commented Oct 2, 2024

Jermolene commented Oct 2, 2024

pmario Oct 31, 2024

pmario Oct 31, 2024

Jermolene Oct 31, 2024

pmario Oct 31, 2024

pmario Oct 31, 2024

Jermolene Oct 31, 2024

pmario Oct 31, 2024 •

edited

Loading

pmario Oct 31, 2024

pmario Nov 1, 2024

CrossEye Nov 3, 2024 •

edited

Loading

netlify bot commented Nov 6, 2024 •

edited

Loading

		@@ -2,6 +2,8 @@

		# test TiddlyWiki5 for tiddlywiki.com

		npm install

	exports.parseInt = function(str) {
	return parseInt(str,10) \|\| 0;
	};

Introduce "Multi Wiki Server" Plugin #7915

Are you sure you want to change the base?

Introduce "Multi Wiki Server" Plugin #7915

Conversation

Jermolene commented Jan 2, 2024 • edited Loading

Introduction

Recipe/bag Model

Usage

Related Work

SQLite Browser Wasm Experiments

Bob.Exe

Progress

vercel bot commented Jan 2, 2024 • edited Loading

pmario commented Jan 3, 2024

Jermolene commented Jan 3, 2024

linonetwo commented Jan 15, 2024

Jermolene commented Jan 15, 2024

linonetwo commented Jan 15, 2024

Jermolene commented Jan 15, 2024

pmario commented Oct 2, 2024

Jermolene commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmario Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CrossEye Nov 3, 2024 • edited Loading

Choose a reason for hiding this comment

netlify bot commented Nov 6, 2024 • edited Loading

✅ Deploy Preview for symphonious-seahorse-84f7f9 ready!

Jermolene commented Jan 2, 2024 •

edited

Loading

vercel bot commented Jan 2, 2024 •

edited

Loading

pmario Oct 31, 2024 •

edited

Loading

CrossEye Nov 3, 2024 •

edited

Loading

netlify bot commented Nov 6, 2024 •

edited

Loading