About JSON operator performance of repeative parse/stringify

[linonetwo]

I'm writing a gamification plugin like habitca, that will save some wiki change events in JSON array tiddler, and transform them to game rewards. So I'm dealing with JSON array using filter expression.

I find a usage in official doc that do the enlist for a JSON array:

[{$:/build-commands-as-json}jsonindexes[]] :map[{$:/build-commands-as-json}jsonextract<currentTiddler>]

(I change it from jsonget to jsonextract)

But each json operator will call $tw.utils.parseJSONSafe on the input, and jsonextract will call JSON.stringify(item) later, which wastes lots of performance. So filter expression is like the Unix shell pipeline, which can only pass strings. While PowerShell can pass objects through the pipeline.

But is it? Can we allow "object mode" for JSON-related operators? (I'm sorry that I didn't ask for this before its PR gets merged, we can't change it now, so I'm just asking about the potential that I write this kind of operator that only works in my plugin.)

[linonetwo]

As a JS plugin developer, I can provide my operators for wikitext users. I'm writing my all-in-one operators that do all the job in one operator to save performance. Just the name is a bit long...

/* eslint-disable @typescript-eslint/strict-boolean-expressions */
import { IFilterOperator } from 'tiddlywiki';
import { IGameEventLogCacheFile } from '../event-queue/GamificationEventLogTypes';

export const enlistJsonFilterTypeGetEvents = ((source, operator): string[] => {
  const typeOfEventToFilter = operator.operand;
  const results: string[] = [];
  source(function(_, jsonString) {
    const jsonArray = $tw.utils.parseJSONSafe(jsonString);
    if (Array.isArray(jsonArray)) {
      (jsonArray as IGameEventLogCacheFile).forEach((item) => {
        if (item.event.type === typeOfEventToFilter) {
          results.push(JSON.stringify(item.event));
        }
      });
    }
  });
  return results;
}) satisfies IFilterOperator;

But can't put too many features into one operator, otherwise it will be not flexible to compose them. So there still have to parse and stringify a few times for an expression.

[Jermolene]

Hi @linonetwo I explored this problem a year or two ago in #6932. The upshot of that work is that I believe we can move to using a polymorphic data type for filter values, but that there would be minor backwards compatibility issues that means that we probably need to defer this until we are planning to break backwards compatibility.