Support for JSON dialects like JSON5, JSONC, Yaml, etc

[AlexRMU]

Add support for JSONC (JSON with comments) and non-strict JSON (JS objects), for example via the check mark in the settings
(of course, technically it's not JSON)

[josdejong]

Yes, I am open to that idea. Maybe also JSON5.

Before we do, I would like to see a little proof of concept first to see how this impacts the editor and what it requires.

[AlexRMU]

Yes, json5 is what we need

REPL

• During formatting/compacting, it does stringify/parse, so all non-json is deleted. You can use minifier/beautifier for js.
• You need to use a patch from fleece for change

[josdejong]

😎 nice to see this repl.

Yeah, so I think what is needed to make the editor work with multiple JSON variants:

• replace the internal data model from plain JSON to some standardized AST. I see golden-fleece mentions "The returned AST is ESTree compliant".
• We'll have to see if this AST can handle large documents. Currently the editor can handle files up to about 500 MB, which in large part because it directly acts on the plain JSON without an intermediate step to turn it into an AST. Maybe we need some hybrid approach.
• Implement syntax highlighting modes for these other JSON variants in the Text mode.

[Sinjai]

Can we use the existing [from what I understand] more-performant AST when possible and only use the new one as-needed, with some prompt/warning for documents >500mb?

@josdejong, if you could suggest starting point(s) for the requirements you detailed, feel free to post them and I can try to make some headway. Otherwise I'll try to dedicate some time regardless if I'm able, but I'm not familiar with e.g. syntax trees; I mostly have experience with object-oriented and shell/nix programming.

[josdejong]

I think we'll first have to do a small experiment to get an idea of how much work it will be (I expect a lot), and if it can be performant.

Good to know: the predecessor of svelte-jsoneditor, jsoneditor, did have a step to first parse JSON into an AST, and then render the AST basically. This AST was not performant, and only useable for JSON documents up to 10 or 50 MB or so. It could be that the AST implementation wasn't the most efficient of course. Anyway, the new library, svelte-jsoneditor is lazy in that regard, and stays really close to the JSON, only creating/keeping some additional state when it actually is going to render it. That is one of the reasons that tree and table mode are so fast.

[Sinjai]

Ah I see, I misread your point.

This is a stupid question, but surely there's a method for formatting JSON5 in a performant way, given that e.g. VSCode does it. Why is this situation different? (Not saying it isn't, to be clear, genuinely interested to know.)

[AlexRMU]

There are few parsers for json5 and they are not very fast (the difference is several or several dozen times).
We will probably have to write our own parser based on the existing one for json.

Useful links:
https://github.com/json5/json5
https://github.com/alemures/fast-json
https://github.com/Rich-Harris/golden-fleece
https://github.com/noahsug/json5-writer
https://github.com/rlidwka/jju
https://github.com/ota-meshi/jsonc-eslint-parser
https://github.com/ttys026/json5-editor
https://github.com/dimfeld/codemirror-json5

[josdejong]

Thanks for the list with JSON5 parsers @AlexRMU, that can be useful.

This is a stupid question, but surely there's a way for formatting JSON5 in a performance way, given that e.g. VSCode does it. Why is this situation different? (Not saying it isn't, to be clear, genuinely interested to know.)

It is a good question @Sinjai . That VSCode can do it doesn't mean that it is trivial to make it working 😉. To give a bit of context: svelte-jsoneditor has three modes, each requires a different approach:

• text mode is relatively easy: in the basis, you split the text into separate lines, and only render the lines that are visible. This mode currently works smoothly for files up to something like 50 or 100 MB.
• tree mode requires parsing the document in full and building up an internal model of the document, and keep state like which objects and arrays are expanded and which part of the array is rendered etc. For a large document (like 500MB), it can cost serious time to create this model, and you have to be careful not to blow up memory. The current solution creates this model lazily, which makes it fast. To prevent the editor from crashing the browser when rendering, you cannot render the full document, but have to keep nested nodes collapsed, and only render for example the first 100 items of an array. There is no simple way though to only render a small piece of the document like in text mode.
• table mode is also relatively easy: you parse the document, and then render only the rows that are visible (similar to text mode)

I expect that support for JSON5 requires parsing the code into a full AST, and I expect that naively parsing an AST is probably slow and may blow up memory for documents of 500MB. I doubt if there is a parser that can load and AST partially and lazily. So that may be a challenge, and indeed we may need to build a custom JSON5/JSON parser. Alternatively, we can accept that JSON5 can only work with small documents, and we can build an hybrid system that can use the JSON5 parser and the fast JSON parser + internal model side by side. To do that, we need to create an interface to interact with the document, and use that everywhere we want to read something from the document or operate on it. I think that may be the best option, but we need to be really careful: the complexity of that can easily explode, it will be a huge refactoring, and it will be a challenge to make it performant. It will require an experiment first to see what difficulties we come across.

I'm also not sure how to render for example comments in tree and table mode: a comment can located anywhere, before and after any opening and closing bracket and between keys and values. How can we keep the locations of comments intact? Both tree and table mode lose any manual indentation. One solution would be to only support text mode for JSON5.

[josdejong]

Moving this idea to the Discussions > Ideas section since we are not actively working on it.