PML 2.0.0 and Quoted Strings

[tajmone]

I've resumed updating the Sublime PML package so that it can support PML 2.0.0.

Question: If I've understood correctly, there is no longer the difference between quoted and unquoted strings, there are only unquoted strings — e.g. in title tags. Is that correct? Can I safely delete all code that used to handle quoted strings, and all associated completions, Symbol indexing, etc.?

---

I've been going through the PML CHANGELOG, in order to fully understand the backward breaking changes, but I soon realized there's more at stake then what mentioned therein — e.g. now that title is no longer an attribute but a node, it means that titles can span multiple lines, which requires some acrobatic tweaking in editor syntaxes that are RegEx based, sine they can only parse one line at the time.

It's really just a matter of time, since I have to ensure that all previously supported nodes and features are back and running again, which means having to fine-comb the whole package, plus some testing of edge cases with the new PML parser.

I hope that the package might be usable again in a week or two, and when this happens I'll just submit it to PackageControl so that users can be able to install it, even if not every PML feature is fully implemented (those that aren't will be simply treated as generic nodes).

The harder part is going to cover lenient parsing, which requires handling all tags individually, compared to shared snippets the syntax could count on before. But eventually it should be covered too, which is the key to provide context-smart completions.

[pml-lang]

If I've understood correctly, there is no longer the difference between quoted and unquoted strings, there are only unquoted strings — e.g. in title tags. Is that correct?

Nothing has been changed regarding "quoted and unquoted strings". Quoted and unquoted attribute values are still supported, as prior to version 2.
Attribute values are quoted, as shown in chapter Attributes of PML's user manual, e.g.:

name = "value"
name = "value with spaces"

However, as explained in chapter Lenient Parsing:

Quotes around attribute values can be omitted if the value does not contain:

• whitespace (a space, tab, carriage return or line feed)
• any of the following characters: [ ] ( ) " '

Hence lenient parsing allows us to alternatively write:

name = value

now that title is no longer an attribute but a node

Yes, this has been changed in version 2.
Prior to version 2 you would write:

[ch ( title = "My Title" )
    text
]

In version 2 you need to write:

[ch [title My Title]
    text
]

Note that the title must no more be quoted, because the title's text is now defined in a standard text node, and no more in an attribute value.
If you wrote:

[ch [title "My Title"]
    text
]

... then the quotes would be rendered in the target HTML document as part of the title (i.e. "My Title").

it means that titles can span multiple lines

Yes. And titles can contain inline nodes, e.g.:

[ch [title Title With [i Italic]
    and [b Bold] Words]
    text
]

The ability to render styled titles was one of the main reasons to no longer use attributes for titles.

Note that the new line in the title of the above example will not be rendered as a new line in the target HTML document, because of the Whitespace Handling rule applied in text nodes:
"Whitespace in text is replaced by a single space character."

Quoted attribute values can also contain new lines, e.g.

foo = "bar
   zar"

Note that whitespace in attribute values is preserved (not replaced), as stated here

I hope that the package might be usable again in a week or two

That's great. Thank you.

The harder part is going to cover lenient parsing

Yes, as discussed in #56

[coemgenuslcp]

from what I gathered so far from the VSCode documentation I don't think it will be possible

😭

[tajmone]

i.e. I don't think it will be possible without using LSP (see #56 on this).

There has been a discussion already on the need and advantages of creating a PML Language Server, which would then support all features (current and upcoming) across all editors that support them.

[pml-lang]

This means that editor syntaxes which index Symbols (e.g. TOC entries, as Sublime PML does) now need to be able to pre-process the symbol before indexing

Yes, the PML converter does this too when the TOC is created. It removes all inline styles. For example, if a title is defined like this:

[title Title With [i Italic]
    and [b Bold] Words]

... then PML converts the text to be used in the TOC to:

Title With Italic and Bold Words

Note: New lines are not removed yet in version 2.1.0, but this has already been fixed in the development branch.

As you can see, PML 2.0 introduced lot's of changes in terms of updating the Sublime syntax

Yes, I see. Your commitment to update the Sublime syntax is very appreciable.

from what I gathered so far from the VSCode documentation I don't think it will be possible.

I think you're right. For VSCode my suggestion would be to create a plugin for PDML, which would be much much easier than creating a plugin for PML, because there are only generic nodes and no lenient parsing rules. A PDML plugin would be useful in its own right (for all kinds of PDML documents), but it would also be helpful for editing PML documents, although of course to a lesser extend than your Sublime plugin. Support for comments and standard attributes (no lenient parsing rules) would be a nice addition and (IMO) easy to implement in VSCode.

[tajmone]

Yes, the PML converter does this too when the TOC is created. It removes all inline styles.

Right ... I had forgot about that, was only thinking of the newline chars and extra padding for alignment that needs to removed.

Anyhow, for now if the formatting tags are preserved in the indexed title is not a huge deal (kind of "literal string indexing"), but of course stripping the inline styles is better — at least in ST, which doesn't support formatting in indexed entries; VSCode might allow preserving those styles through HTML though.

For VSCode my suggestion would be to create a plugin for PDML, which would be much much easier than creating a plugin for PML,

If we're speaking of Basic PDML (i.e. the stripped down core, without even comments delimiters) than it shouldn't take more than an hour. I actually thought about it, but realized it was such a simple task that it might not be in itself a full fledged package — one would have to add at least some features like prettifying and minifying (like for JOSN), serialization to/from JSON, YAML, etc., and some other stuff like that, to make it interesting as a package.

[pml-lang]

If we're speaking of Basic PDML ...

Yes, exactly. This would be a very good start. Just displaying node starts and ends in a different color would (IMO) be useful, even in PML.

And, as said already, support for comments and standard attributes would be a nice addition.

prettifying and minifying (like for JOSN), serialization to/from JSON, YAML

Now that would be AWESOME!