Next features to add for wolfram language

[msollami]

Now that there's some initial support for the wolfram language, there are a few more features we could add and I've added examples of them in the attached pdf. I would like to hear which ones people think are most doable and their thoughts on implementating them.

prism-wolfram-feature-requests.pdf

[RunDevelopment]

Symbol vs User Expressions
Local Variables

Too complex. To track defined/undefined/local variables/arguments we would need a full-blown parser. We can only highlight syntax.

The problem with Prism's regex approach is that we can't highlight anything based on (complex) context. Using lookaheads and lookbehinds is the best we can do to detect context...

Function Arguments & Pattern Names (1)
Named Variables in system symbols
Named and indexed arguments in pure functions

Same here, tracking variables/arguments requires a parser.

Function Arguments & Pattern Names (2)

If it's only the #0 and #foo syntax, then it's doable.

We can simple make two (or one) patterns like this: /#\d+\b/, /#[a-z_]\w*\b/i

Named Messages

Seems doable, { pattern: /(::)\w+/, lookbehind: true }. Maybe we need a (?=\s*=\s*") lookahead?

Approximate Numbers (accuracy and precision)

Ideally, this would be part of the number token so that the whole "0.123``10" is a single number token. That regex will be complex...

I wonder if we could also support bases?

Multiplication Symbol shown in spaces between things

Can this symbol also be used in names? If yes, how does Wolfram distinguish between a×b (multiplication) and a×b (a single variable)?

If it can't be used in variable names, then the regex is simple.

Q: Are there also other symbols like this e.g. for division?

Connect prism with wolfram’s language server protocol?

Why?

[msollami]

Symbol vs User Expressions
Local Variables

Too complex. To track defined/undefined/local variables/arguments we would need a full-blown parser. We can only highlight syntax.

The problem with Prism's regex approach is that we can't highlight anything based on (complex) context. Using lookaheads and lookbehinds is the best we can do to detect context...

Ok, that makes sense.

Function Arguments & Pattern Names (1)
Named Variables in system symbols
Named and indexed arguments in pure functions

Same here, tracking variables/arguments requires a parser.

Agreed

Function Arguments & Pattern Names (2)

If it's only the #0 and #foo syntax, then it's doable.

We can simple make two (or one) patterns like this: /#\d+\b/, /#[a-z_]\w*\b/i

Nice, this one seems easy

Named Messages

Seems doable, { pattern: /(::)\w+/, lookbehind: true }. Maybe we need a (?=\s*=\s*") lookahead?

Ok, I'll try this.

Approximate Numbers (accuracy and precision)

Ideally, this would be part of the number token so that the whole "0.123``10" is a single number token. That regex will be complex...

I wonder if we could also support bases?

That's a great point. I would like to get all of the forms of numbers in that table to highlight as number tokens.
Are any of those obviously not possible or too expensive?

Multiplication Symbol shown in spaces between numbers

Can this symbol also be used in names? If yes, how does Wolfram distinguish between a×b (multiplication) and a×b (a single variable)?

Yes and no. In the WL frontend, when you type \[Cross], it is replaced with the cross symbol × (unicode \:F4A0). Many of these can be variable names or part of them, e.g. \[GothicCapitalB] = 1 is ok, but special reserved symbols that represent operators are not, e.g. a\[Cross]b = 1 doesn't parse.

If it can't be used in variable names, then the regex is simple.

I think you're right. So in WL all expressions need to be separated by a semicolon, otherwise proximity is multiplcation, so ppl find this useful becuase it helps visually identify missing semicolors. The frontend achieves this by adding a ghosted character just for appearance.

Q: Are there also other symbols like this e.g. for division?

Yes, a \[Divide] b. There are many many special charaters. Including custom input aliases and operator forms

Connect prism with wolfram’s language server protocol?

Why?

I realise it's not a parser but it would be cool to highlight errors through an lsp api in the future

[RunDevelopment]

I would like to get all of the forms of numbers in that table to highlight as number tokens. Are any of those obviously not possible or too expensive?

They should all be possible. The resulting regex will be long tho. Just ask if need a little help.

Yes and no. In the WL frontend, when you type [Cross], it is replaced with the cross symbol × (unicode :F4A0). Many of these can be variable names or part of them, e.g. [GothicCapitalB] = 1 is ok, but special reserved symbols that represent operators are not, e.g. a[Cross]b = 1 doesn't parse.

In that case, couldn't we just add all of these special reserved symbols to the list of operators?

There are many many special charaters. Including custom input aliases and operator forms

Maybe we don't need to support the input aliases? Or are they commonly used in code that will be highlighted by Prism?

I realise it's not a parser but it would be cool to highlight errors through an LSP API in the future

I think I still get your idea wrong. Do you want to use Prism as the backend of an LSP API or do want to enhance Prism's highlighting by querying an LSP API?

[msollami-sf]

I would like to get all of the forms of numbers in that table to highlight as number tokens. Are any of those obviously not possible or too expensive?

They should all be possible. The resulting regex will be long tho. Just ask if need a little help.

OK, I'll add them.

Yes and no. In the WL frontend, when you type [Cross], it is replaced with the cross symbol × (unicode :F4A0). Many of these can be variable names or part of them, e.g. [GothicCapitalB] = 1 is ok, but special reserved symbols that represent operators are not, e.g. a[Cross]b = 1 doesn't parse.

In that case, couldn't we just add all of these special reserved symbols to the list of operators?
Good idea, might be long list but I'll source it and we'll see.

There are many many special characters. Including custom input aliases and operator forms

Maybe we don't need to support the input aliases? Or are they commonly used in code that will be highlighted by Prism?

Yes they are more of an editor feature, let's skip this.

I realize it's not a parser but it would be cool to highlight errors through an LSP API in the future

I think I still get your idea wrong. Do you want to use Prism as the backend of an LSP API or do want to enhance Prism's highlighting by querying an LSP API?

So I was thinking more the later. The use-case I had in mind of for web apps with code snippet editors - wherein I could do auto-complete/suggestions/error-highlights (and other lexical things) in addition to syntax highlighting with js. Not possible/appropriate with/for prism yet obviously, but I think you probably know about some great related libraries I should consider.

[RunDevelopment]

I think I still get your idea wrong. Do you want to use Prism as the backend of an LSP API or do want to enhance Prism's highlighting by querying an LSP API?

So I was thinking more the later. The use-case I had in mind of for web apps with code snippet editors - wherein I could do auto-complete/suggestions/error-highlights (and other lexical things) in addition to syntax highlighting with js.

Frankly, for these editor-like applications, I'd recommend use an editor. Prism isn't designed to be used in editors (not that this stops some people from trying). Among other problems, Prism can't do differential highlighting or partial highlighting, so you can't use it with more than ~200 LOC and expect decent performance.

I think you probably know about some great related libraries I should consider

Sorry, I honestly don't. There is Prism live but that's pretty much just a proof-of-concept AFAICS.