Cannot invalidate highlight at startup and/or with manual text settings

[nutsmuggler]

I am experimenting with some code that is derived directly from the iOS sample code.
I am using a custom tree-sitter grammar I am building for the ABC music notation.
For reference, I am using a UITextView in a UIViewController, embedded in a UIViewControllerRepresentable.
Here is the whole code:
https://gist.github.com/nutsmuggler/c1d50e04c894324b1f0cd9b6a1502cfb

Highlighting is working, but only after I edit the text in the text view (a space or a delete are sufficient).

I tried to call the new .invalidate() method of TextViewHighlighter, but it has no effect.
For debugging purposes, I dispatched on the main queue with some delays, to make sure the text was there.
Here it is, it's very dirty but I wanted to see what happened:

    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)
        DispatchQueue.main.asyncAfter(deadline: .now() + 2) {
            let text = self.textView.text
            self.textView.text = "jsk"
            DispatchQueue.main.asyncAfter(deadline: .now() + 2) {
                self.textView.text = text

                self.highlighter.invalidate()
            }
        }
    }

It looks like highlighting invalidations is not triggered when the text is modified programmatically.

[mattmassicotte]

Ok, so I investigated this a bit. Here's the bad news: this appears to be a bug in TextKit 2.

At least under certain conditions, calling NSTextLayoutManager's setRenderingAttributes(, for:) does not result in the displayed text being re-drawn. You can force a text view back into TextKit 1 mode with the following code:

_ = textView.layoutManager

If you do that, things work.

TextKit 2 was totally unusable in macOS 12, but is a lot better in macOS 13. I'm unsure how it fairs in iOS, but it looks like this particular issue is still present on both iOS 16 and macOS 13. It could be that the issue is the approach. Controlling text styling is surprisingly difficult on macOS, and iOS has fewer APIs.

This isn't the first time something like this has come up, and it's tempting me to write specialized TextSystemInterface implementations for each one. With that in place, it may be possible to find a way to make this work correctly.

[nutsmuggler]

Ah, I see.
I had heard about the (ahem) joys of TextKit but never experienced them first hand :)

I tried the workaround, it does work; however, the layout is quite different: the font is drawn at a much smaller size (50-60% I'd say), and there's a noticeable flicker when typing. I imagine this is because of TextKit 1, right?

[mattmassicotte]

Probably not. My experience with TextKit 1 has, at least on macOS, been very positive. TextKit 2 is much simpler, but comes with some major bugs especially on older OSes.

Flickering is caused by latency between computing invalidation, tokens, and applying styles. Do you still have any Dispatch.async's to main floating around? This also could just be lack of testing on iOS. I'm not that familiar with UITextView and it isn't exactly the same as its NS counterpart.

There is also a Dispatch.async inside TextViewHighlighter. However, that was required to keep the implementation simple and reduce the number of dependencies. Removing it is non-trivial.

[mattmassicotte]

Can I help out any more here? The main branch is undergoing very significant changes, and if I can do something to improve the situation I'd like to!