Ideas to fix mobile scroll sync latency issue for DOM to WebGL frameworks - Fixed

[martinlaxenaire]

Edit: this issue seems to be fixed with a little hack shown here: #75 (comment)

TL;DR:
Instead of listening to the scroll on the window element, the idea is to create a scrollable div the size of the viewport, and to listen to the scroll event of that div.

---

Calling to all WebGL devs!

This discussion aims to gather informations, documentation and ideas on how to improve scroll latency issue when it comes to sync WebGL elements with DOM elements on mobile.

This issue is, to my knowledge, common to all implementations of DOM to WebGL whatever the framework used, be it three.js, PixiJS, OGL, Babylon.js, curtainsjs or any other.

Currently, there's a noticeable delay between WebGL meshes and DOM elements positions visible when scrolling on mobile. This is more noticeable when going up then down then up, etc. while touching the screen.

The issue is actually easy to reproduce while trying to sync fixed positioned DOM elements and regular ones on scroll, without having to use any WebGL at all.

Here's a CodePen reproducing the issue:
[**Edit : updated the codepen to better reflect our WebGL usage:

• use a rAF because that's usually where we'll update the meshes world matrices, hence their positions
• using translate transforms to update the boxes positions to avoid triggering a repaint

**end of edit]

• code view: https://codepen.io/martinlaxenaire/pen/JjREzdv
• debug view: https://cdpn.io/martinlaxenaire/debug/JjREzdv/xJAjOeEgJqNk

Any suggestion (except for using a virtual scroll), article, documentation, idea and code example is very welcomed here.

Let's solve this!

[ZachSaucier]

One potential work around to this issue is to try and smooth the positioning over time (tween the position) using a requestAnimationFrame instead of setting it to the exact offset that it should be each time the scroll/resize fires. In summary: move toward the desired position each tick instead of going exactly to the end position. That way it stays smooth even if it's not perfectly synchronized with the DOM elements.

I created a basic demo of this sort of approach here (debug view here). It seems to work well on my iPhone though it is not very smooth on my Android device. Perhaps we can figure out why that is and possibly fix it?

In a production build this functionality could be enabled/disabled easily. It should also be normalized for devices that have a higher-than-normal RAF rate.

[martinlaxenaire]

This is an interesting approach, unfortunately as you said it still looks jumpy/unnatural on my device.
Maybe this could be refined tho?

[ZachSaucier]

@zadvorsky brought up another quite different approach that side steps this issue entirely:
Instead of using a single canvas, tracking the DOM elements, and positioning their virtual boxes to match the DOM positions, instead you could:

1. Render all of your WebGL pieces to a single off screen canvas (so that you can make use of a single WebGL context).
2. Replace each DOM element with a <canvas>.
3. Draw each piece of the off-screen canvas to the respective canvas that's visible.

The OffscreenCanvas API is perfect for this sort of thing but support is meh. You can do the same sort of approach without it though, just not quite as performantly. This is the approach that web components like the model-viewer component use. At this point I think it's a better approach than the approach I suggested above if it can be done well.

[ZachSaucier]

I wouldn't recommend rendering everything with Canvas. Doing so is limiting as to what people can do on their webpage, much harder to make accessible, and touches a bit more than what people would expect from a library like Curtains.

I'd be interested in looking at your implementation using an offscreen canvas.

[martinlaxenaire]

Oh I think I've misunderstood your suggestion.
You mean replacing every WebGL enhanced DOM elements with a <canvas> containing our offscreen WebGL context output cropped to fit the element?

Well that could work, but it needs a decent amount of time to make it work, and I am a bit afraid of the perf impact by drawing on all those 2D canvases.
I'll see if I can find some time to explore this further.

Edit: having a second thought on this, planes that have their vertices displaced might be cropped by the appended canvas, as our planes might overflow their original HTML sometimes...

[ZachSaucier]

Yes, that is what I mean 🙂 Sorry for not being more clear.

[martinlaxenaire]

Well, see my edit above: I think this could break when displacing a plane vertices

[ZachSaucier]

There would definitely need to be some extra spacing at least in some circumstances. But I think this method would the best bet of not having as much of an issue keeping things synced while scrolling on mobile.

[martinlaxenaire]

Before diving too deep into ideas and tests, I was willing to take a little step back to actually see what was happening when we were scrolling on mobile.

Here are some console debug screenshots taken with my Android Pocophone.

First here's what's happening on each frame while scrolling. We can see the whole task process is quite fast (7ms) and the scroll event is called before the rAF call, which is what we were expecting:

Another interesting thing is that we can see the touchmove event is fired at an higher rate than the scroll event/rAF calls.

According to this codepen, the touchmove event is fired approximatively every 10ms on my mobile, whereas my rAF is called every 16.6ms (60FPS).

So I've tried to update the positions inside the touchmove event instead of the rAF (see this codepen debug view), but the result is unfortunately pretty janky. Besides it would need to handle a lot of extra things, like window edges detection, window resize when the nav bar is hidden/shown, inertia that occurs after the touchmove event, and so on...

[luruke]

I believe is a lost war from the start 🙂
From my little knowledge, If you have a frame drop, browser will always prefer to prioritise its own UI then the javascript thread.

[martinlaxenaire]

Well yes, it might indeed not be solvable (at least yet), but it's worth a few tries.

I don't think the issue is coming from framedrops or CPU bottleneck tho. I believe it's more likely coming from a difference between the touch / screen refresh rates and the scroll event / rAF frequencies.
That'd mean that the browser would someway be able to composite things faster while scrolling than its main thread script executions speed would allow (does that sentence even makes sense? 😅)

[ashthornton]

I've managed to solve this with ASScroll.

I'll try to put a demo together at some point that demonstrates it but it will retain native mobile scrolling (i.e. inertia) and give you no visible lag between DOM and WebGL elements. But it should be fairly straightforward to stick into your own DOM/WebGL syncing set up.

I have tested it on many devices and its currently working on two sites we have in development with no reported issues so far.

I haven't tested whether or not the specific CSS I'm using to disable that pesky browser UI showing/hiding as you scroll is a factor in this though. It essentially sets to:

    position: fixed;
    overflow: hidden;
    width: 100%;
    height: 100%;

and the page container (the ASScroll container element) to:

    position: absolute;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    overflow-y: auto;

Like I said, I'm not sure if these CSS rules make a difference in the DOM/WebGL lag issue, but they're included in those sites to stop the annoying canvas resize due to browser UI shifting anyway.

[ashthornton]

Just threw together a quick demo: https://codepen.io/ashthornton/pen/f0db400416fff43ad5096b45f6974de4

Here's the live link so you can view on your touch device without the Codepen UI: https://codepen.io/ashthornton/live/f0db400416fff43ad5096b45f6974de4

The green squares are in WebGL.

So through trying to recreate the setup I have on the mentioned sites this is being used on, I found that the key might be to have an element covering the page but with pointer-events: none applied (the .pointer-blocker element in that Codepen). For some reason removing this in the Codepen brings the lag back.

I'd be interested to see if anyone has any problems with that demo!

And FYI, virtual scroll is not use on touch devices in ASScroll by default.

[martinlaxenaire]

I'm not setting the scroll position, just reading it from the page container and using that to set the Y position in WebGL. You can transform the page element using the window.scrollY value as people have tried in the past but you lose the native mobile scroll behaviour as well as a lot of performance.

Wait, so if I understand well, you're doing exactly what we're all already doing? Just listening to the scroll event and update the meshes positions based on window.pageYOffset?
So the only difference is your pointer-events: none hack?

[ashthornton]

Not using window.pageYOffset but the .scrollTop of the page container.

I don't know if the pointer-events hack is all that is needed, that would need to be tested. Not even sure if having to use .scrollTop is necessary either. I just found the perfect combination once and decided not to mess with it! It's worth breaking it down to find what exactly is needed though.

[ashthornton]

In fact, using window.scrollY or window.pageYOffset (both cause layout thrashing so should be the same perf-wise) may even be more performant than using .scrollTop of the element. I'm using .scrollTop because I'm using the hack to fix the page element so the mobile browser UI doesn't show/hide causing a resize event.

[martinlaxenaire]

Could it be that because you're combining both pointer-events: none and the hack to avoid the page to be resized by the browser navbar, the browser someway internally improves its rendering speed by anticipating resize/click events won't be triggered while scrolling?... 😮

I've tried to update the codepen test reference with the pointer-events hack, but I did not see any improvements:
Debug view: https://cdpn.io/martinlaxenaire/debug/jOMmpmP/VJkxxEGZORek
Editor view: https://codepen.io/martinlaxenaire/pen/jOMmpmP
I'll try to add the fix to avoid the browser UI resize as well. But I'm a bit confused for now.

[ashthornton]

Could it be that because you're combining both pointer-events: none and the hack to avoid the page to be resized by the browser navbar, the browser someway internally improves its rendering speed by anticipating resize/click events won't be triggered while scrolling?... 😮

Possible I suppose! I'd be surprised if the browser was being that smart though :D

Can you provide the editor link for that codepen? I'm not seeing those edits in the one from the first post.

But obviously the biggest difference there is that it's not in a canvas, so I imagine there might be some lag induced through painting DOM elements rather than a single canvas.

[martinlaxenaire]

Here's another interesting approach found in this example by @greggman reported in this tweet.

The idea is to use an absolute positioned canvas and then translate it with the scroll values.
Only downside is when you scroll really fast, you can see the canvas translation is not keeping up with the scroll, cropping the top or bottom edge.
Maybe this CSS fix could help (it will obviously enlarge the rendering size tho, which could impact performance):

canvas {
    position: absolute;
    top: -20vh;
    left: 0;
    width: 100vw;
    height: 140vh;
}

[saminton]

When I tried this technique out I tried to do exactly that. The problem is that having an absolute positioned element larger than the viewport means that the page will continue scrolling until the element is in view. But as the canvas is repositioned via JS it means it's always out of view, thus you end up with an endless scrolling page. May be a way to prevent this behaviour but I ended up giving up due to time limitations before finding a solution.

[martinlaxenaire]

Okay folks,

after having spending a bit more time on this with @ashthornton, it looks like we've found a workaround (or at least we're pretty close from it)!
Editor codepen: https://codepen.io/martinlaxenaire/pen/ExgXXNZ
Debug: https://cdpn.io/martinlaxenaire/debug/ExgXXNZ/PBkNWRqxXYEM

In fact the solution is fairly simple.
Instead of listening to the window scroll, we have to use a scrollable container, get its scrollTop position on scroll and update our positions with that. There's also a little hack so the browser UI is always shown, but that's it.

Let me know if it's working on your side as well.

[saminton]

How strange, on chrome (Desktop) when I first load the page the webgl is lagging behind. But if I change tab and come back it's perfectly synced. I still get the small latency between the webgl and the text on mobile however.

[martinlaxenaire]

Hey @saminton,

that's weird! Could you tell us more about the phone you used for your test?
Not noticing any lag neither on Chrome desktop nor on my Pocophone F1 Android device (Chrome as well).

[mrflix]

For us, moving the scroll into a container resulted in heavy rendering bugs in Desktop Safari:

In the screenshot, the curtains.js canvas is currently hidden, and the DOM elements are wrongly positioned. I have never seen such a bug. The blue area shows where Safari thinks that one image is. And where it actually should be. But in reality it's rendered further up. The caption "Flyer - Toxic Space..." is at the bottom of the image. Moving scroll back to the window fixed this.

[sintj]

Hi guys, I have a problem with stuttering in mobile that seems to be relates to this issue. In this site (in development):https://retrospettivi.netlify.app/ I made the hero banner with three.js. On desktop there is no problem. But on mobile if the site is scrolled down totally, and then scrolled up again, it will freeze for a second. I believe it's related to the canvas and webgl context more than three.js, but still.. Has anyone have an idea to fix or mitigate this?

[abubaker-jameel]

@martinlaxenaire Hi bro,
Hope you are doing great.

If possible can we talk, I need to discuss few things about curtain js.

Thanks