Feature for the UA to handle viewing the system inputs during a session

[AdaRoseCannon]

/facetoface On visionOS when you don't have the hand-tracking feature we show the system input so the user can see their hands and is comfortable. I would like to discuss adding a feature to show the system input even when it otherwise would not be shown.

[mkeblx]

This would be useful, and developers have requested this ability.

I think XRSession would be a good place to put this. A few options for consideration:

(A) Show and hide methods directly on session

let systemInputShown = xrSession.hideSystemInput()
let systemInputShown = xrSession.showSystemInput()

if (!systemInputShown) {
  app.drawOwnInput()
}

(B) Single toggle method with a force param
Similar to elem.classList.toggle(cls, force)

let systemInputShown = xrSession.toggleSystemInput(force)

if (!systemInputShown) {
  app.drawOwnInput()
}

(C) systemInput property with methods

xrSession.systemInput.visible
xrSession.systemInput.show()
xrSession.systemInput.hide()
// xrSession.systemInput.toggle() ?

if (!xrSession.systemInput.visible) {
  app.drawOwnInput()
}

I'm partial to (C). I could see adding more fields on systemInput in the future.

Thoughts?

@AdaRoseCannon

[AdaRoseCannon]

I was thinking something along these lines too, I will try to put my thoughts together shortly, thank you for this.

[mkeblx]

A mockup video of toggling system input display, where systemInput.visible:

• true => passthrough hands (symbolizing system-rendered input)
• false => app rendered 3D hand mesh

Note that, as with examples above, I do think we want this to be dynamic and not just something at session init.

xrSession.addEventListener('select', () => {
  xrSession.systemInput.toggle()
  app.drawInput(xrSession.systemInput.visible)
});

sys-input-toggle-mock.mp4

[toji]

Of the options @mkeblx offered, I like the idea of the systemInput object best, because it offers a very clear way of segmenting the API and testing for availability. (ie: if (xrSession.systemInput) { /* systemInput is available */ })

I'm not sure that we need two separate methods and a visible attribute. The exact shape probably depends on how difficult we think it will be for systems to turn their input visualization on or off. If we figure it can happen within a frame then I think the entire API could simply be xrSession.systemInput.visible = true;. If we think it'll take multiple frames to switch then it may have to be something more like xrSession.systemInput.setVisibility(true).then(() => { /* new visibility applied */ });

[mkeblx]

Even if there would be high confidence most platforms would be able to switch visibility quickly, a Promise could still make sense. A Promise would also allow to handle the hopefully rare case of unable to set the visibility.

xrSession.systemInput.setVisibility(true).then(() => {
}).catch((err) => {
  console.log('Unable to set visibility', err);
});

One question I now have is whether a boolean is enough. What is a system has multiple ways of displaying the system input and would like to expose that to the app level? Some examples off the top of my head: 'hands', 'hand+tool', 'tool', 'hands+arms', 'full body', etc.

I suppose if something like this is needed in the future, it could be exposed via another method (e.g. setVisibilityType)