Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenXR: Add support for binding modifiers #97140

Merged

Conversation

BastiaanOlij
Copy link
Contributor

@BastiaanOlij BastiaanOlij commented Sep 18, 2024

Adds support for OpenXR binding modifiers to the action map.

This allows you to add modifiers to inputs such as applying thresholds or offsetting values, etc.
Binding modifiers can add additional input paths or change the type of an input path.

Todos:

  • Create infrastructure for binding extension system to the action map
  • Make UI changes to the action map UI
  • Implement support for haptic triggers
  • Implement analog threshold extension (may move this into vendor plugin)
    • implement resource class
    • implement UI
  • Implement dpad binding extension
    • implement resource class
    • implement UI

Depends on:

There is now a demo for this project that can be found here:
godotengine/godot-demo-projects#1137

Some improvements that can be done (possibly in follow up PRs):

  • Should show our modifier button icon in a different color if there are modifiers defined.
  • Should find a way to limit the modifiers you can create to those that are applicable.

Contributed by Khronos Group through the Godot Integration Project

@BastiaanOlij BastiaanOlij added this to the 4.x milestone Sep 18, 2024
@BastiaanOlij BastiaanOlij requested a review from a team September 18, 2024 10:15
@BastiaanOlij BastiaanOlij self-assigned this Sep 18, 2024
@AThousandShips AThousandShips changed the title OpenXR: Adding support for binding modifiers OpenXR: Add support for binding modifiers Sep 18, 2024
@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch 3 times, most recently from 45f85cf to 17a009e Compare September 26, 2024 04:57
@BastiaanOlij
Copy link
Contributor Author

Back working on this after dealing with a bout of the flu. Starting to get an interface together but need to change direction slightly.

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch 2 times, most recently from e3aeb3b to a419e71 Compare October 14, 2024 03:29
@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch 2 times, most recently from 0b77271 to 577c7ef Compare October 14, 2024 23:06
@BastiaanOlij
Copy link
Contributor Author

While some of this is a bit of an unknown (but likely) future, I changed our action map structure slightly so we can record binding modifiers both on interaction profiles and on individual bindings.

Right now the dpad modifier is a bit of the odd duckling being the only modifier that is linked to an action set/source path combination and is thus recorded on the interaction profile level.
The analog threshold modifier is linked to an action/source path combo which is recorded as a binding (hence the changes in #98163) and it is very likely that this is going to be true for future modifiers as well.

Right now they are all added to the next chain or XrInteractionProfileSuggestedBinding when calling xrSuggestInteractionProfileBindings however this may change/be enhanced in the future.

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from 577c7ef to 924f137 Compare October 15, 2024 07:13
@BastiaanOlij
Copy link
Contributor Author

UI is finally coming together. Binding modifiers are accessible either on the interaction profile (1) or for individual bindings (2):
image

Either button opens up a popup that allows for creating applicable binding modifiers on that level:
image

I still need to change the dialog for selecting a new binding modifier as it doesn't filter by type, and we only have the two modifiers right now but that will change in the near future.

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from 924f137 to 050d3cf Compare October 15, 2024 07:32
@BastiaanOlij
Copy link
Contributor Author

Some interesting feedback concerning the DPad modifier extension that we need to ensure gets documented well.

The extension actually does two things when enabled.

  1. It adds new binding paths you can use for each thumbstick and thumbpad input. These paths are immediately usable even if you do not add a dpad binding modifier
  2. And it adds the dpad binding modifier, which lets you customise the dpad behaviour

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from 050d3cf to 3db345a Compare October 20, 2024 11:44
@BastiaanOlij
Copy link
Contributor Author

All the new paths have been added to the meta data database. In doing this I've cleaned up a lot of the code to make it easier to manage. Still planning on adding a bit of extra logic in the UI when editing the action map so we only show bindings that are available based on which extensions have been selected in project settings so we don't get a whole bunch the user isn't using.

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch 2 times, most recently from 4fb4d9a to 00c39d2 Compare October 21, 2024 07:44
@BastiaanOlij
Copy link
Contributor Author

Other then needing to do some more testing, this should now all work.

My only remaining issue is that I had to use the EditorInspector class to make the property editors work properly but these introduce the resource section into the UI and I haven't found a way to supress this:

image

These sections make no sense in this context so I hope there is a way to remove them.

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from 00c39d2 to ff9ffb0 Compare October 21, 2024 07:49
@BastiaanOlij BastiaanOlij marked this pull request as ready for review October 21, 2024 07:50
@BastiaanOlij BastiaanOlij requested review from a team as code owners October 21, 2024 07:50
doc/classes/ProjectSettings.xml Outdated Show resolved Hide resolved
@@ -156,6 +175,10 @@ void OpenXRActionMap::remove_interaction_profile(Ref<OpenXRInteractionProfile> p
int idx = interaction_profiles.find(p_interaction_profile);
if (idx != -1) {
interaction_profiles.remove_at(idx);

ERR_FAIL_COND_MSG(p_interaction_profile->action_map != this, "Removing interaction profile that belongs to this action map but had incorrect action map pointer."); // this should never happen!
p_interaction_profile->action_map = nullptr;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to call p_interaction_profile->action_map->remove_interaction_profile(p_interaction_profile) before you reset the action_map?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the design is that an interaction profile always belongs to a single action map. The action_map member is only null momentarily in 2 conditions:

  1. when we're creating a new interaction profile right before it's added to an action map
  2. when we're destructing an interaction profile where it's remove from the action map just before.

The check to see if the actionmap pointer is different is just to catch errors in usage, someone who later on adds code that would attempt to add an interaction profile to multiple action maps, and these checks just erroring out.

Comment on lines 124 to 130
Array OpenXRIPBinding::get_binding_modifiers() const {
Array ret;
for (const Ref<OpenXRBindingModifier> &binding_modifier : binding_modifiers) {
ret.push_back(binding_modifier);
}
return ret;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not directly returning the binding_modifiers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we're converting from a Vector<Ref<OpenXRBindingModifier>> to an Array. I don't think there is an implicit conversion or was that added at some time?

binding_modifiers.remove_at(idx);

ERR_FAIL_COND_MSG(p_binding_modifier->ip_binding != this, "Removing binding modifier that belongs to this binding but had incorrect binding pointer."); // this should never happen!
p_binding_modifier->ip_binding = nullptr;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment; should p_binding_modifier be removed from ip_binding before resetting it?

Comment on lines 28 to 29
Return [code]false[/code] if binding modifiers of this type are recorded on an interaction profile.
Return [code]true[/code] if binding modifiers of this type are recorded on bindings within an interaction profile.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description is a bit confusing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically there are two types of binding modifiers, those recorded on the interaction profile, and those recorded against a specific binding on the interaction profile. This flag simply indicates which type we're dealing with.
This really won't get a lot cleared until more binding modifiers become public.

Comment on lines +24 to +25
If [code]false[/code], when the joystick enters a new dpad zone this becomes true.
If [code]true[/code], when the joystick remains in active dpad zone, this remains true even if we overlap with another zone.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also hard to understand what this field is for at first read.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had real difficulty describing this in text, you really need to look at the diagrams in the API spec and even there the description is overly technical.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are those diagrams publicly available? Can you link to them in this description?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you scroll to the top of the help page you'll notice that on both modifiers, I link to the specification for the modifiers.

For the DPad modifier its this one: https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XR_EXT_dpad_binding

Totally open to suggestions on how to work that more user friendly and embed that into our documentation.

modules/openxr/editor/openxr_action_map_editor.cpp Outdated Show resolved Hide resolved
modules/openxr/editor/openxr_binding_modifier_editor.cpp Outdated Show resolved Hide resolved
@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from 7d8bef8 to ba417e9 Compare November 6, 2024 09:48
Copy link
Contributor

@m4gr3d m4gr3d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logic looks good, although I'd like us to iterate on the documentation as in their current state, they're hard to parse.

@BastiaanOlij Is there a test project that can be used to help validate those changes?

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from ba417e9 to 5ff3002 Compare November 22, 2024 08:33
@BastiaanOlij
Copy link
Contributor Author

Did some more testing with this while addressing some of the feedback, found out that when both modifier extensions are enabled in the settings, one wouldn't work because both extensions require the base modifier extension. Fixed that up so Godot will react properly if multiple wrappers ask for the same extension to be enabled.

@BastiaanOlij
Copy link
Contributor Author

@BastiaanOlij Is there a test project that can be used to help validate those changes?

@m4gr3d, I added a demo project on godot-demo-projects, see link in OP.

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from 5ff3002 to a304355 Compare November 22, 2024 09:11
@BastiaanOlij BastiaanOlij requested a review from m4gr3d November 27, 2024 02:39
Copy link
Contributor

@dsnopek dsnopek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I skimmed the latest code, and mostly just found more little nitpicky stuff.

I haven't tested the binding modifiers at runtime, but I did re-test the editor stuff again. I'm able to add an OpenXRAnalogThresholdModifier, and that seems to be working fine. However, when I try to add an OpenXRDpadBindingModifier I get this error:

ERROR: Condition "!new_binding_modifier->record_on_binding()" is true.
   at: _on_dialog_created (modules/openxr/editor/openxr_binding_modifiers_dialog.cpp:134)

Is that expected? If so, how should dpad modifiers be added?

modules/openxr/openxr_api.cpp Outdated Show resolved Hide resolved
modules/openxr/action_map/openxr_binding_modifier.h Outdated Show resolved Hide resolved
modules/openxr/action_map/openxr_binding_modifier.h Outdated Show resolved Hide resolved
modules/openxr/editor/openxr_binding_modifier_editor.h Outdated Show resolved Hide resolved
modules/openxr/extensions/openxr_dpad_binding_extension.h Outdated Show resolved Hide resolved
modules/openxr/extensions/openxr_dpad_binding_extension.h Outdated Show resolved Hide resolved
doc/classes/ProjectSettings.xml Outdated Show resolved Hide resolved
doc/classes/ProjectSettings.xml Outdated Show resolved Hide resolved
@BastiaanOlij
Copy link
Contributor Author

Is that expected? If so, how should dpad modifiers be added?

DPad modifiers aren't added to individual bindings, they are added to the interaction profile with the binding button on the right.
I need to figure out if I can add additional filters to the standard popup. I didn't want to create two separate base classes but maybe i need to go down that route.

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from a304355 to 2f60056 Compare December 5, 2024 21:31
@dsnopek
Copy link
Contributor

dsnopek commented Dec 6, 2024

@BastiaanOlij:

DPad modifiers aren't added to individual bindings, they are added to the interaction profile with the binding button on the right.
I need to figure out if I can add additional filters to the standard popup. I didn't want to create two separate base classes but maybe i need to go down that route.

I think even just having a more user-friendly error message would be OK. Right now, I'm getting a very programmer-y error and the modifier just fails to be added.

Maybe we just need a popup with an explanation?

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from 2f60056 to da92172 Compare December 8, 2024 08:21
@BastiaanOlij
Copy link
Contributor Author

@BastiaanOlij:

DPad modifiers aren't added to individual bindings, they are added to the interaction profile with the binding button on the right.
I need to figure out if I can add additional filters to the standard popup. I didn't want to create two separate base classes but maybe i need to go down that route.

I think even just having a more user-friendly error message would be OK. Right now, I'm getting a very programmer-y error and the modifier just fails to be added.

Maybe we just need a popup with an explanation?

I decided to go down the subclass path after all. So there is now an OpenXRIPBindingModifier class for binding modifiers that are recorded on interaction profile level, and a OpenXRActionBindingModifier class for binding modifiers that are recorded on individual actions/bindings.

Right now it seems a little overkill but once more binding modifiers become available it will start to make a whole lot more sense I recon.

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from da92172 to f863461 Compare December 8, 2024 23:20
@BastiaanOlij BastiaanOlij requested a review from dsnopek December 10, 2024 00:43
Copy link
Contributor

@dsnopek dsnopek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I re-tested in the editor, adding both threshold and dpad modifiers with haptics in the right places for those. Having the two base classes seems like a nice way to go, both for the editor and making the API simpler (removing the somewhat confusing record_on_binding virtual method).

I skimmed the code again and it looks good to me, but it's a big PR and having reviewed it so many times already, my brain can't pick up on all that much anymore. :-)

I think this is ready to go!

modules/openxr/action_map/openxr_action_map.cpp Outdated Show resolved Hide resolved
modules/openxr/action_map/openxr_action_map.cpp Outdated Show resolved Hide resolved
modules/openxr/action_map/openxr_interaction_profile.cpp Outdated Show resolved Hide resolved
modules/openxr/action_map/openxr_interaction_profile.cpp Outdated Show resolved Hide resolved
modules/openxr/action_map/openxr_action_map.cpp Outdated Show resolved Hide resolved
Comment on lines +68 to +75
int get_binding_modifier_count() const; // Retrieve the number of binding modifiers in this profile path
Ref<OpenXRActionBindingModifier> get_binding_modifier(int p_index) const;
void clear_binding_modifiers(); // Remove all binding modifiers
void set_binding_modifiers(const Array &p_bindings); // Set the binding modifiers (for loading from a resource)
Array get_binding_modifiers() const; // Get the binding modifiers (for saving to a resource)

void add_binding_modifier(const Ref<OpenXRActionBindingModifier> &p_binding_modifier); // Add a binding modifier object
void remove_binding_modifier(const Ref<OpenXRActionBindingModifier> &p_binding_modifier); // Remove a binding modifier object
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
int get_binding_modifier_count() const; // Retrieve the number of binding modifiers in this profile path
Ref<OpenXRActionBindingModifier> get_binding_modifier(int p_index) const;
void clear_binding_modifiers(); // Remove all binding modifiers
void set_binding_modifiers(const Array &p_bindings); // Set the binding modifiers (for loading from a resource)
Array get_binding_modifiers() const; // Get the binding modifiers (for saving to a resource)
void add_binding_modifier(const Ref<OpenXRActionBindingModifier> &p_binding_modifier); // Add a binding modifier object
void remove_binding_modifier(const Ref<OpenXRActionBindingModifier> &p_binding_modifier); // Remove a binding modifier object
int get_binding_modifier_count() const; // Retrieve the number of binding modifiers in this profile path
Ref<OpenXRActionBindingModifier> get_binding_modifier(int p_index) const;
void clear_binding_modifiers(); // Remove all binding modifiers.
void set_binding_modifiers(const Array &p_bindings); // Set the binding modifiers (for loading from a resource)
Array get_binding_modifiers() const; // Get the binding modifiers (for saving to a resource)
void add_binding_modifier(const Ref<OpenXRActionBindingModifier> &p_binding_modifier); // Add a binding modifier object.
void remove_binding_modifier(const Ref<OpenXRActionBindingModifier> &p_binding_modifier); // Remove a binding modifier object.

Comment on lines +120 to +131
Ref<OpenXRIPBinding> find_binding(const Ref<OpenXRAction> &p_action, const String &p_binding_path) const; // Get our binding record
Vector<Ref<OpenXRIPBinding>> get_bindings_for_action(const Ref<OpenXRAction> &p_action) const; // Get our binding record for a given action
void add_binding(const Ref<OpenXRIPBinding> &p_binding); // Add a binding object
void remove_binding(const Ref<OpenXRIPBinding> &p_binding); // Remove a binding object

void add_new_binding(const Ref<OpenXRAction> &p_action, const String &p_paths); // Create a new binding for this profile
void remove_binding_for_action(const Ref<OpenXRAction> &p_action); // Remove all bindings for this action
bool has_binding_for_action(const Ref<OpenXRAction> &p_action); // Returns true if we have a binding for this action

int get_binding_modifier_count() const; // Retrieve the number of binding modifiers in this profile path
Ref<OpenXRIPBindingModifier> get_binding_modifier(int p_index) const;
void clear_binding_modifiers(); // Remove all binding modifiers
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Ref<OpenXRIPBinding> find_binding(const Ref<OpenXRAction> &p_action, const String &p_binding_path) const; // Get our binding record
Vector<Ref<OpenXRIPBinding>> get_bindings_for_action(const Ref<OpenXRAction> &p_action) const; // Get our binding record for a given action
void add_binding(const Ref<OpenXRIPBinding> &p_binding); // Add a binding object
void remove_binding(const Ref<OpenXRIPBinding> &p_binding); // Remove a binding object
void add_new_binding(const Ref<OpenXRAction> &p_action, const String &p_paths); // Create a new binding for this profile
void remove_binding_for_action(const Ref<OpenXRAction> &p_action); // Remove all bindings for this action
bool has_binding_for_action(const Ref<OpenXRAction> &p_action); // Returns true if we have a binding for this action
int get_binding_modifier_count() const; // Retrieve the number of binding modifiers in this profile path
Ref<OpenXRIPBindingModifier> get_binding_modifier(int p_index) const;
void clear_binding_modifiers(); // Remove all binding modifiers
Ref<OpenXRIPBinding> find_binding(const Ref<OpenXRAction> &p_action, const String &p_binding_path) const; // Get our binding record.
Vector<Ref<OpenXRIPBinding>> get_bindings_for_action(const Ref<OpenXRAction> &p_action) const; // Get our binding record for a given action.
void add_binding(const Ref<OpenXRIPBinding> &p_binding); // Add a binding object.
void remove_binding(const Ref<OpenXRIPBinding> &p_binding); // Remove a binding object.
void add_new_binding(const Ref<OpenXRAction> &p_action, const String &p_paths); // Create a new binding for this profile.
void remove_binding_for_action(const Ref<OpenXRAction> &p_action); // Remove all bindings for this action.
bool has_binding_for_action(const Ref<OpenXRAction> &p_action); // Returns true if we have a binding for this action.
int get_binding_modifier_count() const; // Retrieve the number of binding modifiers in this profile path.
Ref<OpenXRIPBindingModifier> get_binding_modifier(int p_index) const;
void clear_binding_modifiers(); // Remove all binding modifiers.

Comment on lines +135 to +136
void add_binding_modifier(const Ref<OpenXRIPBindingModifier> &p_binding_modifier); // Add a binding modifier object
void remove_binding_modifier(const Ref<OpenXRIPBindingModifier> &p_binding_modifier); // Remove a binding modifier object
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
void add_binding_modifier(const Ref<OpenXRIPBindingModifier> &p_binding_modifier); // Add a binding modifier object
void remove_binding_modifier(const Ref<OpenXRIPBindingModifier> &p_binding_modifier); // Remove a binding modifier object
void add_binding_modifier(const Ref<OpenXRIPBindingModifier> &p_binding_modifier); // Add a binding modifier object.
void remove_binding_modifier(const Ref<OpenXRIPBindingModifier> &p_binding_modifier); // Remove a binding modifier object.

Comment on lines +52 to +53
static HashMap<String, String> interaction_profile_editors; // interaction profile path, interaction profile editor
static HashMap<String, String> binding_modifier_editors; // binding modifier class, binding modifiers editor
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
static HashMap<String, String> interaction_profile_editors; // interaction profile path, interaction profile editor
static HashMap<String, String> binding_modifier_editors; // binding modifier class, binding modifiers editor
static HashMap<String, String> interaction_profile_editors; // interaction profile path, interaction profile editor.
static HashMap<String, String> binding_modifier_editors; // binding modifier class, binding modifiers editor.

Comment on lines +95 to +96
for (int i = 0; i < new_binding_modifiers.size(); i++) {
Ref<OpenXRBindingModifier> binding_modifier = new_binding_modifiers[i];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for (int i = 0; i < new_binding_modifiers.size(); i++) {
Ref<OpenXRBindingModifier> binding_modifier = new_binding_modifiers[i];
for (Ref<OpenXRBindingModifier> binding_modifier : new_binding_modifiers) {

@BastiaanOlij
Copy link
Contributor Author

@dsnopek Thanks for checking it again though, I do think it's time we merge this and any further things we find we can deal with later. It's a big PR but for people who don't use binding modifiers it's fairly low risk.

@AThousandShips dang :) That's a lot of cleanup of existing code. Worth doing I guess, I'll go through it later today :)

@BastiaanOlij BastiaanOlij force-pushed the xr_action_map_enhancements branch from f863461 to 0a61ebd Compare December 11, 2024 22:46
@AThousandShips
Copy link
Member

Limited it largely to new code 😅

@akien-mga akien-mga modified the milestones: 4.x, 4.4 Dec 12, 2024
@akien-mga akien-mga merged commit d1b683d into godotengine:master Dec 12, 2024
20 checks passed
@akien-mga
Copy link
Member

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants