Flip mouse bindings for simulate perturb UI?

[yuvaltassa]

This is a poll about the simulate viewer's mouse bindings. You can see these bindings in action in this video.

Right now simulate's camera and perturbation mouse bindings are as follows:

mode	binding	action
camera	left drag	orbit
	right drag	vertical pan
	shift right drag	horizontal pan
perturb (pause)	control left drag	rotate
	control right drag	displace vertical
	control shift right drag	displace horizontal
perturb (play)	control left drag	apply torque
	control right drag	vertical mouse-spring
	control shift right drag	horizontal mouse-spring

So right now left mouse buttons are mapped to rotational motions, while right mouse buttons are mapped to translational motions which is in some sense elegant.

On a mouse, the left button is primary and right is secondary. On touch pads this is even more pronounced with corresponding single-finger and double-finger touches and drags corresponding to left/right buttons. On a touch pad, two finger drag is very rare and inconvenient.

Here's the problem: For cameras, orbiting is indeed more useful than panning (so there is a good match), but for mouse-spring perturbations, applying forces (translation) is more useful than applying torques (rotation). So... should the mapping of the left/right mouse buttons be flipped in perturb mode?

[zalo]

A convention I've found quite useful is to switch the action of a drag based on context:

• If your drag starts on a mocap body or a body with any movable degrees of freedom, then Left-Click Drag is Translation, Right-Click Drag is Rotation
• If your drag starts anywhere else, then Left-Click Drag is Orbit, Right-Click Drag is 2D Panning, and Middle-Click or Scroll-Wheel is Zoom

You can try my proposal (minus Rotating bodies) here: https://zalo.github.io/mujoco_wasm/

While the above demo will also work on iPads/Phones and other multitouch devices, I think there might be a better multitouch camera control mechanism that acts a bit like the way Google/Apple Maps works in oblique mode. (I had an implementation for this somewhere but can't seem to find it now).

[erez-tom]

When the camera is looking sideways, toggling back and forth between the two planes can give you full 3D control in the entire volume. For example, I've seen users (specifically @yuvaltassa ) build elaborate structures from scattered blocks, or move a robot between different parts of a complex arena.

[yuvaltassa]

The current simulate convention is vertical-plane (not image-plane!) and horizontal-plane.

I once tried switching to image plane instead of vertical plane and concluded that vertical plane is much better.

As Tom says, the usefulness becomes apparent when trying to do things like block stacking. The key is that one can switch between the horizontal and vertical plane within the same drag motion, by pressing/letting go of the shift key.

@zalo, I urge you to make a model with 4 boxes and stack them in simulate. If you wish I can make such a model and record an interaction video, but this is something you should try.

[yuvaltassa]

Hi @zalo,

Here is a video showing box stacking with the current UI, in live interaction mode. Here is the model.

[zalo]

Alright, stacking boxes is a compelling argument; I’ll meditate on how best to achieve similar functionality on mobile.

It’ll probably involve the second finger coming in… switching between rotation and pinch-to-zoom semantics depending on whether it also starts on the object.

Also, is there a good reason that drag forces should not come from the center of the body? If rotation is its own input, then off-center drags won’t be needed for it.

Also, have you ever used the “PhysGun” in Garry’s Mod? It’s a compelling mouse based interface that excels at stacking boxes and posing ragdolls…

[yuvaltassa]

simulate used to have mouse-spring forces applied at body CoM, but we recently changed to selection point. It's better but not mind-blowingly better. Try simulate from MuJoCo 2.3.0 on the above model to see what it used to be like. Not completely opposed to reverting.

Also note that the mouspring dynamics are not trivial, pick up the cup in mujoco_wasm and in simulate to get a sense of the difference.

Personally, I'm not familiar with PhysGun 🙂