Quest 3 Motion Controller States

The MotionController offers a way to stream the current pose, button and trackpad states of the Quest 3 motion controller to the python side. To use this mixed reality (XR) feature, you need to setup vuer behind a SSL proxy. I usually use ngrok, which is a paid service. You can also use local tunnel, which is free.

Motion Controller API

Getting the motion controller states: You can get the full pose of the motion controllers by listening to the CONTROLLER_MOVE event. You can add flags left and right to specify which side you want to track.

Haptic Feedback via pulsing: You can pulse the gamepad using the following keys: pulseLeftStrength, pulseLeftDuration, puseLeftHash. The pulseLeftStrength is a number between 0 and 1, and the pulseLeftDuration is the duration of the pulse in milliseconds. The puseLeftHash is a unique identifier for the pulse, that once changed, will trigger a new pulse.

Example Script: Haptic Trigger Pull

The example below shows how to trigger a haptic pulse to reflect the trigger pull value. The user should experience a stronger vibration when the trigger is pulled further.

python

from vuer import Vuer, VuerSession
from vuer.schemas import MotionControllers
from asyncio import sleep

app = Vuer()

prev_bstate = None

@app.add_handler("CONTROLLER_MOVE")
async def handler(
    event,
    session: VuerSession,
):
    global prev_bstate
    bstate = event.value["leftState"]["triggerValue"]

    if prev_bstate != bstate and bstate:
        # pulse the gamepad according to the trigger value
        session.upsert @ MotionControllers(
            key="motion-controller",
            left=True,
            right=True,
            pulseLeftStrength=bstate,
            pulseLeftDuration=100,
            puseLeftHash=f"{datetime.now()}:0.3f",
        )

    prev_bstate = bstate

@app.spawn(start=True)
async def main(session: VuerSession):
    # Important: You need to set the `stream` option to `True` to start
    # streaming the controller movement.
    session.upsert @ MotionControllers(stream=True, key="motion-controller", left=True, right=True)

    while True:
        await sleep(1)

The returned data looks like the following:

typescript

/**
 * Significantly more accurate and stable than controller-tracking.
 */
export type ControllerData = {
  left?:       Matrix4Tuple;      // array with length==16
  right?:      Matrix4Tuple;      // array with length==16
  leftState?:  ControllerStateType;
  rightState?: ControllerStateType;
};

export type ControllerStateType = {
  trigger:    boolean;
  squeeze:    boolean;
  touchpad:   boolean;
  thumbstick: boolean;
  aButton:    boolean;
  bButton:    boolean;

  triggerValue:      number;
  squeezeValue:      number;
  touchpadValue:   [ number, number ];   // X and Y values for the touchpad
  thumbstickValue: [ number, number ]; // X and Y values for the thumbstick
  aButtonValue:      boolean;
  bButtonValue:      boolean;
};

Button and Trackpad States

The webXR Motion Controller API uses the XRInputSource's gamepad.

For detailed API, refer to the link

attribute to get the button and trackpad states. The following code snippet shows how to extract the button and trackpad states from the gamepad:

Buttons	`xr-standard` Mapping	Required
buttons[0]	Primary trigger	Yes
buttons[1]	Primary squeeze button	No
buttons[2]	Primary touchpad	No
buttons[3]	Primary thumbstick	No

Axes	`xr-standard` Mapping	Required
axes[0]	Primary touchpad X	No
axes[1]	Primary touchpad Y	No
axes[2]	Primary thumbstick X	No
axes[3]	Primary thumbstick Y	No

typescript

  const gamepad = inputSource.gamepad;
  const buttons = gamepad?.buttons || [];

  return {
    transform: Array.from(transform) as Matrix4Tuple,
    trigger: buttons[0]?.pressed  || false,
    squeeze: buttons[1]?.pressed  || false,
    touchpad: buttons[2]?.pressed || false,
    thumbstick: buttons[3]?.pressed || false,
    aButton: buttons[4]?.pressed || false,
    bButton: buttons[5]?.pressed || false,

    triggerValue: buttons[0]?.value || 0,
    squeezeValue: buttons[1]?.value || 0,
    touchpadValue: [gamepad?.axes[0] || 0, gamepad?.axes[1] || 0],
    thumbstickValue: [gamepad?.axes[2] || 0, gamepad?.axes[3] || 0],
    aButtonValue: buttons[4]?.pressed || false,
    bButtonValue: buttons[5]?.pressed || false,
  };

Matrix format

All 4x4 transform matrices used in WebGL are stored in 16-element Float32Arrays. The values are stored in the array in column-major order; that is, each column is written into the array top-down before moving to the next column to the right and writing it into the array. Therefore, for the array [a0, a1, a2, …, a13, a14, a15], the matrix looks like this:

                                  ⌈  a0 a4 a8 a12  ⌉
                                  |  a1 a5 a9 a13  |
                                  |  a2 a6 a10 a14 |
                                  ⌊  a3 a7 a11 a15 ⌋

For details, refer to the MDN documentation on XR Rigid Body Transformation