WebXR, Vision Pro, and Multiplayer

Asad Memon

August 8, 2023

I got hands on the Vision Pro device for a day with XCode set up 🎉

We applied for Apple's Developer Labs to get access to Vision Pro at Cupertino. Luckily, we got selected for the first day! Probably one of the earliest non-Apple devs to get hands on Vision Pro.

My main goal was to try a few code demos, mostly explore WebXR on it, make a multiplayer demo using Playroom Kit (opens in a new tab) and try to benchmark the device for a bunch of AI use-cases that we are interested in (like inference time with Stable Diffusion running on-device). Unfortunately, I am not allowed to talk much about my experience. But it was positive overall and I had fun!

Preparing for cross-platform XR revolution

Right now we only have Oculus Quest headsets in the mainstream world, but when Vision Pro arrives, we will likely need to make cross-platform XR stack to make games in since the tech stack is completely different for both platforms. In short, the main options for making a cross-platform XR game are either by using Unity or by WebXR.

We like WebXR because there is no installation required which reduces friction, specially for a multiplayer game. This is the same reason we at Playroom are betting on web-first multiplayer games.

WebXR on Vision Pro

The simulator supports WebXR but right now it only supports the fully immersive mode and not the mixed-reality mode. WebXR is hidden behind a feature flag in Safari on Vision Pro. You need to enable it from Settings:

Developing a quick multiplayer game with WebXR and Playroom Multiplayer SDK on Vision Pro proved fairly seamless! Used three.js and Playroom's built-in Gamepad API (opens in a new tab) to connect game controller to the game. Video recording wasn't allowed with Vision Pro, but here is a video of me playing the game an iPhone using a game controller WebXR Viewer (opens in a new tab).

Code Overview

This is a standard three.js game with xr enabled:

import { XRButton } from 'three/addons/webxr/XRButton.js';
// ...
renderer = new THREE.WebGLRenderer( { antialias: true, alpha: true } );
renderer.setPixelRatio( window.devicePixelRatio );
renderer.setSize( window.innerWidth, window.innerHeight );
renderer.setAnimationLoop( render );
renderer.xr.enabled = true;
document.body.appendChild( renderer.domElement );
document.body.appendChild( XRButton.createButton( renderer ) );

Added Playroom SDK which handles the multiplayer and gamepad API:

const { onPlayerJoin, insertCoin, isHost, myPlayer, Joystick, isStreamScreen } = Playroom;
let players = [];
 
// ...
 
await insertCoin({streamMode: true, allowGamepads: true});
onPlayerJoin((state) => {
  // Create a plane for the player
  const plane = createPlane(scene, state.getProfile().color.hex);
  // Create a joystick for the player
  const joystick = new Joystick(state, {type: "dpad"})
  players.push({state: state, plane, joystick });
});

Game can now be played using game controllers. It just needed to read the gamepad state and update the player's plane position in the main game loop:

function render() {
  renderer.render( scene, camera );
  players.forEach((player) => {
    const playerState = player.state;
    const plane = player.plane;
    const dpad = player.joystick.dpad() || {};
    if (dpad.x === "left"){
      plane.mesh.rotation.y += 0.03;
    }
    if (dpad.x === "right"){
      plane.mesh.rotation.y -= 0.03;
    }
    if (dpad.y === "up"){
      plane.mesh.rotation.z -= 0.03;
    }
    if (dpad.y === "down"){
      plane.mesh.rotation.z += 0.03;
    }
 
    // Move the plane forward towards the direction it is facing
    // ...
  });
}

Limitations

• Do note that WebXR on Vision Pro is not perfect. It doesn't yet support immersive-ar, which means no mixed-reality experiences yet. I hope the support for immersive-ar lands soon.