EXA's built-in mixed-reality feature brings you into the app itself! This allows your image to be illuminated by the ringer lights, avoids the need to composite mixed-reality layers manually via OBS, and lets you do things like... mixed-reality selfies :)



The light from the ringers really improves the illusion of a player standing in the VR scene.

You'll need a few things to use EXA's custom mixed-reality feature. Fortunately, these are mostly the same needs as the "standard" mixed-reality that you get with SteamVR.

• HTC Vive: This custom mixed-reality feature currently works only with the Vive.
  
  
• Low-latency webcam or camera: The Logitech C922 is often recommended for mixed-reality, and it's the only one that I've tested/used so far (I've found that its standard resolution settings provide the least latency). Low latency is important here, because the video is composited directly into the app -- there isn't currently a way to delay/buffer the app's video/audio to compensate for camera latency.
  
  
• Green screen: This is needed for removing the "background" portions of the video, revealing only the player. It helps to have sufficient lighting, as well, to ensure the entire green-screen has a bright and consistent color. Without a green-screen, EXA's mixed-reality would work a bit differently -- anything "behind" the player's plane would be obscured by the background portions of the video.
  
  
• The "mixedreality.cfg" file: This configuration file tells EXA how to position/align the virtual camera, which Vive controller/tracker to follow, which green-screen colors to remove, etc. See full details below.
  
  
• Vive Tracker (Optional): The easiest way to setup mixed-reality is by attaching your real-world camera to a Vive tracker. The virtual camera can then follow the Vive tracker, which ensures that the virtual and real-world cameras are aligned with each other.

The first three minutes of this video provide a visual overview of how this mixed-reality feature looks and operates. It shows how the video can be illuminated by the lights in the scene, how the "video plane" is placed into the scene, and how you can see yourself within the app via the mixed-reality monitor/mirror.

Place the mixedreality.cfg file in the EXA install folder (next to EXA.exe).

Here are example contents of the mixedreality.cfg configuration file:

The camera settings are related to the alignment of the virtual camera to the real-world webcam and the webcam's hardware settings.



The vr settings choose a specific Vive device to follow, and can shift the entire scene. This shifting is an effect similar to teleporting, since it changes the origin of the scene. This is useful if your camera is in a fixed position, and you want to move everything in the scene towards or away from it.




The image settings tell the app how to process the webcam video, including the removal of the green-screen background. Note: for a pixel to be removed, its color/values must fall within all of the provided "cutout" ranges (hue, chroma, brightness).

Hit the "C" key to cycle through all the available webcam devices. The device index and name will appear on-screen. Once you find the one you want, you can update the mixedreality.cfg file with the correct device index.


Hit the "D" key to cycle through all possible Vive devices (with indexes 0 to 15). The device index will appear on-screen. Once you find the one you want, you can update the mixedreality.cfg file with the correct device index.


Hit the "A" key to toggle the "alignment" mode. When this mode is enabled, squeeze the "grip" buttons on the right Vive controller. While squeezed, rotate the controller to fine-tune the rotational offset (the camera.offset.r* values).

It's best to rotate the controller on one X/Y/Z axis at a time. This isn't very intuitive, so it will probably take a moment to figure out how controller rotation translates to "camera.offset" rotation.

Each time you release the "grip" buttons, EXA prints the new "camera.offset" values to its debugging log. Starting in the same folder as the configuration file, you can find this log in the EXA_Data/output_log.txt file. You can update the mixedreality.cfg file with the new offset values.