Limino

Limino is an HCI research and interaction design project exploring the potential of making blended-reality experiences more customizable.

By building a toolkit for designers and developers to support adaptive blending interactions in Mixed Reality applications, we investigated the design considerations and challenges of various ways of adaptive blending.

Navigating the Reality-Virtuality Continuum

Since Milgram and Kishino proposed Reality-Virtuality Continuum three decades ago, a substantial development in the XR space has brought us experiences at different points of the spectrum, from AR navigation on mobile Google Maps to fully immersive VR games on PCVR headsets.

However, in use cases outside of gaming and entertainment, some users may demand higher customizability and more personalization in the way that real and virtual content are blended together. For example, pet owners and parents may prefer staying aware of their pets and kids during semi-immersive MR sessions. Other users may find it useful to reference real-world objects and environments while interacting with virtual content.

This type of dynamic blending has become more practical thanks to the recent introduction of color passthrough and scene understanding in next-generation MR headsets like Pico 4 Pro and Meta Quest Pro.

Types of Blending

Blending, in general, can be categorized into three types. Enhancement refers to overlaying physical content with virtual modifications to improve their forms or functions. Diminishment weakens or obstructs physical content to reduce its significance or replace its functionalities. Unveiling reveals the physical appearance of real-world objects and environments.

The current, and perhaps even next-generation, MR technologies are still not mature enough to support general-purpose high-fidelity enhancement and diminishment for mobile MR experiences. However, unveiling is already somewhat practical with Meta Quest 2 and Pro in the form of passthrough. We believe studying unveiling interactions is not only an important first step towards customizable blending, but the exploration also would inform the design of the previous two categories.

Methods for Adaptive Blending Interactions

This project proposed a set of passthrough-based unveiling interactions for adjusting the blending between real and virtual content based on user needs and circumstances. This dynamically mediated blending is referred to as Adaptive Blending.

Through initial user interviews and product research, we devised three methods to improve the customizability of passthrough interactions. The three methods have their own pros and cons, and correspond to different levels of automation and persistence.

1. Give users control over the size, position and appearance of passthrough
2. Make passthrough visuals reactive to user input in real-time
3. Automate the appearance and disappearance of passthrough based on contextual events

Types of Passthrough Interactions

Based on these three methods, we designed three types of passthrough interactions in our project. Pre-configured Passthrough utilizes the first principle and allows users to configure the passthrough in an Edit Mode. Responsive Passthrough utilizes the second principle and responds to user input dynamically. Automated Passthrough utilizes the third principle and triggers passthrough through context awareness.

Designing Pre-configured and Responsive Passthrough

For pre-configured and responsive passthrough, we further proposed three subcategories of interactions - Casting, Piercing and Fading. They differ not only in their responsiveness but also in the way they are integrated into the scene. Specifically, Casting and Piercing add passthrough objects into the scene while Fading only changes the appearance of existing content.

Under each subcategory, we designed two example interactions to help us understand their use cases and constraints.

Casting

With casting interactions, users can cast a passthrough shadow onto the environment as if they are using a torch. The torch can be attached to different body parts. In our prototype, flashlight and headlight cast passthrough shadows by tracking the hand and head movement respectively.

Piercing

With piercing interactions, users can create, update and remove cutouts that occlude the underneath virtual content with a slice of the live camera stream of the physical world. Passthrough Brush paints strokes of reality on top of the virtual environment using their controller. Passthrough Shape displays the passthrough image on a surface created by the projection from the controllers.

Fading

With fading intersections, users can adjust the opacity of virtual content to reveal the background passthrough behind it. Global Fading decreases the opacity of all virtual content. Object Fading decreases the opacity of particular virtual objects or digital twins.

Designing Context-aware Passthrough

We also experimented with triggering the passthrough interactions based on activity and environmental events.

We identified three use cases that could demonstrate the functions of context-aware passthrough interactions.

Break Time

Break Time is an example of activity awareness that helps users temporarily exit the virtual environment by switching back to reality. The system understands when the user is taking a break from work by detecting the change in the headset position.

Item Searching

Item Searching is an example of activity awareness that assists users in locating items close enough to be reached with a stretched arm but far enough to be outside their peripheral vision. When the system detects the need for item searching, it toggles on the aforementioned headlight interaction. The system understands the user movement by accessing the headset data, such as the location of the headset and controller.

Bystander Interruption

Bystander Interruption is an example of environmental awareness that recognizes bystanders entering the predefined activity boundary and fades out the virtual door overlay to reveal the real-world position of the person. Sensing devices ranging from cameras to microphones can be used for detecting changes in the space, such as a non-MR user entering the room. However, this project employs a wizard-of-oz technique that triggers the fading manually.

Toolkit UI

The toolkit menu can be summoned by pressing the menu button on the controller. And the tools are categorized into responsive tools in the main menu, and configuration tools in the edit mode menu.

Technical Implementation

The prototype was implemented on Unity using the Oculus XR Plugin and SDK with OpenXR backend. We have chosen Meta Quest 2 and Meta Quest Pro as our testing devices due to their availability and community support, but the experience can be ported to any 6DOF MR headsets that support passthrough, spatial anchors, and controller tracking. The implementation of the system comprises the following four parts.

Environment Curation

The environment consists of two parts. The hybrid space resembles the physical room the user is situated in, while the virtual space extends the hybrid space to create a spacious and cozy experience.

Hybrid Space

The hybrid space is generated based on the scene anchors provided by the Presence Platform SDK. These scene anchors capture the world positions and dimensions of the bounding planes and boxes that enclose real-world objects, such as desks, couches, walls, doors, and windows. The users need to define them upfront through the Meta Quest room mapping setting. Digital twins of the mapped real-world objects are then generated based on the scene anchors.

Virtual Space

In addition to the above digital twins, completely virtual items, such as a bigger house, additional furniture and soothing scenery, are also included to support the experience aesthetically.

Compositing Mechanism

The MR workspace is a customizable 3D environment that blends physical and virtual worlds. The intermixing of the two worlds is achieved by compositing different layers together. The position and opacity of the content in each layer contribute to the overall blending of the scene. These layers can be conceptually categorized into four types based on their rendering priority in the depth buffer.

Blending Interactions and Context Awareness

Passthrough content is assigned with passthrough materials rendered with special HLSL shaders that expose the underlying camera live stream. Two different shaders were designed for the project.

For fading interactions, we exposed the opacity control of the entire environment to the users. For piercing interactions, we implemented the passthrough surfaces by assigning passthrough materials and editable behavior to individual planes. The passthrough stroke implementation is based on an Oculus XR Integration sample with support for brush size and opacity adjustment added by us. For casting interactions, the passthrough shadow is projected based on a light volume model that tracks the user movement. The required tracking data was obtained from the built-in sensors through the Presence Platform SDK.

In terms of context awareness, activity detection is based on user motions, such as standing and arms movement, through the built-in sensors and Presence Platform SDK. For example, the application can detect standing by monitoring the vertical position of the headset relative to the floor and arm movement by measuring the distance between the headset and the hands or controllers. For environmental awareness, we relied on the wizard-of-oz method to manually trigger the door fading effect to let us focus on the interaction design.

Bottom Line

With a series of design explorations and proposed design languages in adaptive blending, the project contributes to the overall MR research landscape by categorizing the design space for blending interactions, as well as introducing context awareness as an approach to implement automation for triggering blending. We hope to expand our work in the future and that our work will inspire more explorations in context-aware MR experiences, Mixed initiative MR experiences, and blending interactions.

For more findings and insights from the project, please refer to my thesis.