One of the problems of mixed reality (MR) — where a person is superimposed into a VR setting — is that the faces of those inside the system are obscured by the VR headset: if you watch someone in VR, you can’t see their facial expressions. One way to try and address this is with a digital avatar stand in, but some clever folks from Google Research and Google Daydream Labs have achieved a more realistic solution.
As outlined in a recent Google blog post, Google has created a technique using “a combination of machine learning, 3D computer vision, and advanced rendering techniques” to remove the headset from mixed reality. The process involves making a “dynamic” 3D model of a person’s face, tracking their eye movements and then compiling the visuals in the virtual space: the result is a realistic representation of what the headset wearer may be expressing and where they may be looking, as you can see in the GIF below.
Though the outline of the VR headset doesn’t need to be included, Google said it added it as a way of slightly obscuring the face to avoid an “uncanny valley” effect.
It still looks slightly strange and artificial but it’s a big step for mixed reality. The lack of these visuals has been one of the major caveats of the platform, as it makes interactions in that space more akin to a telephone interaction (i.e. lacking in the social signals provided by facial expressions). This could bring us closer — not there, but closer — to the real reaction of friends or family in a virtual space: and I find the implications of that incredibly exciting.