Face Tracking

Overview

With face tracking, you can create interactive gameplay in your module that uses the position and expression of a player's face. Whether that's trying not to laugh or keeping your head above water, we make it easy to create games that feel magical. ✨

How it works

The FaceTracker uses a machine learning model on the camera feed to determine the co-ordinates of different face landmarks (e.g. nose, left eye, upper lip) and measure the intensity of various facial expressions (e.g. smiling, squinting). In this guide, we'll show you how you can detect faces and use each of these face tracking properties in your module.

Implementation

Here's a step-by-step recipe which walks you through getting started with the FaceTracker. You'll learn how to listen for face detection events, and subscribe to updates for each face tracking property.

Face Landmarks

Each face landmarks update provides an array of positions, with each face landmark indexed by FacePointIndex.

Blend Shape Expressions

Each blend shapes update provides an array of intensity values from 0 to 100, with each face action unit indexed by BlendShapeIndex.

Best Practices

• Start and stop the tracker: Only keep face tracking active while it is used for gameplay in your module
• Frames per second: If you're finding you're having performance issues, you can try reducing the frame rate to 10-15 FPS
• Memory usage: Face tracking can be memory-intensive, so be sure to consider the memory impact of other features in your module
• Unsubscribe when a face is lost: When you receive a FaceTracker.FaceRemoved event, make sure to unsubscribe from its face tracking updates
• Setup once: Make sure you're only setting up the face tracker once, and not every time a screen is shown