Iris landmark tracking in the browser with MediaPipe and TensorFlow.js

Posted by , Software Engineers at Google
enables a wide range of applications, such as hands-free interfaces for assistive technologies and understanding user behavior beyond clicks and gestures. Iris tracking is also a challenging computer vision problem. Eyes appear under variable light conditions, are often occluded by hair, and can be perceived as differently shaped depending on the head’s angle of rotation and the person’s expression. Existing solutions rely heavily on specialized hardware, often requiring a costly headset or a remote eye tracker system. These approaches are ill-suited for mobile devices with limited computing resources.

the release of a new package detecting facial landmarks in the browser. Today, we’re excited to add iris tracking to this package through the . We have deprecated the and the accompanying ). The MediaPipe iris model is able to track landmarks for the iris and pupil using a single RGB camera, in real-time, without the need for specialized hardware. The model also returns landmarks for the eyelids and eyebrow regions, enabling detection of slight eye movements such as blinking.

Above left are predictions from @tensorflow-models/[email protected], above right are predictions from @tensorflow-models/[email protected]. Iris landmarks are in red.

Users familiar with our existing model with only a few code changes, detailed below. faceLandmarksDetection offers three major improvements over facemesh:

1. Iris keypoints detection
2. Improved eyelid contour detection
3. Improved detection for rotated faces

These improvements are highlighted in the GIF above, which demonstrates how the landmarks returned by faceLandmarksDetection and facemesh differ for the same image sequence.

Installation

There are two ways to install the faceLandmarksDetection package:

1. Through script tags:
TEXT Kopieren

<script src="https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]/dist/tf.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/face-landmarks-detection"></script>

2. Through NPM (via the or even an for more details about the API.

   Performance

   FaceLandmarksDetection is a lightweight package containing only ~3MB of weights, making it ideally suited for real-time inference on a variety of mobile devices. When testing, note that TensorFlow.js also provides several different backends to choose from, including WebGL and WebAssembly (WASM) with