TSEC NEWS: 06.05.21 Cron-Job Fehlerhaft nach PHP Update + PWA mobile + Desktop / 04.05.21 - Android App von TSECURITY 28.04.21 - NEUER SERVER // 26.04.21 ++ Download the Electron-App für tsecurity.de // Über 550 Feed-Quellen

❈ MediaPipe 3D Face Transform

Programmierung feedproxy.google.com

Posted by Kanstantsin Sokal, Software Engineer, MediaPipe team

Earlier this year, the MediaPipe Team released the Face Mesh solution, which estimates the approximate 3D face shape via 468 landmarks in real-time on mobile devices. In this blog, we introduce a new face transform estimation module that establishes a researcher- and developer-friendly semantic API useful for determining the 3D face pose and attaching virtual objects (like glasses, hats or masks) to a face.

The new module establishes a metric 3D space and uses the landmark screen positions to estimate common 3D face primitives, including a face pose transformation matrix and a triangular face mesh. Under the hood, a lightweight statistical analysis method called Procrustes Analysis is employed to drive a robust, performant and portable logic. The analysis runs on CPU and has a minimal speed/memory footprint on top of the original Face Mesh solution.

MediaPipe image

Figure 1: An example of virtual mask and glasses effects, based on the MediaPipe Face Mesh solution.


The MediaPipe Face Landmark Model performs a single-camera face landmark detection in the screen coordinate space: the X- and Y- coordinates are normalized screen coordinates, while the Z coordinate is relative and is scaled as the X coordinate under the weak perspective projection camera model. While this format is well-suited for some applications, it does not directly enable crucial features like aligning a virtual 3D object with a detected face.

The newly introduced module moves away from the screen coordinate space towards a metric 3D space and provides the necessary primitives to handle a detected face as a regular 3D object. By design, you'll be able to use a perspective camera to project the final 3D scene back into the screen coordinate space with a guarantee that the face landmark positions are not changed.

Metric 3D Space

The Metric 3D space established within the new module is a right-handed orthonormal metric 3D coordinate space. Within the space, there is a virtual perspective camera located at the space origin and pointed in the negative direction of the Z-axis. It is assumed that the input camera frames are observed by exactly this virtual camera and therefore its parameters are later used to convert the screen landmark coordinates back into the Metric 3D space. The virtual camera parameters can be set freely, however for better results it is advised to set them as close to the real physical camera parameters as possible.

MediaPipe image

Figure 2: A visualization of multiple key elements in the metric 3D space. Created in Cinema 4D

Canonical Face Model

The Canonical Face Model is a static 3D model of a human face, which follows the 3D face landmark topology of the MediaPipe Face Landmark Model. The model bears two important functions:

  • Defines metric units: the scale of the canonical face model defines the metric units of the Metric 3D space. A metric unit used by the default canonical face model is a centimeter;
  • Bridges static and runtime spaces: the face pose transformation matrix is - in fact - a linear map from the canonical face model into the runtime face landmark set estimated on each frame. This way, virtual 3D assets modeled around the canonical face model can be aligned with a tracked face by applying the face pose transformation matrix to them.

Face Transform Estimation

The face transform estimation pipeline is a key component, responsible for estimating face transform data within the Metric 3D space. On each frame, the following steps are executed in the given order:

  • Face landmark screen coordinates are converted into the Metric 3D space coordinates;
  • Face pose transformation matrix is estimated as a rigid linear mapping from the canonical face metric landmark set into the runtime face metric landmark set in a way that minimizes a difference between the two;
  • A face mesh is created using the runtime face metric landmarks as the vertex positions (XYZ), while both the vertex texture coordinates (UV) and the triangular topology are inherited from the canonical face model.

Effect Renderer

The Effect Renderer is a component, which serves as a working example of a face effect renderer. It targets the OpenGL ES 2.0 API to enable a real-time performance on mobile devices and supports the following rendering modes:

  • 3D object rendering mode: a virtual object is aligned with a detected face to emulate an object attached to the face (example: glasses);
  • Face mesh rendering mode: a texture is stretched on top of the face mesh surface to emulate a face painting technique.

In both rendering modes, the face mesh is first rendered as an occluder straight into the depth buffer. This step helps to create a more believable effect via hiding invisible elements behind the face surface.

MediaPipe image

Figure 3: An example of face effects rendered by the Face Effect Renderer.

Using Face Transform Module

The face transform estimation module is available as a part of the MediaPipe Face Mesh solution. It comes with face effect application examples, available as graphs and mobile apps on Android or iOS. If you wish to go beyond examples, the module contains generic calculators and subgraphs - those can be flexibly applied to solve specific use cases in any MediaPipe graph. For more information, please visit our documentation.

Follow MediaPipe

We look forward to publishing more blog posts related to new MediaPipe pipeline examples and features. Please follow the MediaPipe label on Google Developers Blog and Google Developers twitter account (@googledevs).


We would like to thank Chuo-Ling Chang, Ming Guang Yong, Jiuqiang Tang, Gregory Karpiak, Siarhei Kazakou, Matsvei Zhdanovich and Matthias Grundman for contributing to this blog post.


Kompletten Artikel lesen (externe Quelle: http://feedproxy.google.com/~r/GDBcode/~3/7mE5b-Cz_n0/mediapipe-3d-face-transform.html)

Zur Startseite

➤ Weitere Beiträge von Team Security | IT Sicherheit (tsecurity.de)

Object Detection and Tracking using MediaPipe

vom 1022.71 Punkte
Posted by Ming Guang Yong, Product Manager for MediaPipeMediaPipe in 2019MediaPipe is a framework for building cross platform multimodal applied ML pipelines that consist of fast ML inference, classic computer vision, and media processing (e.g. video decoding). Med

MediaPipe 3D Face Transform

vom 942.47 Punkte
Posted by Kanstantsin Sokal, Software Engineer, MediaPipe team Earlier this year, the MediaPipe Team released the Face Mesh solution, which estimates the approximate 3D face shape via 468 landmarks in real-time on mobile devices. In this blog, we intr

High Fidelity Pose Tracking with MediaPipe BlazePose and TensorFlow.js

vom 858.98 Punkte
Posted by Ivan Grishchenko, Valentin Bazarevsky and Na Li, Google Research Today we’re excited to launch MediaPipe's BlazePose in our new pose-detection API. BlazePose is a high-fidelity body pose model designed specifically to support challenging domains like

MediaPipe on the Web

vom 813.78 Punkte
Posted by Michael Hays and Tyler Mullen from the MediaPipe teamMediaPipe is a framework for building cross-platform multimodal applied ML pipelines. We have previously demonstrated building and running ML pipelines as MediaPipe graphs on mobile (Android, iOS) and on edge d

SignAll SDK: Sign language interface using MediaPipe is now available for developers

vom 768.58 Punkte
A guest post by the Engineering team at SignAll | Twitter handle | MediaPipe team Please note that the information, uses, and applications expressed in the below post are solely those of our guest author, SignAll. SignAll SDK: Sign language interface u

Face and hand tracking in the browser with MediaPipe and TensorFlow.js

vom 661.43 Punkte
Posted by Ann Yuan and Andrey Vakunov, Software Engineers at GoogleToday we’re excited to release two new packages: facemesh and handpose for tracking key landmarks on faces and hands respectively. This release has been a collaborative effort between the MediaPipe and TensorFlow.js teams within Go

Control your Mirru prosthesis with MediaPipe hand tracking

vom 655.42 Punkte
Guest post by the Engineering teams at Mirru and Tweag What is the Mirru App? Mirru is a free and open source Android app under development with which one can control robotic prosthetic hands via hand tracking. With our app, a user can instantly mirror grips from

Alfred Camera: Smart camera features using MediaPipe

vom 630.22 Punkte
Guest post by the Engineering team at Alfred Camera Please note that the information, uses, and applications expressed in the below post are solely those of our guest author, Alfred Camera. In this article, we’d like to give you a short overview of Alfred

Giving users and developers more control over focus

vom 557.68 Punkte
Chrome 86 introduces two new features that improve both the user and developer experience when it comes to working with focusThe :focus-visible pseudo-class is a CSS selector that lets developers opt-in to the same heuristic the browser uses when it's

Iris landmark tracking in the browser with MediaPipe and TensorFlow.js

vom 520.46 Punkte
Posted by Ann Yuan and Andrey Vakunov, Software Engineers at Google Iris tracking enables a wide range of applications, such as hands-free interfaces for assistive technologies and understanding user behavior beyond clicks and gestures. Iris tracking is also a challenging computer vision problem. Eyes appear under variable l

MediaPipe KNIFT: Template-based Feature Matching

vom 459.41 Punkte
Posted by Zhicheng Wang and Genzhi Ye, MediaPipe team Image Feature Correspondence with KNIFTIn many computer vision applications, a crucial building block is to establish reliable correspondences between different views of an object or scene, forming the

Instant Motion Tracking with MediaPipe

vom 429.41 Punkte
Posted by Vikram Sharma, Software Engineering Intern; Jianing Wei, Staff Software Engineer; Tyler Mullen, Senior Software Engineer Augmented Reality (AR) technology creates fun, engaging, and immersive user experiences. The ability to perform AR tracki

Team Security Diskussion über MediaPipe 3D Face Transform