How Modiface utilized TensorFlow.js in production for AR makeup try on in the browser

Guest post by Jeff Houghton, Chief Operating Officer - Modiface Inc.

ModiFace has been creating artificial intelligence tech for the beauty industry for over a decade and began working on AR experiences before "Augmented Reality" was a household term. As smartphones hit the market, ModiFace quickly took advantage of the platform to switch from a virtual try on for a 2D image, to a virtual try on for a live 3D video. In 2018, ModiFace was acquired by the L'Oreal Group, and since then we’ve expanded our live virtual try on to be even more accessible by expanding our reach to the web using to find a match for the face, so the smaller the implementation, the more accuracy that was sacrificed. While small, we had pushed the structure to its limit, and it still had a size that would take a while to load on slower networks - especially on mobile devices without WiFi. With less accuracy, edge cases for different face shapes became harder to track. Our original goal of launching virtual try on via the web was to make the experience accessible, and instead, it became an experience that was not as inclusive for everyone.

To improve our accuracy and create a product that we could continuously improve, we turned to more state-of-the art solutions such as using CNNs to find a user’s face. However, current popular architectures (including those we’ve used on other platforms) suffered from the same problems that existed in our previous experience; they were large and computationally expensive.

Our new solution

We focused our development on network structures which aim to maximize accuracy with less available computing power. Using enables you to build your own looks across their entire catalogue.

, hover to try on over 100 different shades.

has instant looks to make finding a set of products super simple.