Optimizing style transfer to run on mobile with TFLite

Posted by , Developer Advocates

that is optimized for mobile, and an sample app that uses the model to stylize any images.

In this article, we will walk you through the journey of optimizing the large TensorFlow model for mobile deployment, and how to use it efficiently in a mobile app with TensorFlow Lite. We hope that you can use our pre-trained style transfer model or leverage our insights for your use cases.

Background

. The original technique, however, was computationally expensive and it can take several seconds to stylize an image even on high-end GPUs. Subsequent work by several authors ( for our sample app. The model can take any content and style image as input, then use a feedforward neural network to generate a stylized output image. This model allows much faster style transfer compared to the technique in

The structure of our style transfer model

The Magenta’s arbitrary style transfer model consists of two subnetworks:

• Style prediction network: converts the style image to a style embedding vector.
  
• Style transform network: applies the style embedding vector on the content image to generate a stylized image.
  

Magenta’s style prediction network has an InceptionV3 backbone, so we replaced it with a MobileNetV2 backbone, which is optimized for mobile. The style transform network consists of several convolution layers. We applied the width multiplier idea from * Benchmarked on Pixel 4 CPU using TensorFlow Lite with 2 threads, April 2020.
* See this using the TensorFlow Model Optimization Toolkit. This is an important technique that is applicable for most mobile deployment of TensorFlow models, as it can shrink the model size up to 4X and speed up model inference with insignificant quality trade-off.
Among the quantization options available that TensorFlow provides, we decided to use * Benchmarked on Pixel 4 CPU using TensorFlow Lite with 2 threads, April 2020.

Deployment to mobile

We implemented an Android app to demonstrate how to use the style transfer model. The app takes a style image, a content image, and outputs an image that mixes the style and content of the input images.
We use the phone's camera to capture the content images with the model for CPU inference, and * Benchmarked on Pixel 4 using TensorFlow Lite, April 2020.Another possible performance gain is to cache the results of the style prediction network if you only plan to support a fixed set of style images in your mobile app. This will make your app smaller as you do not need to include the style prediction network, which accounts for 91% of the total network size. This is the main reason why the process is splitted into two models instead of only one.
The sample can be found on .
It is important that we do not run style transfer on the UI thread as it is computational expensive. We instead use the ViewModel class from AndroidX and a Coroutine to run it on a dedicated background thread and easily update the view. Besides, when running a model using that uses TensorFlow Lite to run style transfer on-device. The model used is very similar to the one above but prioritizes quality over speed and model size. Try it out if you are interested in seeing style transfer in production. . Both model versions, the float16 () and the int8 quantized version (), are available on
Magenta is an open source project powered by TensorFlow. It uses machine learning to make music and art. There are many models that can be converted to TensorFlow Lite, including this style transfer model.

TensorFlow Lite can leverage many different types of hardware accelerator available on devices, including GPUs and DSPs, to speed up model inference.

Vollständiger Original-Bericht

Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf blog.tensorflow.org.

↗ Original-Artikel auf blog.tensorflow.org lesen