We created this end-to-end tutorial to help developers with these objectives:
- Provide a reference for the developers looking to convert models written in TensorFlow 1.x to their TFLite variants using the new features of the latest (v2) converter — for example, the MLIR-based converter, more supported ops, and improved kernels, etc.
(In order to convert TensorFlow 2.x models in TFLite please follow for model saving/conversion, populating metadata; and the Android code on GitHub for details. While this tutorial discusses the steps of how to create the TFLite models , feel free to download them directly from TensorFlow Hub by Xinrui Wang and Jinze Yu. For this tutorial, we used the generator part of White-box CartoonGAN.Create the TensorFlow Lite Model
The authors of White-box CartoonGAN . Here is a step-by-step summary of what we will be covering in this section:- Generate a SavedModel out of the pre-trained model checkpoints.
- Convert SavedModel with post-training quantization using the latest TFLiteConverter.
- Run inference in Python with the converted model.
- Add metadata to enable easy integration with a mobile app.
- Run model benchmark to make sure the model runs well on mobile.
Generate a SavedModel from the pre-trained model weights
The pre-trained weights of White-box CartoonGAN come in the following format (also referred to as checkpoints) -As the original White-box CartoonGAN model is implemented in TensorFlow 1, we first need to generate a single self-contained model file in the SavedModel format using TensorFlow 1.15. Then we will switch to TensorFlow 2 later to convert it to the lightweight TFLite format. To do this we can follow this workflow -TEXT├── checkpoint
├── model-33999.data-00000-of-00001
└── model-33999.index- Create a placeholder for the model input.
- Instantiate the model instance and run the input placeholder through the model to get a placeholder for the model output.
- Load the pre-trained checkpoints into the current session of the model.
- Finally, export to SavedModel.
This is how all of this looks in code in TensorFlow 1.x:Now that we have the original model in the SavedModel format, we can switch to TensorFlow 2 and proceed toward converting it to TFLite.PYTHONwith tf.Session() as sess:
input_photo = tf.placeholder(tf.float32, [1, None, None, 3], name='input_photo')
network_out = network.unet_generator(input_photo)
final_out = guided_filter.guided_filter(input_photo, network_out, r=1, eps=5e-3)
final_out = tf.identity(final_out, name='final_output')
all_vars = tf.trainable_variables()
gene_vars = [var for var in all_vars if 'generator' in var.name]
saver = tf.train.Saver(var_list=gene_vars)
sess.run(tf.global_variables_initializer())
saver.restore(sess, tf.train.latest_checkpoint(model_path))
# Export to SavedModel
tf.saved_model.simple_save(
sess,
saved_model_directory,
inputs={input_photo.name: input_photo},
outputs={final_out.name: final_out}
)Convert SavedModel to TFLite
TFLite provides support for three different .
In order to let theTFLiteConvertertake advantage of this strategy, we need to just passconverter.representative_dataset = representative_dataset_genand removeconverter.target_spec.supported_types = [tf.float16].
So after we generated these different models here’s how we stand in terms of model size -.
These models are available on .Running inference in Python
After you have generated the TFLite models, it is important to make sure that models perform as expected. A good way to ensure that is to run inference with the models in Python before integrating them in mobile applications.
Before feeding an image to our White-box CartoonGAN TFLite models it’s important to make sure that the image is preprocessed well. Otherwise, the models might perform unexpectedly. The original model was trained using BGR images, so we need to account for this fact in the preprocessing steps as well. You can find all of the preprocessing steps in this Again, you can find all of the postprocessing steps in . But in this section, we are going to provide you with some of the important pointers about metadata population for the TFLite models we generated. You can follow running in background and another with an Repeat above steps for the other two tflite models:float16andint8variants.
In summary, here is the average inference time we got from the benchmark tool running on a Pixel 4:| . Model deployment to Android
Now that we have the quantized TensorFlow Lite models with metadata by either following the previous steps (or by downloading the models directly from TensorFlow Hub .
The Android app uses Jetpack Navigation Component for UI navigation and CameraX for image capture. We use the new ML Model Binding feature for importing the tflite model and then Kotlin Coroutine for async handling of the model inference so that the UI is not blocked while waiting for the results.
Let’s dive into the details step by step:- Download Android Studio 4.1 Preview.
- Create a new Android project and set up the UI navigation.
- Set up the CameraX API for image capture.
- Import the .tflite models with ML Model Binding.
- Putting everything together.
Download Android Studio 4.1 Preview
We need to first install Android Studio Preview (4.1 Beta 1) in order to use the new ML Model Binding feature to import a .tflite model and auto code generation. You can then explore the tfllite models visually and most importantly use the generated classes directly in your Android projects.
Download the Android Studio Preview to learn more details about this support library.
There are 3 screens in this sample app:PermissionsFragment.kthandles checking the camera permission.CameraFragment.kthandles camera setup, image capture and saving.CartoonFragment.kthandles the display of input and cartoon image in the UI.
nav_graph.xml defines the navigation of the three screens and data passing betweenCameraFragmentandCartoonFragment.Set up CameraX for image capture
CameraX is a Jetpack support library which makes camera app development much easier.
Camera1 API was simple to use but it lacked a lot of functionality. Camera2 API provides more fine control than Camera1 but it’s very complex — with almost 1000 lines of code in a very basic example.
CameraX on the other hand, is much easier to set up with 10 times less code. In addition, it’s lifecycle aware so you don’t need to write the extra code to handle the Android lifecycle.
Here are the steps to set up CameraX for this sample app:- Update
build.gradledependencies - Use
CameraFragment.ktto hold the CameraX code - Request camera permission
- Update
AndroidManifest.ml - Check permission in
MainActivity.kt - Implement a viewfinder with the CameraX Preview class
- Implement image capture
- Capture an image and convert it to a
Bitmap
CameraSelectoris configured to be able to take use of the front facing and rear facing camera since the model can stylize any type of faces or objects, and not just a selfie.
Once we capture an image, we convert it to aBitmapwhich is passed to the TFLite model for inference. Navigate to a new screenCartoonFragment.ktwhere both the original image and the cartoonized image are displayed.Import the TensorFlow Lite models
Now that the UI code has been completed. It’s time to import the TensorFlow Lite model for inference. ML Model Binding takes care of this with ease. In Android Studio, go to File > New > Other > TensorFlow Lite Model: This import accomplishes two things:- automatically create a ml folder and place the model file .tflite file under there.
- auto generate a Java class under the folder:
app/build/generated/ml_source_out/debug/[package-name]/ml, which handles all the tasks such as model loading, image pre-preprocess and post-processing, and run model inference for stylizing the input image.
When developing applications, it’s important to consider for resources and tools you can use.↗ Original-Artikel auf blog.tensorflow.org lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf blog.tensorflow.org. - Generate a SavedModel out of the pre-trained model checkpoints.
SOCIAL SHARE CARD GENERATOR