PhotoBooth Lite on Raspberry Pi with TensorFlow Lite

Posted by Lucia Li, TensorFlow Lite Intern
Illustration of the Smart Photo Booth application running in real time.We’re excited to showcase the experience of building a Smart Photo Booth application on Raspberry Pi with TensorFlow (we're not open-sourcing the code yet). It can capture smiling faces and record them automatically. Additionally, you can use speech commands to interact with it. Thanks to the . In this to evaluate the performance of the face detection model on Raspberry Pi

Face Detection Latency Comparison

The whole pipeline of detecting smiling faces, including the three steps we mentioned before, cost 48.1ms with one single thread on average, which means we realized real-time smiling face detection.

Face detection

Our face detection model consists of an 8-bit modified MobileNet v1 body and SSD-Lite head with a 0.25 depth multiplier. Its size is only a little larger than 200kB. Why is this model so small? First, the TensorFlow Lite model is based on Flatbuffer, which is smaller in size than the TensorFlow model based on protobuf. Second, we applied an 8-bit quantized model. Third, our modified MobileNet v1 has fewer channels than the original. Similar to most face detection models, our model outputs the position of a bounding box and 6 landmarks including the left eye, right eye, nose tip, mouth center, left ear tragion, and right ear tragion. We also apply non-maximum suppression to filter repeated faces. The inference time of our face detection TensorFlow Lite model is about 30ms. It means our model can detect a face on Raspberry Pi in real time.

Example of the bounding box and 6 landmarks.

Face cropper

The detected face may have various directions and various sizes. To unify them for better classification, we rotated, cropped, and resized the original image. The input of this function is the positions of the 6 landmarks we get from the face detection model. With 6 landmarks, we can compute the rotation Euler angles and resize ratios. Through this, we can get a 128x128 standard face. The figure below shows an example of our face cropper function. The blue bounding box is the output of the face detection model, while the red bounding box is our calculated cropping bounding box. We duplicated the borderline for the pixels outside the image.

Face Cropper Illustration

Face attribute classification

Our face attribute classification model is also an 8-bit quantized MobileNet model. With a 128x128 standard face as the input, the model outputs a float variable from 0 to 1 to predict the smiling probability. The model also outputs a 90-d vector to predict age from 0 to 90. Its inference times on Raspberry Pi can reach around 30ms.

How to recognize speech commands?

Real-time speech commands recognition can also be divided into three steps:

1. Pre-processing: we use a sliding window to store the latest 1s audio data, with 512 frames different from the last recording.
   
2. Inference: given a 1s audio input, we can apply a speech command recognition model to get probabilities for four categories (“yes”/“no”/“silence”/“unknown”).
   
3. Post-processing: we average current inference result with previous ones. When the average probability of one word exceeds one threshold, we decide that a speech command is detected.

The three steps are explained in detail below.

Pre-processing:

We use we used can be found publicly in many TensorFlow examples. It is composed of audio_spectrogram, MFCC, 2 convolutional layers, and 1 fully-connected layer. The input of this model is 1s of audio data with a sampling rate of 16kHz. The dataset is public, or you can train it yourself. This dataset contains 30 categories of speech command data. Since we only need “yes” and “no”, we disregard all other categories labeled as “unknown”. Additionally, we used other methods to improve the latency performance:

1. We cut half the channels. The TensorFlow Lite model size is about 1.9 MB after compression.
   
2. We used 4 output channels of the last fully-connected layer than the usual 12 as we only need 4 categories.
   
3. We use multi-threads for inference.

In training, we set the background volume to 0.3 to improve the noise tolerance of our model. We also set the silence percentage as 25% and the unknown percentage as 25% to balance the training set.

Post-processing:

Audio Stream Post-processing

Since the audio data we get may only cover half of the words, the single prediction result is not as precise. We stored previous results with recording time that is no more than 1.5s earlier to get average prediction results. It significantly improves the real-time performance of keyword detection. The amount of previous results we keep depends highly on our inference time. For example, the inference time of our model on Raspberry Pi is about 160ms, which means we can keep 9 previous results at most.

Whats next?

We hope to open source code for this example soon in the TensorFlow Lite Github repository. For more information about how to start with TensorFlow Lite, please see . Please let us know what you think or share your TensorFlow Lite use case with us.

Acknowledgements

Lucia Li, Renjie Liu, Tiezhen Wang, Shuangfeng Li, Lawrence Chan, Daniel Situnayake, Pete Warden.