Using TensorFlow.js to deploy a radiology-based image search application for radiologists

A guest post by Erwin John T. Carpio, MD

As a doctor and radiologist from the Philippines, I’ve always wanted to learn how to develop and apply machine learning (ML from here on) models to my field of practice. However, machine learning was like a foreign language to me when I began. I had limited programming experience, and without a formal computer science background, I felt this field would be beyond my reach, especially when I tried to look at existing research. But that was soon about to change as I started my learning journey and I discovered this field was much more accessible than I initially thought. (version 1) is a web app that uses your device’s camera to and Using RadLens version 1.
Left: An X-ray image of the forearm is scanned using the phone's camera in real time.
Center: RadLens classifies the fracture type as Monteggia or Galeazzi
Right: Click the hyperlink to browse a Google Image Search for images of the detected fracture type for cross reference.The hyperlink to Google Image Search is important, especially as RadLens is trained on only a small dataset. It makes using this tool interactive, and I can assess the accuracy as I go and use any mistakes it may make to help guide my future work. I can explore similar cases and verify if the model is correct, or decide if additional training data needs to be gathered, or if other medical professionals need to be consulted.

Building RadLens

Instead of developing an AI model that is as accurate as a radiologist, I instead decided to focus on small models that help me search for references faster. To build the first version of RadLens (for fractures), my initial prototypes were coded in Python using Tensorflow, taking advantage of a technique called , a website by Google that allows your computer to recognize your own images, sounds, & poses. You can even upload training data using the UI if you wish so training happens live in the web browser. I used the models produced by Teachable Machine to create my 2nd prototype for A .

Looking forward

Most of today’s ML solutions for healthcare come prepackaged, and while robust, have many limitations. These systems have huge file sizes and limited deployment as the model must stay within a central IT system. In addition, they can be very expensive so only large hospitals and clinics can afford to use them. Since they are already pre-trained and pre-packaged, it can be hard for local radiologists to retrain them for use cases that may be more attuned to the local practice’s needs. I essentially want to put cost effective ML into the palm of the local radiologist. Building a proof of concept system was more within reach than I initially thought.
In the future, I am hoping to further improve upon the web app by adding object detection to highlight the found fracture or ossicle with a visible bounding box on the image itself. Currently, the app performs image classification only, which detects the presence but does not show the location.
I have learned that the spread of ML as a technology can both be horizontal and vertical. Horizontal applications are broad and widespread. These are usually made possible through the efforts of larger teams of AI experts, as they traverse the wide canvas of computer vision in medicine. I hope to spark interest in the vertical spread of AI development as it becomes more customized to the individual use cases of specific radiologists around the world. I can think of no better way of doing that right now than with the web and TensorFlow.js to easily enable people to try and experiment with the possibilities of using machine learning in their niche areas.