, Senior Developer Relations Engineer, TensorFlowWelcome to part 2 of my dual approach to content moderation! In this post, I show you how to implement content moderation using machine learning in a server-side environment. If you'd like to see how to implement this moderation client-side, demo code. The website in the Firebase demo showcases content moderation through a basic guestbook using a server-side content moderation system implemented through a , a NoSQL database. The to determine if the text written to the database is inappropriate, and then remove it from the database if needed. With this model, you can evaluate text on different labels of unwanted content, including identity attacks, insults, and obscenity. You can try out .
Server-side moderation
The Firebase text moderation example I used as my starting point doesn't include any machine learning. Instead, it checks for the presence of profanity from a list of words and then replaces them with asterisks using the bad-words npm package. I thought about blending this approach with machine learning (more on that later), but I decided to just wipe the slate clean and replace the code of the Cloud Function altogether. Start by navigating to the Cloud Functions folder of the Text Moderation example:cd text-moderation/functions |
const functions = require('firebase-functions'); const toxicity = require('@tensorflow-models/toxicity'); exports.moderator = functions.database.ref('/messages/{messageId}').onCreate(async (snapshot, context) => { const message = snapshot.val(); // Verify that the snapshot has a value if (!message) { return; } functions.logger.log('Retrieved message content: ', message); // Run moderation checks on the message and delete if needed. const moderateResult = await moderateMessage(message.text); functions.logger.log( 'Message has been moderated. Does message violate rules? ', moderateResult ); }); |
Deploy the code
To deploy the Cloud Function, you can use the .You can view your
Don't just eliminate - moderate!
One of the things I like about the original Firebase moderation sample is that it sanitizes the text rather than just deleting the post. You could run text through the sanitizer before checking for toxic language through the text toxicity model. If the sanitized text is deemed appropriate, then it could overwrite the original text. If it still doesn't meet the standards of decent discourse, then you could still delete it. This might save some posts from otherwise being deleted.What's in a name?
You've probably noticed that my moderation functionality doesn't extend to the name field. This means that even a halfway-clever troll could easily get around the filter by cramming all of their expletives into that name field. That's a good point and I trust that you will use some type of moderation on all fields that users interact with. Perhaps you use an authentication method to identify users so they aren't provided a field for their name. Anyway, you get it: I didn't add moderation to the name field, but in a production environment, you definitely want moderation on all fields.Build a better fit
When you test out real-world text samples on your website, you might find that the text toxicity classifier model doesn't quite fit your needs. Since each social space is unique, there will be specific language that you are looking to include and exclude. You can address these needs by training the model on new data that you provide.If you enjoyed this article and would like to learn more about TensorFlow.js, then there are a ton of things you can do:
- Check out the .
- Play around with TensorFlow.js projects on Glitch.
SOCIAL SHARE CARD GENERATOR