MTTM: Metamorphic Testing for Textual Content Moderation Software

02/11/2023
by   Wenxuan Wang, et al.
0

The exponential growth of social media platforms such as Twitter and Facebook has revolutionized textual communication and textual content publication in human society. However, they have been increasingly exploited to propagate toxic content, such as hate speech, malicious advertisement, and pornography, which can lead to highly negative impacts (e.g., harmful effects on teen mental health). Researchers and practitioners have been enthusiastically developing and extensively deploying textual content moderation software to address this problem. However, we find that malicious users can evade moderation by changing only a few words in the toxic content. Moreover, modern content moderation software performance against malicious inputs remains underexplored. To this end, we propose MTTM, a Metamorphic Testing framework for Textual content Moderation software. Specifically, we conduct a pilot study on 2,000 text messages collected from real users and summarize eleven metamorphic relations across three perturbation levels: character, word, and sentence. MTTM employs these metamorphic relations on toxic textual contents to generate test cases, which are still toxic yet likely to evade moderation. In our evaluation, we employ MTTM to test three commercial textual content moderation software and two state-of-the-art moderation algorithms against three kinds of toxic content. The results show that MTTM achieves up to 83.9 finding rates (EFR) when testing commercial moderation software provided by Google, Baidu, and Huawei, respectively, and it obtains up to 91.2 testing the state-of-the-art algorithms from the academy. In addition, we leverage the test cases generated by MTTM to retrain the model we explored, which largely improves model robustness (0 accuracy on the original test set.

READ FULL TEXT
research
08/18/2023

An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software

The exponential growth of social media platforms has brought about a rev...
research
05/23/2023

Validating Multimedia Content Moderation Software via Semantic Fusion

The exponential growth of social media platforms, such as Facebook and T...
research
02/19/2019

Fusing Visual, Textual and Connectivity Clues for Studying Mental Health

With ubiquity of social media platforms, millions of people are sharing ...
research
08/22/2023

LEAP: Efficient and Automated Test Method for NLP Software

The widespread adoption of DNNs in NLP software has highlighted the need...
research
05/08/2020

Beyond Accuracy: Behavioral Testing of NLP models with CheckList

Although measuring held-out accuracy has been the primary approach to ev...
research
04/26/2022

Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective

Metamorphic testing has recently been used to check the safety of neural...
research
09/12/2023

Catch You Everything Everywhere: Guarding Textual Inversion via Concept Watermarking

AIGC (AI-Generated Content) has achieved tremendous success in many appl...

Please sign up or login with your details

Forgot password? Click here to reset