Facilitating Pornographic Text Detection for Open-Domain Dialogue Systems via Knowledge Distillation of Large Language Models

Pornographic content exists in various media formats, including video, images, and text. Many researchers are making efforts to develop accurate and robust classifiers to filter or detect such large volumes of data in order to control the distribution of pornographic content online. However, most efforts are focused on detecting pornographic images [10, 11, 12, 13] and videos [14, 15, 16], with little research conducted on pornographic text detection [17], let alone dialogues.

The rise of large language models, exemplified by systems such as ChatGPT and GPT-4, has generated considerable interest in their potential for efficient and high-quality data annotation. In the realm of natural language understanding, these large language models are employed to categorize text, such as agriculture [20] and banking [21], while in natural language generation, they aid in producing output sequences. For instance, aiming to tackle the data scarcity in mental health support, SmileChat [8] is a large-scale, diverse, and high-quality dialogue dataset, comprising 55,165 dialogues in total, produced using ChatGPT.

Pornographic text in dialogues refers to written material that contains explicit descriptions or depictions of sexual acts, organs, or behavior intended to arouse sexual excitement. This type of text typically includes explicit language that is intended to elicit sexual arousal or titillation. Pornographic text may vary widely in its content and intensity, ranging from mild descriptions of sexual encounters to more extreme and explicit depictions of taboo or fetishistic acts.

We collect data from several popular social media platforms in the wild, enabling people to engage in profound discussions about life, aspirations, and philosophy with well-known virtual figures for role-playing dialogues.

In open-domain human-machine interaction conversations, while we acknowledge the users’ right to express themselves freely, it is crucial to monitor the appropriateness of user inputs. Ensuring that dialogue systems do not generate pornographic content for users is a crucial task. To address this issue, we propose extracting the dialogue into two data formats: utterance-level and context-level content. For utterance-level content, we split the dialogue into utterances, consisting of $\{u_{i}\}_{1}^{n}$. For context-level content, we divide the dialogue into single-turn sessions, consisting of $\{u_{i}^{\mathrm{U}},u_{i}^{\mathrm{C}}\}_{1}^{n}$, where users initiate the conversation and chatbots respond. $u$ denotes the utterance. $\mathrm{U}$ and $\mathrm{C}$ denote the user and chatbot, respectively.

To avoid the issue of data leakage in the validation and test sets, where a sample in validation or tests occurs in the training set, and to ensure the reliability of model evaluation, we conduct a data deduplication operation prior to data labeling.

Data annotation via knowledge distillation of a large language model (LLM) refers to prompting a teacher LLM to generate a label $y$ for a given input $x$ as the seed knowledge. In this paper, we mainly focus on the zero-shot paradigm, where the LLM is only provided with the input $x$ and an instruction $I$ without any labeled examples or demonstrations. This direct approach to knowledge extraction from teacher LLMs is simple yet effective and has been widely used across various tasks and applications. It only requires having a dataset of input data, which is then fed into the LLM to obtain the desired label $y$. This process can be formulated as follows:

where $\oplus$ denotes the operation of text concatenation, $\mathcal{X}$ denotes the unlabeled dataset, and $p_{T}$ represents the teacher LLM.

Next, we present our algorithm for knowledge distillation of large language models in Algorithm 1, detailing each stage subsequently.

Table I presents examples of pornographic and non-pornographic content in our dataset. Furthermore, Table II presents the statistics of our proposed dataset. For individual utterances, 6,729 out of 76,621 samples in the training set belong to pornographic content, representing 8.78%. However, in single-turn dialogue sessions, 6,558 out of 45,490 samples in the training set are classified as pornographic, accounting for 14.42%. Combining both types of data, there are a total of 13,287 samples out of 122,111 in the training set designated as pornographic content, comprising 10.88%.

To better monitor and detect pornographic text input by users or generated by dialogue systems, we approach the task as a text classification problem. We assemble our dataset as $\mathcal{D}=\{(x_{i},y_{i})\}_{1}^{n}$, where $x_{i}$ has two data formats, as illustrated in $\lx@sectionsign$III-C. At the level of individual utterances, $x_{i}=u$ represents an utterance produced by a user or a dialogue system. At the context level, we focus on whether the model response is pornographic, conditioned on user input. For context-aware detection, we denote $x_{i}=\{\mathrm{[user]}\ u^{\mathrm{U}}\ \mathrm{[SEP]}\ \mathrm{[chatbot]}\ u^{\mathrm{C}}\}$, where $u_{1}$ and $u_{2}$ stand for a single utterance produced by a user and a dialogue system, respectively. $y_{i}$ is the label of the $i$-th sample. For distinguishing context-level content detection, we add two speaker tokens, [user] and [chatbot], and place a [SEP] token between two utterances.

We apply the pre-trained model, BERT [2], which is a popular language model used widely in various tasks in natural language processing, to train a text classification model. In this paper, we fine-tune the entire bert-base-cased¹¹1https://huggingface.co/bert-base-cased model. The output features $h$ of the top layer of the BERT model can be represented as $z=\left[z_{c},z_{1},z_{2},...,z_{n}\right]$, where $z_{c}$ is the representation of the class-specific token [CLS].

The mechanism of pornographic text detection is presented in Figure 5. To facilitate detecting pornographic text for a dialogue system, we train a fully-connected feed-forward neural network (FFNN) with a softmax activation function to identify content categories based on a pre-trained language model. Specifically, we feed $z_{c}$ into a feed-forward neural network with a default model dropout rate of 0.1 for the final prediction. Our optimized objective function is

where $C$ represents the output dimension, which is defined as the union of the label spaces from the training, validation, and test sets, while $y_{d}$ corresponds to the golden label.

During the fine-tuning process, we utilize the Adam optimizer [22] with momentum values $[\beta_{1},\beta_{2}]=[0.9,0.999]$. The learning rate is initialized at $2e-5$ and decays using a linear scheduler. The batch size is set to 16, with a maximum sequence length of 512. We use five commonly used random seeds, including 42, 43, 44, 45, and 46. The warm-up ratio and dropout are both 0.1. The weight decay is 0.01. The training epoch is 10 and we update the model parameters in each batch. We employ the standard cross-entropy loss [23] to train our model and retain the checkpoint when the accuracy is best in the validation set.

We employ the widely used metrics of precision, recall, and F1-score to evaluate the performance of models for each category. Additionally, we utilize macro precision, recall, F1-score and accuracy to evaluate the overall performance of the models.

We present the classification results of the BERT model in Table III. In summary, we observe that the trained classifier can better identify the pornographic category, achieving a macro-precision of 89.20%, a macro-recall of 89.40%, a macro-F1 score of 89.20%, and an average accuracy of 89.60%. These results demonstrate that despite a significant label imbalance, classification performance is satisfactory.

In the predominant normal category, the model prediction displays a certain bias towards it, leading to slightly higher precision, recall, and F1 scores compared to the overall values. Referring to the results in Table III, we observe precision, recall, and F1 scores of 92.80%, 90.40%, and 91.40%, respectively.

Conversely, for the category representing a smaller amount of pornography, the model prediction exhibits a bias towards disadvantage, resulting in slightly lower precision, recall, and F1 scores compared to the overall values. From Table III, we observe precision, recall, and F1 scores of 85.20%, 88.80%, and 86.80%, respectively.

We expect the classifier to accurately predict the pornography category rather than misclassify it as the normal category. At this point, our focus is on recall, aiming to minimize false negatives in the recall value. Simultaneously, we also hope the classifier does not predict too many normal categories as pornography, as this would diminish the model’s robustness. In practical applications, we prioritize minimizing false negative predictions. Therefore, to some extent, we expect the trained classifier’s recall value to be higher than precision.

We present several case studies in Table IV to deeply analyze the performance of model predictions.