NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task–Next Sentence Prediction

by   Yi Sun, et al.
NetEase, Inc

Using prompts to utilize language models to perform various downstream tasks, also known as prompt-based learning or prompt-learning, has lately gained significant success in comparison to the pre-train and fine-tune paradigm. Nonetheless, virtually all prompt-based methods are token-level, meaning they all utilize GPT's left-to-right language model or BERT's masked language model to perform cloze-style tasks. In this paper, we attempt to accomplish several NLP tasks in the zero-shot scenario using a BERT original pre-training task abandoned by RoBERTa and other models–Next Sentence Prediction (NSP). Unlike token-level techniques, our sentence-level prompt-based method NSP-BERT does not need to fix the length of the prompt or the position to be predicted, allowing it to handle tasks such as entity linking with ease. Based on the characteristics of NSP-BERT, we offer several quick building templates for various downstream tasks. We suggest a two-stage prompt method for word sense disambiguation tasks in particular. Our strategies for mapping the labels significantly enhance the model's performance on sentence pair tasks. On the FewCLUE benchmark, our NSP-BERT outperforms other zero-shot methods on most of these tasks and comes close to the few-shot methods.


ELECTRA is a Zero-Shot Learner, Too

Recently, for few-shot or even zero-shot learning, the new paradigm "pre...

OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Model

To enrich language models with domain knowledge is crucial but difficult...

On the Role of Bidirectionality in Language Model Pre-Training

Prior work on language model pre-training has explored different archite...

Zero-shot Entity Linking with Efficient Long Range Sequence Modeling

This paper considers the problem of zero-shot entity linking, in which a...

A Few-shot Approach to Resume Information Extraction via Prompts

Prompt learning has been shown to achieve near-Fine-tune performance in ...

Mediators in Determining what Processing BERT Performs First

Probing neural models for the ability to perform downstream tasks using ...

Template-free Prompt Tuning for Few-shot NER

Prompt-based methods have been successfully applied in sentence-level fe...

Code Repositories


The code for our paper "NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task —— Next Sentence Prediction"

view repo

1 Introduction

GPT-2 (up to 1.5B radford2019language) and GPT-3 (up to 175B brown2020language) are ultra-large-scale language models with billions of parameters that have recently demonstrated outstanding performance in various NLP tasks. Compared with previous state-of-the-art fine-tuning methods, they can achieve competitive results without any or with just a limited quantity of training data. Although studies have shown that scaling up the model improves task-agnostic and few-shot performance, some studies have shown that by constructing appropriate prompts for the model, models like BERT devlin2018bert or RoBERTa liu2019roberta can achieve similar performance despite having a parameter count that is several orders of magnitude smaller schick2021it; schick2021exploiting; wang2021entailment.

Figure 1: Prompts for various NLP tasks of NSP-BERT.

Since then, the area of natural language processing has seen a fresh wave of developments, including the introduction of a new paradigm known as

prompt-based learning or prompt-learning, which follows the ”pre-train, prompt, and predict” liu2021pretrain process. In zero-shot and few-shot learning, prompt-learning has achieved a lot of success. Not only does it achieve outstanding performance, prompt-learning better integrates pre-training and downstream tasks and brings NLP tasks closer to human logic and habits.

Figure 2: (Left) MLM task for token-level prompt-learning. (Right) NSP task for sentence-level prompt-learning.

The input text for the classification task, for example, “The Italian team won the European Cup.”, should be assigned to one of the candidate labels, such as Gaming, Sports, or Finance. At this point, the template “This is [MASK] news.” will be added to the original text, and the model will be asked to predict the missing word or span. The model’s output will then be mapped to the candidate labels. We could utilize the pre-training tasks of several types of language models (LM) to predict the abovementioned templates, including but not limited to Left-to-right LM (GPT series radford2018improving; radford2019language; brown2020language), Masked LM (BERT devlin2018bert, RoBERTa liu2019roberta), prefix LM (UniLM dong2019unified; bao2020unilmv2) and Encoder-decoder LM (T5 raffel2019exploring, BART lewis2020bart).

Although most research on prompt-learning has been conducted, the majority of the pre-training tasks used in prompt-learning are token-level, requiring the labels to be mapped to a fixed-length token span schick2021it; schick2021exploiting; cui2021template. On the one hand, when the number of labels grows rapidly, this necessitates a lot of human labor. On the other hand, tasks with variable-length options make Left-to-right LM (L2R LM) or masked LM (MLM) difficult to cope with. The length of each candidate entity’s description, for example, varies significantly in the entity linking task.

At the same time, we observed that there is an original sentence-level pre-training object in vanilla BERT——NSP (Next Sentence Prediction), which is a binary classification task that predicts whether two sentences appear consecutively within a document or not. Many models, like RoBERTa liu2019roberta and many others conneau2019cross; yang2019xlnet; joshi2020spanbert, have questioned and abandoned this task during pre-training. Nevertheless, based on the task’s features and object, we believe it is appropriate to use in prompt-learning.

Unlike most prior work, we present NSP-BERT, a sentence-level prompt-learning method. The paper’s main contributions can be summarized as follows:

  • We propose the use of NSP, a sentence-level pre-training task for prompt-learning. On the FewCLUE benchmark, NSP-BERT has achieved the SOTA performance among zero-shot models without using any task-specific training data. Its performance is comparable to that of several few-shot learning methods.

  • Based on the features of the downstream tasks, we propose two alternative label/answer mapping methods that significantly improved prompt-learning performance in the sentence-pair task.

  • We suggest to use soft-position and two-stage prompt construction methods to alleviate the problem that sentence-level prompt-based models are not sensitive to token positions, which further improves the performance of NSP-BERT on word sense disambiguation tasks.

  • We demonstrate that a simple sentence-level contrastive learning pre-training task on interactive models can fit prompt-based learning well and solve various NLP tasks. It is very inspiring for zero-shot and few-shot learning.

2 Related Work

Many studies on prompt-learning for zero-shot or few-shot have been conducted. This section focuses on models at different levels and optimization methods.

2.1 Token-Level and Sentence-Level

Token-Level Prompt-Learning

Token-level pre-training tasks, such as MLM (Shown in the left part of Figure 2) jiang2020how; schick2021it; schick2021exploiting or L2R LMradford2019language; brown2020language; cui2021template, are commonly used in token-level prompt-learning approaches. Although the expected answer may be in the form of tokens, spans, or sentences in token-level prompt-learning, the predicted answer is always generated token by token. Tokens are usually mapped to the whole vocabulary or a set of candidate words petroni2019language; cui2021template; han2021ptr; adolphs2021query; hu2021knowledgeable. Take PET model schick2021it; schick2021exploiting as an example, the sentiment classification input/label pair is reformulated to “x: [CLS] The Italian team won the European Cup. This is [MASK] news. [EOS], : Sports”.

Sentence-Level Prompt-Learning

Sentence-level methods concentrate on the relationship between sentences, with the model’s output usually mapped to a relationship space. As far as we know, EFL wang2021entailment is the only sentence-level model. It reformulates NLP tasks into sentence entailment-style tasks. For example, the sentiment classification input/label pair is reformulated to “: [CLS] The Italian team won the European Cup. [SEP] This is Sports news.[EOS], : Entail”. The output of model is Entail or Not Entail. The EFL model can perform well on few-shot learning but not on Zero-shot tasks unless it is trained on labeled natural language inference (NLI) datasets like MNLI williams2018a.

2.2 Optimization methods

Automated Prompt

Manually designed prompts are highly unstable. Sometimes it is necessary to be familiar with the particular task and language model in order to construct a high-quality prompt. As a result, several studies attempt to automatically search for and generate prompts. LM-BFF gao2021making model use conditional likelihood to automatically select labels words, and use T5 raffel2019exploring to generate templates. AutoPrompt shin2020autoprompt uses a gradient-guided search to create prompts. Compared to the discrete prompt search methods mentioned above, P-tuning liu2021gptunderstands

employs trainable continuous prompt embeddings, with P-tuning, GPTs achieve comparable and sometimes better performance to similar-sized BERTs in supervised learning.

Training Strategy

There are many optimization methods in prompt-learning. ADAPET tam2021improving uses more supervision by decoupling the losses for the label tokens and a label-conditioned MLM objective over the full original input. PTR han2021ptr incorporates logic rules to compose task-specific prompts with several simple sub-prompts. zhao2021calibrate

pointed out that there are 3 types of bias (majority label bias, recency bias and common token bias) in GPT. By using content-free inputs (e.g. “N/A”) to calibrate the model’s output probabilities, the performance of GPT-2 and GPT-3 has been substantially improved.

3 Framework of NSP-BERT

In this section, we will introduce the framework of NSP-BERT.

3.1 Next Sentence Prediction

The next sentence prediction is one of the two basic pre-training tasks (the other is MLM) of the vanilla BERT model devlin2018bert (Shown in the right part of Figure 2). This task inputs two sentences A and B into BERT at the same time to predict whether sentence B comes after sentence A in the same document. During specific training, for of the time, B is the actual next sentence that follows A (IsNext), and for the other of the time, we use a random sentence from the corpus (NotNext).


Let denote the model trained on a large-scale corpus. This model is trained on both MLM task and NSP task at the same time. and denote sentence A and sentence B, respectively. The model’s input is , and denotes the output probability of model’s NSP head (Eq. 2). , where

is the hidden vector of

[CLS] and is a matrix learned by NSP task,

. The loss function of NSP task

, where .


NSP is a self-supervised task that is simple and weak. We believe the task is more likely to judge whether two sentences are from the same document since the negative sample is randomly picked from another unrelated document. In other words, rather of determining the order of two phrases, the NSP task may determine if they have the same topic and express the same semantics.

The NSP task is quite similar to a contrastive learning task, as shown in Figure 3. So, does the NSP just compare sentence similarities or does it have the ability to reason logically? The following are the major reasons why we believe NSP has logical reasoning ability:

  • The NSP task is interactive. Tokens in one sentence could interact with their own tokens while also interacting with tokens in the other sentence.

  • The NSP task is trained alongside the MLM task. The MLM task provides a training basis for the self-attention mechanism of the entire model.

Figure 3: Conceptual comparison between End-to-End representation-based contrastive learning of sentence embeddings (CSE) and All-to-All interaction-based next sentence prediction (NSP). Except that the output of the model is not the representation of the sentence, the NSP task uses a weak self-supervision method to train the BERT.

NSP-BERT is a true prompt-based learner, not a sentence similarity matcher, as determined by the above two points. This will be confirmed in our experiments. The model performs better the closer the template is to a fluent and logical natural language sentence.

3.2 Prompts in NSP-BERT

NSP-BERT, like other prompt-based learning methods, requires the construction of appropriate templates for various tasks. Since NSP-BERT does not rely on the training data of any downstream tasks, the template’s building form must closely match the original NSP task. In this section, we’ll show how to construct templates for different tasks.

Figure 4: The comparison of markers (Left) and two-stage prompt (Right), examples in coreference resolution and entity linking/typing tasks.

Single Sentence Task

Samples must be classified into different topics in the single sentence task. Sentiment analysis, for example, is the classification of texts into various sentiment trends. Suppose that the training dataset of a single sentence classification task

, is the th sentence in the total samples, and the label of is , which can be mapped to , where , is the number of topics in this dataset. For each , it will be mapped to a template . And the input of the model will be,


the probability when the label of sample is is:


Sentence Pair Task

The sentence pair tasks aim to identify the relationship between two sentences. Such as OCNLI ocnli, BUSTM BUSTM and CSL clue used in this paper. This type of dataset contains samples, each with 2 sentences and . The relationship between them is , which can be mapped to , where , is the number of relationship types. Its input is the same as Eq. 1. The output of the NSP model is shown in Eq. 5. (We do not directly associate the output of the NPS model directly with the labels here.)


Cloze-Style Task

The cloze-style task is to give a sentence with blanks, and the model must find the most appropriate tokens or spans to fill in the blanks. The dataset . For each sample, there is a sentence with a [BLANK], and there are candidates to be chosen. For each option , there is a template corresponding to it. Given the input


the output of model is:


As shown in Figure 5, in cloze-style task (such as ChID), if we use the regular position embeddings (hard-position embeddings), the candidate word coin can’t perceive the [BLANK] and the context. Inspired by liu2020kbert and sun2020colake, we adopt the soft-position index which allows NSP to work similarly like MLM. We align the position index of the candidate word coin with [BLANK].

Figure 5: Soft-position and hard-position in NSP-BERT.

Word Sense Disambiguation

In a fully supervised training scenario, we may add markers before and after the word to identify the word to be disambiguated huang2019glossbert; soares2019matching; wu2019enriching, as illustrated in Figure 4. Because there is no downstream tasks training data for sentence-level prompt-learning, it is impossible to identify the target word’s position by markers. We propose a Two-Stage Prompt construction method to indicate the target word using natural language descriptions in our NSP-BERT.

  • Stage 1: Prompt the target word at the end of sentence A. This stage’s purpose is to provide enough context for the target word.

  • Stage 2: Prompt the description of the candidate word sense in sentence B.

Feed the two-stage prompt into the language model, and it will determine if the sentence is fluent and reasonable. Let and denote the first and the second part of the prompt. The model’s input is:

Figure 6: Two answer mapping methods candidates-contrast method (Left) and samples-contrast method (Right).

3.3 Answer Mapping

It’s easy to observe that not all probability outputs in the above tasks are directly linked with labels. This is because not all datasets can provide contrastive candidate objections (sentiments/topics/idioms/entities). Pre-trained language models, on the other hand, are not susceptible to negative inference kassner2020negated, the NSP model is no exception. As a result, we propose two answer mapping methods, candidates-contrast answer mapping and samples-contrast answer mapping, for different situations.


For datasets with multiple candidates, such as candidate sentiments (EPRSTMT), candidate topics (TNEWS, IFLYTEK and CSLDCP), candidate idioms (ChID) and candidate entities (DuEL). For the above datasets, there is a template (or ) corresponding to the label (or ). As show in Figure 6. We take the highest probability output by among the candidates as the final output answer where the condition is IsNext:



For datasets with no candidate for contrast (such as OCNLI, BUSTM, CSL and CLUEWSC), we propose the samples-contrast answer mapping method (Figure 6), this procedure is summarized in Algorithm 1. Considering the fairness of the comparative experiment, we consider two preconditions. One is that a complete development set and a test set can be obtained at the same time; the other is that only the development set can be obtained, and the test samples must be predicted one by one or batch by batch during testing. In our experiment, we use the development set to determine the thresholds of probability, and use these thresholds to predict the test set.

Input: Test set , where .
Parameter: Oder {“ascending”, “descending”}, distribution of labels .

1:  for  do
3:  end for
4:   rank(, )
5:   divide ()
6:  for  do
7:      where
8:  end for
Algorithm 1 Samples-Contrast Answer Mapping

4 Experiment

4.1 Tasks and Datasets


We evaluate our model mainly on FewCLUE xu2021fewclue, a Chinese Few-shot Learning Evaluation Benchmark, which contains 9 NLU tasks in Chinese, with 4 single sentence tasks, 3 sentence pair tasks and 2 reading comprehension tasks. The details of the datatsets is shown in Appendix. The number of samples in every training set is few, each label corresponds to 8 or 16 samples. We report the accuracy on all 9 tasks following FewCLUE.


In order to further verify the ability of NSP-BERT for word sense disambiguation, the entity linking dataset DuEL2.0 was added. In particular, we divide DuEL2.0 into two parts. In the first part, the entity linking part, there are 26586 samples. All the samples’ mention can be mapped to single or multiple entities in the knowledge base, and each mention can be linked to 5.37 entities on average. In the second part, the entity typing part, there are 6465 samples. Those samples’ mention cannot be found in the knowledge base, but they will be divided into their corresponding upper entity types. There are a total of 24 upper entity types.

4.2 Baselines

Refer to the FewCLUE xu2021fewclue 222https://github.com/CLUEbenchmark/FewCLUE, we mainly choose 3 training scenarios, fine-tuning, few-shot and zero-shot.

Single Sentence Sentence Pair Others
Human 82.50 90.0 68.0 71.0 66.0 90.3 88.0 84.0 87.1 98.0
BERT 39.60 61.9 25.6 40.5 22.6 33.6 54.1 50.5 15.0 50.3
RoBERTa 42.80 63.2 35.7 49.3 32.8 33.5 55.5 50.0 15.7 49.6
PET 57.37 87.2 56.9 53.7 35.1 43.9 64.0 55.0 61.3 59.2
ADAPET 50.90 89.0 43.3 54.8 36.3 37.0 69.7 52.1 22.2 53.9
P-tuning 59.91 88.3 56.0 54.2 57.6 41.9 60.9 62.9 59.3 58.1
LM-BFF 55.80 84.6 53.6 56.3 46.1 43.1 54.1 51.2 61.3 51.8
EFL 56.54 85.6 46.7 53.5 44.0 67.5 67.6 61.6 28.2 54.2
GPT-Zero 43.40 57.5 26.2 37.0 19.0 34.4 50.0 50.1 65.6 50.3
PET-Zero 45.10 85.2 12.6 26.1 26.6 40.3 50.6 52.2 57.6 54.7
NSP-BERT 55.96 86.9 47.6 51.0 41.6 37.4* 63.4* 64.4* 52.0 59.4*
Table 1: Main results on FewCLUE benchmark. We report the accuracy on all 9 tasks and calculate the average accuracy as the score of FewCLUE benchmark. The asterisk * indicates the using of samples-contrast answer mapping method.


Standard fine-tuning of the pre-trained language model on the FewCLUE training set. The models are fine-tuned with cross entropy loss and using the BERT-style model’s hidden vector of [CLS] with a classification layer , where , is the number of labels.


In few-shot scenario, we choose token-level model PET schick2021it; schick2021exploiting and its opitmized models ADAPET tam2021improving, P-tuning liu2021gptunderstands and LM-BFFgao2021making. We also choose sentence-level model EFL wang2021entailment. All few-shot models are trained on FewCLUE’s training set.


In zero-shot scenario, there are two ways to realize, one is GPT-Zero using L2R LM radford2018improving; radford2019language; brown2020language, the other is PET-Zero using MLM schick2021it; schick2021exploiting.

4.3 Experiment Settings

For the baselines, we follow the settings in FewCLUE. The BERT model is using RoBERTa-wwm-ext cui2019pre; cui2020revisiting 333https://github.com/ymcui/Chinese-BERT-wwm , a Chinese RoBERTa-base with whole-word-mask, which is expected to have better performance on cloze-style tasks. The GPT model is NEZHA-Gen wei2019nezha 444https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA-Gen-TensorFlow.

Because of the need to utilize the model pre-trained by the NSP task, none of the RoBERTa models are suitable for our NSP-BERT. So we adopt the vanilla BERT trained by UER using MLM and NSP zhao2019uer 555https://github.com/dbiir/UER-py . The pre-training corpus is a large mixed corpus in Chinese. Along with the basic model (=12, =768, =12, Total Parameters=110M), we conduct experiments using UER-BERTs of various scales (tiny, small, and big) to validate the effect of varied scale models on NSP-BERT. Meanwhile, we use models trained by other organizations (Google666https://github.com/google-research/bert and HFLfootnote 3), to evaluate the robustness of our optimization methods.

4.4 Main Results

The table 1 reports the main results on FewCLUE. Our NSP-BERT model outperformed all other zero-shot learning methods on 7 out of 9 datasets. Its performance is comparable to the best few-shot methods currently available.

When using the same size model, it outperforms GPT-Zero (based on L2R LM) and PET-Zero (based on MLM) significantly on the single sentence classification tasks (CSLDCP, TNEWS and IFLTEK). It demonstrates NSP’s remarkable ability to distinguish across sentence topics. Nonetheless, as discussed in the previous section, the sentence-level prompt-learning methods have a number of drawbacks when used with cloze-style tasks, and NSP-BERT is no exception. This demonstrates that we have a gap in ChID when compared to token-level methods. The following (§ 4.5) section attempts to narrow the gap using soft-position methods.

Figure 7: The performance of the samples-contrast answer mapping method under different preconditions on OCNLI, BUSTM, CSL and CLUEWSC. Batch size , when the batch size is ( and for OCNLI), the result is a random guess, when the batch size is , indicating that the entire test set is obtained at one time. Thresholds means that the thresholds are obtained through the dev set, and then used for the prediction of the test set.

4.5 Optimizations and Analysis

Soft-Position for Cloze-Style Task

As seen in Table 2, while the soft-position approach clearly has an effect on ChID, it exhibits some instability. This could be because the model was not fine-tuned sufficiently on the training set.

ORG Models Hard-Position Soft-Position
(Dev / Test) (Dev / Test)
Googlefootnote 6 BERT-Chinese 46.04 / 48.15 41.09 / 48.65↑
HFLfootnote 3 BERT-wwm 46.53 / 47.56 45.05 / 46.05↓
BERT-wwm-ext 51.49 / 52.90 53.47 / 54.55↑
UERfootnote 5 BERT-mixed 47.52 / 50.85 53.96 / 52.00↑
Table 2: Accuracy of NSP-BERT with soft-position compared with hard-position on cloze-style task ChID. The models we adopted are from different organizations.

Two-Stage Prompt

In §3.2, we introduced a two-stage prompt method for word sense disambiguation tasks. We compare its effect with a one-stage prompt on dataset DuEL2.0. We divide the dataset into two parts, entity linking and entity typing. In the entity typing part, we do not remove the Other type. Our model has satisfactory performance on DuEL2.0 without relying on any training data, especially for entity linking, NSP-BERT can handle entity descriptions of different lengths well, which is something that models such as PET can hardly achieve.

ORG Models Entity Linking Entity Typing
One-S Two-S One-S Two-S
Googlefootnote 6 BERT-Chinese 60.77 66.99↑ 24.08 31.18↑
HFLfootnote 3 BERT-wwm 57.86 66.64↑ 23.99 28.64↑
BERT-wwm-ext 59.03 66.82↑ 24.25 31.71↑
UERfootnote 5 BERT-mixed 61.16 69.66↑ 31.35 40.04↑
Baselines GPT-Zero / / 28.48
PET-Zero / / 40.46
Table 3: Results (Acc.) of NSP-BERT on DuEL2.0 with one-stage prompt (One-S) and two-stage prompt (Two-S). We do not remove the type Other in entity typing.
Figure 8: Sketch of accuracy for different scales of models. X-axis represents the tasks in FewCLUE and the y-axis represents the baselines (GPT-Zero and PET-Zero) and NSP-BERT at different model scales (tiny, small, base and large).

Size of Models

Shown in Figure 8, we compared the impact of the models’ scale on FewCLUE. The average accuracy of tiny, small, base and large BERT models are 47.35, 49.69, 56.95 and 57.0 respectively, when the baselines GPT-Zero and PER-Zero are 43.40 and 45.10.

Figure 9: When prompts become more fluent and logical, the accuracy of NSP-BERT improves.

Influence of Prompt’s Logic and Fluency

The biggest difference between NSP-BERT and contrast learning is that the prompts in NSP-BERT need to be close to natural language habits. As shown in Figure 9, based on the 3 prompt templates, according to the logic, , the accuracy increased significantly, on 4 datasets (EPRSTMT, TNEWS, CSLDCP and IFLYTEK).

5 Conclusion

In this paper, we introduce NSP-BERT, which uses an unexpected pre-training task called Next Sentence Prediction (NSP) of BERT to perform various NLP tasks using prompts. As a sentence-level prompt-learning method, NSP-BERT not only can achieve SOTA results on multiple tasks, but it also has an impressive improvement over prior zero-shot methods (GPT and PET). NSP-BERT can accomplish non-fixed length tasks that are difficult to be solved by token-level methods, such as entity linking tasks with variable-length entity descriptions. Although it is based on sentence-level, our method is different from the traditional sentence embedding similarity contrast learning since it must be prompted by natural language. The model’s performance improves as the prompt becomes more logical and fluent. Our NSP-BERT is inspiring for prompt-based learning owing to our experiments show that a simple pre-training task can efficiently solve various downstream tasks without any task-specific training data. In future work, it is essential to extend NSP-BERT to the few-shot scenario.


Appendix A Models

a.1 Parameters of Models

Model Total Parameters
GPT 12 768 12 102M
RoBERTa 12 768 12 102M
BERT-Tiny 3 384 6 14M
BERT-Small 6 512 8 31M
BERT-Base 12 768 12 102M
BERT-Large 24 1024 16 327M
Table 4: The parameters of different models used in our experiment. Denote the number of layers as , the hidden size as , and the number of self-attention heads as .

a.2 Probability Formula


Denote the token in position as , the original text as , the prompt as , the label which will be predicted as , and it will be replaced by . When ignoring special tokens such as and , the input of PET-Zero is:


The output probability for label is:



For Left-2-Right language model, the prompt is , and tokens will input one by one, when the current token of prompt is , the condition input is :


The output probability for label is:



For our NSP-BERT, the prompt will be inputed at once:


The output probability for label is:


Appendix B More Details

ORG Models Template 1 Template 2 Template 3
(Dev/Test) (Dev/Test) (Dev/Test)
Goo. BERT-Chinese 70.63/72.30 75.63/79.84 76.88/83.11
HFL BERT-wwm 68.13/69.34 72.50/81.48 76.25/81.97
BERT-wwm-ext 53.75/51.80 75.00/81.31 81.88/83.61
UER BERT-Tiny 68.13/76.56 75.00/80.82 81.88/80.33
BERT-Small 85.00/87.70 82.50/87.70 87.50/86.72
BERT-Base 60.00/54.59 78.75/80.98 88.13/86.89
BERT-Large 78.13/82.79 83.75/82.62 84.38/84.43
Table 5: Accuracy on EPRSTMT.
ORG Models Template 1 Template 2 Template 3
(Dev/Test) (Dev/Test) (Dev/Test)
Goo. BERT-Chinese 45.00/43.18 48.91/51.39 51.73/52.38
HFL BERT-wwm 44.63/41.79 51.00/50.75 49.09/50.05
BERT-wwm-ext 45.72/41.14 52.09/50.90 52.10/51.94
UER BERT-Tiny 38.80/36.62 39.25/36.37 41.07/38.56
BERT-Small 38.98/38.81 39.80/40.35 41.80/42.19
BERT-Base 41.26/41.84 46.99/48.66 50.64/51.00
BERT-Large 45.17/42.79 48.72/48.31 54.28/53.83
Table 6: Accuracy on TNEWS.
ORG Models Template 1 Template 2 Template 3
(Dev/Test) (Dev/Test) (Dev/Test)
Goo. BERT-Chinese 40.03/40.36 43.96/45.12 43.96/46.02
HFL BERT-wwm 42.89/45.07 44.92/46.52 45.60/47.31
BERT-wwm-ext 38.10/39.18 40.18/42.32 41.30/42.21
UER BERT-Tiny 24.03/25.73 27.37/29.60 25.68/28.81
BERT-Small 28.48/30.72 29.35/31.45 29.78/31.78
BERT-Base 39.80/40.53 44.87/45.80 45.26/47.59
BERT-Large 44.73/42.83 44.00/44.34 45.89/46.92
Table 7: Accuracy on CSLDCP.
ORG Models Template 1 Template 2 Template 3
(Dev/Test) (Dev/Test) (Dev/Test)
Goo. BERT-Chinese 31.97/31.33 39.18/34.53 41.59/37.56
HFL BERT-wwm 31.25/29.96 38.02/34.19 40.64/37.05
BERT-wwm-ext 29.86/28.30 36.20/33.16 39.83/35.05
UER BERT-Tiny 32.70/32.65 31.97/34.13 33.65/34.59
BERT-Small 32.27/32.42 35.54/34.65 35.25/34.76
BERT-Base 36.41/36.59 42.39/40.19 43.12/41.62
BERT-Large 37.73/36.94 44.28/42.60 44.87/42.42
Table 8: Accuracy on IFLYTEK.
Entity Linking Ave. Entities Entity Tpying Types
26586 5.37 6465 24
Table 9: Since the DuEL2.0’s test set is not public, we use the dev set to test our model. The the number of the original text lines is 10000. According to the predicted target (entities in knowledge base or upper types), we manually divide it into two parts, entity linking and entity typing.
Dataset Dev Test
1 2 4 8 16 32 64 128 All Threshold
OCNLI 37.50 33.33 33.33 35.75 36.51 36.90 37.26 37.50 36.83 36.90 37.38
BUSTM 62.50 50.00 56.09 67.79 59.59 59.93 61.06 61.40 61.85 63.43 63.43
CSL 64.38 50.00 58.91 62.09 62.79 62.86 62.79 63.07 63.00 63.85 64.41
CLUEWSC 57.23 50.00 53.69 54.30 54.51 54.71 55.53 56.56 56.56 58.61 59.43
Table 10: The performance of the samples-contrast answer mapping method under different preconditions on OCNLI, BUSTM, CSL and CLUEWSC. Batch size , when the batch size is ( and for OCNLI), the result is a random guess, when the batch size is , indicating that the entire test set is obtained at one time. Thresholds means that the thresholds are obtained through the dev set, and then used for the prediction of the test set.
Corpus Train Dev Test Pub Labels Task Metrics Source
Single Sentence Tasks
EPRSTMT 32 32 610 2 Sentiment Analysis Acc. E-commerce Review
TNEWS 240 240 2010 15 Short Text Classification Acc. News Title
CSLDCP 536 2068 1784 67 Long Text Classification Acc. Academic CNKI
IFLYTEK 928 690 1749 119 Long Text Classification Acc. App Description
Sentence Pair Tasks
OCNLI 32 32 2520 3 Natural Language Inference Acc. 5 genres
BUSTM 32 32 1772 2 Short Text Matching Acc. AI Virtual Assistant
CSL 32 32 2828 2 Keyword Recognition Acc. Academic CNKI
ChID 42 42 2002 7 Chinese Idiom Cloze Test Acc. Novel, Essay News
CLUEWSC 32 32 976 2 Coreference Resolution Acc. Chinese Fiction Books
Table 11: Task descriptions and statistics of FewCLUE, we omit the unlabeled dataset because it is not used. Test Pub indicates the number of samples in the public test set. The 5 text genres of OCNLI are government documents, news, literature, TV talk shows and telephone conversations.
Strategies Single Sentence Task Sentence Pair Task Others
Prompt Prefix
Answer C-C
Mapping S-C
Table 12: Strategies adopted on the 10 datasets in our experiment. The prefix means to put the prompt in front of the original text, and the suffix is the opposite. C-C means candidates-contrast answer mapping method, and S-C means samples-contrast answer mapping method.
Task Prompt Label Names
EPRSTMT Template 1: The screen stopped working. [SEP] [label].
Template 2: The screen stopped working. [SEP] I am [label].
Template 3: The screen stopped working. [SEP] I am very [label] about this shopping.
2 labels:
Positive (Happy); Negative (Sad)
TNEWS Template 1: La Liga: Atletico Madrid VS Espanyol. [SEP] [label].
Template 2: La Liga: Atletico Madrid VS Espanyol. [SEP] [label] news.
Template 3: La Liga: Atletico Madrid VS Espanyol. [SEP] This is a piece of [label] news.
15 labels:
Education; Finance; House; Travel; Technology; Sports; Game; Culture; Car; Story; Entertainment; Military; Agriculture; World; Stock.
CSLDCP Template 1: Grove Mountains (GRV) 020043 is a special chondrite…. [SEP] [label].
Template 2: Grove Mountains (GRV) 020043 is a special chondrite…. [SEP] [label] paper.
Template 3: Grove Mountains (GRV) 020043 is a special chondrite…. [SEP] This is a paper about [label].
67 labels:
Materials Science and Engineering; Crop Science; Stomatology; Pharmacy; Pedagogy; Water Conserv-ancy Engineering; Theoretical Economics; Food Sci-ence and Engineering; Animal Science/Veterinary Science ; …
IFLYTEK Template 1: GooglePlay is Google’s official application market… [SEP] [label].
Template 2: GooglePlay is Google’s official application market… [SEP] [label] app.
Template 3: GooglePlay is Google’s official application market… [SEP] It’s a [label] app.
119 labels:
Taxi; Map Navigation; Free WIFI; Car Rental; Same City Service; Express Logistics; Wedding; House-keeping; Public Transportation; Government Affairs; Community Services; Fleece; Magic; Xian Xia; Card; Flying Air Combat; Shooting Game; Leisure Puz; …
OCNLI The two people came back from Japan the day before yesterday. [SEP] The two of them stayed in Japan for a week. 3 labels:
Contradiction; Neutral; Entailment.
BUSTM Sing me a song. [SEP] Play a song for us. 2 labels:
Matched; Unmatched.
ChID This means that in the near future, HJT heterojunction cells may usher in an explosion, and photovoltaic cells may also usher in a [BLANK] opportunity period from PERC to HJT. [SEP] historically revolutionary. 7 candidates (Each sample has different candidates):
stand ready; historically revolutionary; absolutely irreconcilable; far away; return to the original owner; waves and clouds; strut.
CLUEWSC The phone on the desk rang. It hung up after two minutes. It means [SEP] phone. 2 labels:
True; False.
Franklin drafted the Declaration of Independence. Franklin is [SEP] he is the founding Fathers of the United States… 5.37 entities per sample:
Entity 1: The founding Fathers of the United States. American politician, physicist and social activist.
Entity 2: American female swimmer, good at short backstroke and freestyle, nicknamed ”female flying fish”.
Entity 3: British captain and Arctic explorer, served on the Bellerophon in the early years and participated in the Battle of Trafalgar.
Franklin drafted the Declaration of Independence. Franklin is [SEP] he is a person… 24 types:
Event; Person; Work; Location; Time and Calendar; Brand; Natural and Geography; Game; Biological; Medicine; Food; Software; Vehicle; Website; Disease and Symptom; Organization; Awards; Education; Culture; Constellation; Law and Regulation; Virtual-Things; Diagnosis and Treatment; Other.
Table 13: The prompts used for all tasks. Since there are two options for the prompt, prefix and suffix, we select the most suitable one through the development set. The original datasets are all in Chinese, in order to facilitate understanding, we have performed a certain conversion. Especially for the ChID dataset, since idioms are a relatively specific linguistic phenomenon in Chinese, most idioms are composed of 4 tokens, so we only use the general cloze-sytle task to show its Prompt. For dataset with a lot of labels, due to space considerations, we have omitted some of them. The underlined part is the prompt template, otherwise it is the original text.