Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension

04/21/2019 ∙ by Kai Sun, et al. ∙ Tencent cornell university 0

With an ultimate goal of narrowing the gap between human and machine readers in text comprehension, we present the first collection of Challenging Chinese machine reading Comprehension datasets (C^3) collected from language and professional certification exams, which contains 13,924 documents and their associated 23,990 multiple-choice questions. Most of the questions in C^3 cannot be answered merely by surface-form matching against the given text. As a pilot study, we closely analyze the prior knowledge (i.e., linguistic, domain-specific, and general world knowledge) needed in these real world reading comprehension tasks. We further explore how to leverage linguistic knowledge including a lexicon of common idioms and proverbs and domain-specific knowledge such as textbooks to aid machine readers, through fine-tuning a pre-trained language model (Devlin et al.,2019). Our experimental results demonstrate that linguistic knowledge may help improve the performance of the baseline reader in both general and domain-specific tasks. C^3 will be available at



There are no comments yet.


page 1

page 2

page 3

page 4

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Machine reading comprehension (MRC) tasks, which aim to teach machine readers to read and understand a reference material (e.g., a document), and evaluate the comprehension ability of machines by letting them answer questions relevant to the given content Poon et al. (2010); Richardson et al. (2013), have attracted substantial attention of both academia and industry.

An increasing number of studies focus on developing MRC datasets that contain a significant number of questions that require prior knowledge in addition to the given context  Richardson et al. (2013); Mostafazadeh et al. (2016); Lai et al. (2017); Ostermann et al. (2018); Khashabi et al. (2018); Talmor et al. (2018); Sun et al. (2019a). Therefore, they can serve as good test-beds for evaluating progress towards goals of teaching machine readers to use different kinds of prior knowledge for better text comprehension and narrowing the performance gap between human and machine readers in real-world settings such as language or subject examinations. However, progress on these kind of tasks is mostly limited to English Storks et al. (2019) because of the unavailability of large-scale datasets in other languages.

To study the prior knowledge needed to better comprehend written and oral texts in Chinese, we propose the first collection of Challenging Chinese multiple-choice machine reading Comprehension datasets (C) that contain both general and domain-specific tasks. For the general-domain task: Given a reference document that can be either written or oral (i.e., a dialogue), select the correct answer option from all options associated with a question. Besides, we present a challenging task that has not been explored in the literature: Given a counseling oral text (a third-person narrative or a dialogue) mostly about life concerns and an additional domain-specific reference corpus, select the correct answer option(s) from associated options of a question. Compared to relevant datasets Ostermann et al. (2018); Sun et al. (2019a), besides the oral text, we also need additional domain-specific knowledge for answering questions. However, it is relatively difficult to link the content in less formal oral language to the corresponding well-written facts, explanations, or definitions in the domain-specific reference corpus. For all the mentioned tasks, we collect questions from language and professional certification exams designed by experts (Section 3.1).

We observe three kinds of prior knowledge are required for in-depth understanding of the written and oral texts to answer most of the questions in both general and domain-specific reading comprehension tasks: linguistic knowledge, domain knowledge, and general world knowledge that is further broken down into eight types such as arithmetic, connotative, and cause-effect (Section 3.2). Around of general-domain questions and all the domain-specific questions require knowledge beyond the given context. We further investigate the utilization of linguistic knowledge including a lexicon of common Chinese idioms and proverbs (Section 4.2), general world knowledge in the form of graphs Speer et al. (2017) (Section 4.3), and domain-specific knowledge such as textbooks to improve the comprehension ability of machine readers, via fine-tuning a pre-trained language model Devlin et al. (2019) (Section 4.1

). Experimental results show that general-domain lexicons and general world knowledge graph generally improve the baseline performance on both general and domain-specific tasks. Experiments also demonstrate that for domain-specific questions, typical methods such as enriching the given text with retrieved sentences from additional domain-specific corpora actually hurt the performance of the baseline, which indicates the great challenge in finding external knowledge relevant to informal oral texts (Section 

5.2). We hope our observations and proposed challenging datasets may inspire further research on knowledge acquisition and utilization for Chinese or cross-language reading comprehension.

2 Related Work

Task Reference Document Reference Corpus Domain Language
Chinese English
C-1A mixed-genre general N/A RACE Lai et al. (2017)
C-1B dialogue general N/A DREAM Sun et al. (2019a)
C-2A narrative psychological N/A N/A
C-2B dialogue psychological N/A N/A
Table 1: Comparison of C and existing representative large-scale multiple-choice reading comprehension tasks.

2.1 Machine Reading Comprehension and Question Answering

We discuss tasks in which texts are written in English. Much of the early work focuses on constructing large-scale extractive MRC datasets: Answers are spans from the reference document Hermann et al. (2015); Hill et al. (2016); Bajgar et al. (2016); Rajpurkar et al. (2016); Trischler et al. (2017); Joshi et al. (2017). As a question and its answer are usually in the same sentence, deep neural models Devlin et al. (2019); Radford et al. (2019) have outperformed human performance on many such tasks. To increase the difficulty of MRC tasks, researchers have explored ways including adding unanswerable Trischler et al. (2017); Rajpurkar et al. (2018) or conversational Reddy et al. (2018); Choi et al. (2018) questions that might require reasoning Zhang et al. (2018a), designing free-form answers Kočiskỳ et al. (2018) or (question, answer) pairs that cover the content of multiple sentences or documents Welbl et al. (2018); Yang et al. (2018). Still, questions usually provide sufficient information to find answers in the given context.

There are also a variety of non-extractive machine reading comprehension Richardson et al. (2013); Mostafazadeh et al. (2016); Lai et al. (2017); Khashabi et al. (2018); Talmor et al. (2018); Sun et al. (2019a) and question answering tasks Clark et al. (2016, 2018); Mihaylov et al. (2018), mostly in multiple-choice forms. For question answering tasks, there is no reference document provided for each question. Instead, a reference corpus is provided, which contains a collection of domain-specific textbooks or/and related encyclopedia articles. For these tasks, besides the given reference documents or corpora, knowledge from other resources may be necessary to solve a significant percentage of questions.

Besides these standard tasks, we are aware that there is a trend of formalizing tasks such as relation extraction Levy et al. (2017), word prediction Chu et al. (2017), and judgment prediction Long et al. (2018) as extractive or non-extractive machine reading comprehension problems, which are beyond the scope of this paper.

We compare our proposed tasks with similar datasets in Table 1. C-2A and C-2B can be regarded as new challenging tasks that have never been studied before.

2.2 Chinese Machine Reading Comprehension and Non-Extractive Question Answering

We have seen the construction of span-style Chinese machine reading comprehension datasets Cui et al. (2016, 2018b, 2018a); Shao et al. (2018), using Chinese news reports, books, and Wikipedia articles as source documents, similar to their English counterparts CNN/Daily Mail Hermann et al. (2015), CBT/BT Hill et al. (2016); Bajgar et al. (2016), and SQuAD Rajpurkar et al. (2016), in which all answers are extractive spans from the provided reference documents.

Previous work also focus on non-extractive question answering tasks Cheng et al. (2016); Guo et al. (2017a, b); Zhang and Zhao (2018); Zhang et al. (2018b); Hao et al. (2019), in which questions are usually collected from examinations. Another kind of non-extractive question answering tasks He et al. (2017) is based on search engines (similar to English MS MARCO Nguyen et al. (2016)): Researchers collect questions from query logs and ask crowdsourcers to generate answers.

Compared to the tasks mentioned above, we focus on Chinese machine reading comprehension tasks that require prior knowledge to facilitate the understanding of the given text.

3 Data

In this section, we describe how we construct C (Section 3.1), and we analyze the data (Section 3.2) and knowledge needed (Section 3.3).

3.1 Collection Methodology and Task Definitions

1928年,经徐志摩介绍,时任中国公学校长的胡适聘用了沈从文做讲师,主讲大学一年级的现代文学选修课。 In 1928, recommended by Hsu Chih-Mo (1897-1931), Hu Shih (1891-1962), who was the president of the previous National University of China, employed Shen Ts’ung-wen (1902-1988) as a lecturer of the university who was in charge of teaching the optional course of modern literature.
当时,沈从文已经在文坛上崭露头角,在社会上也小有名气,因此还未到上课时间,教室里就坐满了学生。上课时间到了,沈从文走进教室,看见下面黑压压一片,心里陡然一惊,脑子里变得一片空白,连准备了无数遍的第一句话都堵在嗓子里说不出来了。 At that time, Shen already made himself conspicuous in the literary world and was a little famous in society. For this sake, even before the beginning of class, the classroom was crowded with students. Upon the arrival of class, Shen went into the classroom. Seeing a dense crowd of students sitting beneath the platform, Shen was suddenly startled and his mind went blank. He was even unable to utter the first sentence he had rehearsed repeatedly.
他呆呆地站在那里,面色尴尬至极,双手拧来拧去无处可放。上课前他自以为成竹在胸,所以就没带教案和教材。整整 10 分钟,教室里鸦雀无声,所有的学生都好奇地等着这位新来的老师开口。沈从文深吸了一口气, 慢慢平静了下来,原先准备好的东西也重新在脑子里聚拢,然后他开始讲课了。不过由于他依然很紧张,原本预计一小时的授课内容,竟然用了不到 15 分钟就讲完了。 He stood there motionlessly, extremely embarrassed. He wrung his hands without knowing where to put them. Before class, he believed that he had had a ready plan to meet the situation so he did not bring his teaching plan and textbook. For up to 10 minutes, the classroom was in perfect silence. All the students were curiously waiting for the new teacher to open his mouth. Breathing deeply, he gradually calmed down. Thereupon, the materials he had previously prepared gathered in his mind for the second time. Then he began his lecture. Nevertheless, since he was still nervous, it took him less than 15 minutes to finish the teaching contents he had planned to complete in an hour.
接下来怎么办?他再次陷入了窘境。无奈之下,他只好拿起粉笔在黑板上写道:我第一次上课,见你们人多,怕了。 What should he do next? He was again caught in embarrassment. He had no choice but to pick up a piece of chalk before writing several words on the blackboard: This is the first time I have given a lecture. In the presence of a crowd of people, I feel terrified.
顿时,教室里爆发出了一阵善意的笑声,随即一阵鼓励的掌声响起。得知这件事之后,胡适对沈从文大加赞赏,认为他非常成功。 Immediately, a peal of friendly laughter filled the classroom. Presently, a round of encouraging applause was given to him. Hearing this episode, Hu heaped praise upon Shen, thinking that he was very successful.
有了这次经历,在以后的课堂上,沈从文都会告诫自己不要紧张,渐渐地,他开始在课堂上变得从容起来。 Because of this experience, Shen always reminded himself of not being nervous in his class for years afterwards. Gradually, he began to give his lecture at leisure in class.
Q1   第2段中, “黑压压一片”指的是: Q1   In paragraph 2, “a dense crowd” refers to
A.    教室很暗 A.    the light in the classroom was dim.
B.    听课的人多 B.    the number of students attending his lecture was large.
C.    房间里很吵 C.    the room was noisy.
D.    学生们发言很积极 D.    the students were active in voicing their opinions.
Q2   沈从文没拿教材,是因为他觉得: Q2   Shen did not bring the textbook because he felt that
A.    讲课内容不多 A.    the teaching contents were not many.
B.    自己准备得很充分 B.    his preparation was sufficient.
C.    这样可以减轻压力 C.    his mental pressure could be reduced in this way.
D.    教材会限制自己的发挥 D.    the textbook was likely to restrict his ability to give a lecture.
Q3   看见沈从文写的那句话,学生们: Q3   Seeing the sentence written by Shen, the students
A.    急忙安慰他 A.    hurriedly consoled him.
B.    在心里埋怨他 B.    blamed him in mind.
C.    受到了极大的鼓舞 C.    were greatly encouraged.
D.    表示理解并鼓励了他 D.    expressed their understanding and encouraged him.
Q4   上文主要谈的是: Q4   The passage above is mainly about
A.    中国教育制度的发展 A.    the development of the Chinese educational system.
B.    紧张时应如何调整自己 B.    how to make self-adjustment if one is nervous.
C.    沈从文第一次讲课时的情景 C.    the situation where Shen gave his lecture for the first time.
D.    沈从文如何从作家转变为教师的 D.    how Shen turned into a teacher from a writer.
Table 2: A sample C-1A problem (left) and its English translation (right) (: the correct answer option).
F: How is it going? Have you bought your ticket?
M: There are so many people at the railway station. I have waited in line all day long. However, when my turn comes, they say that there is no ticket left unless the Spring Festival is over.
F: It doesn’t matter. It is all the same for you to come back after the Spring Festival is over.
M: But according to our company’s regulation, I must go to the office on the 6th day of the first lunar month. I’m afraid I have no time to go back after the Spring Festival, so could you and my dad come to Shanghai for the coming Spring Festival?
F: I am too old to endure the travel.
M: It is not difficult at all. After I help you buy the tickets, you can come here directly.
Q1   What is the relationship between the speakers?
A.    father and daughter
B.    mother and son
C.    classmates
D.    colleagues
Q2   What difficulty has the male met?
A.    his company does not have a vacation.
B.    things are expensive during the Spring Festival.
C.    he has not bought his ticket.
D.    he cannot find the railway station.
Q3   What suggestion does the male put forth?
A.    he invites the female to come to Shanghai.
B.    he is going to wait in line the next day.
C.    he wants to go to the company as soon as possible.
D.    he is going to go home after the Spring Festival is over.
Table 3: English translation of a sample problem from C-1B (: the correct answer option). We show the original problem in Chinese in Appendix A (Table 8).
Metric C-1A C-1B C-2A C-2B
Min./Avg./Max. # of options per question 2 / 3.7 / 4 3 / 3.8 / 4 4 / 4 / 4 4 / 4 / 4
Min./Avg./Max. # of correct options per question 1 / 1 / 1 1 / 1 / 1 1 / 1.9 / 4 1 / 1.8 / 4
Min./Avg./Max. # of questions per reference document 1 / 1.9 / 6 1 / 1.2 / 6 2 / 10.0 / 20 1 / 6.4 / 22
Avg./Max. option length (in characters) 6.5 / 45 4.4 / 31 5.6 / 39 6.5 / 36
Avg./Max. question length (in characters) 13.5 / 57 10.9 / 34 22.5 / 97 26.0 / 91
Avg./Max. reference document length (in characters) 180.2 / 1,274 76.3 / 1,540 395.9 / 995 440.1 / 1,651
character vocabulary size 4,120 2,922 2,093 2,075
non-extractive correct option (%) 81.9 78.9 91.4 95.2
(sub-)documents that contain proverbs or idioms (%) 25.4 7.8 70.9 33.6
# of (sub-)documents / # of questions
  Training 3,138 / 6,013 4,885 / 5,856 143 / 1,414 225 / 1,216
  Development 1,046 / 1,991 1,628 / 1,825 45 / 469 41 / 406
  Testing 1,045 / 2,002 1,627 / 1,890 49 / 492 52 / 416
  All 5,229 / 10,006 8,140 / 9,571 237 / 2,375 318 / 2,038
Table 4: The overall statistics of C. For C-2A and C-2B, the reference documents refer to the sub-documents, with the exception that we regard an option that does not appear in the full reference document as non-extractive.

We collect general-domain problems from Hanyu Shuiping Kaoshi (HSK) and Minzu Hanyu Kaoshi (MHK), which are designed to evaluate the Chinese listening and reading comprehension ability of second-language learners such as international students, overseas Chinese, and ethnic minorities. We collect domain-specific problems from Psychological Counseling Examinations (a national qualification test that certifies level two and level three psychological counselors in China), which focus on accessing the acquisition and retention of subject knowledge. We include problems from both real and practice exams, and all of them are freely accessible online for public usage.

Each general-domain problem consists of a reference document and a series of questions. Each question is associated with several answer options, exactly one of which is correct. The goal is to select the correct option. According to the reference document type, we divide the collected general-domain problems into two sub-tasks: C-1A and C-1B. In C-1B, a dialogue serves as the reference document. The rest of the problems belong to the C-1A. We show a sample problem for each type in Table 2 and Table 3, respectively.

Each domain-specific problem comprises a reference document mostly about life concerns (e.g., social, work, family, school, and emotional or physical health), which contains one or multiple sub-documents. Every sub-document is followed by a series of questions designed mainly for this sub-document. Each question is associated with several answer options, at least one of which is correct. The goal is to select all correct answer options. An answerer is allowed to read the complete reference document and utilize the relevant knowledge in an additional reference corpus such as a psychological counseling textbook (Section 5.2) to reach the correct answer options. Similarly, domain-specific problems are divided into sub-tasks C-2A and C-2B. For each problem in C-2B, its reference document is made up of one or multiple dialogues (in chronological order). C-2A contains the rest problems in which reference documents are third-person narratives. See a sample problem for each type in Appendix A (Table 12 and Table 13) due to limited space.

We remove duplicate problems and randomly split the data (13,924 documents and 23,990 questions in total) at the problem level, with training, development, and test.

3.2 Data Analysis

We summarize the overall statistics of C in Table 4. We observe some differences, which may be relevant to the difficulty level of questions, exist between general-domain (i.e., C-1A and C-1B) and domain-specific tasks (i.e., C-2A and C-2B). For example, the percentage of non-extractive correct answer options in domain-specific tasks (C-2A: ; C-2B: ) is much higher than that in general-domain Chinese (C-1A: ; C-1B: ) and English language exams (RACE Lai et al. (2017): ; DREAM Sun et al. (2019a): ). Besides, the average document/question length of C-2A and C-2B is much longer than that of C-1A and C

-1B. The differences are probably due to the fact that domain-specific exam designers (experts) assume that most of the participants are high-proficiency native readers who obtained at least a bachelor’s degree in psychology, education, or sociology, while C

-1A and C-1B are designed for those less proficient second-language learners.

Chinese idioms and proverbs, which are widely used in both written and oral language, play an essential role in Chinese learning and understanding because of their conciseness in forms and expressiveness in meaning Lewis et al. (1998); Yang and Xie (2013). We notice that a significant percentage of reference documents in C (especially C-2A in Table 4) contain at least one idiom or proverb. As the meaning of such an expression may not be predicted from the meanings of its constituent parts, we require culture-specific background knowledge Wong et al. (2010). For example, to answer Q in Table 2, we need to know that the bolded idiom “成竹在胸” means “has a ready plan to meet the situation” instead of its literal meaning “chest-have-fully developed-bamboo” derived from a story about a painter who has a complete image of the bamboo in mind before drawing it. Therefore, the frequent use of idioms and proverbs in C may impede comprehension of human readers as well as pose challenges for machine readers. We will introduce details about how we attempt to teach machine readers idioms and proverbs in Section 4.2.

Metric C-1A C-1B C-1 C-2A C-2B C-2
Matching 12.0 14.3 13.2 0.0 0.0 0.0
Prior knowledge 88.0 85.7 86.8 100.0 100.0 100.0
Linguistic 49.0 30.7 39.8 33.3 6.7 20.0
Domain-specific 0.7 1.0 0.8 91.7 98.3 95.0
General world 50.7 64.0 57.3 71.7 80.0 75.8
Arithmetic 3.0 4.7 3.8 0.0 0.0 0.0
Connotation 1.3 5.3 3.3 0.0 8.3 4.2
Cause-effect 14.0 6.7 10.3 6.7 1.7 4.2
Implication 17.7 20.3 19.0 6.7 10.0 8.3
Part-whole 5.0 5.0 5.0 0.0 0.0 0.0
Precondition 2.7 4.3 3.5 0.0 0.0 0.0
Scenario 9.6 24.3 17.0 61.7 68.3 65.0
Other 3.3 0.3 1.8 0.0 0.0 0.0
Single sentence 50.7 22.7 36.7 1.7 3.3 2.5
Multiple sentences 47.0 77.0 62.0 83.3 81.7 82.5
Independent 2.3 0.3 1.3 15.0 15.0 15.0
# of annotated questions 300 300 600 60 60 120
Table 5: Distribution (%) of types of required knowledge based on a subset of test and development sets of C.

3.3 Categories of Knowledge

Because there is no prior work discussing the required knowledge in Chinese machine reading comprehension, we carefully analyze a subset of questions randomly sampled from the development and test sets of C (Table 4) and arrive at the following three kinds of prior knowledge.

Linguistic: To answer a given question (e.g., Q in Table 2 and Q in Table 3), we require lexical/grammatical knowledge include but not limited to: idioms, proverbs, negation, antonymy, synonymy, and sentence structures.

Domain-Specific: This kind of world knowledge consists of, but not limited to, facts about domain-specific concepts, their definitions and properties, and relations among these concepts Grishman et al. (1983); Hansen (1994).

General World: It refers to the general knowledge about how world works, sometimes called commonsense knowledge. We focus on the sort of world knowledge that an encyclopedia would assume readers know without being told Lenat et al. (1985); Schubert (2002) instead of the factual knowledge such as properties of famous entities. We further break down general world knowledge into eight types, some of which (marked with ) are similar to the categories for recognizing textual entailment summarized by lobue2011types.

  • Arithmetic: This includes numerical computation and analysis (e.g., comparisons).

  • Connotation: This includes knowledge about implicit and implied sentiments Feng et al. (2013); Van Hee et al. (2018).

  • Cause-effect: The occurrence of a event A causes the occurrence of event B. See Q in Table 2 for an example.

  • Implication: This category indicates the implicit inference from the content explicitly described in the text, which cannot be reached by paraphrasing sentences using linguistic knowledge. For example, Q and Q in Table 2 belong to this category.

  • Part-whole: We require knowledge that object A is a part of object B. Relations such as member-of, stuff-of, and component-of between two objects also fall into this category Winston et al. (1987); Miller (1998).

  • Scenario: It includes knowledge about human behaviors or activities, which may involve corresponding time and location information. We also consider knowledge about the profession, education, personality, and mental or physical health of the involved participant as well as the relations among the participants, indicated by the behaviors or activities described in texts. For example, we put Q in Table 2 in this category as “friendly laughter” may express “understanding”.

  • Precondition: If had event A not happened, event B would not have happened Ikuta et al. (2014); O’Gorman et al. (2016).

  • Other: Knowledge that belongs to none of the above categories.

As shown in Table 5, compared to narrative-based (C-2A) or dialogue-based (C-1B and C-2B) tasks, we tend to require more linguistic knowledge and less general world knowledge to answer questions designed for well-written texts in C-1A. In C-2, not surprisingly, of questions require domain-specific knowledge, and we notice that a higher percentage () of questions require general world knowledge especially the scenario-based knowledge () compared to that in C-1. Besides, we require multiple sentences to answer most of the domain-specific questions, which also reflects the difficulty of these kind of tasks.

4 Approaches

4.1 Reading Comprehension Model

We follow the framework of discriminatively fine-tuning pre-trained language models on machine reading comprehension tasks Radford et al. (2018). We use the Chinese BERT-Base model (denoted as ) released by bert2018 as the pre-trained language model.

Given a reference document , a question , and an answer option , we construct the input sequence by concatenating a [CLS] token, tokens in , a [SEP] token, tokens in , a [SEP] token, tokens in , and a [SEP] token, where [CLS] and [SEP]

are the classifier token and sentence separator token in BERT, respectively. We add an embedding

A to every token before the first [SEP] token (inclusive) and a B embedding to every other token, where A and B are pre-trained segmentation embeddings in BERT. We denote the final hidden state for the first token in the input sequence as . For C-1A and C-1B, we introduce a classification layer and obtain the unnormalized log probability of being correct by . For C-2A and C-2B, we introduce a classification layer and obtain the probabilities of answer option being correct and incorrect by . We refer readers to bert2018 for more details.

4.2 Proverb and Idiom Knowledge

As mentioned in Section 3.2, Chinese proverbs and idioms are usually difficult to understand without enough background knowledge. Teachers generally believe that these expressions can be learned effectively via a proverb or idiom dictionary Lewis et al. (1998). Thus, we consider infusing the linguistic knowledge in a lexicon of proverbs and idioms into the baseline reader.

We propose to introduce an additional fine-tuning stage: Instead of directly fine-tuning on C, we first fine-tune on multiple-choice proverb and idiom problems that are automatically generated based on proverb and idiom dictionaries and then fine-tune the resulting model on C. Specifically, we generate two types of problems: (1) Given an explanation of a proverb/idiom, choose the corresponding proverb/idiom; (2) Given a proverb/idiom, select the corresponding explanation. To generate distractors (wrong answer options), we first sort all entries (i.e., proverbs and idioms) in alphabetical order and assume two entries are more likely to be closer in meaning if they share more characters. For type (1) problems, we treat the entry close to the correct entry as distractors. For type (2) problems, distractors are explanations of entries close to the given entry. See examples of generated problems in Table 9 in Appendix A. When fine-tuning on the generated problems, we regard the given explanation (for type (1) problems) or given entry (for type (2) problems) as the reference document and leave the question context empty.

4.3 General World Knowledge Graph

A graph of general world knowledge such as ConceptNet Speer et al. (2017) is useful to help us understand the meanings behind the words and therefore may bridge knowledge gaps between human and machine readers. For instance, relational triples under relation categories Causes and PartOf in ConceptNet may be helpful for us to solve questions in C, which fall into cause-effect and part-whole subcategories of the general world knowledge defined in Section 3.3.

We propose to introduce an additional fine-tuning stage to incorporate general world knowledge. We first fine-tune on multiple-choice problems, which are automatically generated based on ConceptNet. We then fine-tune the resulting model on C. Let denote a relational triple in ConceptNet: and are Chinese words or phrases; represents the relation type (e.g., Causes) between and . For each relation type , we introduce two special tokens [r] and [r] to represent and its reverse relation type, respectively. We convert each into two problems: (1) Given and [r], choose . (2) Given and [r], choose . Distractors are formed by randomly picked Chinese words or phrases in ConceptNet. See examples of generated problems in Table 10 in Appendix A. During the fine-tuning stage on the generated problems, we regard the given word or phrase as the reference document and the given relation type token as the question.

5 Experiment

Method C-1A C-1B C-2A C-2B Average
Dev Test Dev Test Dev Test Dev Test Dev Test
Random 27.8 27.8 26.4 26.6 6.7 6.7 6.7 6.7 16.9 17.0
63.0 62.6 62.3 62.1 36.7 26.2 34.7 31.3 49.2 45.6
Domain-specific Knowledge:
+ IR from Textbooks 35.2 26.0 32.5 30.0
+ Core Knowledge Problems 34.8 27.0 35.5 29.6
+ Textbook Pre-Training 36.2 29.7 33.7 28.1
Linguistic Knowledge:
+ Proverb and Idiom Look-Up 62.6 63.2 62.8 62.2 37.1 27.6 32.3 30.0 48.7 45.8
+ Proverb and Idiom Problems 63.6 63.5 63.9 64.0 38.8 30.9 38.4 32.2 51.2 47.7
General World Knowledge:
+ Graph-Structured Knowledge 63.8 65.0 63.9 64.5 37.7 28.9 34.7 32.0 50.0 47.6
Cross Genre:
64.8 65.1 65.4 64.3 37.1 28.0 36.7 33.7 51.0 47.8
Human Performance 96.0 93.3 98.0 98.7
Table 6: Performance of baseline and models absorbing linguistic knowledge and general world knowledge in accuracy (%) on the C dataset (: performance based on a subset of test and development sets of C).

5.1 Experimental Settings

We set the learning rate to , the batch size to , and the maximal sequence length to . We truncate the longest sequence among , , and (Section 4.1) when the input sequence length exceeds . The embeddings of relation type tokens are initialized randomly (Section 4.3). For C-2A and C-2B, we regard each sub-document as . When we introduce one additional fine-tuning stage before fine-tuning on the target C

task, we first fine-tune on the additional task for one epoch. For all experiments, we fine-tune on the target C

task(s) for eight epochs. We run every experiment five times with different random seeds and report the best development set performance and its corresponding test set performance.

C-1A | C-1B C-1A | C-1B
Matching 100.0 | 74.1 100.0 | 100.0
Prior knowledge 57.6 | 60.2 95.7 | 97.6
Single sentence 65.8 | 79.4 97.0 | 97.0
Multiple sentences 55.6 | 57.8 94.0 | 98.0
Table 7: Performance comparison in accuracy (%) based on the coarse-grained question categories.

5.2 Baseline Results and Discussion

We report the baseline performance in Table 6 and discuss the following aspects of our observations.

Domain-specific knowledge: For domain-specific questions in C-2, we explore three ways to introduce domain-specific knowledge.

First, we follow previous work Sun et al. (2019b) for English question answering tasks. Given a reference document , a question , and an answer option , we use Lucene McCandless et al. (2010) to retrieve top sentences from two psychological counselling textbooks by using the concatenation of and as a query. In comparison, we append the retrieved sentences to the ending of to form the new input sequence.

In another attempt, we collect 4,544 multiple-choice question answering problems111We will release them along with C. on core knowledge of psychological counseling from Psychological Counseling Examinations (the same source as problems in C-2A and C-2B). Each problem is composed of a question and four answer options, at least one of which is correct. See an example in Table 11 in Appendix A. We first fine-tune on the core knowledge problems and then fine-tune the resulting model on C-2A/C-2B. In the first stage, we leave the reference document context empty as no context is provided.

In the third method, we run pre-training steps on two psychological counselling textbooks starting from the checkpoint and use the resulting model as the new pre-trained model.

However, none of the above methods outperforms the baseline. It remains a challenge to leverage expert knowledge to improve the performance on C-2 for future investigation.

Gap between machine and human: We show human performance on the same subset of C-1 used for analysis of the required knowledge in Section 3.3. We do not report the human performance on C

-2 due to the wide variance in the human expertise with psychological counseling. We see a significant gap between the automated approach and human performance on C

-1, especially on questions that require prior knowledge or multiple sentences than questions that can be answered by surface matching or only involve content from a single sentence (Section 3.3).

5.3 Impact of Linguistic Knowledge and General World Knowledge

Linguistic Knowledge: We generate problems based on proverbs/idioms and their explanations. By introducing an additional fine-tuning stage on the generated proverb and idiom problems, we see consistent gain in accuracy over all C tasks compared to the baseline, with an absolute improvement of 2% in average accuracy (Table 6).

We also compare with an alternative approach of imparting proverb and idiom knowledge. For each reference document , we use the same lexicon as used in proverb and idiom problem generation to look up proverbs and idioms in for their explanations. Let and denote proverbs/idioms in and their corresponding explanations, respectively. We replace with the concatenation of and when constructing the input sequence. However, this approach does not yield promising results.

General World Knowledge: We generate 737,534 problems based on ConceptNet. By introducing an additional fine-tuning stage on the generated problems, we see gain in accuracy over most C tasks compared to the baseline (Table 6). We notice that C-1 benefits more than C-2 from this general world knowledge graph introduced by the proposed approach.

5.4 Other Attempt: Cross Genre Training

We also fine-tune on sub-tasks from similar domains but in different genres simultaneously, instead of fine-tuning it on each of the four C sub-tasks separately. We observe that trained on the combination of C-1A or C-1B consistently outperforms the same model trained solely on C-1A or C-1B. We have a similar observation on C-2.

6 Conclusion

We present the first collection of Challenging Chinese multiple-choice machine reading Comprehension datasets (C) collected from real-world exams, requiring linguistic, general or domain-specific knowledge to answer questions based on the given oral or written text. We study the prior knowledge needed in these challenging reading comprehension tasks and further explore how to utilize linguistic, general world, and domain-specific knowledge to improve the comprehension ability of machine readers through fine-tuning BERT. Experimental results show that linguistic and general world knowledge may help the reader baseline perform better in both general and domain-specific reading comprehension tasks.


Appendix A Appendices

: 怎么样?买到票了吗?
: 火车站好多人啊,我排了整整一天的队,等排到我了,他们说没票了,要等过了年才有。
: 没关系,过了年回来也是一样的。
: 公司初六就上班了,我怕过了年来不及。要不今年您和我爸来上海过年吧?
: 我这老胳膊老腿的不想折腾了。
: 一点儿不折腾,等我帮你们买好票,你们直接过来就行。
Q1   说话人是什么关系?
A.    父女
B.    母子
C.    同学
D.    同事
Q2   男的遇到了什么困难?
A.    公司不放假
B.    过年东西贵
C.    没买到车票
D.    找不到车站
Q3   男的提出了什么建议?
A.    让女的来上海
B.    明天再去排队
C.    早点儿去公司
D.    过了年再回家
Table 8: A sample problem from C-1B (: the correct answer option).
Sample Problem 1:
A.    一人向隅,满坐不乐
B.    一人善射,百夫决拾
C.    一人得道,鸡犬升天
D.    一人传虚,万人传实
Sample Problem 2:
A.    犹言龙腾虎卧。比喻笔势。
B.    比喻高举远逝。
C.    比喻文笔、书法纵逸雄劲。
D.    比喻超逸雄奇。
Table 9: Examples of generated Chinese proverb/idiom problems (: the correct answer option).
Sample Problem 1:
Q    [MotivatedByGoal]
A.    增广见识
B.    打开电视机
C.    刚搬家
D.    美好的生活
Sample Problem 2:
Q    [MotivatedByGoal]
A.    想偷懒的时候
B.    努力赚钱
C.    抢妹妹的食物
D.    龟起来
Table 10: Examples of generated problems based on general relational knowledge (: the correct answer option).
对求助者形成初步印象的工作程序包括( )。
A.    对求助者心理健康水平进行衡量
B.    对求助者心理问题的原因作解释
C.    对求助者的问题进行量化的评估
D.    对某些含混的临床表现作出鉴别
Table 11: An example of problems on core knowledge of psychological counseling (: the correct answer option).
General information: a female, at the age of 24, unmarried, a cashier.
The help seeker’s self-narration: In the past two years, I have always felt everything is dirty, especially money. For this sake, I wash my hands so frequently that the skin of them has peeled off. Even so, I do not feel at ease. I know that this is not good for me but I cannot help doing so.
Case Introduction: Ten years ago, the help seeker went to hospital to pay a visit to her classmate. After she went home, she ate an apple without washing her hands and was scolded by her parents after being noticed by them. They warned her that she would fall ill if she eats things without washing her hands. For this sake, she was worried and suffered from insomnia for two days. Since then, whenever she has come home from school, she has remembered to wash her hands earnestly. Little by little, this episode has gone by. Two years ago, she became a cashier dealing with money every day. She always believes that money is dirty for it is covered with numerous bacteria. So she washes her hands repeatedly after work. In spite of being clearly aware that her hands are quite clean, she is still unable to control her mentality and always afraid that “in case they might not washed clean”? Much time has been spent in her perplexity. She is so vexed that she has been unable to go to work recently. A month ago, she began to worry that she was likely to suffer from a mental disorder, which has made her dispirited and sleepless. As a result, she often suffers from a headache and frequently sees doctors. However, she is unwilling to take the medicine prescribed by doctors because of many side effects written in the instructions. Her parents think that she does not have mental illness and accompany her to seek for psychological counseling.
According to the parents of the help seeker, she is the only child in her family and her parents are strict with her. During her childhood, she was not permitted to go out to play by herself. Since her parents were busy, she was sent to her grandmother’s home to be taken care of. Her grandmother tightly protected her and did not allow her to play with other little friends. Every day, she must have meals and go to bed on time. Later, she was at school. She was very meticulous in her study and her academic results were excellent. Yet her classmate relation was ordinary. She listened to her parents and was careful about what she did. She was a little coward. After failing to pass the postgraduate entrance examination after her graduation from college, she once suffered from depression and insomnia. Fortunately, everything has gone well with her since she was employed. As a cashier, she has to deal with money every day and always thinks that money is dirty. Consequently, she has washed her hands more and more frequently. Recently, she has been so depressed that she is unable to work.
Q1   The main symptoms of the help seeker are ( )
A.    fear     B.    depression     C.    obsession      D.    compulsive behavior
Q2   Which of the following symptoms does not happen to the help seeker ( )
A.    palpitation     B.    insomnia      C.    headache     D.    nausea
Q3   The main behavioral symptoms of the help seeker are ( )
A.    repeated behavior     B.    repeated counseling     C.    repeated examination     D.    repeated weeping
Q4   Which of the following behavioral symptoms does not happen to the help seeker ( )
A.    worry     B.    depression     C.    frequent headaches     D.    nervousness and fear
Q5   The course of disease on the part of the help seeker is ( )
A.    one month     B.    two years     C.    three months     D.    ten years
Q6   The psychological characteristics of the help seeker include ( )
A.    strict family education     B.    cowardliness and overcaution
C.    failure in passing the postgraduate entrance examination     D.    headaches and insomnia
Q7   The reasons for judging whether the help seeker has a normal mentality are (  )
A.    whether she has self-consciousness      B.    see doctors of her own accord
C.    severity of symptom     D.    impairment of social function
Q8   The causes of the help seeker’s psychological problem exclude ( )
A.    personality factors     B.    cognitive factors     C.    examination stress     D.    stress imposed by her parents
Q9   The causes of the formation of the help seeker’s personality may be ( )
A.    parental control     B.    tight protection given by her grandmother
C.    work environment     D.    failure in passing the postgraduate entrance examination
Q10   The help seeker’s characteristics of personality include ( )
A.    excessive demands on herself     B.    excessive pursuit of perfection
C.    excessive sentimentality     D.    excessively self-righteous
Q11   The crux of the psychological problem on the part of the help seeker lies in ( )
A.    self-inferiority and parental criticism     B.    fear and being afraid of falling ill
C.    anxiety and excessive demands on herself     D.    depression and the loss of work
Q12   The life events affecting the help seeker include ( )
A.    separation from her parents during her childhood
B.    being unable to play by herself when she was a child
C.    insomnia caused by her failure in passing the postgraduate entrance examination
D.    being scolded for eating things without washing her hands
Table 12: English translation of a sample problem (Part 1) from C-2A (: the correct answer option). We show the original problem in Chinese in Table 14.
Q13   The grounds for the help seeker’s mental disease exclude ( )
A.    free of depressive symptom     B.    free of hallucination
C.    free of delusional disorder     D.    free of thinking disorder
Q14   The definite diagnose on the part of the help seeker needs to have a further understanding of the following materials ( )
A.    her body conditions      B.    characteristics of her inner world
C.    her economic conditions     D.    her inter-personal communication
Q15   While offering the counseling service to the help seeker, what a counselor should pay attention to are ( )
A.    job change     B.    behavior change     C.    mood change     D.    cognitive change
Table 12: English translation of a sample problem (Part 2) from C-2A (: the correct answer option). We show the original problem in Chinese in Table 14.
General information: a help seeker, male, at the age of 24, graduation from a university, waiting for employment.
Case introduction: Two years have passed since the help seeker graduated from a university. Coerced by his parents, he once went to a job fair. However, shortly after he arrived at the job fair, he went away without saying one sentence. He confesses that he is not good at expressing himself. Seeing that other people are skilled at promoting themselves, he feels that he is inferior to others in this regard. Because of the lack of work experience, he is afraid that he cannot be competent for a job. Therefore, he always has no self-confidence, only to stay at home.
The information of the help seeker observed and understood by a psychological counselor includes introverted personality, poor independence, no interest in making friends and low mood. The following is a section of talk between the psychological counselor and the help seeker:
Psychological Counselor: What aspect would you like to receive my help?
Help Seeker: I am afraid to go to a job fair.
Psychological Counselor: Have you been there?
Help Seeker: Yes, I have. But I only wandered around before going away.
Psychological Counselor: You only wandered about without saying anything. Can you be called a candidate?
Help Seeker: I’m afraid to tell my intention in case I might be declined. What’s more, I’m worried that I am not competent at a job.
Psychological Counselor: Just now, you’ve said that you are eager to land a job but now you tell me that you have participated in almost no job fair. There seems to be a contradiction. Can you explain it?
Help Seeker: Almost two years have passed since I graduated from university. Yet I have not landed a job. I am really anxious. But I really have no idea how I can prepare for a job fair.
Psychological Counselor: As a university graduate, you’d better consider how to attend a job fair.
Help Seeker

: (keeps silent for a moment) I have to go to a job fair and tell employers that I need a job and I must discuss with them about work nature, conditions, wage and so on.

Psychological Counselor: For what reasons have you not done these?
Help Seeker: I’m afraid that they may not accept me if I go there.
Psychological Counselor: You mean that as long as you go there, employers will be scrambling for employing you. In other words, if you go to the State Council to apply for a job, you will make it; if you go to the municipal government to seek a job, you will still realize your intention and if you go to an enterprise to apply for a post, you will succeed all the same.
Help Seeker: Ah… it seems that I don’t think so (shaking his head).
Psychological Counselor: Since time is limited, so much for this counseling. Please go home to think it over. Next time, we can continue to discuss this issue.
Q1   The major causes that trigger the help seeker’s psychological problem include ( )
A.    personality factors     B.    inter-personal stress     C.    cognitive factors     D.    economic pressures
Q2   At the beginning of the counseling, the approaches to asking questions adopted by the psychological counselor are ( )
A.    to question intensely     B.    to ask open-ended questions
C.    to ask indirectly about something     D.    to ask closed questions
Q3   The psychological counselor says, “You only wandered about without saying anything. Can you be called a candidate? ” He indicates his ( ) attitude to the help seeker.
A.    reprobation     B.    query     C.    enlightenment     D.    encouragement
Q4   By saying “There seems to be a contradiction. Can you explain it? ” the psychological counselor adopts the following tactic ( )
A.    guidance     B.    confrontation     C.    encouragement     D.    explanation
Q5   Silence phenomenon has several major types except for ( )
A.    suspicion type     B.    intellectual type     C.    vacant type     D.    resistant type
Q6   “I’m afraid that they may not accept me if I go there.” This sentence reflects the help seeker’s ( )
A.    pessimistic attitude     B.    overgeneralization     C.    absolute requirement     D.    extreme awfulness
Q7   The psychological counselor says, “You mean that as long as you go there, …” What tactic is employed in this paragraph?
A.    Aristotle’s sophistry     B.    “mid-wife” argumentation
C.    technique of rational-emotion imagination     D.    rational analysis report
The help seeker is the same person in the passage above.
Here is the section of the second talk between the psychological counselor and the help seeker:
Psychological Counselor: From the perspective of psychology, what triggers your emotional reaction is not some events that have happened externally but your opinions of these events. It is necessary to change your emotion instead of external events. That is to say, it is necessary to change your opinions and comment on these events. In fact, people have their opinions of things. Some of their opinions are reasonable while others are not reasonable. Different opinions may lead to different emotional results. If you have realized that your present emotional state has been caused by some unreasonable ideas in your mind, perhaps you are likely to control your emotion.
Table 13: Translation of a sample problem (Part 1) from C-2B (: the correct answer option). We show the original problem in Chinese in Table 15.
Help Seeker: I see to some extent. Previously, I thought that if I was to apply for a job, I should be employed, which is rather unreasonable. However, I have a poor self-confidence. I am afraid that I can be regarded as being stupid and incapable. I am also afraid that if I was employed, I could not be qualified for my work.
Psychological Counselor: Let me give an example for you. If you look downward from upstairs, I will feel that you are taller than anyone else. However, if you look upward by lying on your stomach, you will feel that anyone else is taller than you. Therefore, as far as the same height is concerned, it is possible that you are taller or shorter than anyone else. Are you clear?
Help Seeker: (silent) I see. But I cannot do so.
Psychological Counselor: Are you going to stay at home to be visited by employers who drive BMW? Therefore, I advise you to collect some relevant newspaper at home and find out the posts related to your specialty before casting your resume. Is that OK?
Help Seeker: Yeah, yeah, I understand what you have said.
Q8   According to the pyschological counselor, “from the perspective of psychology, …”, what stage is shown in this paragraph? ( )
A.    diagnostic stage     B.    corrective stage     C.    realization stage     D.    educational stage
Q9   The objective that cannot be attained by rational emotive therapy is ( )
A.    self-compassion     B.    self-direction     C.    self-criticism     D.    self-acceptance
Q10   According to ABC Theory, which of the following is not the major cause leading to the help seeker’s psychological problem? ( )
A.    setback and poor experience     B.    distorted cognition     C.    evaluative concept     D.    irrational belief
Q11   In this case, according to ABC Theory, the help seeker’s A includes ( )
A.    is afraid to land a job     B.    is forced to land a job     C.    is unwilling to land a job     D.    fails to land a job
Q12   In this case, according to ABC Theory, the help seeker’s C includes ( )
A.    difficulty in landing a job     B.    conflict with his parents     C.    emotive anxiety     D.    fails to land a job
Q13   Which of the following type of help seekers is suitable for using the rational emotive therapy? ( )
A.    young and of higher education level     B.    bigotry     C.    old and of lower educational level     D.    autism
Q14   In the rational emotive therapy, the roles a psychological counselor plays are ( )
A.    mentor     B.    persuader     C.    analyst     D.    arguer
Q15   The active attention paid to the help seeker by the psychological counselor should be embodied in ( )
A.    recounts by himself that he is not good at expression     B.    wants to land a job
C.    emphasizes that he cannot do so     D.    takes part in a job fair
Q16   By saying “But I cannot do so.”, what the help seeker expresses are ( )
A.    dependence     B.    impedance     C.    empathy     D.    loquacity
Q17   The most serious mistake made by the psychological counselor in this paragraph is ( )
A.    guides the help seeker how to attend a job fair
B.    understands the rational emotive therapy in a wrong way
C.    the example given by him clearly misleads the help seeker
D.    the counseling service offered by him is heavily biased towards solving concrete problems
Q18   Taking into consideration the contents of the talk, we can judge that they belong to ( )
A.    the talk intended for collecting related information
B.    the talk intended for conducting psychological assessment
C.    the talk intended for fixing the scheme of psychological therapy
D.    the talk intended for attaining the objective of consultation
Table 13: Translation of a sample problem (Part 2) from C-2B (: the correct answer option). We show the original problem in Chinese in Table 15.
Q1   求助者的症状主要有( )。
A.    恐惧情绪     B.    抑郁情绪     C.    强迫观念     D.    强迫行为
Q2   求助者没有出现的生理症状是( )。
A.    心慌     B.    失眠     C.    头痛     D.    恶心
Q3   求助者的主要行为症状是( )。
A.    反复动作     B.    反复询问     C.    反复检查     D.    反复哭泣
Q4   求助者没有出现的行为症状是( )。
A.    担心害怕     B.    情绪低落     C.    经常头痛     D.    紧张恐怖
Q5   对该求助者病程的判定是( )。
A.    一个月     B.    两年     C.    三个月     D.    十年
Q6   该求助者的心理特点包括( )。
A.    家教严格     B.    胆小怕事     C.    考研失败     D.    头痛失眠
Q7   判断求助者心理正常与否的依据是( )。
A.    有否自知力     B.    主动求医     C.    症状严重程度     D.    社会功能受损
Q8   求助者心理问题的原因不包括( )。
A.    性格因素     B.    认知因素     C.    考试压力     D.    父母压力
Q9   求助者性格形成的原因可能是( )。
A.    父母管教     B.    外婆严格保护     C.    工作环境     D.    考研失败
Q10   求助者的性格特点包括( )。
A.    过分要求自己     B.    过分追求完美     C.    过分多愁善感     D.    过分自以为是
Q11   求助者心理问题的关键点是( )。
A.    自卑和父母批评     B.    恐惧和害怕得病     C.    焦虑和要求过高     D.    抑郁和失去工作
Q12   影响求助者的生活事件包括( )。
A.    小时候与父母分离     B.    小时候不能单独玩     C.    考研落榜后失眠     D.    不洗手吃东西被训
Q13   排除求助者精神病性的依据包括( )。
A.    无抑郁症状     B.    无幻觉症状     C.    无妄想症状     D.    无思维障碍
Q14   对该求助者做出明确的诊断,还需要了解的资料包括( )。
A.    躯体情况     B.    内心世界特点     C.    经济情况     D.    人际交往情况
Q15   对该求助者咨询时需关注的是( )。
A.    工作改变     B.    行为改变     C.    情绪改变     D.    认知改变
Table 14: A sample problem from C-2A (: the correct answer option).
Q1   引发求助者心理问题的主要原因包括( )。
A.    性格因素     B.    人际压力     C.    认知因素     D.    经济压力
Q2   心理咨询师在咨询开始阶段使用的提问方式是( )。
A.    直接逼问     B.    开放式提问     C.    间接询问     D.    封闭式提问
Q3   心理咨询师“你去招聘会转一圈…那叫应聘吗?”表明其对求助者( )。
A.    指责     B.    质疑     C.    启发     D.    鼓励
Q4   心理咨询师“这似乎存在着矛盾,你能解释一下吗?”表明其使用的技术是( )。
A.    指导     B.    对峙     C.    鼓励     D.    解释
Q5   沉默现象的几种主要类型中并不包括( )。
A.    怀疑型     B.    理智型     C.    茫然型     D.    反抗型
Q6   “我怕去了人家不要我”反映出求助者存在( )。
A.    悲观的态度     B.    过分概括     C.    绝对化要求     D.    糟糕至极
Q7   心理咨询师说“你的意思是说你去了,……”本段所使用的是( )。
A.    亚里士多德诡辩术     B.    产婆术辩论法     C.    合理情绪想象技术     D.    合理分析报告
Q8   心理咨询师说“按照心理学的观点,……”这段呈现的是( )。
A.    诊断阶段     B.    修通阶段     C.    领悟阶段     D.    教育阶段
Q9   合理情绪疗法无法帮助求助者达到的目标是( )。
A.    自我关怀     B.    自我指导     C.    自我批评     D.    自我接受
Q10   根据ABC理论,该求助者心理问题的主要原因不是( )。
A.    挫折和遭遇     B.    歪曲的认知     C.    评价性观念     D.    不合理信念
Q11   在该案例中,根据ABC理论,求助者的A包括( )。
A.    不敢找工作     B.    被逼找工作     C.    不愿找工作     D.    找不到工作
Table 15: A sample problem from C-2B (: the correct answer option) (Part 1).
Q12   在该案例中,根据ABC理论,求助者的C包括( )。
A.    难找工作     B.    与父母冲突     C.    情绪焦虑     D.    找不到工作
Q13   适合使用合理情绪疗法的求助者范围是( )。
A.    年轻、文化水平较高     B.    偏执     C.    年老、文化水平较低     D.    自闭
Q14   在合理情绪疗法中,咨询师所扮演的角色是( )。
A.    指导者     B.    说服者     C.    分析者     D.    辩论者
Q15   心理咨询师对求助者积极关注,应体现在关注其( )。
A.    自述不善于表达     B.    想找工作     C.    强调自己做不到     D.    参加应聘
Q16   求助者说“可我做不到啊”所表现的是( )。
A.    依赖     B.    阻抗     C.    移情     D.    多话
Q17   心理咨询师在本段咨询中出现的最严重的错误是( )。
A.    指导求助者如何应聘     B.    错误理解合理情绪疗法
C.    举例明显误导求助者     D.    咨询偏向解决具体问题
Q18   纵观本次谈话的内容,可以判定其属于( )。
A.    摄入性会谈     B.    鉴别性会谈      C.    治疗性会谈     D.    咨询性会谈
Table 15: A sample problem from C-2B (: the correct answer option) (Part 2).