DeepAI AI Chat
Log In Sign Up

EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models

by   Zhengwei Tao, et al.
Peking University

Events serve as fundamental units of occurrence within various contexts. The processing of event semantics in textual information forms the basis of numerous natural language processing (NLP) applications. Recent studies have begun leveraging large language models (LLMs) to address event semantic processing. However, the extent that LLMs can effectively tackle these challenges remains uncertain. Furthermore, the lack of a comprehensive evaluation framework for event semantic processing poses a significant challenge in evaluating these capabilities. In this paper, we propose an overarching framework for event semantic processing, encompassing understanding, reasoning, and prediction, along with their fine-grained aspects. To comprehensively evaluate the event semantic processing abilities of models, we introduce a novel benchmark called EVEVAL. We collect 8 datasets that cover all aspects of event semantic processing. Extensive experiments are conducted on EVEVAL, leading to several noteworthy findings based on the obtained results.


page 1

page 2

page 3

page 4


Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners

The emergent few-shot reasoning capabilities of Large Language Models (L...

Event Extraction: A Survey

Extracting the reported events from text is one of the key research them...

Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

Prior research has explored the ability of computational models to predi...

Drafting Event Schemas using Language Models

Past work has studied event prediction and event language modeling, some...

Combining Event Semantics and Degree Semantics for Natural Language Inference

In formal semantics, there are two well-developed semantic frameworks: e...

A Semantic Approach for Automatic Structuring and Analysis of Software Process Patterns

The main contribution of this paper, is to propose a novel semantic appr...

VideoLLM: Modeling Video Sequence with Large Language Models

With the exponential growth of video data, there is an urgent need for a...