DeepAI AI Chat
Log In Sign Up

EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models

05/24/2023
by   Zhengwei Tao, et al.
Peking University
0

Events serve as fundamental units of occurrence within various contexts. The processing of event semantics in textual information forms the basis of numerous natural language processing (NLP) applications. Recent studies have begun leveraging large language models (LLMs) to address event semantic processing. However, the extent that LLMs can effectively tackle these challenges remains uncertain. Furthermore, the lack of a comprehensive evaluation framework for event semantic processing poses a significant challenge in evaluating these capabilities. In this paper, we propose an overarching framework for event semantic processing, encompassing understanding, reasoning, and prediction, along with their fine-grained aspects. To comprehensively evaluate the event semantic processing abilities of models, we introduce a novel benchmark called EVEVAL. We collect 8 datasets that cover all aspects of event semantic processing. Extensive experiments are conducted on EVEVAL, leading to several noteworthy findings based on the obtained results.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/24/2023

Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners

The emergent few-shot reasoning capabilities of Large Language Models (L...
10/07/2022

Event Extraction: A Survey

Extracting the reported events from text is one of the key research them...
07/22/2021

Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

Prior research has explored the ability of computational models to predi...
05/24/2023

Drafting Event Schemas using Language Models

Past work has studied event prediction and event language modeling, some...
11/02/2020

Combining Event Semantics and Degree Semantics for Natural Language Inference

In formal semantics, there are two well-developed semantic frameworks: e...
10/02/2012

A Semantic Approach for Automatic Structuring and Analysis of Software Process Patterns

The main contribution of this paper, is to propose a novel semantic appr...
05/22/2023

VideoLLM: Modeling Video Sequence with Large Language Models

With the exponential growth of video data, there is an urgent need for a...