Retrofitting Structure-aware Transformer Language Model for End Tasks

09/16/2020
by   Hao Fei, et al.
0

We consider retrofitting structure-aware Transformer-based language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme. Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases. By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks.

READ FULL TEXT
research
04/13/2023

LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model

Universally modeling all typical information extraction tasks (UIE) with...
research
11/02/2017

Neural Language Modeling by Jointly Learning Syntax and Lexicon

We propose a neural language model capable of unsupervised syntactic str...
research
04/15/2021

Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling

The goal of semantic role labelling (SRL) is to recognise the predicate-...
research
09/09/2022

Multi-grained Label Refinement Network with Dependency Structures for Joint Intent Detection and Slot Filling

Slot filling and intent detection are two fundamental tasks in the field...
research
06/07/2019

Analyzing the Structure of Attention in a Transformer Language Model

The Transformer is a fully attention-based alternative to recurrent netw...
research
05/11/2023

Musketeer (All for One, and One for All): A Generalist Vision-Language Model with Task Explanation Prompts

We present a sequence-to-sequence vision-language model whose parameters...
research
12/15/2021

Oracle Linguistic Graphs Complement a Pretrained Transformer Language Model: A Cross-formalism Comparison

We examine the extent to which, in principle, linguistic graph represent...

Please sign up or login with your details

Forgot password? Click here to reset