LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehension

12/16/2022
by   Wenyue Hua, et al.
0

The application of Natural Language Processing (NLP) to specialized domains, such as the law, has recently received a surge of interest. As many legal services rely on processing and analyzing large collections of documents, automating such tasks with NLP tools emerges as a key challenge. Many popular language models, such as BERT or RoBERTa, are general-purpose models, which have limitations on processing specialized legal terminology and syntax. In addition, legal documents may contain specialized vocabulary from other domains, such as medical terminology in personal injury text. Here, we propose LegalRelectra, a legal-domain language model that is trained on mixed-domain legal and medical corpora. We show that our model improves over general-domain and single-domain medical and legal language models when processing mixed-domain (personal injury) text. Our training architecture implements the Electra framework, but utilizes Reformer instead of BERT for its generator and discriminator. We show that this improves the model's performance on processing long passages and results in better long-range text comprehension.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/15/2022

Toward Improving Attentive Neural Networks in Legal Text Processing

In recent years, thanks to breakthroughs in neural network techniques es...
research
12/25/2021

CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction

Acronyms and long-forms are commonly found in research documents, more s...
research
09/01/2022

Unsupervised Simplification of Legal Texts

The processing of legal texts has been developing as an emerging field i...
research
08/08/2023

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore

The legality of training language models (LMs) on copyrighted or otherwi...
research
01/16/2023

Towards an Automatic Consolidation of French Law

We present preliminary results about Legistix, a tool we are developing ...
research
11/01/2019

BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding

Fine-tuning language models, such as BERT, on domain specific corpora ha...
research
03/28/2020

Orchestrating NLP Services for the Legal Domain

Legal technology is currently receiving a lot of attention from various ...

Please sign up or login with your details

Forgot password? Click here to reset