WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain

10/31/2022
by   Raj Sanjay Shah, et al.
9

Pre-trained language models have shown impressive performance on a variety of tasks and domains. Previous research on financial language models usually employs a generic training scheme to train standard model architectures, without completely leveraging the richness of the financial data. We propose a novel domain specific Financial LANGuage model (FLANG) which uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective. Additionally, the evaluation benchmarks in the field have been limited. To this end, we contribute the Financial Language Understanding Evaluation (FLUE), an open-source comprehensive suite of benchmarks for the financial domain. These include new benchmarks across 5 NLP tasks in financial domain as well as common benchmarks used in the previous research. Experiments on these benchmarks suggest that our model outperforms those in prior literature on a variety of NLP tasks. Our models, code and benchmark data are publicly available on Github and Huggingface.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/18/2023

BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark

To advance Chinese financial natural language processing (NLP), we intro...
research
03/30/2023

BloombergGPT: A Large Language Model for Finance

The use of NLP in the realm of financial technology is broad and complex...
research
06/15/2020

FinBERT: A Pretrained Language Model for Financial Communications

Contextual pretrained language models, such as BERT (Devlin et al., 2019...
research
04/21/2019

Few-shot NLG with Pre-trained Language Model

Natural language generation (NLG) from structured data or knowledge is e...
research
10/13/2021

Towards Efficient NLP: A Standard Evaluation and A Strong Baseline

Supersized pre-trained language models have pushed the accuracy of vario...
research
04/29/2020

GePpeTto Carves Italian into a Language Model

In the last few years, pre-trained neural architectures have provided im...

Please sign up or login with your details

Forgot password? Click here to reset