Dual-View Distilled BERT for Sentence Embedding

04/18/2021
by   Xingyi Cheng, et al.
0

Recently, BERT realized significant progress for sentence matching via word-level cross sentence attention. However, the performance significantly drops when using siamese BERT-networks to derive two sentence embeddings, which fall short in capturing the global semantic since the word-level attention between two sentences is absent. In this paper, we propose a Dual-view distilled BERT (DvBERT) for sentence matching with sentence embeddings. Our method deals with a sentence pair from two distinct views, i.e., Siamese View and Interaction View. Siamese View is the backbone where we generate sentence embeddings. Interaction View integrates the cross sentence interaction as multiple teachers to boost the representation ability of sentence embeddings. Experiments on six STS tasks show that our method outperforms the state-of-the-art sentence embedding methods significantly.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/27/2019

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new ...
research
06/15/2016

Siamese CBOW: Optimizing Word Embeddings for Sentence Representations

We present the Siamese Continuous Bag of Words (Siamese CBOW) model, a n...
research
10/07/2022

DABERT: Dual Attention Enhanced BERT for Semantic Matching

Transformer-based pre-trained language models such as BERT have achieved...
research
05/18/2016

Modelling Interaction of Sentence Pair with coupled-LSTMs

Recently, there is rising interest in modelling the interactions of two ...
research
04/17/2021

ASBERT: Siamese and Triplet network embedding for open question answering

Answer selection (AS) is an essential subtask in the field of natural la...
research
11/24/2020

Picking BERT's Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis

As the name implies, contextualized representations of language are typi...
research
03/28/2018

Siamese Cookie Embedding Networks for Cross-Device User Matching

Over the last decade, the number of devices per person has increased sub...

Please sign up or login with your details

Forgot password? Click here to reset