DeepAI
Log In Sign Up

Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding

10/08/2022
by   Haoming Jiang, et al.
7

E-commerce query understanding is the process of inferring the shopping intent of customers by extracting semantic meaning from their search queries. The recent progress of pre-trained masked language models (MLM) in natural language processing is extremely attractive for developing effective query understanding models. Specifically, MLM learns contextual text embedding via recovering the masked tokens in the sentences. Such a pre-training process relies on the sufficient contextual information. It is, however, less effective for search queries, which are usually short text. When applying masking to short search queries, most contextual information is lost and the intent of the search queries may be changed. To mitigate the above issues for MLM pre-training on search queries, we propose a novel pre-training task specifically designed for short text, called Extended Token Classification (ETC). Instead of masking the input text, our approach extends the input by inserting tokens via a generator network, and trains a discriminator to identify which tokens are inserted in the extended input. We conduct experiments in an E-commerce store to demonstrate the effectiveness of ETC.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/03/2022

Graph Enhanced BERT for Query Understanding

Query understanding plays a key role in exploring users' search intents ...
03/23/2020

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Masked language modeling (MLM) pre-training methods such as BERT corrupt...
09/15/2022

Context-Aware Query Rewriting for Improving Users' Search Experience on E-commerce Websites

E-commerce queries are often short and ambiguous. Consequently, query un...
07/17/2021

Neural Search: Learning Query and Product Representations in Fashion E-commerce

Typical e-commerce platforms contain millions of products in the catalog...
08/12/2022

Pre-training Tasks for User Intent Detection and Embedding Retrieval in E-commerce Search

BERT-style models pre-trained on the general corpus (e.g., Wikipedia) an...
12/15/2020

Pre-Training Transformers as Energy-Based Cloze Models

We introduce Electric, an energy-based cloze model for representation le...
11/29/2020

Generative Pre-training for Paraphrase Generation by Representing and Predicting Spans in Exemplars

Paraphrase generation is a long-standing problem and serves an essential...