DeepAI
Log In Sign Up

Training Large-Scale News Recommenders with Pretrained Language Models in the Loop

02/18/2021
by   Shitao Xiao, et al.
9

News recommendation calls for deep insights of news articles' underlying semantics. Therefore, pretrained language models (PLMs), like BERT and RoBERTa, may substantially contribute to the recommendation quality. However, it's extremely challenging to have news recommenders trained together with such big models: the learning of news recommenders requires intensive news encoding operations, whose cost is prohibitive if PLMs are used as the news encoder. In this paper, we propose a novel framework, SpeedyFeed, which efficiently trains PLMs-based news recommenders of superior quality. SpeedyFeed is highlighted for its light-weighted encoding pipeline, which gives rise to three major advantages. Firstly, it makes the intermedia results fully reusable for the training workflow, which removes most of the repetitive but redundant encoding operations. Secondly, it improves the data efficiency of the training workflow, where non-informative data can be eliminated from encoding. Thirdly, it further saves the cost by leveraging simplified news encoding and compact news representation. Extensive experiments show that SpeedyFeed leads to more than 100× acceleration of the training process, which enables big models to be trained efficiently and effectively over massive user data. The well-trained PLMs-based model from SpeedyFeed demonstrates highly competitive performance, where it outperforms the state-of-the-art news recommenders with significant margins. SpeedyFeed is also a model-agnostic framework, which is potentially applicable to a wide spectrum of content-based recommender systems; therefore, the whole framework is open-sourced to facilitate the progress in related areas.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/15/2021

Empowering News Recommendation with Pre-trained Language Models

Personalized news recommendation is an essential technique for online ne...
10/12/2021

Aspect-driven User Preference and News Representation Learning for News Recommendation

News recommender systems are essential for helping users to efficiently ...
10/04/2022

IGNiteR: News Recommendation in Microblogging Applications (Extended Version)

News recommendation is one of the most challenging tasks in recommender ...
01/12/2022

GateFormer: Speeding Up News Feed Recommendation with Input Gated Transformers

News feed recommendation is an important web service. In recent years, p...
04/15/2021

Two Birds with One Stone: Unified Model Learning for Both Recall and Ranking in News Recommendation

Recall and ranking are two critical steps in personalized news recommend...
07/04/2020

Birds of a Feather Flock Together: Satirical News Detection via Language Model Differentiation

Satirical news is regularly shared in modern social media because it is ...
02/14/2022

UA-FedRec: Untargeted Attack on Federated News Recommendation

News recommendation is critical for personalized news distribution. Fede...

Code Repositories