Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation

by   Ruihong Qiu, et al.

Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.


Equivariant Contrastive Learning for Sequential Recommendation

Contrastive learning (CL) benefits the training of sequential recommenda...

Graph Masked Autoencoder for Sequential Recommendation

While some powerful neural network architectures (e.g., Transformer, Gra...

Multi-level Cross-view Contrastive Learning for Knowledge-aware Recommender System

Knowledge graph (KG) plays an increasingly important role in recommender...

Addressing the Rank Degeneration in Sequential Recommendation via Singular Spectrum Smoothing

Sequential recommendation (SR) investigates the dynamic user preferences...

AdaptiveRec: Adaptively Construct Pairs for Contrastive Learning in Sequential Recommendation

This paper presents a solution to the challenges faced by contrastive le...

GUESR: A Global Unsupervised Data-Enhancement with Bucket-Cluster Sampling for Sequential Recommendation

Sequential Recommendation is a widely studied paradigm for learning user...

Contrastive Learning with Bidirectional Transformers for Sequential Recommendation

Contrastive learning with Transformer-based sequence encoder has gained ...

Please sign up or login with your details

Forgot password? Click here to reset