Generative Slate Recommendation with Reinforcement Learning

by   Romain Deffayet, et al.

Recent research has employed reinforcement learning (RL) algorithms to optimize long-term user engagement in recommender systems, thereby avoiding common pitfalls such as user boredom and filter bubbles. They capture the sequential and interactive nature of recommendations, and thus offer a principled way to deal with long-term rewards and avoid myopic behaviors. However, RL approaches are intractable in the slate recommendation scenario - where a list of items is recommended at each interaction turn - due to the combinatorial action space. In that setting, an action corresponds to a slate that may contain any combination of items. While previous work has proposed well-chosen decompositions of actions so as to ensure tractability, these rely on restrictive and sometimes unrealistic assumptions. Instead, in this work we propose to encode slates in a continuous, low-dimensional latent space learned by a variational auto-encoder. Then, the RL agent selects continuous actions in this latent space, which are ultimately decoded into the corresponding slates. By doing so, we are able to (i) relax assumptions required by previous work, and (ii) improve the quality of the action selection by modeling full slates instead of independent items, in particular by enabling diversity. Our experiments performed on a wide array of simulated environments confirm the effectiveness of our generative modeling of slates over baselines in practical scenarios where the restrictive assumptions underlying the baselines are lifted. Our findings suggest that representation learning using generative models is a promising direction towards generalizable RL-based slate recommendation.


page 1

page 2

page 3

page 4


Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology

Most practical recommender systems focus on estimating immediate user en...

Representation Learning in Low-rank Slate-based Recommender Systems

Reinforcement learning (RL) in recommendation systems offers the potenti...

HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

Discrete-continuous hybrid action space is a natural setting in many pra...

Learning 3D Navigation Protocols on Touch Interfaces with Cooperative Multi-Agent Reinforcement Learning

Using touch devices to navigate in virtual 3D environments such as compu...

On Modeling Long-Term User Engagement from Stochastic Feedback

An ultimate goal of recommender systems (RS) is to improve user engageme...

Advantage Amplification in Slowly Evolving Latent-State Environments

Latent-state environments with long horizons, such as those faced by rec...

PinnerFormer: Sequence Modeling for User Representation at Pinterest

Sequential models have become increasingly popular in powering personali...

Please sign up or login with your details

Forgot password? Click here to reset