Generative Inverse Deep Reinforcement Learning for Online Recommendation

Introduction

Deep reinforcement learning (DRL) is promising for recommendation systems, given its ability to learn optimal strategies from interactions for generating recommendations that best fit users’ dynamic preferences. DRL-based recommendation systems cover three categories: deep Q-learning based methods, policy gradient based methods, and hybrid methods. Deep Q-learning aims to find the best step by maximizing a Q-value over all possible actions. Zheng et al. (2018) first introduced DRL into recommendation systems for news recommendation; then, Chen et al. (2018) introduced a robust Q-learning to handle dynamic environments for online recommendation. However, Q-learning based methods suffer the agent stuck problem, i.e., Q-learning requires the maximise operation over the action space, which becomes infeasible when the action space is extremely large. Policy gradient based methods can mitigate the agent stuck problem Chen et al. (2019a). Such methods use average reward as the guideline yet they may treat bad actions as good actions, making the algorithm hard to converge Pan et al. (2019). In comparison, hybrid methods combine the advantages of Q-learning and policy gradient. As a popular algorithm among hybrid methods, actor-critic network Konda and Tsitsiklis (2000) adopts policy gradient on an actor network and Q-learning on a critic network to achieve nash equilibrium on both networks. Until now, actor-critic network has been widely applied to DRL-based recommendation systems Chen et al. (2019b); Liu et al. (2020); Zhao et al. (2020); Chen et al. (2020).

Despite differences, all existing DRL-based methods rely on well-designed context-dependent reward functions, as shown in the general workflow of a DRL-based recommendation systems in Fig 1 (a). However, in many cases, the reward function cannot be easily defined, due to dynamic environments and various factors that affect user’s interest Shang et al. (2019). Thus, the existing methods may suffer limited generalization ability. Besides, they generally take into account both user’s preference and user’s actions (e.g., click-through, rating or implicit feedback) in the reward function, under the assumptions that the reward is determined by the current chosen item and user’s action is unaffected by the recommended items. Such assumptions, however, no longer hold for online recommendation Shang et al. (2019). Another challenge is that the usage of reinforcement learning in finding the recommendation policy from scratch might be time-consuming Finn et al. (2016); Levine and Koltun (2012). In a recommendation problem, the state space (i.e., all candidate items in which users might be interested) and action space (i.e., all the actions for candidate items) might be huge, traditional reinforcement learning based methods iterate all the possible combinations to figure out the best policy which is arduous.

Targeting at the above challenges, we aim to enable the agent to infer a reward function from user’s behaviors via inverse reinforcement learning and learn the recommendation policy directly from user behaviors in an efficient and adaptive way, as shown in Fig 1 (b). To this end, we propose a generative inverse reinforcement learning approach for adaptively inferring an implicit reward function from user behaviors. Specifically, the approach can measure the performance of the current recommendation policy and update the current policy with the expert policy in the discriminator, thus alleviating the need of defining reward function for online recommendation. In particular, we transform inverse deep reinforcement learning (DRL) into a generator to augment a diverse set of state-action pairs. Under this generative strategy, our method can achieve better generalization ability under complex online recommendation conditions. In a nutshell, we make the following contributions:

We propose generative inverse reinforcement learning to automatically learn reward function for online recommendation. To the best of our knowledge, this is the first work to decouple the reward function and the agent for online recommendation.
We design a novel actor-discriminator network module that takes a discriminator as the critic network and a novel actor-critic network as the actor network, to implement the proposed framework. The module is model-free and can be easily generalized to a variety of scenarios.
We conduct experiments on a virtual online platform, VirtualTB, to demonstrate the feasibility and effectiveness of the proposed approach. Our proposed method achieves a higher click-through-rate that several state-of-the-art methods.

Problem Formulation

Online Recommendation Online recommendation differs from offline recommendation in dealing with real-time interactions between users and the recommendation system. The system needs to analyse user’s behavior and updates the recommend policy dynamically. The objective is to find a solution that best reflects those interactions and apply it to the recommend policy.

Reinforcement Learning-based Recommendation

Reinforcement Learning based recommendation systems learn from interactions through an Markov Decision Process (MDP). Given a recommendation problem consisting of a set of users

U = {u_{0}, u_{1}, \dots u_{n}}

, a set of items

I = {i_{0}, i_{1}, \dots i_{m}}

and user’s demographic information

D = {d_{0}, d_{1}, \dots, d_{n}}

, MDP can be represented as a tuple

(S, A, P, R, γ)

, where

S

denotes the state space, which is the combination of the subsets of

I

;

A

is the action space, which represents agent’s selection during recommendation based on the state space

S

;

P

is the set of transition probabilities for state transfer based on the action received;

R

is a set of rewards received from users, which are used to evaluate the action taken by the recommendation system, with each reward being a binary value to indicate user’s click;

γ

is a discount factor

γ \in [0, 1]

for balancing the future reward and current reward.

Given a user $u_{0}$ and the state $s_{0}$ observed by the agent (or the recommendation system), which includes a subset of item set $I$ and user’s demographic information $d_{0}$ , a typical recommendation iteration for user $u_{0}$ goes as follows. First, the agent makes an action $a_{0}$ based on the recommend policy $π_{0}$ under the observed state $s_{0}$ and receives the corresponding reward $r_{0}$ . Then, the agent generates a new policy $π_{1}$ based on the received reward $r_{0}$ and determines the new state $s_{1}$

based on the probability distribution

p (s_{n e w} | s_{0}, a_{0}) \in P

. The cumulative reward after

k

iterations is as follows:

r_{c} = \sum k = 0 γ^{k} r_{k}

Inverse Reinforcement Learning-based Online Recommendation We propose inverse reinforcement learning without predefining a reward function for online recommendation. We aim to optimize the current policy $π : S \to A$ to make recommendations that are most suitable for the user.

Figure 2: The proposed framework. The actor-critic network generates the policy $π$ based on current state $s_{t}$ . The discriminator takes state-action pair $(s_{t}, a_{t})$ and expert state-action pair $(s_{t}, a_{E}^{t})$ as input. The goal of the actor-critic network is to generate $π$
which can let discriminator classify it as an expert policy. The discriminator aims to distinguish the policy that is generated by actor-critic network. The input of GAE is the Value from the critic network and the
$c (s, a)$ from discriminator which details will be presented later. We circled one update episode which happened on agent.

Specifically, we model online recommendation as an MDP with a finite state set $S$ , a set of actions $A = {a_{0}, a_{1}, \dots, a_{t}}$ , transition probabilities $P (S, A)$ , and a discount factor $γ$ . Suppose there exist an expert policy $π_{E}$ that can master any state $s \in S$ . The recommendation turns into the optimization problem of finding the policy $π$ that best approximates the expert policy $π_{E}$ across the cost function class $C$ , by the following objective function Abbeel and Ng (2004):

minimize π max c \in C E_{π} [c (s, a)] - E_{π_{E}} [c (s, a)]

(1)

The cost function class $C$ is restricted to convex sets defined by the linear combination of a few basis functions { $f_{1}, f_{2}, \dots, f_{k}$

}. Hence, the corresponding feature vector for the state-action pair

(s, a)

can be represented as

f (s, a) = [f_{1} (s, a), f_{2} (s, a), \dots, f_{k} (s, a)]

. The expectation

E_{π} [c (s, a)]

is defined as (on

γ

-discounted infinite horizon):

E_{π} [c (s, a)] = E [\infty \sum t = 0 γ^{t} c (s_{t}, a_{t})]

(2)

Methodology

The overall structure of our proposed approach is illustrated in Fig 2. It consists of three main components: policy approximation, policy generation, and discriminative actor-critic network. Policy approximation provides the theoretical approach to approximating the learned recommendation policy with the expert policy $π_{E}$ ; policy generation increases the diversity of the recommendation policy; and discriminative actor-critic network

constitutes the main structure of our approach. Besides the above components, we will present the optimization method and the corresponding training algorithm, which aims to limit the update step in optimizing the loss function to ensure that a new policy achieves better performance than the old one.

Policy Approximation

The policy approximation component aims to make the learned recommendation policy $π$ and expert policy $π_{E}$ as similar as possible. To this end, we look for a cost function that delivers the best expert policy $π_{E}$ among all the policies on the latent space. According to Eq.(1), the cost function class $C$ is convex sets, which have a linear format Abbeel and Ng (2004) and a convex format Syed et al. (2008), respectively:

	$C_{l i n e a r} = {\sum i w_{i} f_{i} : ∥ w ∥_{2} \leq 1}$		(3)
	$C_{c o n v e x} = {\sum i w_{i} f_{i} : \sum i w_{i} = 1, \forall i s.t. w_{i} \geq 0}$		(4)

The corresponding objective functions are as follows:

			(5)
	$max j \in {1, \dots, k} E_{π} [f_{j} (s, a)] - E_{π_{E}} [f_{j} (s, a)]$		(6)

In particular, Eq.(5) minimizes the $l_{2}$ distance between the state-action pairs, known as feature expectation matching Abbeel and Ng (2004). Eq.(6) minimizes the worst-case excess cost among the functions $f_{k}$ Syed and Schapire (2008). An issue with such methods is the ambiguity in Eq.(1) that many candidate policies $π$ can approximate the expert $π_{E}$ if we only compare the features Ziebart et al. (2008). We resolve the ambiguity by introducing the following $γ$ -discounted causal entropy Bloem and Bambos (2014) into Eq.(1):

	$H (π) ≜$	$E_{π} [- log π (a \| s)]$
	$=$	$E_{s_{t}, a_{t} \sim π} [- \infty \sum t = 0 γ^{t} log π (a_{t} \| s_{t})]$		(7)

We thereby rewrite Eq.(1) into

minimize π - H (π) - E_{π_{E}} [c (s, a)] + max c \in C E_{π} [c (s, a)]

(8)

and define the reinforcement learning process according to Ziebart et al. (2008):

a r g m i n π \in Π - H (π) + E_{π} [c (s, a)]

(9)

Suppose $Π$ is the policy set. We define the loss function $c (s, a)$ such that the expert policy has a lower cost and other policies have higher cost. Referring to Eq.(8), we define the maximum causal entropy inverse reinforcement learning Ziebart et al. (2010) as follows:

maximize c \in C (min π \in Π - H (π) + E_{π} [c (s, a)]) - E_{π_{E}} [c (s, a)]

(10)

Policy Generation

We regard policy generation as the problem of matching two occupancy measures and solve it by training a Generative Adversarial Network (GAN)

Goodfellow et al. (2014). The occupancy measure

ρ

can be defined as:

ρ_{π} (s, a) = π (s | a) \infty \sum t = 0 γ^{t} P (s_{t} = s | π)

(11)

Since the generator aims to generate the policy as similar to the expert policy as possible, we use GAIL Ho and Ermon (2016) to bridge inverse reinforcement learning and GAN by making an analogy from the occupancy matching to distribution matching. Specifically, we introduce a GA regularizer to restrict the entropy function:

ψ_{G A} (c) = {\begin{matrix} E_{π_{E}} [g (c (s, a))] & c < 0 \infty & otherwise \end{matrix}

(12)

where $g (x)$ is defined as:

g (x) = {\begin{matrix} - x - log (1 - exp (x)) & c < 0 \infty & otherwise \end{matrix}

(13)

By introducing the GA regularizer, we can directly measure the difference between the policy and expert policy without needing to know the reward function. We use the loss function from the discriminator as $c (s, a)$ in Eq.(10). We represent the negative log loss for the binary classification to distinguish the policy $π$ and $π_{E}$ via state-action pairs. The optimal of Eq.(14) is equivalence to the Jensen-Shannon divergence Nguyen et al. (2009):

	$ψ_{G A} (ρ_{π} - ρ_{π_{E}}) = max D \in (0, 1)^{S \times A} E_{π} [log D (s, a)]$
	$+ E_{π_{E}} [log (1 - D (s, a))]$		(14)

	$D_{J S} (ρ_{π}, ρ_{π_{E}}) = D_{K L} (ρ_{π} ∥ (ρ_{π} + ρ_{π_{E}}) / 2) +$
	$D_{K L} (ρ_{π_{E}} ∥ (ρ_{π} + ρ_{π_{E}}) / 2)$		(15)

Finally, We obtain the inverse reinforcement learning definition by substituting the GA regularizer into Eq.(8):

minimize π - λ H (π) + ψ_{G A} (ρ_{π} - ρ_{π_{E}})      D_{J S} (ρ_{π}, ρ_{π_{E}})

(16)

where $λ$ is a factor with $λ \geq 0$ . Note that Eq.(16) has the same goal as the GAN, i.e., finding the squared metric between distributions. More specifically, we have the following equivalence for Eq.(16):

	$minimize π - λ H (π) + ψ_{G A} (ρ_{π} - ρ_{π_{E}}) \equiv min π max D L_{D}$
	$L_{D} = E_{π} [log D (s, a)] + E_{π_{E}} [log (1 - D (s, a))] - λ H (π)$		(17)

Discriminative Actor-Critic Network

The discriminative actor-critic network aims to map online recommendation into an inverse reinforcement learning framework.

Specifically, we take advantage actor-critic network, a variant of the actor-critic to constitute the main structure of our approach. Within this network, the actor uses the policy gradient to update the policy, and the critic uses Q-learning to evaluate the policy and provides feedback Konda and Tsitsiklis (2000).

Given user’s profile at timestamp $t$ (i.e., the item list ${i_{0}, \dots, i_{t - 2}, i_{t - 1}}$ ) and the optional demographic information $D$ (which is used to generate the state $s_{t}$

), the environment embeds user’s recent interest and user’s features into the latent space via neural network

Chen et al. (2020); Liu et al. (2020). Once the actor network gets the state

s_{t}

from the environment, it feeds the state to a network with two fully-connected layers with ReLU as the activation function. The final layer outputs the target policy function

π

parameterized by

θ

, which will be updated together with discriminator

D

. Then, the critic network takes the input from the actor network with current policy

π_{θ_{t}}

, which can be used for sampling to get the trajectory

(s_{t}, a_{t})

. We concatenate the state-action pair and feed it into two fully-connected layers with ReLU as the activation function. The output of the critic network is a value

V (s_{t}, a_{t}) \in I R

, which will be used to calculate the advantage, which is a value used for optimization (to be discussed later).

As aforementioned, the discriminator $D$ is the key component of our approach. To build an end-to-end model and better approximate the expert policy $π_{E}$ , we parameterized the policy as $π_{θ}$ and clip the output of the discriminator so that $D : S \times A \to (0, 1)$ with weight $w$ . The loss function of $D$ is $L_{D}$ . Besides, we use Adam Kingma and Ba (2014) to optimize weight $w$ (the optimization for $θ$ will be introduced later). Here, the discriminator $D$ can be treated as a local cost function to guide the policy update. Specifically, the policy will move toward expect-like regions (divided by $D$ ) in the latent space by minimizing the loss function $L_{D}$ , i.e., finding a point $(π, D)$ for Eq.(17) such that the equation output is minimal.

Policy Optimization

We use the actor-critic network as a policy network to be trained jointly with the discriminator. Therefore, the actor-critic network needs to update the policy parameter $θ$ based on the discriminator. During this process, we aim to limit the agent’s step size to ensure the new policy is better than the old one. Specifically, we use trust region policy optimization (TRPO) Schulman et al. (2015a) to update the policy parameter $θ$ and formulate the TRPO problem as follows:

	$max θ \frac{1}{T} T \sum t = 0 [\frac{π_{θ} (a_{t} \| s_{t})}{π_{θ_{o l d}} (a_{t} \| s_{t})} A_{t}]$
	$subject to D_{K L}^{θ_{o l d}} (π_{θ_{o l d}}, π_{θ}) \leq η$		(18)

where $A_{n}$

is the advantage function calculated by Generalized Advantage Estimation (GAE)

Schulman et al. (2015b) below:

	$A_{t} = \infty \sum l = 0 (γ λ_{g})^{l} δ_{t + l}^{V}$
	$where δ_{t + l}^{V} = - V (s_{t}) + \infty \sum l = 0 γ^{l} r_{t + l}$		(19)

where the reward $r_{t + l}$ is the $l$ -step’s test reward at timestamp $t$ . The reward $r_{t + l}$ have two components which are reward returned by environment and the bonus reward calculated by Discriminator $D$ by using $log (D (s_{t}, a_{t}))$ . Considering the massive computation load of updating the TRPO via optimizing Eq.(18), we use Proximal Policy Optimization (PPO) Schulman et al. (2017) with the objective function below, to update the policy:

E_{t} [min (\frac{π_{θ} (a_{t} | s_{t})}{π_{θ_{o l d}} (a_{t} | s_{t})} A_{t}, clip (\frac{π_{θ} (a_{t} | s_{t})}{π_{θ_{o l d}} (a_{t} | s_{t})}, 1 - ϵ, 1 + ϵ) A_{t})]

(20)

where $ϵ$ is the clipping parameter, which represents the maximum percentage of change that can be updated at a time.

The training procedure involves two components: the discriminator and the actor-critic network. The training algorithm is illustrated in Algorithm 1. Specifically, for the discriminator, We use Adma as the optimizer to find the gradient for Eq.(17) for weight $w$ :

E_{π} [\nabla_{w} log (D_{w} (s, a))] + E_{π_{E}} [\nabla_{w} log (1 - D_{w} (s, a))]

(21)

input : Expert Policy

π_{E}

, current state

s

1 Sampling expert trajectories

(s, a_{E}) \sim π_{E}

;

2 Initialize discriminator parameter

w_{0}

;

3 Initialize policy parameter

θ_{0}

;

4 Initialize clipping parameter

ϵ

;

5 for $i = 0, 1, \dots$ do

6 Sampling trajectories

(s, a) \sim π_{θ_{i}}

;

7 Update the parameter

w_{i}

by gradient on Eq.(21) ;

8 for $k = 0, 1, \dots$ do

9 Get the trajectories

(s, a)

on policy

π_{θ} = π (θ_{k})

;

10 Estimate advantage

A_{t}

using Eq.(19);

Compute the Policy Update

θ_{k + 1} = a r g m a x θ Eq. (???)

By taking

K

step of minibatch SGD (via Adma)

end for

θ_{i} \leftarrow θ_{K}

end for

Algorithm 1 Training algorithm for our model

Experiments

We report experimental evaluation of our model on a real-world online retail environment, VirtualTB Shi et al. (2019), on OpenAI gym¹¹1https://gym.openai.com.

Virtual TaoBao

VirtualTB is a dynamic environment to test the feasibility of the recommendation agent. It enables a customized agent to interact with it and achieve the corresponding rewards. On VirtualTB, each customer has 11 static attributes as the demographic information and are encoded into a 88-dimensional space with binary values. The customers have multiple dynamic interests that are encoded into a 3-dimensional space and may change over interaction process. Each item has several attributes, e.g,. price and sales volume, and are encoded into a 27-dimensional space.

Baseline methods

IRecGAN Bai et al. (2019): An online recommendation method that employs reinforcement learning and GAN.
PGCR Pan et al. (2019): A policy Gradient based method for contextual recommendation.
GAUM Chen et al. (2019c): A deep Q-learning based method that employs GAN and cascade Q-learning for recommendation.
KGRL Chen et al. (2020): Actor-Critic based method for interactive recommendation, a variant of online recommendation.

Note that GAUM and PGCR are not designed for online recommendation, and KGRL requires knowledge graph as side information, which is unavailable to the gym environment. Hence, we only keep the network structure and put those network into the VirtualTB platform for testing.

Evaluation Metric and Experimental Environment

The experiments are conducted in the OpenAI gym environment where the reward can be readily obtained for each episode. Since each episode may have different number of steps, it leads to the difficulty in determining when users will end the session. For this reason, we choose click-through-rate to represent the performance which is defined as:

C T R = \frac{r_{e p i s o d e}}{10 * N_{s t e p}}

(22)

where $10$ means that user is interested in all 10 items which are recommended in a single page, $r_{e p i s o d e}$ is the reward received per episode and $N_{s t e p}$ is number of step included in one episode.

The model is implemented by using PyTorch

Paszke et al. (2019) and the experiments are carried out on a server which consists of two 12-core/ 24-thread Intel (R) Xeon (R) CPU E5-2697 v2 CPUs, 6 NVIDIA TITAN X Pascal GPUs, 2 NVIDIA TITAN RTX, with a total 768 GiB memory.

Expert Policy Acquisition

In this part, we introduce the strategy on acquiring the expert policy for VirtualTB. There is no official expert policy in VirtualTB. Obviously, it is unrealistic to manually create the expert policy from this virtual environment where the source data is not available. Hence, we follow the similar strategy as in Gao et al. (2018) to generate a set of expert policy from a pre-trained expert policy network. We design an actor-critic network $M$ with the same actor and critic network structure as our model, but without advantage. The critic network from $M$ is used to calculate the $Q$ -value by adopting deep Q-learning. We adopts the Deep Deterministic Policy Gradients (DDPG) Lillicrap et al. (2015) to train $M$ .

		GAE: $λ_{g}$
		0.94	0.95	0.96	0.97	0.98	0.99
PPO: $ϵ$	0.05	0.630 $\pm$ 0.063	0.632 $\pm$ 0.064	0.633 $\pm$ 0.062	0.630 $\pm$ 0.059	0.626 $\pm$ 0.060	0.629 $\pm$ 0.059
	0.10	0.632 $\pm$ 0.062	0.635 $\pm$ 0.060	0.636 $\pm$ 0.061	0.636 $\pm$ 0.058	0.634 $\pm$ 0.061	0.633 $\pm$ 0.060
	0.15	0.633 $\pm$ 0.060	0.635 $\pm$ 0.061	0.639 $\pm$ 0.061	0.640 $\pm$ 0.057	0.639 $\pm$ 0.059	0.638 $\pm$ 0.061
	0.20	0.634 $\pm$ 0.060	0.636 $\pm$ 0.060	0.641 $\pm$ 0.063	0.643 $\pm$ 0.061	0.643 $\pm$ 0.063	0.641 $\pm$ 0.058
	0.25	0.631 $\pm$ 0.061	0.635 $\pm$ 0.059	0.636 $\pm$ 0.060	0.637 $\pm$ 0.060	0.636 $\pm$ 0.061	0.634 $\pm$ 0.059
	0.30	0.630 $\pm$ 0.059	0.631 $\pm$ 0.061	0.632 $\pm$ 0.060	0.630 $\pm$ 0.059	0.630 $\pm$ 0.058	0.629 $\pm$ 0.050

Table 1: CTR for Different Parameter Settings for GAE and PPO with 95% Confidence Interval

Hyper Parameters Setting

For the policy network $M$ , we set the DDPG parameters as: $γ = 0.95, τ = 0.001$ , size of hidden layer is $128$ , the size of reply buffer is $1, 000$ and the number of episode is set to $200, 000$ . For Ornstein-Uhlenbeck Noise, scale is $0.1$ , $μ = 0, θ = 0.15, σ = 0.2$ . For our approach, number of episode is set to $100, 000$ , hidden size of the advantage actor-critic network is $256$ , hidden size for discriminator is $128$ , learning rate is $0.003$ , factor $λ$ is $10^{- 3}$ , mini batch size is $5$

and the epoch of PPO is

4

. For the generalized advantage estimation, we set the discount factor

γ

0.995

λ_{g} = 0.97

and

ϵ = 0.2

. For fair comparison, all those baseline methods are training under the same condition. For easy recognition, we set one iteration as 100 episodes.

Results

Fully results are reported in Fig 3. Our approach generally outperforms all four state-of-the-art methods. Specifically, our method gets a best result over all those baseline methods after $500$ iterations. It demonstrates the feasibility of the proposed approach.A possible reason for the poor performance of KGRL is that KGRL maintains a local knowledge graph inside the model and actively interacts with the environment. Because the experiments are conducted on an online platform which does not provide the side information for KGRL to generate its knowledge graph. Hence, KGRL performs poorer than other baseline methods.

Impact of Key Parameters

We are interested in how the control parameter $λ$ on GAE and the clipping parameter $ϵ$ on PPO affect the performance. These two key parameters significantly affect the generalized advantage estimation and proximal policy optimization process. The $λ$

is used to make a compromise between bias and variance which normally is selected from

[0, 95, 1)

with step 0.01. The clipping parameter

ϵ

is used to determine the number of percentage need to be clipped, normally smaller than

0.3

to control the optimization step size. The results are reported in Table 1. For fairness, we report CTR at iteration 2000. Observe that when

λ = 0.97

and

ϵ = 0.2

, the model achieves the best result which is

0.643 \pm 0.061

. These two values are also used as our default setting. More details about the model

M

can be found in the Supplementary Materials.

Ablation Study

In this part, we investigate the effect of the GAE. We use two different optimization strategies to optimize the proposed model which are DDPG and Adaptive KL Penalty Coefficient. The Adaptive KL Penalty Coefficient is the simplified version of the PPO which can be defined as:

L (θ) = E_{t} [\frac{π_{θ} (a_{t} | s_{t})}{π_{θ_{o l d}} (a_{t} | s_{t})} A_{t} - β KL [π_{θ_{o l d}} (\cdot | s_{t}), π_{θ} (\cdot | s_{t})]]

(23)

The update rule for $β$ is:

	${\begin{matrix} β \leftarrow β / b & d < d_{t a r g e t} * a β \leftarrow β * b & d >= d_{t a r g e t} * a \end{matrix}$
	$where d = E_{t} [KL [π_{θ_{o l d}} (\cdot \| s_{t}), π_{θ} (\cdot \| s_{t})]]$

The parameter $a = 1.5$ , $b = 2$ and $d_{t a r g e t} = 0.01$ are determined by experiments where the selection process are reported in Supplementary Materials due to the space limit. The result of the ablation study can be found on Fig 3 (c).

Discussion

This study provides a new approach for reinforcement learning based online recommendation, without the need of defining reward function. In this way, our work is feasible to be applied in various real-world recommendation scenarios, where the reward function is hard to manually define or highly domain-dependent. The proposed method offers a fundamental support for inverse reinforcement learning based recommendation system. By providing a few user behaviors, the proposed method can extract an adaptive unknown reward function so as to automatically find out the optimal strategies to generating recommendations best fitting users’ interest. Our empirical evaluation testify its competitive performance against reinforcement learned based existing state-of-the-art methods. Our model has implication and potentially accelerates the progress in applying reinforcement learning in practice where a complex environment exists.

Related Work

We briefly review previous studies related to deep reinforcement learning (DRL)-based recommendation. All those methods are MDP-based or partial observable MDP-based (POMDP). POMDP based methods can be further divided into three categories: value function estimation Hauskrecht (1997), policy optimization Poupart and Boutilier (2005), and stochastic sampling Kearns et al. (2002). Due to the high computational and representational complexity of the POMDP based methods, MDP-based methods are relatively more popular in academia.

MDP-based DRL methods for recommendation can be concluded in three different approaches: deep Q-learning based, policy gradient based and Actor-Critic based methods. Zheng et al. (2018) adopts the deep Q-learning into the news recommendation by using user’s historical record as the state. Zou et al. (2020) improves the structure of Deep Q-learning to achieve more robust results. Pan et al. (2019) applies the policy gradient to learn the optimal recommendation strategies. Wang et al. (2020) introduces the knowledge graph into the policy gradient for sequential recommendation. However, Q-learning may get stuck because of the max operation, and policy gradient requires a large scale data to boost the converge speed and it will only update once per episode. Hence, Actor-Critic uses the Q-value to conduct the policy gradient per step instead of episode. Zhao et al. (2017) adopts the actor-critic methods to conduct the list-wise recommendation in a simulated environment. Chen et al. (2020); Zhao et al. (2020) utilize the knowledge graph as the side information embedded into the state-action space to increase model’s capability on actor-critic network. In addition, Liu et al. (2020) proposes to produce recommendations via learning a state-action embedding within the DRL framework.

Furthermore, Chen et al. (2019c) integrates the generative adversarial network with reinforcement learning structure to generate user’s attribute so that more side information would be available to boost the reinforcement learning based recommendation system’s performance. Shang et al. (2019) proposes a multi-agent based DRL method for environment reconstruction which take the environmental co-founders into account.

Conclusion and Future Work

In this paper, we propose a new approach InvRec for online recommendation. Our proposed approach are designed to overcome the drawback due to the inaccurate reward function. The proposed model is built upon advantage actor-critic network with the generate adversarial imitation learning. We evaluate our method on the online platform VirtualTB and our model achieves a good performance. We also compared our method with a few state-of-the-art methods in three different categories: Deep Q-Learning based, policy gradient based and actor-critic network based methods. The results demonstrate that the proposed approach’s feasibility and superior performance.

This study provides a good initial attempt about the application of deep inverse reinforcement learning on online recommendation system. However, there are remains a few shortcomings which are not addressed in this paper such as the sample inefficiency problem for the imitation learning Kostrikov et al. (2019). Low sample inefficiency will lead to longer training time and the need of a larger dataset. Random sampling also would affect the performance. The possible solutions would be using the off-policy methods instead of on-policy, finding an optimal sampling strategy such that agent will get the same expert trajectories when facing the state which comes up before.

References

P. Abbeel and A. Y. Ng (2004) Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the twenty-first international conference on Machine learning
, pp. 1. Cited by: Problem Formulation, Policy Approximation, Policy Approximation.
X. Bai, J. Guan, and H. Wang (2019) A model-based reinforcement learning with adversarial training for online recommendation. In Advances in Neural Information Processing Systems, pp. 10735–10746. Cited by: 1st item.
M. Bloem and N. Bambos (2014) Infinite time horizon maximum causal entropy inverse reinforcement learning. In 53rd IEEE Conference on Decision and Control, pp. 4911–4916. Cited by: Policy Approximation.
H. Chen, X. Dai, H. Cai, W. Zhang, X. Wang, R. Tang, Y. Zhang, and Y. Yu (2019a) Large-scale interactive recommendation with tree-structured policy gradient. In
Proceedings of the AAAI Conference on Artificial Intelligence
, Vol. 33, pp. 3312–3320. Cited by: Introduction.
M. Chen, A. Beutel, P. Covington, S. Jain, F. Belletti, and E. H. Chi (2019b) Top-k off-policy correction for a reinforce recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 456–464. Cited by: Introduction.
S. Chen, Y. Yu, Q. Da, J. Tan, H. Huang, and H. Tang (2018) Stabilizing reinforcement learning in dynamic environment with application to online recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1187–1196. Cited by: Introduction.
X. Chen, C. Huang, L. Yao, X. Wang, W. Liu, and W. Zhang (2020) Knowledge-guided deep reinforcement learning for interactive recommendation. arXiv preprint arXiv:2004.08068. Cited by: Introduction, Discriminative Actor-Critic Network, 4th item, Related Work.
X. Chen, S. Li, H. Li, S. Jiang, Y. Qi, and L. Song (2019c) Generative adversarial user model for reinforcement learning based recommendation system. In International Conference on Machine Learning, pp. 1052–1061. Cited by: 3rd item, Related Work.
C. Finn, S. Levine, and P. Abbeel (2016) Guided cost learning: deep inverse optimal control via policy optimization. In International conference on machine learning, pp. 49–58. Cited by: Introduction.
Y. Gao, H. Xu, J. Lin, F. Yu, S. Levine, and T. Darrell (2018) Reinforcement learning from imperfect demonstrations. External Links: Link Cited by: Expert Policy Acquisition.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680. Cited by: Policy Generation.
M. Hauskrecht (1997) Incremental methods for computing bounds in partially observable markov decision processes. In AAAI/IAAI, pp. 734–739. Cited by: Related Work.
J. Ho and S. Ermon (2016) Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565–4573. Cited by: Policy Generation.
M. Kearns, Y. Mansour, and A. Y. Ng (2002) A sparse sampling algorithm for near-optimal planning in large markov decision processes. Machine learning 49 (2-3), pp. 193–208. Cited by: Related Work.
D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: Discriminative Actor-Critic Network.
V. R. Konda and J. N. Tsitsiklis (2000) Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008–1014. Cited by: Introduction, Discriminative Actor-Critic Network.
I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson (2019) Discriminator-actor-critic: addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, External Links: Link Cited by: Conclusion and Future Work.
S. Levine and V. Koltun (2012) Continuous inverse optimal control with locally optimal examples. arXiv preprint arXiv:1206.4617. Cited by: Introduction.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971. Cited by: Expert Policy Acquisition.
F. Liu, H. Guo, X. Li, R. Tang, Y. Ye, and X. He (2020) End-to-end deep reinforcement learning based recommendation with supervised embedding. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 384–392. Cited by: Introduction, Discriminative Actor-Critic Network, Related Work.
X. Nguyen, M. J. Wainwright, M. I. Jordan, et al. (2009) On surrogate loss functions and f-divergences. The Annals of Statistics 37 (2), pp. 876–904. Cited by: Policy Generation.
F. Pan, Q. Cai, P. Tang, F. Zhuang, and Q. He (2019) Policy gradients for contextual recommendations. In The World Wide Web Conference, pp. 1421–1431. Cited by: Introduction, 2nd item, Related Work.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019)
Pytorch: an imperative style, high-performance deep learning library
. In Advances in neural information processing systems, pp. 8026–8037. Cited by: Evaluation Metric and Experimental Environment.
P. Poupart and C. Boutilier (2005) VDCBPI: an approximate scalable algorithm for large pomdps. In Advances in Neural Information Processing Systems, pp. 1081–1088. Cited by: Related Work.
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015a) Trust region policy optimization. In International conference on machine learning, pp. 1889–1897. Cited by: Policy Optimization.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2015b) High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. Cited by: Policy Optimization.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: Policy Optimization.
W. Shang, Y. Yu, Q. Li, Z. Qin, Y. Meng, and J. Ye (2019) Environment reconstruction with hidden confounders for reinforcement learning based recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 566–576. Cited by: Introduction, Related Work.
J. Shi, Y. Yu, Q. Da, S. Chen, and A. Zeng (2019) Virtual-taobao: virtualizing real-world online retail environment for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, pp. 4902–4909. Cited by: Experiments.
U. Syed, M. Bowling, and R. E. Schapire (2008)
Apprenticeship learning using linear programming
. In Proceedings of the 25th international conference on Machine learning, pp. 1032–1039. Cited by: Policy Approximation.
U. Syed and R. E. Schapire (2008) A game-theoretic approach to apprenticeship learning. In Advances in neural information processing systems, pp. 1449–1456. Cited by: Policy Approximation.
P. Wang, Y. Fan, L. Xia, W. X. Zhao, S. Niu, and J. Huang (2020) KERL: a knowledge-guided reinforcement learning model for sequential recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 209–218. Cited by: Related Work.
K. Zhao, X. Wang, Y. Zhang, L. Zhao, Z. Liu, C. Xing, and X. Xie (2020) Leveraging demonstrations for reinforcement recommendation reasoning over knowledge graphs. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 239–248. Cited by: Introduction, Related Work.
X. Zhao, L. Zhang, L. Xia, Z. Ding, D. Yin, and J. Tang (2017) Deep reinforcement learning for list-wise recommendations. arXiv preprint arXiv:1801.00209. Cited by: Related Work.
G. Zheng, F. Zhang, Z. Zheng, Y. Xiang, N. J. Yuan, X. Xie, and Z. Li (2018) DRN: a deep reinforcement learning framework for news recommendation. In Proceedings of the 2018 World Wide Web Conference, pp. 167–176. Cited by: Introduction, Related Work.
B. D. Ziebart, J. A. Bagnell, and A. K. Dey (2010) Modeling interaction via the principle of maximum causal entropy. Cited by: Policy Approximation.
B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey (2008) Maximum entropy inverse reinforcement learning.. In Aaai, Vol. 8, pp. 1433–1438. Cited by: Policy Approximation.
L. Zou, L. Xia, P. Du, Z. Zhang, T. Bai, W. Liu, J. Nie, and D. Yin (2020) Pseudo dyna-q: a reinforcement learning framework for interactive recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 816–824. Cited by: Related Work.

Generative Inverse Deep Reinforcement Learning for Online Recommendation

Pitfalls of learning a reward function online

Deep Reinforcement Learning with Function Properties in Mean Reversion Strategies

Cost-Sensitive Portfolio Selection via Deep Reinforcement Learning

Goal-Oriented Next Best Activity Recommendation using Reinforcement Learning

Where to go: Agent Guidance with Deep Reinforcement Learning in A City-Scale Online Ride-Hailing Service

Neural Model-Based Reinforcement Learning for Recommendation

Deep Reinforcement Learning for Imbalanced Classification

Introduction

Problem Formulation