1 Introduction
Modelfree Reinforcement Learning (MFRL) has succeeded in several domains, including video game playing [mnih2015human; mnih2016a3c] and robot control [lillicrap2016ddpg; schulman2015trust]. However, high sample complexity prevents applying MFRL to most complex real world applications: MFRL directly optimizes the agent’s policy from interactions with the environment usually requiring millions of data samples [mnih2015human; mnih2016a3c; lillicrap2016ddpg].
Contrary to MFRL, modelbased reinforcement learning (MBRL) is typically more dataefficient. However, when the perfect model is inaccessible, a forward dynamics model of the environment must be approximated [deisenroth2011pilco; kurutach2018modelensemble; nagabandi2017neural; kamthe2018gpmpc; chua2018deep]. Despite the learning of the forward dynamics model, MBRL often requires less interactions with the environment compared to learning a policy directly. Nevertheless, a major disadvantage of MBRL is that, due to model approximation errors [nagabandi2017neural], MBRL commonly cannot achieve the performance of MFRL at convergence.
Prior work takes advantage of both MFRL and MBRL to some extent. levine2013guided show that generating training samples for MFRL by modelbased trajectory optimization may increase the performance of MFRL, while gu2016continuous find in contrast that insufficient exploration of MBRL may impair the performance of the resultant policy and nagabandi2017neural show that naive exploration for training forward dynamics models prevents accuracy of the acquired forward dynamics model. Apart from generating samples using MBRL, silver2016mastering; tamar2016value; oh2017value; lowrey2018plan combine MFRL with online modelbased planning (i.e. MBRL) to improve the performance of the reinforcement learning (RL) agent, but either is restricted in discrete state and action spaces, relies on the assumption that the state space has 2D structure, or assumes a perfect forward dynamics model. In sum, despite the notable performance of the conjunction of MFRL with modelbased planning, the strong assumptions prohibit applications on more complex tasks.
We propose a novel framework that unifies MFRL and MBRL through Model Predictive Control (MPC). Our approach leverages the merits of MFRL and MBRL at both training and testing time. At training time, we utilize the exploratory policy of MFRL to collect more diverse training samples in the environment than MBRL [gu2016continuous; nagabandi2017neural] and thus prevent the impacts of insufficient exploration on policies and forward dynamics models. Next, at testing time, we combine samplingbased MPC (a modelbased online planning approach) with MFRL to further increase the performance beyond training. Different from the contemporary works [silver2016mastering; lowrey2018plan], our approach does not rely on any assumption of perfect forward dynamics models or state/action space. We use an approximated forward dynamics model that can be applied in arbitrary state/action spaces. Furthermore, we jointly leverage the value function and the policy to yield remarkable MPC planning performance and dataefficiency. Finally, we propose a softgreedy approach that improves action selection in planning. To our best knowledge, we are the first to combine MFRL and MPC, on the least assumptions of state/action space.
We demonstrate the effectiveness of our approach on wellknown challenging continuous control tasks in MuJoCo [todorov2012mujoco]. The experimental results show that our approach leads to better performance in less data than that of independently using MFRL or MPC, particularly in complex tasks, and thus confirm that our approach of combining MBRL and MFRL is effective. Furthermore, the results show that our approach favors the quality of forward dynamics model training. In addition, we provide empirical analysis to verify the limitations of contemporary approaches.
The contributions of this paper are the following:

We show that using an MFRL policy to collect training data improves the quality of the trained forward dynamics model.

We show that using an MFRL policy can enhance MPC‘ s performance.

We show advantages and limitations of using value function in MPC.

We provide a comprehensive evaluation of each design decision for combining MPC and MFRL.
2 Background
In order to elaborate the motivation and the implementation of the proposed method, this section starts with an introduction to Reinforcement Learning (RL) [sutton1998introduction], then explains MFRL, and finally discusses recently proposed MPC approaches for MBRL.
2.1 Reinforcement Learning (RL)
In this paper, we study a standard deterministic discrete time RL problem, consisting of a 4tuple . denotes the state and the action space. is a taskspecific reward function encoding the task objective. is the true forward dynamics model. At each time step , an RL agent perceives the current state , takes the control action and observes the next state and the immediate reward . The training objective of RL is to search for an optimal controller that selects for each such that the expected return is maximized, where represents the horizon of the task and denotes the constant discount factor. Note that while this paper adheres to deterministic cases for simplicity our framework can be easily extended to a stochastic formulation.
2.2 Modelfree Reinforcement Learning (MFRL)
We discuss both the training and evaluation phases in MFRL. First, using data collected by an exploratory policy , the training phase learns a possibly stochastic control policy that infers which maximizes for , and a value function estimating expected returns for . and denote parameters. In this paper, we adhere to an appealing MFRL approach, policy gradient [sutton2000policy] which updates along the direction .
is for variance reduction
[williams1992simple]. A common choice for is the value function estimate . can be obtained by various value function approximation approaches [schulman2015high; sutton1998introduction; bertsekas2005dynamic]. Finally, the evaluation phase simply samples control actions using : .2.3 Model Predictive Control for Modelbased Reinforcement Learning (MPCMBRL)
MPC has long been prevalent in robotic control [garcia1989model; tassa2014control] and recently been applied to MBRL [nagabandi2017neural; kamthe2018gpmpc; chua2018deep; williams2017information]. MPCMBRL simply trains a forward dynamics model to plan the control action at each time step during evaluation.
The training phase trains an approximated forward dynamics model since the real forward dynamics model is usually inaccessible [nagabandi2017neural; kamthe2018gpmpc]. MPCMBRL collects an initial dataset consisting of a series of transitions by using uniformly random exploration in the environment: , then optimizing
by minimizing the following loss function:
(1) 
where is a batch of transitions sampled from . Next, MPCMBRL appends data from online execution of MPCplanning to : [nagabandi2017neural; chua2018deep], and trains the forward dynamics model according to Eq. 1 again.
In the evaluation phase, MPCMBRL computes at each time step the control action for the current state , with modelbased planning consisting of three stages: trajectory sampling, trajectory evaluation, and action selection. Trajectory sampling stage simulates a set of trajectories: , where denotes the number of simulated trajectories, denotes the index of a trajectory within , and indicates the planning horizon. can be obtained by sequentially applying
(2) 
where denotes the simulated state at planning step within , is the action applied to , and denotes a given action distribution. Next, trajectory evaluation stage evaluates each trajectory with the taskspecific reward function and the terminal reward function :
(3) 
where denotes the simulated accumulated rewards associated with , and and denote the state and action sequences extracted from respectively. Although is typically ignored (i.e. ), we use to illustrate our approach in Section 3. Finally, action selection stage selects the first control action of the action sequence which yielded highest value w.r.t. Eq. 3:
(4) 
3 The approach: Model Predictive Control with Modelfree Reinforcement Learning (MPCMFRL)
Typical MFRL algorithms and MBRL methods do not fully utilize the beneficial information that can be extracted from the environment. MFRL ignores the information encapsulated in the forward dynamics of the environment, while MPC neglects the utility of policies and value functions. Contrary to both, our approach jointly leverages forward dynamics, policies, and value functions, and therefore is more likely to have better performance in the case of dearth of data. Section 3.1 details the proposed training method and Section 3.2 the proposed online hybrid MPCMFRL approach.
3.1 The training phase: learning a policy, value function, and a forward dynamics model
We jointly train the control policy , the value function , and the forward dynamics model using the same data at the same time. The training phase iteratively executes the following three steps until convergence. Firstly, we gather a trajectory using the exploratory policy (if using onpolicy RL, must be set as ). collects data by interacting with the environment: (as Section 2.1 describes). Secondly, we update and by , as Section 2.1 describes. In this paper, we use Trust Region Policy Optimization (TRPO) [schulman2015trust], a trust region policy gradient method, for training since TRPO has been successful in several domains and is a theoretic sounded policy gradient algorithm, but other MFRL methods [lillicrap2016ddpg; schulman2017proximal] could be used instead. Finally, we append to the training dataset : ( is initialized as ), and optimizes the forward dynamics model by minimizing the loss function defined in Eq. 1 with batches sampled in . Algorithm 5 in supplementary material details the training scheme. Note that though this paper focuses on training a deterministic forward dynamics model, probabilistic models [chua2018deep]
[kamthe2018gpmpc] can be easily applied in our approach.Jointly training , , and poses the following advantages. First and foremost, can collect more extensive data for the forward dynamics model than does uniform random exploration and MPC onpolicy data aggregation [nagabandi2017neural]. Optimized to maximize reward, can collect successful experiences which uniform random exploration cannot. Also, MPC simply exploits rewards while an MFRL exploratory policy balances exploitation and exploration and thereby guarantees data diversity. Second, interacting with an environment using requires less computation time than collecting the training dataset using MPCplanning [nagabandi2017neural; chua2018deep]. Finally, our joint training procedure maximize the utility of each interaction with the environment, comparing to simply learning a policy (MFRL) or a forward dynamics model (MBRL).
3.2 The evaluation phase: planning the control action using the policy, the value function, and the forward dynamics model
Our approach at the evaluation time is built on the top of MPC framework defined in Section 2.3 and improves each stage by MFRL. We use MFRL‘s control policy for trajectory sampling, MFRL‘s value function for trajectory evaluation, and softgreedy approach for action selection. Fig. 1 illustrates the planning process and Algorithm 1 summarizes the approach. Next, we discuss how our method improves trajectory sampling, trajectory evaluation, and action selection.
Trajectory sampling.
By replacing the action distribution in Eq. 2 with the MFRL control policy , our method can more efficiently sample trajectories of high value than uniform random sampling [richards2005robust] and Cross Entropy Method (CEM) [rubinstein1999cross]. We sample as follows:
(5) 
The MFRL control policy improves trajectory sampling in the following aspects. To begin with, can readily result in highvalue trajectories since is trained to maximize the expected return . Additionally, even before convergence can restrain the search space in trajectory sampling and thus allows trajectories of high value to be sampled more likely than in uniform sampling [richards2005robust]. Moreover, in contrast to CEM [rubinstein1999cross] which uses simulated data for optimization, our method is not susceptible to forward dynamics approximation error since is trained with real experience. Finally, our method does not require costly computations since our method does not rely on online iterative optimization like CEM [rubinstein1999cross].
Trajectory evaluation.
Similar to lowrey2018plan, our approach substitutes the terminal reward function (Eq. 3) with the MFRL value function to resolve the shortsighted planning problem mentioned in Section 2.3. Trajectory evaluation becomes:
(6) 
The MFRL value function in Eq.6 estimates the expected return of a given state (see Section 2.2) and therefore prevents shortsighted planning even with a short planning horizon . More importantly, planning with a short horizon avoids compounding errors in simulation of long horizon and saves computation time as well.
Different from lowrey2018plan, we use an approximated dynamics model rather than a perfect one. The assumption of a perfect dynamics model is unrealistic in most cases, especially on complex tasks, where the perfect dynamics model is inaccessible. In addition, estimating the expected return of a simulated state could be risky since the value function trained with real experience is unlikely to be accurate in the states absent in the training data. Moreover, jointly using value function estimation and a plain random action sampling in simulation of MPCplanning could magnify this problem since those unconstrained or weakly constrained action sampling may induce lots of unreachable states (e.g. states that violate physical constraints) where the value function cannot estimate accurately on.
However, our approach can alleviate the above problem by sampling actions with an MFRL policy in trajectory sampling. MFRL policies can prevent agents from performing actions that could lead to unreachable states since MFRL polices are trained to maximize to expected reward of the task and thus are less likely to perform those infeasible actions.
Action selection.
If we simply take the control action as the sampled action that yielded the max expected return (Eq. 4), the agent may be overly optimistic to the simulated results and is likely to impair the performance. Thus, we propose a softgreedy approach to alleviate the impact of approximation errors in the forward dynamics model . Our softgreedy approach takes the control action as the average over the best action sequences w.r.t. . Formally, we describes as the follows:
(7) 
where sorts all action sequences according to in descending order.
Our proposed softgreedy action selection approach can prevent biasing toward the best action obtained from imperfect simulation. Averaging has been shown to be able to alleviate the inherent bias of the maxoperator (Eq. 4) [everitt2011miscellaneous]. Preventing bias due to the maxoperator allows MPCMFRL to operate with inaccurate forward dynamics models caused by, for example, underfitting or overfitting forward dynamics models [burnham2003model].
4 Experiments
The experiments are designed to answer the main question of whether MPCMFRL successfully leverages MPC to bridge the gap between MBRL and MFRL, overcoming the drawbacks of prior works. Moreover, we perform additional experiments to answer the following more detailed questions: (1) Do we obtain a more accurate forward dynamics model by collecting training samples using an exploratory policy of MFRL? (2) Does an MFRL control policy enhance the planning performance of MPC? (3) Can the value function favor the performance of MPC planning even in an approximated forward dynamics model? (4) Does the proposed softgreedy action selection improve performance under forward dynamics model approximation errors? Next, we shortly introduce the experimental setup, and then discuss the experimental results.
4.1 Experimental setup
Benchmark tasks.
We use standard continuous control Mujoco benchmark tasks of varying difficulty from OpenAI gym [gym] varying in dimensions of and : Swimmer , Reacher , HalfCheetah , and Ant .
Implementation of MPCMFRL.
The MFRL control policy , the MFRL value function , and the approximated forward dynamics model
are implemented as neural networks.
is modeled as a multivariate Gaussian distribution, where
, where andare the mean vector and the covariance matrix conditioned on learned parameters
and current state . We optimize by TRPO [schulman2015trust], while updating and using the Adam optimizer [kingma2014adam].Baselines.
For comparison against both MFRL and MPCMBRL methods we select the following baselines (see supplementary material for details):

MF (S): TRPO that uses stochastic actions for evaluation.

MF (D): TRPO that uses deterministic actions for evaluation (i.e. ).

MPCRandom
: MPC that samples actions from a uniform distribution for trajectory rollouts (Eq.
2) and uniform random exploration with onpolicy data aggregation (see Section 2.3) for training the forward dynamics model. 
MPCCEM: MPC that uses the same model training approach as MPCRandom, while using CEM [rubinstein1999cross] for trajectory sampling since CEM has been shown to work well with MPC in prior work [chua2018deep].
4.2 Evaluation procedure
In order to assess performance and dataefficiency of each method, we evaluate offline [chua2018deep]
each method periodically w.r.t. number of training samples used. At offline evaluation time, we fix all model parameters and measure the average total reward over 10 episodes. We then report the mean and bootstrapped confidence interval of the best average total reward (denoted "Average return" in figures below) so far over 5 distinct random seeds.
4.3 The results of overall performance
Fig. 2 shows the performance of each method w.r.t. the number of samples. MPCMFRL achieves better performance than all baseline methods: MPCMFRL exceeds the performance of MPCMBRL while being more dataefficient than MFRL. Moreover, the improvement is particularly significant in the more challenging tasks like Ant and HalfCheetah. To conclude, this result verifies the effectiveness of MPCMFRL on common benchmark tasks.
Interestingly, Fig. 2 shows that MPCCEM loses to MPCRandom in Antv2 and HalfCheetahv2. Also, we find that even though MPCCEM obtains the highest expected return in simulation (Eq. 3), the expected return in the real environment is surprisingly low. Prior works of MBRL [sutton2012dyna; kurutach2018modelensemble] suggest that training a policy using fictitious data impairs performance. Thus, the poor performance of MPCCEM in complex tasks could be caused by optimizing the policy using simulated data of high approximation errors.
4.4 The results of improved exploration for training
We show that our training approach improves the training quality of forward dynamics models (Section 2.3
) by comparing various training schemes w.r.t. testing error of the forward dynamics model and evaluation performance. Fig.
(a)a shows the testing error of the forward dynamics models trained by different schemes. Half of the testing set consists of data from Random+MPC and half from Policy (see Section C in supplementary material for more details). Policy reduces the testing error faster and more than Random+MPC, which shows that collecting data by an MFRL policy can increase the accuracy of forward dynamics models. In addition to favoring accuracy, Fig. (b)b further shows MPCMFRL (Policy) outperforms MPCMFRL (Random+MPC), especially in later stages of evaluation. To summarize, these results verify our approach is superior to prior approaches in terms of accuracy of forward dynamics models and evaluation performance, also showing that accuracy of forward dynamics models greatly affects the performance.4.5 The results of sampling trajectories by an MFRL policy
Fig. (a)a shows MPCMFRL () outperforms MPCMFRL () in all tasks and therefore suggests that the MFRL control policy can more readily sample trajectories of high value in the real environment than the baselines. Furthermore, MPCMFRL () still surpasses MPCMFRL () even in the early stages where the pure modelfree variants of MPCMFRL (i.e. MF (S/D)) perform poorly compared to MPCMFRL (). This result confirms that even though the MFRL control policy has not yet converged, our method can still enhance MPC planning performance, and also show the improvement of dataefficiency against MFRL. Note that we ignore comparison with CEM due to its poor performance in Fig. 2.
4.6 The strength and the limitations of trajectory evaluation with a value function
We investigate the strength and limitations of our trajectory evaluation approach in this section. Fig. (b)b shows that MPCMFRL () outperforms MPCMFRL () with the same planning horizon and thereby verifies that our approach (Eq. 6) is effective. However, on the contrary to the results in the prior work [lowrey2018plan], Fig. (b)b shows that MPCMFRL () approximates MPCMFRL (), which suggests that compounding errors in a longer planning horizon may impair the performance of value function in an approximated dynamics model. In addition, we find that in the experiments of Section 4.5, the terminal reward (i.e. ) in simulated trajectories of MPCMFRL () are similar than MPCMFRL (). This observation implies that a uniform random sampling may sample states that lead overstimation of value function in approximated dynamics model. The detail investigation are left as future works.
4.7 The results of softgreedy action selection
This section verifies that our softgreedy action selection approach improves performance under an approximate forward dynamics model, followed by studying the effectiveness with models of varying complexity. Fig. (a)a shows that MPCMFRL (w SG) outperforms MPCMFRL (w/o SG), suggesting that softgreedy action selection increases performance of an approximated forward dynamics model. Fig. (b)b shows that MPCMFRL (w SG) surpasses MPCMFRL (w/o SG) in all model complexities, thereby suggesting that our softgreedy action selection is superior to classical greedy action selection in both simple and complex models.
5 Related work
The classic Dyna framework [sutton1990integrated; silver2008sample; kurutach2018modelensemble; kalweit2017uncertainty; clavera2018model] learn a forward dynamics model for simulating experiences to train an MFRL agent, while we focus on the interplay of MFRL and MPC.
Guided Policy Search (GPS) [levine2013guided; chebotar2017path; levine2014learning] uses modelbased controllers such as iLQR [tassa2012synthesis] and iLQG [todorov2005generalized] to generate training samples for MFRL policy. The followup works [zhang2016learning; nagabandi2017neural] further adopt MPC to provide supervision for an policy. In contrast, our approach concentrates on improving MPC performance with MFRL.
gu2016continuous, feinberg2018model, and buckman2018sample assist value function learning by modelbased approaches. gu2016continuous collect training samples for value function learning using a modelbased controller. feinberg2018model and buckman2018sample leverage modelbased rollout to compute targets of value function training, thus accelerating value function learning. On the contrary, our method focuses on leveraging value function to ameliorate the shortsighted planning of MPC.
silver2016mastering, weber2017imagination, oh2017value, and tamar2016value add planning to an MFRL policy. silver2016mastering and oh2017value, however, either rely on discrete state and action spaces or a perfect forward dynamics model. tamar2016value assume that the state space has 2D structure. weber2017imagination learn planning by an MFRL approach, hence inheriting the datainefficiency of MFRL. In contrast, our approach can be applied to an arbitrary type of state and action space and is more dataefficient than MFRL.
pong*2018temporal perform MPC planning solely with a goalconditioned stateaction value function. lowrey2018plan improve longterm planning by evaluating sampled trajectories using a value function. pong*2018temporal and lowrey2018plan do not utilize MFRL policies to enhance planning performance. Planning with only a value function [pong*2018temporal] inherits the datainefficiency of value function learning in MFRL, while our approach combines simulated rewards and a value function, thereby mitigating this problem. lowrey2018plan assume a perfect forward dynamics model, whereas our work does not rely on such an unrealistic assumption.
6 Conclusion
We propose the MPCMFRL framework which leverages the advantages of MFRL and MPC to achieve MFRL‘s level of performance while being as dataefficient as MBRL. Moreover, MPCMFRL allows the agent to continually improve performance with more environment interactions. On the other hand, our novel MPCMFRL framework brings promising future work to light as well. An application of MPCMFRL on real robotics systems, for example, can be an appealing direction since MPCMFRL shows superior dataefficiency particularly crucial for real robots. Another direction could be guiding an MFRL policy by MPC. One can pretrain an MFRL policy in several tasks, then online adapt to a new task by MPC or modelbased policy search.
Comments
There are no comments yet.