Multi-Objective Deep Q-Learning with Subsumption Architecture

04/21/2017
by   Tomasz Tajmajer, et al.
University of Warsaw
0

In this work we present a method for using Deep Q-Networks (DQNs) in multi-objective tasks. Deep Q-Networks provide remarkable performance in single objective tasks learning from high-level visual perception. However, in many scenarios (e.g in robotics), the agent needs to pursue multiple objectives simultaneously. We propose an architecture in which separate DQNs are used to control the agent's behaviour with respect to particular objectives. In this architecture we use signal suppression, known from the (Brooks) subsumption architecture, to combine outputs of several DQNs into a single action. Our architecture enables the decomposition of the agent's behaviour into controllable and replaceable sub-behaviours learned by distinct modules. To evaluate our solution we used a game-like simulator in which an agent - provided with high-level visual input - pursues multiple objectives in a 2D world. Our solution provides benefits of modularity, while its performance is comparable to the monolithic approach.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

01/22/2020

pymoo: Multi-objective Optimization in Python

Python has become the programming language of choice for research and in...
05/15/2020

A Distributional View on Multi-Objective Policy Optimization

Many real-world problems require trading off multiple competing objectiv...
05/17/2017

Identification and Off-Policy Learning of Multiple Objectives Using Adaptive Clustering

In this work, we present a methodology that enables an agent to make eff...
02/22/2022

Behaviour-Diverse Automatic Penetration Testing: A Curiosity-Driven Multi-Objective Deep Reinforcement Learning Approach

Penetration Testing plays a critical role in evaluating the security of ...
03/02/2019

Lexicographically Ordered Multi-Objective Clustering

We introduce a rich model for multi-objective clustering with lexicograp...
04/06/2018

Multi-Objective Convolutional Neural Networks for Robot Localisation and 3D Position Estimation in 2D Camera Images

The field of collaborative robotics and human-robot interaction often fo...
04/13/2022

Modularity benefits reinforcement learning agents with competing homeostatic drives

The problem of balancing conflicting needs is fundamental to intelligenc...

1 Introduction

Many recent works on Reinforcement Learning focus on single-objective methods such as Deep Q-learning (Mnih et al., 2013, 2015). As those methods provide great performance in task such as playing video games, many real-life problems require satisfying multiple objectives simultaneously. In single objective reinforcement learning the agent receives a single reward each time it performs an action. In multi-objective reinforcement learning (MORL) the agent receives multiple rewards - one for each objective. In particular, agents dealing with complex environments, such as autonomous robots or agents playing real-time video games, need to pursue multiple, often conflicting objectives.

To have a graspable example, lets consider an autonomous cleaning robot, which is able to clean floors, navigate through obstacles and autonomously return to charging station. The observable aggregated behaviour of such robot may be decomposed into three sub-behaviours: collision avoidance (ca), floor cleaning (fc) and recharging (rg). We may describe the objectives of the robot for each identified sub-behaviour in a multi-objective manner, or we can aggregate the sub-behaviours and define a single objective. In the former case, the robot-agent will receive a set of three rewards ([, , ]) after each action. If the robot collides with a wall, it receives a negative reward related to collision avoidance (), yet the rewards related to floor cleaning and recharging do not depend on this event. However, in single-objective case, the robot will receive only one reward value ([]) dependent on any of the three sub-behaviours. In case of collision, the the single-objective robot will receive a negative reward, but it will be indistinguishable from any negative reward provided with respect to other sub-behaviours such as depletion of batteries.

In single objective scenarios, we may find an optimal policy for which the sum of rewards collected by the agent is the highest possible. Methods such as Q-learning should converge to optimal policies (Sutton & Barto, 1998). However, for multi-objective problems, many such optimal policies may exist, depending on the trade-offs between satisfying particular objectives (Roijers et al., 2013).

Autonomous agents, such as our example cleaning robot, are not really independent - they usually have a purpose defined by another agent: human. This aspect is often neglected in the literature, but is significant when considering practical applications of intelligent agents in robotics, automation or even when designing AIs for video games (always winning AI is not the one that many humans would like to play against). Our cleaning robot may follow a policy for which collision avoidance has greater importance than floor cleaning - in such case the robot should focus on avoiding collisions even at the cost of worse performance at floor cleaning. It is however for the user of such robot to decide, what should be the proportion between carefulness and cleanliness. The user may even want to fully disable some functions (behaviours) of the robot. Yet, state of the art reinforcement learning methods, such as Deep Q-Learning, do not allow to modify the behaviour of the agent after it was trained.

We see that when considering practical applications it is desired to have a multi-objective reinforcement learning method with the following features available post-learning: 1) ability to select the sub-set of pursued objectives and 2) ability to change the impact of particular objectives on the overall policy of the agent. As we will show later, the method presented in this paper posses those features.

Multi-objective problems may be approached using single-policy or multi-policy methods. The simplest single-policy method uses a scalarization function (Moffaert et al., 2013), which converts multiple objectives into a single objective. Scalarization methods utilize a weight matrix to obtain a single score from multiple action-value functions. Some techniques assign linear priorities to objectives (Barrett & Narayanan, 2008; Vamplew et al., 2011). This allows to obtain a single optimal policy with respect to objectives ordered by those priorities.

In contrast to single-policy methods, multi-policy MORL methods are used for find a set of policies. Their aim is to approximate the Pareto front of policies (Roijers et al., 2013). In multi-policy methods, the preference of objectives does not need to be set a priori as a Pareto optimal policy for any preference may be obtained at runtime (Moffaert & Nowé, 2014).

A natural approach in MORL is to use separate learning modules for each objective (Sprague & Ballard, 2003). Modularity allows to decompose the problem into components that are to some extent independent (Raicevic, 2006); modularity may be required for providing features desired in practical applications that were listed earlier. Some works deal with transforming complex single-objective problems to many simpler objectives (Brys et al., 2014). Such methods may be used to benefit from modular approach while solving single-objective problems.

Although Deep Q-Networks gained much attention in recent years, not many works consider the use of DQNs in multi-objective problems. Recently authors of (Mossalam et al., 2016) proposed a multi-policy learning framework that utilizes Deep Q-Networks.

Learning behaviours in embodied agents, such as robots, is a problem well fitted for reinforcement learning methods. In

embodied artificial intelligence

, the idea of parallel, loosely coupled processes (Pfeifer & Scheier, 2001) is proposed as a principle for designing embodied agents. It states, that the control logic for embodied agents should consists of many independent components dedicated for particular aspects of the agent’s behaviour. The aggregated behaviour of an agent emerges from cooperation or competence among those components.

In this work we will present a method for combining multiple Deep Q-Networks for solving multi-objective problems. We will introduce decision values used for more advanced scalarization of multiple Q-functions. Furthermore we will combine decision values with user define priorities, to have an architecture that can dynamically adapt its behaviour with respect to user’s preferences.

In section 2 we will briefly describe single- and multi- objective reinforcement learning. Next, in section 3 we will describe how many separate DQNs may be used together and we will define decision values. In section 4 we will present a simple 2D game - a virtual environment including an autonomous agent that has a local (situated) sensory inputs and may pursue different objectives. Finally in the last section we will evaluate our solution and present the results of our experiments.

2 Background

2.1 Single Objective Reinforcement Learning

In the single-objective reinforcement learning an agent interacts with the environment by perceiving the state and performing an action for each step . The actions are chosen by the agent according to some policy . After performing an action, the agent receives a reward . Then the agent observes the next state and the process repeats. The goal of the agent is to maximize the expected discounted reward , where is the discount factor.

In Q-learning actions are selected based on , which represents the expected discounted reward for performing action in state . For given state s, is the optimal action. The policy of an agent, denoted by

, is the probability of selecting action

in state . If the agent always selects the optimal action, then we say that it follows an optimal policy . Knowing the

allows to create an optimal policy simply by selecting the action with the highest Q-value. Deep Q-learning utilizes Deep Neural Networks for approximating

values, thus enabling this method to be used in many real-world applications. Deep Q-Networks (Mnih et al., 2015) may be used used with high-level visual inputs such as those provided by video games.

2.2 Multi-Objective Reinforcement Learning

We may consider a more complex reinforcement learning scenario in which multiple objectives are pursued by the agent. Let be the set of objectives of an agent. We may assign a priority to each objective such that will have lower priority than when . For further analysis we will assume that , so that priorities may be interpreted as weights.

The agent, instead of a single reward, receives a vector of rewards at each time-step

with respect to each objective , i.e: , where corresponds to objective . For each objective and step we may define the discounted return as:

(1)

Moreover, for each objective there is a Q-function that represents the expected discounted return , i.e: . We may define a vector of Q-functions, which includes for each objective :

(2)

The function may be used by the agent to determine the optimal action with respect to objective at time-step , given state :

(3)

The vector consists of actions optimal with respect to particular objectives at a given time-step . Because at each step, the agent may perform only a single action, a method of reducing to a single action is required.

A common method for selecting a single action is the scalarization (Moffaert et al., 2013) of using some scalarization function and a weight vector . Typically a linear scalarization is applied, so that:

(4)

Then may be used as in equation 3 to select an action. The weight wector in this case corresponds to priorities assigned to particular objectives.

In the further sections of this paper, we will show how to apply scalarization in Deep Q-Networks and we will introduce Decision Values to dynamically adjust the weights for improved performance of the agent. For simplicity, further in the text we will use the index to note that a particular value or function is defined for any objective , and by we will define the number of objectives.

3 Using multiple DQNs

We have considered an agent that have multiple objectives, receives rewards with respect to those objectives and has a separate Q-function for each objective. In this section we will describe how to merge q-values obtained from Deep Q-Networks for different objectives and how the impact of particular DQNs on the behaviours of the agent may be controlled by using Decison Values. Finally we will describe the learning process utilizing DQNs with Decision Values. We will refer to our method as to Multi-Objective Deep Q-Network with Decision Values (MODQN-DV).

3.1 Combining Q-values

In case of multi-objective agent, we may use a separate DQN as an approximator for each in the vector. Such agent would be controlled by multiple Deep Q-Networks working in parallel. Each DQN provides a list of q-values and we want to use q-values from all DQNs to select a single action that will be performed by the agent

Let us define a vector that consists of q-values provided by for each possible action and a single objective , i.e.:

(5)

In the single-objective case the optimal action would be equal to for such that . For multi-objective case we can use scalarization to sum up all vectors and then select the action corresponding to the maximal value of such scaled q-value vector. In this approach, q-values may be interpreted as votes of certain DQN, which are summed-up and the highest-voted action is selected. We need to stress here that simply adding the vectors does not produce a meaningful result yet. The q-values produced by different Q-functions are not scaled. In general q-values may be any real numbers. If we want them to represent votes for particular actions, each vector needs to be rescaled to . Many approaches for scaling the vector may be applied. In our experiments we use the following scaling function for which is mapped to and to :

(6)

The scalarized q-vector is then defined as:

(7)

Now, using the rescaled vectors we can sum them up and select one action with the highest total q-value. For example, let have actions , weight vector , objectives , and corresponding q-vectors and . Adding them will result in vector , for which the second element is the maximal, thus the corresponding action should be selected.

3.2 Decision Value

Figure 1: Three Deep Q-Networks are working in parallel based on the same sensory input. Each DQN corresponds to different task pursued by the agent. Each DQN has an additional decision value output which acts as a dynamic weight used while summing up q-vectors from particular DQNs. User defined priorities are also used for weighting the decision from particual DQNs.

The scalarization allows to combine outputs from multiple DQNs. However, such a combination does not guarantee a meaningful action selection. Let us return to previous to examples and consider a vacuum cleaner approaching a wall; actions correspond to turning left, going straight, and turning right respectively. If the vacuum cleaner perform the action proposed in it will turn right, alternatively if it uses then it will turn left. Using the sum will however lead to going straight forward and hitting the wall. So while both DQNs suggested a meaningful action, their sum is not meaningful at all. We see that using constant weights while summing q-values does not provide a solution for this problem.

To solve this issue, we would need to dynamically choose which q-value vectors are more important in a particular state. In other words, we would like to have a meta-policy for choosing the actual policy of the agent. However, as the agent pursues many objectives, it is hard to define this meta-policy with respect to all objectives. To overcome this problem we propose to indicate the value of the decision provided by each DQN with respect to corresponding objective pursued by the agent.

The proposed decision values may be indicated independently by each DQN based on the current state and used as additional weights while summing up q-value vectors. Going back to the previous example: let assume that is the output from DQN associated with collision avoidance and is the output from DQN associated with cleaning. As the robot approaches a wall, the decision regarding collision avoidance is clearly more important than the decision regarding cleaning. This is because if the robot does not make any decision, it will collide with the wall and receive a negative reward with respect to collision avoidance objective. However, not making the decision will not affect cleaning objective (assuming that the cleanliness of the floor in front of him is not different than in other places). Thus, at this particular state the value of is higher than the value of and should be summed with a higher weight.

We may define the decision value signal , and by denote the decision value associated with . Now the scalarized q-vector would use decision values instead of constant weights:

(8)

We may additionally include the external preferences indicated by values of priorities assigned to objectives as introduced in 2.2. This way the q-values will be scaled both by dynamic decision values and static priorities. Moreover, for technical reasons, we need to add , which is a vector containing very small random values. This will ensure that in a rare cases when all decision values are equal to , a random action will be chosen. Finally the scaled, decision value- and priority- weighted q-value vector denoted by is equal to:

(9)

3.3 Acquiring values of decisions

Now, as we have a method of applying decision values in the scalarization of multiple objectives, let us explain in more details how decision values are defined and how they can be learned by reinforcement learning.

First we should consider how objectives of an agent are defined. Again let us refer to the vacuum cleaning robot example. If the agent had only two objectives: a) to seek dirt and b) to avoid colliding with obstacles, then we could define two reward/terminal states: state A - state in which dirt is collected, state B - state in which the robot is colliding with something. There is a notable difference between those two states. In the first case, the agent should be rewarded positively, but in the the latter case, it should be rewarded negatively. Moreover, if the agent is not in any of those states, it should be not rewarded at all. We can describe the first objective as being attractive (as it attracts the agent by positive rewards) and the second as being repulsive (as it repulses the agent by negative rewards). Many problems in robotics, games or other fields of AI may be presented using a set of attractive or repulsive objectives. In particular some problems may be decomposed into such set of objectives to promote more granular learning and control. Such decomposition is usually simpler and more intuitive compared to more advanced reward shaping techniques.

Let us consider an agent moving in a state-space with attractive and repulsive states. As the agent approaches one of those states, it becomes more critical to perform an action that will either move the agent towards such state or away from it. The value of the decision made with respect to an objective near a rewarding state rises as the distance to this state becomes shorter. This is a simple and intuitive heuristic: if an agent pursues multiple equally weighted objectives, then it probably should focus most on the objective that is already very close to being accomplished.

We can thus create a decision reward - the reward provided to the agent for performing a decision - which would be simply the absolute value of the reward provided with respect to an objective: . Now we can define the decision value as a state-value function (Sutton & Barto, 1998), returning the value of the state under policy , with respect to the decision rewards of a particular objective:

(10)

Such defined decision value will provide high values around rewarding states (either positive or negative) and low values in states which are far from rewarding states. In any state, the decision value will provide the importance of particular objective. The proposed decision value function will hence provide values representing the chances of achieving a rewarding state (with respect to some objective ) given the current state and following policy . Where policy is the policy provided by the Q-function for a particular objective.

It is important to note, that the decision value, as defined, can not be directly used for scalarization, because its value may be any positive number. Moreover, the range of the values provided for different objectives may be very broad. To overcome this problem, the decision value needs to be scaled to be in range as noted in section 3.2. However, the unscaled decision value is needed during learning as it will be shown in the next section. We will therefore denote the unscaled decision value by and define the scaled decision value by as follows:

(11)

Where

is the sigmoid function;

and are derived during learning: is an approximation of the mean value of , while

is an approximation of the standard deviation of

.

3.4 Learning

Have defined decision values, we may move to the method of learning such values along with learning policies for particular objectives. We use Deep Q-Networks to approximate the values of Q-functions. Following the state of the art in this field a provides the approximated function , where are the learnable parameters of the neural network. As in our model we use multiple DQNs, there is a function for a related to objective . Each

is optimised iteratively, using the following loss function for each iteration

:

(12)

As introduced in (Mnih et al., 2013), there are in fact two neural networks involved in the learning process of a single DQN. The on-line network is updated at each iteration, while the target network is updated only each iterations. Moreover experience replay is used to further improve the learning process. The agent stores experienced states, actions and rewards in a replay memory for each respectively. Then at each iteration, each is trained using a sample of past experiences selected uniformly at random from the corresponding replay memory . Those samples are used as mini-batches for gradient descent optimization.

The Decision Value may be updated using TD-learning (Sutton & Barto, 1998) similarly as for any state value function, by using the following update rule:

(13)

As we use a neural network for approximating , we may define the loss function as follows:

(14)

The decision value is provided by an additional output of the DQN and the learning procedure is analogical to Q-function. Moreover the decision value requires scaling, for which the parameters and need to be learned. If we include and in the neural network parameters , then the additional loss function for the decision value scaling would be defined as:

(15)

The neural network is optimized using a combined loss function for Q-values, decision values and scaling of the decision values:

(16)

4 Evaluation

4.1 Cleaner - a 2D game-like virtual environment

Figure 2: Cleaner - a game-like virtual environment with agent pursuing multiple-objectives. The environment consists of the agent, walls, obstacles, recharge area and dirt. The agent perceives the environment by a visual input (a view from the top limited to a square located in the front of the agent). Agent may move forward and turn; its area of movement is limited by walls. Agent has three objectives: avoid walls, consume dirt and recharge.

To evaluate the solution presented in this paper we created Cleaner - a simple game-like virtual environment, simulating the behaviour of an autonomous vacuum cleaner. The environment consists of an agent, walls, recharge areas and dirt consumable by the agent. Cleaner is presented in Figure 2. The agent is a circular object that may move around the map by performing one of three actions: move forward, turn left and turn right. The map is a continuous space. The agent perceives the environment only by visual sense, i.e. a x pixel (width and height) rectangle situated in front of him. This visual input is converted to gray-scale (8bit). Agent’s world (white) is surrounded by walls and filled with obstacles (black rectangles) which agent can not pass. Agent may pick up dirt and recharge itself. Dirt is indicated by three small coaxial circles (black), while recharging field is indicated by a gray rectangle. Dirt re-spawns at random positions on the map after being consumed by the agent. The quantity of dirt, recharge fields and obstacles is constant during the episode. Eater is a simplified simulation of a mobile robot moving on a flat surface (e.g. floor) with a video camera attached at the top of the robot pointed towards the floor.

The agent has a battery level , which is decreased at each time step by . The battery level may be increased when the agent enters the recharging area by each step. An episode ends when the agent’s energy level drops to or when steps pass. The agent starts each game with initial battery level . The position of dirt, recharge fields and obstacles as well as the initial position of the agent are chosen randomly at the start of the episode.

The agent has three objectives: (ca) collision avoidance, (fc) cleaning and (rg) recharging.

The rewards for particular objectives are as follows: objective (ca): for collision, otherwise; objective (fc): for collecting dirt, otherwise; objective (rg): for for each step when , while charging and otherwise.

In all experiments described in this chapter, the game options were as follows: , . The size of the agent sight rectangle is  px,  px. The quantity of food is 20. The numer of obstacles varies randomly from 1 to 5, and the number of charging areas varies randomly from 1 to 3.

4.2 MODQN-DV implementation

Parameter Value
learning steps 1000000
replay memory size
target network update rate
learning rate
start value
end value
end step 100000
discount
batch size
optimizer Adam
Table 1:

MODQN-DV learning hyperparameters

Our implementation of the MODQN-DV was based on the baseline DQN implementation (Dhariwal et al., 2017)

developed by OpenAI using TensorFlow

(Abadi et al., 2015). We expanded the standard DQN with additional decision value outputs and mechanism for scalarizing q-values from multiple DQNs. Each single DQN in a MODQN-DV consist of a convolution network with three convolution layers and no pooling layers, followed by a fully connected layer and the output layer. Dueling (Wang et al., 2016) and double q-learning (Hasselt, 2010)

were used. The additional decision value output is a single neuron linear layer connected to the state score layer used for dueling.

The parameters of the convolution network were kept default as provided in the baselines implementation. The size of the fully connected layer in our models is set to 128, and the size of the input image is our case is 50x50x1, thus the q-values are provided based only on an image input from a single state. The memory replay was modified to store rewards with respect to all objectives separately. The prioritized experience replay(Schaul et al., 2015) was not used in our implementation. The hyperparameters used for training DQNs during evaluation are presented in table 1.

During training of the MODQN-DV, loss functions are used as specified in section 3.4. DQNs for all objectives are trained simultaneously and scaled decision values are used for scalarization during learning.

4.3 Experiments

1 1 1 -88.4 47.6 -35.0 -75.9
1 0 0 -51.9 24.0 -46.2 -74.1
41.4% -49.6% -32.0% 2.37%
0 1 0 -303.0 50.0 -40.3 -293.3
-242.7% 5.1% -15.2% -286.74%
0 0 1 -311.8 20.6 -35.9 -327.2
-252.6% -56.7% -2.7% -331.32%
0.5 0.3 0.2 -45.7 42.9 -39.2 -42.1
48.4% -9.9% -12.2% 44.55%
0.5 0.2 0.3 -68.4 38.3 -39.5 -69.6
22.7% -19.5% -12.9% 8.27%
0.2 0.5 0.3 -143.7 51.3 -33.2 -125.6
-62.5% 7.9% 4.9% -65.63%
0.3 0.5 0.2 -90.0 50.2 -34.7 -74.4
-1.7% 5.6% 0.9% 1.93%
0.2 0.3 0.5 -140.6 45.2 -34.7 -130.1
-59.0% -4.9% 0.6% -71.54%
0.3 0.2 0.5 -123.1 42.4 -33.8 -114.5
-39.2% -10.9% 3.3% -51.01%
(a) with decision values enabled
1 1 1 -61.0 51.3 -28.8 -38.5
1 0 0 -77.6 32.0 -45.0 -90.6
-27.2% -37.6% -56.6% -135.26%
0 1 0 -518.2 33.3 -58.5 -543.4
-749.1% -35.0% -103.4% -1310.52%
0 0 1 -126.9 31.4 -27.8 -123.3
108.0% -38.7% 3.3% -220.12%
0.5 0.3 0.2 -35.7 47.7 -35.7 -23.7
41.6% -7.0% -24.1% 38.54%
0.5 0.2 0.3 -40.3 45.2 -32.6 -27.7
34.0% -11.9% -13.4% 28.08%
0.2 0.5 0.3 -236.2 49.8 -37.8 -224.2
-287.0% -2.8% -31.5% -482.04%
0.3 0.5 0.2 -218.9 50.3 -38.7 -207.3
-258.7% -1.8% -34.5% -438.10%
0.2 0.3 0.5 -86.7 41.9 -29.4 -74.2
-42.1% -18.3% -2.1% -92.71%
0.3 0.2 0.5 -80.8 40.7 -29.1 -69.3
-32.4% -20.7% -1.3% -79.83%
(b) with decision values disabled
Table 2: Evaluation results
Figure 3: The sum of rewards (smoothed) collected by MODQN-DV in cleaner over episodes of training. The plot shows data from 6 different runs.

To evaluate our method we conducted a series of experiments utilizing MODQN-DV and the cleaner environment. In particular we compared the performance of multiple Deep Q-Networks for case a) where decision values were enabled for scalarization and case b) where the decision values were disabled. This comparison gave us a clear indication of the impact of decision values on the performance. We will refer to case (a) as MODQN-DV (b) as to MODQN.

The experiment for both cases (a) and (b) were conducted as follows. First the DQNs were trained using the implementation and parameters as provided in section 4.2 and table 1. In case (a) the decision values were trained and used for scalarization. In case (b) decision values were disabled during training and their values were forcefully set to . During training, the user defined priorities for objectives were set to in all cases (all objectives were weighted equally during scalarization). For each case the training procedure was repeated 6 times and all trained neural networks were saved. As show in figure 3, the learning of MODQN-DV is stable over time.

Next, the trained MODQN-DV and MODQN networks were used for evaluation with 10 different sets of user defined priorities as provided in tables 2(a) and 2(b). In a single evaluation, 100 episodes were played. The same sequence of randomly generated map layouts were used for each run. The sum of collected rewards were recorded for each run. For each set of priorities, 6 runs performed by 6 separately trained MODQN-DV and MODQN instances were averaged.

4.4 Results

The results presented in the tables 2(a) and 2(b) are averaged sums of collected rewards with respect to each objective, namely: for objective (ca) - collision avoidance, for objective (fc) - cleaning and for objective (rg) - recharging. is the sum of the sums of rewards - it indicates the total performance of the agent. Priorities correspond to objectives (ca), (fc) and (cg).

The set of priorities: was used as the baseline for evaluation (also those priories were used during training). For each row in the tables 2(a) and 2(b) there is an additional row marked as with values showing the percentage of gain or loss of collected rewards with respect to the baseline value for each case. The green and red colours of the cells indicate if the reward gain for a particular set of priorities was better compared to the corresponding case in the second table.

The aim of the evaluation was to test how the overall performance of the agent changes when priorities are different from the initial values used during training. As we can see in table 2(a) on 7 of 9 cases, the use of MODQN-DV helped to preserve (or even increased) the overall performance of the agent in comparison to MODQN. Moreover, in almost all cases, the performance of the agent with respect to the objective with the highest priority (marked in bold in the tables) increased when decision values were used. However, it should be noted that in the baseline case, the overall performance of the agent using decision values was lower compared to the case without decision values. The results show that the proposed solution has a significant impact on the performance when priorities are modified post-training.

It is also worth noticing how the decision values change as the agent moves. As expected, the decision value for a particular objective rises when the agent approach a state where it could receive a reward. For instance, the value of collision avoidance rises significantly when the agent is very close to a wall or an obstacle. Moreover, the decision value drops when the agent is in a state far from receiving a reward. For example, if the agent does not perceive any walls or obstacles, then the collision avoidance decision value is lower than average. The agent thus usually selects the action, which is related to the most promising objective at a particular state.

5 Conclusions

In this paper, we presented a method for using multiple Deep Q-Networks to approach multi-objective problems called Multi Objective Deep Q-Networks with Decision Values (MODQN-DV). We introduced decision values to DQNs in order to improve the scalarization of outputs from multiple DQNs. Our method requires only slight modification of existing DQN architectures, while it introduces a number of benefits: 1) it enables the decomposition of problems in to smaller sub-problems, for which independent DQNs may be trained simultaneously, 2) it provides a method for robust manipulation of priorities after the training, which also allows to completely disable DQNs responsible for particular behaviour/objective, 3) it allows to add new objectives to already trained agent without the need of retraining and to tune their impact on the behaviour of the agent.

In the experimental part, we shown that in most cases MODQN-DV improves the performance of the agent, that uses a different set of priorities compared to the training phase. The results are promising, however more tests should be performed using other benchmarks.

In this paper, we also introduce cleaner - a benchmark for multi-objective reinforcement learning problems that provides visual state representation. The authors are not aware of any other existing multi-objective benchmark that would be comparable to atari games benchmark or other provided by OpenAI.

In future work we want to improve the performance of MODQN-DV; one possible improvement is the use of common convolutional layers for all DQNs. It is particularly interesting to use MODQN-DV in very complex environments, such as video games. Recently published Starcraft 2 learning environment may be a good choice for further tests of MODQN-DV architecture, as strategy games may be perceived as multi-objective problems.

References

  • Abadi et al. (2015) Abadi, Martín, Agarwal, Ashish, Barham, Paul, Brevdo, Eugene, Chen, Zhifeng, Citro, Craig, Corrado, Greg S., Davis, Andy, Dean, Jeffrey, Devin, Matthieu, Ghemawat, Sanjay, Goodfellow, Ian, Harp, Andrew, Irving, Geoffrey, Isard, Michael, Jia, Yangqing, Jozefowicz, Rafal, Kaiser, Lukasz, Kudlur, Manjunath, Levenberg, Josh, Mané, Dan, Monga, Rajat, Moore, Sherry, Murray, Derek, Olah, Chris, Schuster, Mike, Shlens, Jonathon, Steiner, Benoit, Sutskever, Ilya, Talwar, Kunal, Tucker, Paul, Vanhoucke, Vincent, Vasudevan, Vijay, Viégas, Fernanda, Vinyals, Oriol, Warden, Pete, Wattenberg, Martin, Wicke, Martin, Yu, Yuan, and Zheng, Xiaoqiang.

    TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.

    URL http://tensorflow.org/. Software available from tensorflow.org.
  • Barrett & Narayanan (2008) Barrett, Leon and Narayanan, Srini. Learning all optimal policies with multiple criteria. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pp. 41–47, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-4. doi: 10.1145/1390156.1390162. URL http://doi.acm.org/10.1145/1390156.1390162.
  • Brys et al. (2014) Brys, T., Harutyunyan, A., Vrancx, P., Taylor, M. E., Kudenko, D., and Nowe, A. Multi-objectivization of reinforcement learning problems by reward shaping. In 2014 International Joint Conference on Neural Networks (IJCNN), pp. 2315–2322, July 2014. doi: 10.1109/IJCNN.2014.6889732.
  • Dhariwal et al. (2017) Dhariwal, Prafulla, Hesse, Christopher, Klimov, Oleg, Nichol, Alex, Plappert, Matthias, Radford, Alec, Schulman, John, Sidor, Szymon, and Wu, Yuhuai. Openai baselines. https://github.com/openai/baselines, 2017.
  • Hasselt (2010) Hasselt, Hado V. Double q-learning. In Lafferty, J. D., Williams, C. K. I., Shawe-Taylor, J., Zemel, R. S., and Culotta, A. (eds.), Advances in Neural Information Processing Systems 23, pp. 2613–2621. Curran Associates, Inc., 2010. URL http://papers.nips.cc/paper/3964-double-q-learning.pdf.
  • Mnih et al. (2013) Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra, Daan, and Riedmiller, Martin. Playing atari with deep reinforcement learning. In

    NIPS Deep Learning Workshop

    . 2013.
  • Mnih et al. (2015) Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, Feb 2015. ISSN 0028-0836. URL http://dx.doi.org/10.1038/nature14236. Letter.
  • Moffaert et al. (2013) Moffaert, K. Van, Drugan, M. M., and Nowé, A. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 191–199, April 2013. doi: 10.1109/ADPRL.2013.6615007.
  • Moffaert & Nowé (2014) Moffaert, Kristof Van and Nowé, Ann. Multi-objective reinforcement learning using sets of pareto dominating policies. Journal of Machine Learning Research, 15:3663–3692, 2014. URL http://jmlr.org/papers/v15/vanmoffaert14a.html.
  • Mossalam et al. (2016) Mossalam, Hossam, Assael, Yannis M., Roijers, Diederik M., and Whiteson, Shimon. Multi-objective deep reinforcement learning. CoRR, abs/1610.02707, 2016. URL http://arxiv.org/abs/1610.02707.
  • Pfeifer & Scheier (2001) Pfeifer, Rolf and Scheier, Christian. Understanding Intelligence. MIT Press, Cambridge, MA, USA, 2001. ISBN 026266125X.
  • Raicevic (2006) Raicevic, Peter. Parallel reinforcement learning using multiple reward signals. Neurocomputing, 69(16–18):2171 – 2179, 2006. ISSN 0925-2312. doi: http://doi.org/10.1016/j.neucom.2005.07.008. URL http://www.sciencedirect.com/science/article/pii/S0925231205003036. Brain Inspired Cognitive SystemsSelected papers from the 1st International Conference on Brain Inspired Cognitive Systems (BICS 2004)1st International Conference on Brain Inspired Cognitive Systems (BICS 2004).
  • Roijers et al. (2013) Roijers, Diederik M., Vamplew, Peter, Whiteson, Shimon, and Dazeley, Richard. A survey of multi-objective sequential decision-making. J. Artif. Int. Res., 48(1):67–113, October 2013. ISSN 1076-9757. URL http://dl.acm.org/citation.cfm?id=2591248.2591251.
  • Schaul et al. (2015) Schaul, Tom, Quan, John, Antonoglou, Ioannis, and Silver, David. Prioritized experience replay. CoRR, abs/1511.05952, 2015.
  • Sprague & Ballard (2003) Sprague, Nathan and Ballard, Dana. Multiple-goal reinforcement learning with modular sarsa(o). In Proceedings of the 18th International Joint Conference on Artificial Intelligence, IJCAI’03, pp. 1445–1447, San Francisco, CA, USA, 2003. Morgan Kaufmann Publishers Inc. URL http://dl.acm.org/citation.cfm?id=1630659.1630892.
  • Sutton & Barto (1998) Sutton, Richard S. and Barto, Andrew G. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
  • Vamplew et al. (2011) Vamplew, Peter, Dazeley, Richard, Berry, Adam, Issabekov, Rustam, and Dekker, Evan. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine Learning, 84(1):51–80, Jul 2011. ISSN 1573-0565. doi: 10.1007/s10994-010-5232-5. URL https://doi.org/10.1007/s10994-010-5232-5.
  • Wang et al. (2016) Wang, Ziyu, Schaul, Tom, Hessel, Matteo, Van Hasselt, Hado, Lanctot, Marc, and De Freitas, Nando. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, pp. 1995–2003. JMLR.org, 2016. URL http://dl.acm.org/citation.cfm?id=3045390.3045601.