Robustifying Reinforcement Learning Agents via Action Space Adversarial Training

07/14/2020
by   Kai Liang Tan, et al.
10

Adoption of machine learning (ML)-enabled cyber-physical systems (CPS) are becoming prevalent in various sectors of modern society such as transportation, industrial, and power grids. Recent studies in deep reinforcement learning (DRL) have demonstrated its benefits in a large variety of data-driven decisions and control applications. As reliance on ML-enabled systems grows, it is imperative to study the performance of these systems under malicious state and actuator attacks. Traditional control systems employ resilient/fault-tolerant controllers that counter these attacks by correcting the system via error observations. However, in some applications, a resilient controller may not be sufficient to avoid a catastrophic failure. Ideally, a robust approach is more useful in these scenarios where a system is inherently robust (by design) to adversarial attacks. While robust control has a long history of development, robust ML is an emerging research area that has already demonstrated its relevance and urgency. However, the majority of robust ML research has focused on perception tasks and not on decision and control tasks, although the ML (specifically RL) models used for control applications are equally vulnerable to adversarial attacks. In this paper, we show that a well-performing DRL agent that is initially susceptible to action space perturbations (e.g. actuator attacks) can be robustified against similar perturbations through adversarial training.

READ FULL TEXT VIEW PDF

Authors

page 1

page 2

page 3

page 4

page 5

page 6

11/13/2020

Query-based Targeted Action-Space Adversarial Policies on Deep Reinforcement Learning Agents

Advances in computing resources have resulted in the increasing complexi...
06/11/2020

Robustness to Adversarial Attacks in Learning-Enabled Controllers

Learning-enabled controllers used in cyber-physical systems (CPS) are kn...
09/05/2019

Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents

Robustness of Deep Reinforcement Learning (DRL) algorithms towards adver...
03/19/2020

Robust Deep Reinforcement Learning against Adversarial Perturbations on Observations

Deep Reinforcement Learning (DRL) is vulnerable to small adversarial per...
05/27/2020

The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems

Modern algorithms in the domain of Deep Reinforcement Learning (DRL) dem...
08/10/2017

Resilient Linear Classification: An Approach to Deal with Attacks on Training Data

Data-driven techniques are used in cyber-physical systems (CPS) for cont...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Data-driven and learning-based methods are increasingly being applied to cyber-physical systems (CPS) with ubiquitous sensing and advancements in data analytics algorithms. Recent studies have demonstrated the feasibility of deep reinforcement learning (DRL) paradigms being applied on CPS as a controller [18, 37, 33]

. The success of these RL paradigms are mainly attributed to the advent of deep neural networks (DNN) that act as expressive decision-making policies. Consequently, adversarial attacks on CPS are inevitable as studies reveal the vulnerability of DNN to adversarial attacks, hence compromising the reliability of RL-based controllers. Success of adversarial attacks in breaking DNNs questions their validity, especially in life- and safety-critical applications such as self-driving cars 

[31]. The threat caused by attacks on DNNs was first detected while white-box attacks (i.e. attacks that are crafted based on a prior knowledge about the model architecture, hyper-parameters, etc.) were being studied [5, 9, 3, 32, 17]. Since then, it has also been shown that transfer attacks (i.e. attacks that were crafted for one DNN architecture and mounted on a different DNN architecture) are capable of breaking DNNs [24].

Researchers have proposed different methods to defend against adversarial attacks. The most popular method amongst them is adversarial training, where the DNN is trained with an adversarially perturbed dataset [17, 20, 7]. A more stable adversarial training decouples the min-max problem in the robust optimization problem and solves the problem using Danskin’s theorem [6]

. This requires finding the worst case perturbation at each training epoch and updating the model parameters using the dataset which has been augmented by the corresponding worst case attacks. Several methods such as the fast gradient sign method (FGSM) 

[9], Carlini-Wagner (CW) [5], projected gradient descent (PGD) [20], tradeoff-inspired Adversarial defense via surrogate-loss minimization (TRADES) [38], and Stochastic Saddle-point Dynamical System (SSDS) [7] approach are a few such examples which were proposed to find the worst case attack.

In contrast to the defense schemes proposed for learning-based methods, other methods rooted in classical control theory have also been developed to counter adversarial perturbations applied on classical controllers [23, 8]. In control theory, the notions of robust control and resilient control have been extensively studied. While robust controllers attempt to remain stable and perform well under various (bounded) uncertainties [39], resilient controllers try to bring back the system to a gracefully degraded operating condition after an adversarial attack [36]. However, similar notions of robustness and resilience have not been studied well for RL-based controllers, except a few recent works [34, 11].

In this study, we investigate the possibility of developing DRL-based controllers that are robust against adversarial perturbations (within specific attack budgets) to the action space (e.g., actuators). Specifically, we develop an algorithm that trains a robust DRL agent via action space adversarial training based on our previous work on gradient-based optimization on action space attacks (MAS-attacks) [19]. We would also like to highlight that developing a resilient architecture that is able to recover from adversarial perturbations may not be tractable as DRL algorithms are inherently trajectory-driven. This is because these complex nonlinear models tend to diverge significantly when they encounter an undesirable trajectory.

Ii Related Works

While the robustification of DRL agents under state space attacks and mitigation of actuator attacks in CPS have been studied in both control and ML literature, robustification of DRL agents against actuator attacks have been relatively less studied. In this section, we divide our literature review into control-theory based defense schemes, adversarial attack and defense on DNN, and robustification DRL agents.

Ii-a Control-Theory Based Defense

Methods to mitigate or improve resiliency of classic controllers against actuation attacks have been extensively studied. For example, authors of [23]

proposed a distributed attack compensator which has the capability to recover agents that are under attacks by estimating the nominal behaviors of the individual agent in a multi-agent setting. On the other hand, numerous studies have also developed theoretical bounds on a system’s ability to recover from adversarial perturbations and designed corresponding solutions such as decoupling state estimates from control  

[8] and  [13] combining a filter, perturbation compensator, and performance controller to re-stabilize a system. Additionally, it has been shown that actuation attacks may go undetected if the attacks are deployed at a higher frequency than sensor sampling frequencies [15], but these attacks can be mitigated with controllers with multi-rate formulations [14].

Ii-B Adversarial Attacks and Defenses

After [32] exposed the vulnerabilities of DNNs to adversarial attacks, a defense strategy was proposed by [3]

, which defines a regularization term in the classifier. Concurrently, 

[9] introduced Fast Gradient Sign Method (FGSM) which was used by [17] to craft a defense strategy against iterative FGSM attacks. Departing from the notion of white-box attacks, [24] showed the vulnerability of DNNs to transfer attacks in which no prior knowledge about the model architecture is needed. Before adversarial training was popularized as a defense method [20, 30, 9], researchers proposed several other defense approaches including defining a network robustness metric [1] and using de-noising auto-encoders to form Deep Constructive Networks [10]. After adversarial training gained popularity, [25]

proposed defensive distillation as another powerful method for defense, although 

[5]

devised multiple loss functions to produce attacks that could break this defense mechanism. Since DRL algorithms employ the use of DNNs as policy functions, those models are also found vulnerable to the attacks described above 

[2].

Ii-C Robust Deep Reinforcement Learning

In the context of training a DRL agent to be robust against state space attacks where the input to the DRL agent is perturbed, [26] and [21] demonstrated that a DRL agent that is robust to parameter and environmental variations can be obtained by adversarially training the agent.. In another study, [11] proposed a meta-learning framework where a meta-RL agent has access to two sub-policies. The meta-agent learns to switch between a policy that maximizes reward during nominal conditions and another policy that mitigates and copes with adversarially perturbed states by observing advantage estimates of both policies during deployment. [27] used a different robustifying scheme by formulating a zero-sum min-max optimization problem where the DRL agent is trained in the presence of another DRL agent that adversarially perturbs the system. Similarly, [34] demonstrated that DRL agents can be robustified against disturbance forces by training the agent with some noise perturbation in a min-max formulation.

Iii Methodology

Fig. 1: Robustifying DRL agent by perturbing the original agent’s action. The perturbation is generated by a white-box adversary, where the adversary has access to the agent’s network architecture and parameters.

We provide a brief overview of DRL algorithms, followed by discussion of robust learning from a robust optimization standpoint. Finally, we formulate the robust RL methodology by combining the RL and robust learning formulation.

Iii-a Reinforcement Learning

In RL, the goal of an agent is to maximize its cumulative future rewards. A typical setup involves an agent interacting with an environment for a finite number of steps, or until a termination condition is met. Upon termination of an episode, the environment resets to an initial state for the agent and repeats the process again. At each step in the environment , the agent receives a state and reward , then selects an action from the agent’s policy . denotes the finite number of steps for an environment, denotes all possible states for an environment, denotes the cumulative reward for one episode, and denotes all possible actions conditioned upon the state. Every state observation is quantified with a reward value indicating how valuable that state is for the agent. Specifically, given a finite number of time steps , the agent’s goal is to learn an optimal policy which maximizes the cumulative discounted rewards :

(1)

where gamma . A -value of makes the agent nearsighted (prefer short-horizon rewards), while a value of makes the agent farsighted (prefer long horizon rewards). Since being in a specific state is a direct result of previous state and action, the agent’s policy will evolve over time to refine the understanding of good and bad trajectories (sequence of state-action combination). Ultimately, the goal is to optimize the agent’s policy such that the mapping between state and action is optimal.

There are two known methods to optimize a policy, namely action-value and policy gradients. Action-value methods optimize the action value for each state-action pair as shown in Eq. 2. Examples of action-value methods include Deep Q-Network (DQN) [22] and DoubleDQN (DDQN) [35].

(2)

Policy gradient methods optimize a policy parameterized by theta , where is directly optimized to maximize the expected reward function :

(3)

denotes the stationary distribution of Markov chain 

[12] for . Examples of policy gradient methods include Trust Region Policy Optimization (TRPO) [28] and Proximal Policy Optimization (PPO) [29].

Iii-B Robust Optimization Classical Formulation

The robust optimization problem can be defined as [20]:

Fig. 2: The goal of the agent is to land at the goal. Annotated directional arrows indicate thrust direction of the lunar lander
(4)

where is the dataset under a data distribution with set of labels, . The loss function (e.g. cross-entropy loss) with additive perturbations is denoted by with as the model parameters (decision variables). Here the saddle point problem is looked at as a composition of an inner maximization and an outer minimization, where the inner maximization tries to find a specific perturbation for each data point such that the overall loss is maximized. In parallel, the outer minimization aims to achieve the model parameters which minimize the corresponding adversarial loss.

The next step is to define the attack model.We introduce a specific perturbation for each data point , where is the set of allowed perturbations (). This set acts as a normalization for the perturbation power. For example, ball around x is a popular way to define the perturbation budget [9]. There are several attack methods to find the corresponding adversary for each data point (i.e. FGSM, PGD, etc.). In this paper, we use PGD for finding the adversaries [20] which uses as the step-size, and can be formulated as:

(5)

For training a neural network, Stochastic Gradient Descent method is used for solving the outer minimization problem at the maximizer point of the inner problem. This approach is valid as Danskin’s theorem proves that the gradients at the inner maximizer act as a valid descent direction for outer loss minimization 

[6].

Iii-C Robust Reinforcement Learning Agents

To achieve a robust DRL agent, the robust optimization is formulated as a DRL problem. In this paper, we focused on white-box attacks in action-space. As such, the robust optimization problem can be written as:

(6)

where are state and action pairs. The reward function with additive perturbations is denoted by where are updated based on the policy () in each iteration, with as the model parameters. Note that this formulation is different from the classical formulation (Eq. 4) in which model parameters are an explicit input to the loss function, whereas in RL the reward function is not an explicit function of model parameters.

The attack formulation in RL is based on our previous work with MAS-attack [19]. MAS-attacks are derived from white-box attacks in action space, where the perturbations for each are computed based on complete knowledge of the policy of the agent, (See Fig. 1). Each perturbation is bounded by , by projecting the perturbations back into . As MAS-attack uses PGD to iteratively find the perturbations, we can formulate the attack as:

(7)

where is the iteration number, is the step size and is the action distribution obtained from the agent’s policy . Here, we note that while the robust optimization formulation was formulated using the reward function, in reality, the reward function is unknown to the DRL agent or the virtual adversary. Hence, the reward function is approximated by the reward-maximizing action distribution for gradient computations [19]. After obtaining the adversarial perturbations and adding it to the nominal actions, we train the DRL agent using standard policy gradient methods, which solves the outer maximization problem in the formulation above. Another distinction we would like to note is that while a -step PGD is usually used to compute the adversarial attacks, with being fixed, we do not adhere to that procedure. Instead, we define a tolerance and keep iterating through the PGD process until the adversarial actions saturate. This ensures that the computed adversarial action actually corresponds to bad action rather than being approximated by a fixed -number of gradient steps. The adversarial training algorithm to robustify the DRL agent is shown in Alg. 1.

Input: episodes , episodic limit , step size , convergence criteria , budget
Initialize state , policy
for  do
       for  do
             Get action distribution from
             Sample from
            
             Sample from
             while  do
                  
                  
             end while
            )
            
             Step through environment with
       end for
      if Time to update then
             Update agent’s policy
       end if
      
end for
Algorithm 1 MAS-Adversarial Training

Iv Experiments

Iv-a Environment Setup

Experiments were conducted in OpenAI gym’s Lunar Lander environment [4], where the goal is to land the lander safely while minimizing thruster usage. The state space of the environment consist of eight continuous values: x-y coordinates of the lander, velocity in x-y components, angle and angular velocity of the lander, and a boolean contact variable for left-right lander legs (e.g

. 1 for contact, 0 for no contact). The action space available are two continuous vectors [-1,1]. First vector controls the up-down engine, where values within [-1,0] turns off the engine while values within (0,1] maps to 50% - 100% of engine power. Second vector controls left-right engines, where [-1,-0.5] and [0.5,1] controls left and right engine respectively.

For this environment, the agent’s goal is to maximize reward. Given an arbitrary starting point (as seen in Fig. 2

), the RL agent has to land on the landing pad without crashing. The agent receives positive rewards (

e.g. between 100 to 140) for landing. The agent will incur negative rewards if the lander moves away from the landing pad. Each successive landing leg contact gives 10 rewards each. Firing the up-down engine cost -0.3 rewards each step. The episode is terminated if the lander crashes or is at rest, which gives -100 and 100 rewards respectively.

Iv-B Deep Reinforcement Learning Training and Parameters

For this experiment, we trained a PPO agent with an Actor-Critic architecture [16]

. In this architecture, both the actor and the critic share the same network of multi-layer perceptrons made up of dense layers. The actor has an additional dense layer which outputs the policy in the form of a Gaussian model and the critic estimates the value of the action chosen by the actor. To encourage exploration, an entropy term is also added to the loss function which consists of the actor’s loss and the critic’s loss. The loss function is then jointly optimized using Adam optimizer.


V Results and Discussion

Fig. 3: Training plots on robustly trained DRL agents with MAS-attacks ( and projection). Seven agents was trained for each projection method. Each rewards are averaged across all agents, followed by a moving average.

We conducted two different experiments to study the difference in robustifying DRL agents by using and projection methods with MAS-attack budget of 1 and step size of 3 within Algorithm 1.

V-a Convergence of Robust Agent

In this section, we analyzed the convergence plots (Fig. 3) of the robust agent empirically. The robust agent was trained with and perturbations at every step for 15000 episodes. For each projection method, we trained seven DRL agents with the same network architecture and parameters across different seeds. After 15000 episodes, training rewards are averaged across seven agents, followed by a moving window of 100 episodes to obtain results shown in Fig. 3. Agents trained with converged faster to a higher reward at approximately 4500 episodes as compared to training with , which stabilized around 9000 episodes. This reveals that it is slightly harder to robustify the agent against as compared to due to attacks being more distributed along action dimensions compared to .

width=center 3* Environment Nominal Adversarial Agent Nominal Robust Nominal Robust Reward 3* Environment Nominal Adversarial Agent Nominal Robust Nominal Robust Reward

TABLE I: Summary statistics of agent’s rewards
Fig. 4: A comparison of reward distributions between a nominally trained agent and adversarially trained agent for both and MAS attacks. We observe that the distribution of rewards for a nominal agent shifts to the left and have long tails in the regions with negative rewards when subjected to attacks. In comparison, the distributions of rewards remain similar when the robustly trained agents are subjected to attacks.

V-B Performance Comparison

We compare the performance of the robustly trained DRL agents against the nominally trained DRL agent. Specifically, we tested both DRL agents under two environment settings; nominal and adversarial environment. A nominal environment denotes scenarios where the agent is not attacked, while the agent is attacked in adversarial environments. Fig. 4 shows a histogram of all four possible scenarios for each projection attack. For both projection attacks, the nominal agent tested in the nominal environment is identical. The other three scenarios show slightly different results due to different projection attacks. We summarize the reward distribution of each scenario in Table I. Rewards within the range of 2.5 to 3.0 are successful landing experiments, where the DRL agent learns to land on the landing pad and shuts off it’s engines. Rewards between 1.0 to 2.5 are DRL agents that lands successfully but continuously loses rewards as it fails to turn off the engine. Rewards below 1.0 are experiments where the lunar lander crashed or flew out of frame.

V-B1 Nominal environment

In nominal environments, the nominal agent’s performance is identical across both and plots in Fig. 4 which is expected, as the agent is not attacked. As a result, the agent has a mean reward of . This indicates the nominal agent successfully lands and turns off its engine.

Next, we evaluate both the robust agent trained with and projection attacks in the nominal environment. The rewards for the robust agent in and are and respectively. We hypothesize that the nature of crafted attacks are more evenly distributed across action dimension, hence it is harder to train and test against attacks where else distribute attacks into one dimension. These results are counterintuitive to the notion of robustifying a DRL agent, where the expected results of a robust agent should be higher than the nominal agent in the nominal environment. Interestingly, the same behavior has been observed when robustifying DNN used for image classification as demonstrated in [20].

V-B2 Adversarial environment

In the adversarial environment, the nominal agent’s performance in both and projection attacks dropped significantly as anticipated. The nominally trained agent’s policy was trained in environments with no perturbations. The rewards for both and projection attacks are and respectively. Hence, both projection attacks successfully minimized the nominal agent’s reward. In both and attacks, we observed a high frequency of rewards obtained within the range of 1.0 and 1.5, which corresponds to scenarios where the lander landed but failed to turn off its engine.

For the robust agent, the performance of both and trained agent increased when compared to the nominally tested robust agent counterpart. The rewards for and trained agents are and respectively. Similar to counter-intuitive observations made earlier, we note that the expected results should be a decrease in rewards compared to the robust agent in the nominal environment. These counter intuitive results reveal an important characteristic of adversarial training defense schemes. We can expect that an agent that has been adversarially trained will perform well when tested in an adversarial environment, but at the cost of a slightly reduced performance when tested in nominal situations.

Although it is not a direct comparison, it is interesting to note that the robust agent trained with projected attacks in the adversarial environment outperforms the nominal agent in the nominal environment. This is likely because the nominal agent can only explore and maximize its reward with familiar trajectories seen during training. For the robust agent, the agent’s policy explored many more trajectories with the help of adversarial perturbations. Therefore, it has likely found other trajectories with much higher rewards that the nominal agent did not explore.

Vi Conclusions

Deep RL based controllers are increasingly popular as they demonstrate a potential for controlling complex CPS. Adversarial attacks on these controllers are emerging, which requires these controllers to be robustified against these attacks. In this work, we formulate the problem of robustifying a DRL agent as a robust optimization problem. We adversarially trained a DRL agent that is subjected to action space perturbations and demonstrate that it still performs robustly in the presence of actuator perturbations. In some cases, it even improved the performance of the agent in the absence of attacks. Hence, we show that it is beneficial to adversarially train a DRL agent. Future direction includes extending this work to different attack models and experimenting with transferability of attacks and defense results.

Acknowledgment

This work was supported in part by NSF grant CNS-1845969.

References

  • [1] O. Bastani, Y. Ioannou, L. Lampropoulos, D. Vytiniotis, A. Nori, and A. Criminisi (2016) Measuring neural net robustness with constraints. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.), pp. 2613–2621. External Links: Link Cited by: §II-B.
  • [2] V. Behzadan and A. Munir (2017) Vulnerability of deep reinforcement learning to policy induction attacks. In

    International Conference on Machine Learning and Data Mining in Pattern Recognition

    ,
    pp. 262–275. Cited by: §II-B.
  • [3] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, and F. Roli (2013) Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387–402. Cited by: §I, §II-B.
  • [4] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba (2016) Openai gym. arXiv preprint arXiv:1606.01540. Cited by: §IV-A.
  • [5] N. Carlini and D. Wagner (2017) Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57. Cited by: §I, §I, §II-B.
  • [6] J. M. Danskin (1966) The theory of max-min, with applications. SIAM Journal on Applied Mathematics 14 (4), pp. 641–664. Cited by: §I, §III-B.
  • [7] Y. Esfandiari, K. Ebrahimi, A. Balu, N. Elia, U. Vaidya, and S. Sarkar (2019)

    A saddle-point dynamical system approach for robust deep learning

    .
    arXiv preprint arXiv:1910.08623. Cited by: §I.
  • [8] H. Fawzi, P. Tabuada, and S. Diggavi (2014-06) Secure estimation and control for cyber-physical systems under adversarial attacks. IEEE Transactions on Automatic Control 59 (6), pp. 1454–1467. External Links: Document, ISSN Cited by: §I, §II-A.
  • [9] I. J. Goodfellow, J. Shlens, and C. Szegedy (2014) Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Cited by: §I, §I, §II-B, §III-B.
  • [10] S. Gu and L. Rigazio (2014) Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068. Cited by: §II-B.
  • [11] A. Havens, Z. Jiang, and S. Sarkar (2018) Online robust policy learning in the presence of unknown adversaries. In Advances in Neural Information Processing Systems, pp. 9916–9926. Cited by: §I, §II-C.
  • [12] W. Hendricks (1972) The stationary distribution of an interesting markov chain.

    Journal of Applied Probability

    9 (1), pp. 231–233.
    Cited by: §III-A.
  • [13] X. Huang and J. Dong (2018-12) Reliable control policy of cyber-physical systems against a class of frequency-constrained sensor and actuator attacks. IEEE Transactions on Cybernetics 48 (12), pp. 3432–3439. External Links: Document, ISSN Cited by: §II-A.
  • [14] H. Jafarnejadsani, H. Lee, N. Hovakimyan, and P. Voulgaris (2018-12) A multirate adaptive control for mimo systems with application to cyber-physical security. In 2018 IEEE Conference on Decision and Control (CDC), Vol. , pp. 6620–6625. External Links: Document, ISSN Cited by: §II-A.
  • [15] J. Kim, G. Park, H. Shim, and Y. Eun (2016-12) Zero-stealthy attack for sampled-data control systems: the case of faster actuation than sensing. In 2016 IEEE 55th Conference on Decision and Control (CDC), Vol. , pp. 5956–5961. External Links: Document, ISSN Cited by: §II-A.
  • [16] V. R. Konda and J. N. Tsitsiklis (2000) Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008–1014. Cited by: §IV-B.
  • [17] A. Kurakin, I. Goodfellow, and S. Bengio (2016) Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236. Cited by: §I, §I, §II-B.
  • [18] N. Lazic, C. Boutilier, T. Lu, E. Wong, B. Roy, M. Ryu, and G. Imwalle (2018) Data center cooling using model-predictive control. In Advances in Neural Information Processing Systems, pp. 3814–3823. Cited by: §I.
  • [19] X. Y. Lee, S. Ghadai, K. L. Tan, C. Hegde, and S. Sarkar (2019) Spatiotemporally constrained action space attacks on deep reinforcement learning agents. arXiv preprint arXiv:1909.02583. Cited by: §I, §III-C, §III-C.
  • [20] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2017) Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. Cited by: §I, §II-B, §III-B, §III-B, §V-B1.
  • [21] A. Mandlekar, Y. Zhu, A. Garg, L. Fei-Fei, and S. Savarese (2017) Adversarially robust policy learning: active construction of physically-plausible perturbations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932–3939. Cited by: §II-C.
  • [22] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. (2015) Human-level control through deep reinforcement learning. Nature 518 (7540), pp. 529. Cited by: §III-A.
  • [23] A. Mustafa and H. Modares (2018) Attack analysis and resilient control design for discrete-time distributed multi-agent systems. CoRR abs/1801.00870. External Links: Link, 1801.00870 Cited by: §I, §II-A.
  • [24] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami (2017) Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506–519. Cited by: §I, §II-B.
  • [25] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami (2016) Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582–597. Cited by: §II-B.
  • [26] A. Pattanaik, Z. Tang, S. Liu, G. Bommannan, and G. Chowdhary (2018) Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2040–2042. Cited by: §II-C.
  • [27] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta (2017) Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2817–2826. Cited by: §II-C.
  • [28] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015) Trust region policy optimization. In International conference on machine learning, pp. 1889–1897. Cited by: §III-A.
  • [29] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §III-A.
  • [30] U. Shaham, Y. Yamada, and S. Negahban (2018) Understanding adversarial training: increasing local stability of supervised models through robust optimization. Neurocomputing 307, pp. 195–204. Cited by: §II-B.
  • [31] C. Sitawarin, A. N. Bhagoji, A. Mosenia, M. Chiang, and P. Mittal (2018) Darts: deceiving autonomous cars with toxic signs. arXiv preprint arXiv:1802.06430. Cited by: §I.
  • [32] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus (2013) Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199. Cited by: §I, §II-B.
  • [33] K. L. Tan, S. Poddar, S. Sarkar, and A. Sharma (2019) Deep reinforcement learning for adaptive traffic signal control. In ASME 2019 Dynamic Systems and Control Conference, Cited by: §I.
  • [34] C. Tessler, Y. Efroni, and S. Mannor (2019) Action robust reinforcement learning and applications in continuous control. arXiv preprint arXiv:1901.09184. Cited by: §I, §II-C.
  • [35] H. Van Hasselt, A. Guez, and D. Silver (2016) Deep reinforcement learning with double q-learning. In

    Thirtieth AAAI conference on artificial intelligence

    ,
    Cited by: §III-A.
  • [36] D. Wei and K. Ji (2010-08) Resilient industrial control system (rics): concepts, formulation, metrics, and insights. In 2010 3rd International Symposium on Resilient Control Systems, Vol. , pp. 15–22. External Links: Document, ISSN Cited by: §I.
  • [37] H. Zhang, H. Jiang, Y. Luo, and G. Xiao (2017-05) Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method. IEEE Transactions on Industrial Electronics 64 (5), pp. 4091–4100. External Links: Document, ISSN Cited by: §I.
  • [38] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan (2019) Theoretically principled trade-off between robustness and accuracy. CoRR abs/1901.08573. External Links: Link, 1901.08573 Cited by: §I.
  • [39] K. Zhou, J. C. Doyle, K. Glover, et al. Robust and optimal control. Vol. 40. Cited by: §I.