Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysis

07/29/2022
by   Tao Li, et al.
7

Meta reinforcement learning (meta RL), as a combination of meta-learning ideas and reinforcement learning (RL), enables the agent to adapt to different tasks using a few samples. However, this sampling-based adaptation also makes meta RL vulnerable to adversarial attacks. By manipulating the reward feedback from sampling processes in meta RL, an attacker can mislead the agent into building wrong knowledge from training experience, which deteriorates the agent's performance when dealing with different tasks after adaptation. This paper provides a game-theoretical underpinning for understanding this type of security risk. In particular, we formally define the sampling attack model as a Stackelberg game between the attacker and the agent, which yields a minimax formulation. It leads to two online attack schemes: Intermittent Attack and Persistent Attack, which enable the attacker to learn an optimal sampling attack, defined by an ϵ-first-order stationary point, within 𝒪(ϵ^-2) iterations. These attack schemes freeride the learning progress concurrently without extra interactions with the environment. By corroborating the convergence results with numerical experiments, we observe that a minor effort of the attacker can significantly deteriorate the learning performance, and the minimax approach can also help robustify the meta RL algorithms.

READ FULL TEXT

page 15

page 18

research
03/31/2022

Robust Meta-Reinforcement Learning with Curriculum-Based Task Sampling

Meta-reinforcement learning (meta-RL) acquires meta-policies that show g...
research
08/07/2022

A Game-Theoretic Perspective of Generalization in Reinforcement Learning

Generalization in reinforcement learning (RL) is of importance for real ...
research
03/27/2020

Adaptive Reward-Poisoning Attacks against Reinforcement Learning

In reward-poisoning attacks against reinforcement learning (RL), an atta...
research
09/02/2020

Yet Meta Learning Can Adapt Fast, It Can Also Break Easily

Meta learning algorithms have been widely applied in many tasks for effi...
research
02/25/2019

Adversarial Reinforcement Learning under Partial Observability in Software-Defined Networking

Recent studies have demonstrated that reinforcement learning (RL) agents...
research
04/08/2023

Evolving Reinforcement Learning Environment to Minimize Learner's Achievable Reward: An Application on Hardening Active Directory Systems

We study a Stackelberg game between one attacker and one defender in a c...
research
07/06/2020

Meta-Learning through Hebbian Plasticity in Random Networks

Lifelong learning and adaptability are two defining aspects of biologica...

Please sign up or login with your details

Forgot password? Click here to reset