Motivation: Safety is a desirable and necessary design constraint for any control system; specially when operated in a shared environment with a decision maker (DM). Arguably, most environments have associated spatial risks, whose source can vary from hard constraints (e.g. moving obstacles) to softer constraints (e.g. wind conditions). Different DMs can perceive these risks differently, leading to notions of perceived risks and perceived safety from these risks.
It is well known from psychophysics  and behavioral economics  research that humans as DMs have fundamental non-linear perception leading to non-rational decision making in risky situations. In such cases, existing methods assuming perfect knowledge or rational and coherent treatment (as in expected risk and Conditional Value at Risk (CVaR)) of risks may not suffice, which can lead to loss of trust or discomfort among DMs. This motivates the need of richer and more inclusive modeling of risk perception to capture a variety of DMs and use them for safe control design. This work aims to bridge the gap between behavioral decision making and safety using Cumulative Prospect Theory (CPT) as a risk perception model, and Control Barrier Functions (CBFs) for safe control design.
Related Work: Safe control system design has been tackled using various frameworks such as artificial potential functions , barrier certificates  and, more recently, control barrier functions (CBFs) . CBFs have gained popularity due to their Lyapunov-like properties, rigorous safety guarantees and ease of application. They have been successfully used in optimization , stabilization  and data-driven control frameworks . CBFs were traditionally used in static scenarios, more recently, they have been used to deal with moving obstacles  and multi-agent systems .
Uncertainty has been mainly handled using robustness measures , stochastic control , or chance constraints . Very few works have considered the notion of risk perception explicitly in a control system [13, 14]. All these works use CVaR to quantify risk perception, which only captures linear and rational risk-averse behavior. CPT on the other hand is a more expressive (see ), non-linear and non-rational perception theory which is yet to be applied in the context of safety for a control system. Moreover, CPT has been successfully used in engineering applications like path planning , traffic routing , and network protection .
Contributions: We first adapt the notion of non-rational risk perception to the context of safety for control systems. With this, we capture a larger spectrum of DM’s risk profile, extending the existing literature. We support this claim theoretically by defining the notion of “inclusiveness” and proving that CPT is the most inclusive risk perception model out of the other popular models: CVaR and ER. We then use the CPT value function to construct a class of CBFs to guarantee safety according to a DM’s perceived risk and define the notion of perceived safety. Additionally, we find sufficient geometric conditions on the control input to maintain the validity of our proposed RPA CBF and compare them among the three risk perception models (RPMs). Then, we design a QP-based RPA controller to guide an agent to a desired goal safely w.r.t. perceived risks. Thus we extend the literature with more inclusive safe control design. Practically, we consider 2D simulations with moving obstacles and show the effectiveness of the proposed RPA controller along with the practical translation of the inclusiveness heirarchy.
This work provides a framework to incorporate and compare a wide range of RPMs to generate a variety of RPA controls. We would also like to clarify that the validation of CPT models using user studies for typical control scenarios is beyond the scope of this work.
Ii Risk perception formalism and problem setup
Here, we introduce some notation111The
Euclidean norm in is denoted by . We use as
the expectation operator on a random variable. The set
as the expectation operator on a random variable. The setis a ball of radius centered at . and a formal notion of risk perception, starting with a concise description of CPT and CVaR (see  and  for more details). Later, we describe our problem statement.
Risk Perception: By risk perception, we refer to the notion of attaching a value (risk) to a random cost output. Formally, let
be a discrete sample space endowed with a probability distribution
. We model environmental cost via a real-valued, discrete random variable, taking possible values, , , and such that , with . We Let be the set of such random cost variables and a value function which associates a value (risk) to a random cost variable.
A value function can be defined in many ways, resulting in different risk perceptions. Here, an RPM is characterized as a parameterized family of value functions. In what follows, we consider three popular RPMs: Expected Risk (ER) Conditional Value at Risk222The CVaR model uses a class of value functions parameterized by to represent expectation over a fraction () of the worst-case outcomes. Thus the CVaR value with is the worst-case outcome of , . While, with CVaR value equals ER (). (CVaR)  and Cumulative Prospect Theory (CPT) .
CPT captures non-rational decision making, and was introduced in [18, 20]. In CPT, outcomes are first weighed using a non-linear utility function , with , modeling a DM’s perceived cost. The parameters
represent “risk aversion” and “risk sensitivity”, respectively. In addition, a non-linear probability weighing function, given by and , is used to model uncertainty perception. Here, uncertainty sensitivity is tuned via the parameters . CPT also suggests that probabilities are perceived via decision weights , which are calculated in a cumulative fashion. Defining a partial sum function as , and , we have
With this, assigning the parameter for CVaR and for CPT, the value functions of ER (), CVaR () and CPT () of a DM are defined as:
In CPT, can be varied to generate different value functions pertaining to various risk profiles of DMs (from risk-taker to risk-averse). We refer to [15, 18] for more details on the parameter choices in CPT. Risky Environment: Consider a compact state space containing dynamic spatial sources of risk at and an agent or robot at a state . The relative state space is . Our starting point is an uncertain cost field , that aims to quantify objectively the (negative) consequences of being at relative to a known risk source at . More precisely, is a discrete RV which can take possible values, , for . We assume that
has associated mean and standard deviation functionsand , respectively. We assume that are continuously differentiable in their domains. Given , an associated spatial-risk function is given by , , where belongs to any of the previous RPMs defined in (1) above. When clear from the context, we will identify . The larger is at , the higher the perceived risk of being at
Dynamic systems: We aim to control an agent modeled as a control-affine dynamic system:
where and and are locally Lipschitz. We also consider a dynamic risk
with a locally Lipschitz . We focus on moving obstacles as the source of risk, but the approach can be extended to other scenarios. We also assume that a asymptotically stable controller has been designed to guide the agent to a goal state in the absence of risk sources. We wish to drive the agent to a goal safely, while avoiding risky areas. Formally, we define safety considering a perceived spatial risk function as follows:
We now state the problems we address in this work:
(RPA safe sets) Given a risky environment , endowed with an uncertain cost , design perceived safety sets considering RPMs from (1). Characterize and constrast the properties of these sets among the three RPMs.
(RPA safe controls) Under previous conditions, design a controller , nominally deviating from a stable state feedback controller , such that the agent reaches the goal safely (Definition 1) and examine feasibility of .
Iii Perceived Safety using various RPMs
This section compares various RPMs, solving Problem 1. Given an uncertain field cost , we apply the different risk perception models (see Section II) to obtain the corresponding fields, . With this, let us define the following sets:
In particular, these sets depend on the choice of from (1). Given , we define the range set associated with wrt as the set 333When clear from the context, we will just denote .. Fix a model and a risk source at . The total safe set of wrt is given as (resp. the total risky set of wrt is ). Thus, given , the set (resp. covers all the states in that safe (resp. unsafe) according to a RPM .
(Inclusiveness and Strict Inclusiveness). Consider two RPMs and , a threshold , and a risk source at . Let the sets and be the total safe and risky sets of and wrt and a spatial cost , respectively. We say that is more inclusive than () if either and holds, or and holds, for all and costs . If and both hold, then is strictly more inclusive than ().
In particular, if , then results into a wider range of safety and risky sets for a given environment than .
Now we compare the inclusiveness of CPT, CVaR and ER via their respective value functions. We start by comparing the range space of these RPMs.
Consider a threshold , a risk source at , and two RPMs with range spaces , respectively. If , and if there exists an such that or for any , and any , then . In addition, if there are such that and , , and any , then .
Fix . Since , , there is s.t. , . Thus, and . Assume s.t. or hold for all . This implies either or . Inclusiveness follows from Definition 2. In parallel, .
Consider the CPT, CVaR and ER risk models, with associated range sets and . Then, it holds that , .
Fix . Note that . By choosing with and we have , . Note that only if then for all . When , with any other valid choice of parameters in CVaR we obtain . We can find such that , . Hence, and .
For CVaR, and , where is the worst-case outcome of . Since increases in , . Choosing for , leads to . Taking , with , we get ; hence, .
The previous results now lead to the following.
Let be a discrete random field cost. Consider the ER, CVaR and CPT risk perception models with risk value functions , , and , respectively. For any threshold and risk source , and holds. If the cost outcomes are strictly lower-bounded by 1, then and . If in fact , then .
From Lemma 2, and . As in Lemma 2, take , for some . Choosing , with , we get , for any , and . Thus, from Lemma 1, we have and . Now assume for all . Taking with , we get for any and . Now, take with , we have . Since , , then , implying ando , . From Lemma 1, we get and .
Finally, assume . There is such that . Since the lower bound of is , there is no s.t. . Hence from Lemma 1 and the first part of this result, we get . The above arguments show CPT can produce a larger variety of safe and risky sets leading to richer risk perception. This is illustrated via simulations in Section V.
Additional properties of RPMs: In addition to the notion of inclusiveness, we now characterize the versatility of a RPM in the context of perceived safety.
(Versatility of a RPM). Consider a compact space , a risk source , and a discrete random field cost , with range in . Let be a compact interval. An RPM is said to be versatile if for any for a given . If , then is most versatile in .
The above definition implies that an RPM is versatile, if it has a risk-perception functionthat perceives any states having costs less than as safe, . Further, is most versatile when it contains risk-perception functions that capture a range of perceptions from most risk averse (only states having costs are safe) to the least risk-sensitive (every state including states having the highest cost as safe). With this, we will look at versatility of the three RPMs.
Consider a compact space , with a risk source , and associated discrete random field cost . Then, CPT can capture the most risk averse perception, i.e. the set is considered safe.
Under the setting of Lemma 3, CPT can capture the least risk sensitive perception (the set is considered safe), if , for , and over . Consequently, CPT is most versatile in .
For with we have . Now, choosing , since , , and , we get . Thus, from (4a) and Definition 3, the first result follows. Take now and . Observe that is continuous in and is continuous in . By the intermediate value theorem s.t. , and a s.t. . Hence, from Lemma 3, CPT is most versatile in .
Under the assumptions of Lemma 3, with and , CVaR is versatile and ER is versatile. Hence neither are most versatile RPMs.
This result trivially follows from the range spaces and in the proof of Lemma 2.
Iv Control design with Risk-Perception-Aware-CBFs
Definition 4 (Rpa-Cbf)
where is a extended class function. Since is non-decreasing, implies and from (4a), . Thus, indicates that is perceived as safe w.r.t. .
The RPA control input can be now computed via:
The above problem captures the notion of minimally modifying a stable controller to ensure safety of the system. Next we will analyze the feasibility conditions for the proposed controller and compare it across the proposed models.
Feasibility analysis and comparison: We first describe a
construction of finite outcomes of from and
called “truncated-Gaussian cost” which will be used for
analysis. Assume that is distributed as a truncated
Gaussian444This truncation reassigns the probability
mass s.t. using an appropriate
. Then, given , we approximate by means of discrete values ,
, with probability calculated from the
CDF of at each . That is, , and , for .
Now, we show conditions on for the set to be
non-empty for a given risk function . We first define a few
constants and variables to help us compare the feasibility conditions
of the three RPMs. Let be the relative
angle555 recall angle between two vectors
recall angle between two vectorsis given by between and , and and . Now define , and . Also we define constants , and . Consider , and . The following holds.
For first part, rearranging terms in (8) we get:
where . For the RPA-CBF to be valid, the set needs to be non-empty. Due to the dynamics of the agent and obstacle, and have dynamics:
Using the chain rule, we get the time derivative of:
For the last part, the expressions are obtained by substituting the respective risk functions and evaluating the partial derivatives and (part of ). Thus we need to show the following hold true:
For ER we get and . For CVaR, since
is assumed to belong to a truncated Gaussian distribution, we can use the closed form expression of CVaR (13) for a Gaussian distribution to calculate the partials and .
From (13), it is easy to see that CVaR is linear in and . With this, we get and .
For CPT, the expression is obtained by substituting the CPT risk function and evaluating the partial derivatives and . Constructing truncated Gaussian costs from and , we get outcomes and corresponding probabilities resulting in constant throughout. In this way, from (1c), the CPT value of a random cost with mean and is given by:
With this expression, we can proceed to calculate the partial derivatives and . From (14), we get
From (8), observe that the RHS is independent of and the LHS is independent of and the RPM. This separation makes it easier to compare various RPMs and their associated feasibility conditions.
Next, we remark on the uncertainty perception of each RPM, which will be used in the subsequent proposition to compare the size of control sets respectively generated by each of the RPMs.
Remark 1 (Uncertainty perception among RPMs)
The ER model is insensitive to uncertainty as . In this way, CVaR is averse to uncertainty as for all . With CPT, can be tuned to get both uncertainty insensitive and uncertainty averse behavior, additionally, it can also produce uncertainty liking behavior (when ). 666The first two properties follow by choosing as in Theorem 1. The latter property can be obtained by tuning the uncertainty perception parameters and . Since the chosen distribution is symmetric, we can examine the relation between and for . If we have (for example when is concave) or (when is convex), then we have , or , respectively. A concave () implies that unlikely outcomes are viewed to be more probable compared with the more certain outcomes. This results into an “uncertainty averse behavior”, which is reflected in the positive sign of . Conversely, a convex () leads to an “uncertainty liking behavior” with ..
We finally compare the the flexibility provided by each model via the corresponding control sets .
In order to compare the feasibility of the sets from (5) for the three RPMs, we can compare their respective feasibility conditions (8). Consider , and . Since the LHS in (8) remains the same for any RPM and its parameter choice, to prove the proposition, it is sufficient to show that and . These inequalities follow from the choice of