Log In Sign Up

RTAMT: Online Robustness Monitors from STL

by   Dejan Nickovic, et al.

We present RTAMT, an online monitoring library for Signal Temporal Logic (STL) and its interface-aware variant (IA-STL), providing both discrete- and dense-time interpretation of the logic. We also introduce RTAMT4ROS, a tool that integrates RTAMT with Robotic Operating System (ROS), a common environment for developing robotic applications. We evaluate RTAMT and RTAMT4ROS on two robotic case studies.


page 1

page 2

page 3

page 4


Study of Signal Temporal Logic Robustness Metrics for Robotic Tasks Optimization

Signal Temporal Logic (STL) is an efficient technique for describing tem...

Context-Aware Temporal Logic for Probabilistic Systems

In this paper, we introduce the context-aware probabilistic temporal log...

Arithmetic-Geometric Mean Robustness for Control from Signal Temporal Logic Specifications

We present a new average-based robustness score for Signal Temporal Logi...

Online Monitoring of Metric Temporal Logic using Sequential Networks

Metric Temporal Logic (MTL) is a popular formalism to specify patterns w...

Vision-based Obstacle Removal System for Autonomous Ground Vehicles Using a Robotic Arm

Over the past few years, the use of camera-equipped robotic platforms fo...

Stochastic Robustness Interval for Motion Planning with Signal Temporal Logic

In this work, we present a novel robustness measure for continuous-time ...

Creative Action at a Distance: A Conceptual Framework for Embodied Performance With Robotic Actors

Acting, stand-up and dancing are creative, embodied performances that no...

1 Introduction

Robotic applications are complex autonomous cyber-physical systems (CPS). Robotic Operating System (ROS) [12] provides a meta-operating system that helps development of robotic applications. Verification remains a bottleneck, as existing techniques do not scale to this level of complexity, thus making static safety assurance a very costly, if not impossible, activity. Run-time assurance (RTA) is an alternative approach for ensuring the safe operation of robotic CPS that cannot be statically verified. RTA allows the use of untrusted components in a system that implements a safe fallback mechanism for (1) detecting anomalies during real-time system operations and (2) invoking a recovery mechanism that brings the system back to its safe operation. Runtime verification (RV) provides a reliable and rigorous way for finding violations in system executions and consequently represents a viable solution for the monitoring RTA component.

Formal specifications play an important role in RV and enable formulating system properties. Signal Temporal Logic (STL) [9] is a formal specification language used to describe CPS properties. It admits robustness semantics that measure how far is an observed behavior from satisfying/violating a specification.

We introduce rtamt111, an online STL monitoring library. rtamt supports standard STL and its interface-aware extension (IA-STL) [5] as specification languages. It provides automated generation of online robustness monitors from specifications under both discrete and continuous interpretation of time. We also present rtamt4ros222, an extension that integrates rtamt to ROS, thus enabling the use of specification-based RV methods in robotic applications. We assess the library on two robotic applications.

1.0.1 Related Work

Several tools support offline monitoring of STL with qualitative (AMT2.0 [10]) and quantitative semantics (S-TaLiRo [1] and Breach [4]). Reelay [13] implements past Metric Temporal Logic (MTL) monitors over discrete and continuous time and with qualitative and quantitative semantics. RVROS [7] is a specification-agnostic monitoring framework for improving safety and security of robots using ROS. To the best of our knowledge, rtamt/rtamt4ros is the only tool that implements online robustness STL monitors with both future and past operators and ROS support.

2 RTAMT Design and Functionality

robustnessIA-STL IA-STL input signalrtamt
Figure 1: RTAMT architecture.

The main functionality of rtamt is the automatic generation of online robustness monitors from declarative specifications. Given an input signal in the form of a sequence of (time, value) pairs and a specification, rtamt computes at different points in time how robust is the observed signal to the specification, i.e. how far is it from satisfying or violating it. The library consists of 3 major parts: (1) specifications expressed in a declarative specification language, (2) a front-end with an Application Programming Interface (API) to parse specifications and generate the monitor, and (3) a back-end that implements the actual evaluation algorithm used by the monitor. The rtamt library uses a modular architecture depicted in Figure 1. It uses ANTLR4 parser generator to translate textual (IA-)STL specifications into an abstract parse tree (APT) data structure used to build the actual monitor. The front-end implements the Application Programming Interface (API) and the pre-processing steps such as the translation of bounded-future (IA-)STL to past (IA-)STL in Python. The back-end implements the monitoring algorithms in Python (for discrete-time and dense-time interpretation) and C++ (for discrete-time interpretation). The library is compatible with both Python 2.7 and 3.7.

Specification language in rtamt is STL with infinity-norm quantitative semantics [3]. The library supports four variants of the specification language – standard STL and interface-aware STL [5] interpreted over discrete and dense time. IA-STL extends STL with an input/output signature of the variables and provides two additional semantic interpretations: (1) output robustness and (2) input vacuity. Output robustness measures robustness of output signals with respect to some fixed input. Input vacuity measures how vacuously is a specification satisfied with input signals only. rtamt accepts as input bounded-future STL (bfSTL) that restricts the use of the future temporal operators (eventually, always and until) to bounded intervals. See Appendix 0.A for (IA-)STL definitions.

Parsing and preprocessing follows a two-step procedure. The first step consists in translating the specification given in a textual form to an abstract parse tree (APT). The translation uses ANTLR4 to generate a Python parser for the (IA-)STL grammar. This translation is stll not suitable for online monitors – the specification may have future temporal operator that would require clair-voyant monitoring capability. Hence, we implement the pastification procedure [8] that translates the bfSTL formula into an equi-satisfiable past STL formula , which uses only past temporal operators and postpones the formula evaluation from time index , to the end of the (bounded) horizon where all the inputs necessary for computing the robustness degree are available (see Appendix 0.B).

Monitoring consists of evaluating in online fashion the past specification according to its quantitative semantics, interpreted in discrete or dense time 333Due to pastification, rtamt only needs to evaluate past temporal operators..

Discrete-time monitors follow a time-triggered approach in which sensing of inputs and output generation are done at a periodic rate. This choice is motivated by [6], which shows that by weakening/strengthening real-time specifications, discrete-time evaluation of properties preserves important properties of dense-time interpretation. This approach admits an upper bound on the use of computation resources. rtamt implements two back-ends for STL monitors – one in Python (for rapid prototyping) and one in C++ (for efficiency). rtamt uses Boost.Python library to integrate the Python front-end with the C++ backend.

Dense-time monitors follow an event-driven approach. Their implementation combines the incremental evaluation approach from [11] with the optimal streaming algorithm to compute the min and max of a numeric sequence over a sliding window from [2]. Unlike their discrete-time counterparts, continuous-time monitors do not have bounds on memory requirements.

Figure 2: Integration of RTAMT to ROS.

Integration of RTAMT to ROS ROS supports several messaging approaches, including the subscriber and publisher pattern. A publisher categorizes a message into a class (called topic in ROS) and sends it withoutknowing who will read the message. A subscriber subscribes to a topic and receives its associated messages, without knowing who sent the message444Unless the publisher encodes its identity into the message itself.. The messages are received and processed in callback() functions. Common ROS applications associate a callback() function per subscribed variable.

rtamt4ros, illustrated in Figure 2, integrates rtamt into ROS using rospy. The integration is non-intrusive and provides the user with a generic and transparent monitoring solution for (IA-)STL specifications. The ROS system under observation is implemented with ROS nodes, which interact by publishing and receiving ROS messages on dedicated topics. To publish values of a variable of type on a topic , ROS node associates and to . Similarly, we declare in the STL specification variables that we want to monitor, declare their types and associate them to ROS subscription/publication topics using annotations. Variable names, their types and associated topics are specification-dependent. rtamt4ros implements a dynamic subscription/publication mechanism that uses the concepts of introspection and reflection (the ability to passively infer the type of an object and actively change its properties at runtime). Given a (IA-)STL specification, rtamt4ros infers all the specification variables and dynamically creates their associated subscribers and publishers. The use of reflection allows us to associate a single callback() function to all specification variables, by passing the variable object as an argument to the function. We use the callback() function only to collect input data and the main ROS loop to make robustness monitor updates.

3 Experiments and Use Scenario

We now present experiments performed using rtamt and rtamt4ros. We apply rtamt and rtamt4ros on two ROS case studies: Simple Two Dimensional Simulator (STDR) and Toyota’s Human Support Robot (HSR) platform [14, 15]. We use the STDR example to show step-by-step usage of the rtamt and rtamt4ros for online monitoring of robotic applications. We note that rtamt is versatile and could be used for instance for for offline monitoring and non-robotic applications (see Appendix 0.C). We then evaluate the computation time requirements of the library. The experiments were performed on a Dell Latitude 7490 with an i7-8650U processor and 16 GB of RAM, running Ubuntu 16.04 on a virtual machine.

Online monitoring of robotic applications: STDR is a ROS-compliant environment for easy multi-robot 2D simulation (see Figure 4). We use a simple robot controller with commands encoded as ROS Twist messages that expresses velocity in free space consisting of its linear and angular parts. The robot state is encoded as a ROS Odometry

message that represents an estimate of the position (pose) and velocity (twist) in free space. We than use the

rtamt4ros and rtamt to monitor its low-level requirement stating that every step in the command must be followed by the observed response. The specification spec.stl requires that at all times the distance between the linear velocity on the dimension of the command and the robot is smaller than . The user first needs to import data types used in the specification (lines 1-3). Then, it declares variables used in specification, with their data type and (optionally) their input/output signature (lines 4, 6 and 8). Special comments in lines 5 and 7 are annotations that provide additional information about variables - in this case they associate variables to ROS topics. Finally, line 9 defines the IA-STL property.

1from geometry_msgs.msg import Twist
2from nav_msgs.msg import Odometry
3from rtamt_msgs.msg import FloatMessage
4input Twist cmd
5@ topic(cmd, robot0/cmd_vel)
6output Odometry robot
7@ topic(res, robot0/odom)
8output FloatMessage out
9out.value = always(abs(cmd.linear.x - robot.twist.twist.linear.x) <= 0.5)

To monitor the IA-STL specification spec.stl with rtamt/rtamt4ros , it suffices to run the following command in the ROS environment.

1roscore rtamt4ros --stl spec.stl --period 100 --unit ms
Figure 3: STDR simulator.
Figure 4: HSR service robotics application.

HSR is a robot with 8 degrees of freedom (DoF), combining 3 DoF of its mobile base, 4 DoF of the arm and 1 DoF of the torso lift (see Figure 

4). The robot is equipped with ROS modules for localization, path planning and obstacle avoidance. We used this example to experiment with system-level properties in a multi-agent environment. We were interested in particular in monitoring the following requirements: (1) no-collision requirement stating that two robots are never closer than some distance from each other, and (2) when robot 2 is closer than distance from robot 1, then robot 2 two goes in at most seconds within distance of some location . For this industrial application, we present an abstracted formalization of the above requirements.

1out1 = always (abs(rob1.pos - rob2.pos) < d)
2out2 = abs(rob1.pos - rob2.pos) < d implies
3        eventually[0:T](rob1.pos - L) < d

This experiment demonstrates the use of the library in a sophisticated ROS/Gazebo environment in an industrial case study. The addition of monitors is orthogonal to the development of the application and the monitors are non-intrusive.

bound C++ (s) Python (s)
100 0.00014 0.00023
1k 0.0002 0.00085
10k 0.0008 0.029
100k 0.0047 0.31
1M 0.046 72
Table 1: Timing requirement per single monitor update.

Timing figures: For online monitors, the most important quantitative measure is the computation time of a single monitoring update step. We compared the difference in timing requirements between the C++ and the Python implementation of the discrete-time monitoring algorithm. We used for the experiment the STL specification out = always[0:k] (a+b > -2) where is the upper bound on the timing modality of the always operator that we varied between and million. Table 1 summarized the results of the experiment. The outcomes clearly demonstrate the efficiency of the C++ back-end, especially for large upper bounds in temporal modalities.

4 Conclusions

In this paper, we presented rtamt  a library for generating online monitors from declarative specifications and rtamt4ros, its ROS extension, demonstrating their usability and versatility two robotic case studies.


  • [1] Y. Annpureddy, C. Liu, G. E. Fainekos, and S. Sankaranarayanan (2011) S-taliro: A tool for temporal logic falsification for hybrid systems. In Tools and Algorithms for the Construction and Analysis of Systems - 17th International Conference, TACAS 2011, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2011, Saarbrücken, Germany, March 26-April 3, 2011. Proceedings, pp. 254–257. Cited by: §1.0.1.
  • [2] A. Donzé, T. Ferrère, and O. Maler (2013) Efficient robust monitoring for STL. In CAV 2013, pp. 264–279. Cited by: §2.
  • [3] A. Donzé and O. Maler (2010) Robust satisfaction of temporal logic over real-valued signals. In FORMATS 2010, pp. 92–106. Cited by: §2.
  • [4] A. Donzé (2010) Breach, A toolbox for verification and parameter synthesis of hybrid systems. In CAV 2010, pp. 167–170. Cited by: §1.0.1.
  • [5] T. Ferrère, D. Nickovic, A. Donzé, H. Ito, and J. Kapinski (2019) Interface-aware signal temporal logic. In HSCC 2019, pp. 57–66. Cited by: §1, §2.
  • [6] T. A. Henzinger, Z. Manna, and A. Pnueli (1992) What good are digital clocks?. In ICALP 1992, pp. 545–558. Cited by: §2.
  • [7] J. Huang, C. Erdogan, Y. Zhang, B. Moore, Q. Luo, A. Sundaresan, and G. Rosu (2014-09) ROSRV: runtime verification for robots. In RV 2014, pp. 247–254. Cited by: §1.0.1.
  • [8] O. Maler, D. Nickovic, and A. Pnueli (2007) On synthesizing controllers from bounded-response properties. In CAV 2007, pp. 95–107. Cited by: §2.
  • [9] O. Maler and D. Nickovic (2004) Monitoring temporal properties of continuous signals. In FORMATS/FTRTFT 2004, pp. 152–166. Cited by: §1.
  • [10] D. Nickovic, O. Lebeltel, O. Maler, T. Ferrère, and D. Ulus (2018) AMT 2.0: qualitative and quantitative trace analysis with extended signal temporal logic. In TACAS 2018, pp. 303–319. Cited by: §1.0.1.
  • [11] D. Nickovic and O. Maler (2007) AMT: A property-based monitoring tool for analog systems. In FORMATS 2007, pp. 304–319. Cited by: §2.
  • [12] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng (2009)

    ROS: an open-source robot operating system

    In ICRA workshop on open source software, Vol. 3, pp. 5. Cited by: §1.
  • [13] D. Ulus (2019) Online monitoring of metric temporal logic using sequential networks. CoRR abs/1901.00175. External Links: Link, 1901.00175 Cited by: §1.0.1.
  • [14] T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Y. Asahara, and K. Murase (2018) Development of the research platform of a domestic mobile manipulator utilized for international competition and field test. In IROS 2018, pp. 7675–7682. Cited by: §3.
  • [15] T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Y. Asahara, and K. Murase (2019) Development of human support robot as the research platform of a domestic mobile manipulator. ROBOMECH Journal 6 (1), pp. 4. Cited by: §3.

Appendix 0.A Signal Temporal Logic

We consider Signal Temporal Logic () with both past and future operators with quantitative semantics and interpreted over discrete time. Let be a set of real-valued variables. A valuation For maps a variable to a real value. Given , we denote by the set of valuations over variables in . A signal over is a sequence of (time, valuation) pairs, where and for all in . We note that for discrete-time interpretation

. For dense-time interpretation, we assume constant interpolation.

The syntax of a STL formula over is defined by the following grammar:

where and is of the form or such that and are in and .

The quantitaive semantics of a STL formula with respect to a signal is described via the robustness degree indicating how far is the signal from satisfying or violating the specification at time index .

From this basic definition of STL we use standard rules to derive the other Boolean and temporal operators:

We consider in this paper two subsets of STL

  • Past Signal Temporal Logic (pSTL) that forbids the use of , and

  • Bounded-Future Signal Temporal Logic (bfSTL) that restricts the use of to intervals with both and in .

0.a.1 Interface-aware Signal Temporal Logic

In this section, we consider STL with input/output signature. Given a set of real-valued variables, let be two subsets of such that . Let be an STL formula and a signal trace. We define the -robustness relative to , denoted by induction, where the only difference from the definition of is the case on numeric predicates:

where for all , if , otherwise.

An interface-aware signal temporal logic (IA-STL ) specification over is a tuple , where , and is a STL formula. We define two different notions of relative reobustness for IA-STL specifications: (1) output robustness, denoted , as the -robustness relative to , and (2) input vacuity, denoted , the -robustness relative to .

Appendix 0.B From Bounded-Future STL to Past pSTL 

Monitoring specifications with future temporal formulas is challenging because the evaluation at time index may depend on the observed inputs at some time indices . bfSTL specifications have a bounded future horizon that can be syntactically computed from the formula structure. For such specifications, the online monitoring challenge can be addressed by postponing the formula evaluation from time index , to the end of the (bounded) horizon where all the inputs necessary for computing the robustness degree are available. In this section, we briefly sketch this procedure, called pastification.

We first define the temporal depth of as the syntax-dependent upper bound on the actual depth of the specification that is inductively computed as follows:

The pastification operation on the formula with past and bounded future and its bounded horizon is defined recursively as follows:

Formally, we say that for an arbitrary formula , signal and time index , .

Appendix 0.C Other Use Scenarios

In this paper, we focused on the use of rtamt combined with rtamt4ros in the context of robotic applications using ROS. However, rtamt is a much more versatile library and can be used in many other (non-robotic) applications. We illustrate the versatility of rtamt with two additional applications: (1) general offline anaysis of time series, and (2) model-based development of CPS applications with MATLAB/Simulink.

0.c.1 Offline Analysis of Time Series with rtamt

We use a simplified real-valued request-grant protocol to illustrate (1) interface-aware STL semantics, (2) programmatic interface to rtamt and (3) offline monitoring capability of rtamt. We say that a request is issued whenever the value of the signal req is greater or equal to . Similarly, the grant is issued whenever the value of the signal gnt is greater or equalt to . The specification requires that every request is eventually followed within time units by a grant. Figure 5 illustrates different signals: (a) satisfies the requirement, (b) vacuously satisfies the requirement (request is never issued), (c) and (d) violate the requirement. We demonstrate the discrete-time offline monitoring by iterating over the input sequence and applying the online monitor in each iteration and collecting the output.

Figure 5: Examples of request-grant signals.

IA-STL specification of the request-grant requirement from Section 1 is the formula , where . The text below expresses using the rtamt STL grammar.

1import rtamt
4data = read_csv(’req-gnt.csv’)
6spec = rtamt.STLIOSpecification(is_pure_python=True) = IA-STL Request-Grant Specification
8spec.declare_var(’req’, float’)
9spec.declare_var(’gnt’, float’)
10spec.declare_var(’out’, float’)
11spec.set_var_io_type(’req’, input’)
12spec.set_var_io_type(’gnt’, output’)
13spec.spec = out = always((req>=3) implies (eventually[0:5](gnt>=3)))’
14spec.iosem = output-robustness
16    spec.parse()
17except rtamt.STLParseException as err:
18    print(’STL Parse Exception: {}’.format(err))
19    sys.exit()
21for i in range(len(data1[’ gnt’])):
22    rob = spec.update(i, [(’req’, data1[’ req’][i][1]), (’gnt’, data1[’ gnt’][i][1])])

We first note that we instantiate the class STLIOSpecification to create an IA-STL monitor. We set the argument is_pure_python to True if we want to create a pure Python monitor. This flag is set to False for monitors using the C++ back-end. We observe that we use the method set_var_io_type(var_name, var_io_type) to declare a variable as input or output. We use the attribute iosem to define the IA-STL semantics that we want to use – the available options are standard, output-robustness and input-vacuity.

Consider signals from Figure 5 and the request-grant specification. Standard robustness is , while output robustness is – standard robustness gives the same treatment to input and output signals resulting the input dominating robustness computation. Standard robustness measures in this case how good is the input, while output measures how bad is the grant. We also have that while . Output robustness value tells us that the input is vacuous, while the input vacuity measures how vacuous it is.

The dense-time monitor for the same input signal and specification can be invoked by instantiating the STLIOCTSpecification instead of STLIOSpecification. We note that the signature of the update method is different for discrete-time and dense-time monitors:

Discrete-time update(time_index, [ (var_name, value) … ])
Dense-time update([[var_name, [[time, value] …] …])

0.c.2 Model-based design of CPS with rtamt

We now introduce a case study that we use as a running example to illustrate our approach step by step. We consider the Aircraft Elevator Control System [mosterman-fdir] to illustrate model-based development of a Fault Detection, Isolation and Recovery (FDIR) application for a redundant actuator control system.

Left Elevator
Figure 6: Aircraft Elevator Control System [mosterman-fdir].

Figure 6 shows the architecture of an aircraft elevator control system with redundancy, with one elevator on the left and one on the right side. Each elevator is equipped with two hydraulic actuators. Both actuators can position the elevator, but only one shall be active at any point in time. There are three different hydraulic systems that drive the four actuators. The left (LIO) and right (RIO) outer actuators are controlled by a Primary Flight Control Unit (PFCU1) with a sophisticated input/output control law. If a failure occurs, a less sophisticated Direct-Link (DL/PFCU2) control law with reduced functionality takes over to handle the left (LDL) and right (RDL) inner actuators. The system uses state machines to coordinate the redundancy and assure its continual fail-operational activity.

This model has one input variable, the input Pilot Command, and two output variables, the position of left and right actuators, as measured by the sensors. This is a complex model that could be extremely hard to analyze in case of failure. In fact, the model has signals, from which are internal variables that are instrumented ( real-valued, Boolean and enumerated - state machine - variables) and any of them, or even a combination of them, might be responsible for an observed failure.

When the system behaves correctly, the intended position of the aircraft required by the pilot must be achieved within a predetermined time limit and with a certain accuracy. This can be captured with several requirements. One of them says that whenever Pilot Command cmd goes above a threshold , the actuator position measured by the sensor must stabilize (become at most units away from the command signal) within time units. This requirement is formalized in STL with the following specification:

1out = always((cmd >= m) -> eventually[0:T]always[0,t](abs(cmd - pos) < n)
Figure 7: Expected behavior of AECS.
Figure 8: Failure of the AECS.

Figures 8 and 8 show the correct and faulty behavior of the system, respectively. The control system clearly stops following the reference signal after seconds.

We use rtamt to do sensitivity analysis and falsification testing, two common approaches supported by tools like Breach and S-TaLiRo. Sensitivity of the model robustness to its STL requirements is studied by uniformly varying input parameters, simulating the model for each combination and monitoring the simulation outcomes against the requirements. Figure 10 shows a heat-map visualizing the outcomes.

Given an STL specification and the system model, falsification testing aims to identify an input signal that results in the violation of the requirement. It does so by solving a search problem over input parameter variables to identify a trace that violates the specification. The outcome is shown in Figure 10.

Figure 9: Sensitivity analysis of AECS.
Figure 10: Falsification testing of AECS.

Finally, we integrate rtamt to MATLAB Simulink to evaluate specifications during simulation in an online fashion. We create a custom online monitoring Simulink block that integrates rtamt to a Level-2 MATLAB S-Function. At every simulation step, the monitoring block reads the relevant Simulink signal values and computes robustness for that simulation step. The robustness value is given back to the simulation environment. This approach has the advantage of being able to stop the simulation upon detection of property violation. This can save significant simulation time, especially when property violations are detected early in the simulation.

Appendix 0.D Reproducing Examples

All the examples shown in the paper and its Appendix, except the proprietary Toyota HSR robotic application, can be found on the repositories, together with instructions on how to run them.

Example Repository Dependencies Directory
STDR rtamt4ros ROS, STDR stdr_control
Timing rtamt none examples/timing
Offline req-gnt rtamt none examples/offline_monitors
Sensitivity analysis rtamt MATLAB/Simulink examples/sensitivity_analysis_simulink
Falsification testing rtamt MATLAB/Simulink examples/falsification_testing_simulink
Online Simulink rtamt MATLAB/Simulink examples/online_monitos_matlab_simulink