An IoT Analytics Embodied Agent Model based on Context-Aware Machine Learning

by   Nathalia Nascimento, et al.
University of Waterloo

Agent-based Internet of Things (IoT) applications have recently emerged as applications that can involve sensors, wireless devices, machines and software that can exchange data and be accessed remotely. Such applications have been proposed in several domains including health care, smart cities and agriculture. However, despite their increased adoption, deploying these applications in specific settings has been very challenging because of the complex static and dynamic variability of the physical devices such as sensors and actuators, the software application behavior and the environment in which the application is embedded. In this paper, we propose a modeling approach for IoT analytics based on learning embodied agents (i.e. situated agents). The approach involves: (i) a variability model of IoT embodied agents; (ii) feedback evaluative machine learning; and (iii) reconfiguration of a group of agents in accordance with environmental context. The proposed approach advances the state of the art in that it facilitates the development of Agent-based IoT applications by explicitly capturing their complex and dynamic variabilities and supporting their self-configuration based on an context-aware and machine learning-based approach.



page 1

page 2

page 3

page 4


Machine Learning-based Variability Handling in IoT Agents

Agent-based IoT applications have recently been proposed in several doma...

A Reference Model for IoT Embodied Agents Controlled by Neural Networks

Embodied agents is a term used to denote intelligent agents, which are a...

Context-Aware Environment Monitoring to Support LPWAN-based Battlefield Applications

The use of IoT-related technologies is growing in several areas. Applica...

Hybrid-Learning approach toward situation recognition and handling

The success of smart environments largely depends on their smartness of ...

A Unified Knowledge Representation and Context-aware Recommender System in Internet of Things

Within the rapidly developing Internet of Things (IoT), numerous and div...

Complex Systems Science meets 5G and IoT

We propose a new paradigm for telecommunications, and develop a framewor...

FaiR-IoT: Fairness-aware Human-in-the-Loop Reinforcement Learning for Harnessing Human Variability in Personalized IoT

Thanks to the rapid growth in wearable technologies, monitoring complex ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Based on the Google Trends tool, the Internet of Things (IoT) [1] is emerging as a topic that is highly related to robotics and machine learning. In fact, the use of learning agents has been proposed as an appropriate approach to modeling IoT applications [2]. These types of applications address the problems of distributed control of devices that must work together to accomplish tasks [1]. This has caused agent-based IoT applications to be considered for several domains, such as health care, smart cities, and agriculture. For example, in a smart city, software agents can autonomously operate traffic lights [2], driverless vehicles [3] and street lights [4].

Agents that can interact with other agents or the environment in which the applications are embedded are called embodied agents [5, 6]. The first step in creating an embodied agent is to design its interaction with an application’s sensors and actuators, that is, the signals that the agent will send and receive [6]. As a second step, the software engineer provides this agent with a behavior specification compatible with its body and with the task to be accomplished. However, to specify completely the behaviors of a physical system at design-time and to identify and foster characteristics that lead to beneficial collective behavior is difficult. To mitigate these problems, many approaches [5, 7, 4]

have proposed the use of evolving neural networks to enable an embodied agent to learn to adapt their behavior based on the dynamics of the environment.

The ability of a software system to be configured for different contexts and scenarios is called variability [8]. According to Galster et al. [8], achieving variability in software systems requires software engineers to adopt suitable methods and tools for representing, managing and reasoning about change. However, the number and complexity of variation points [9] that must be considered while modeling agents for IoT-based systems is quite high [10]. Thus, “current and traditional agent development processes lack the necessary mechanisms to tackle specific management of components between different applications of the IoT, bearing in mind the inherent variability of these systems” [10].

In this paper, we propose a self-configurable IoT agent approach in which a machine learning procedure assists a software developer in developing embodied agents for the Internet of Things. The approach involves: (i) a variability model for IoT agents; (ii) feedback-evaluative machine-learning; and (iii) reconfiguration of a group of agents in accordance with environmental context.

We provide more details about our proposed approach in Section III. To evaluate the proposed approach, we present an illustrative example in Section IV. This section presents the experimental setup, results, and evaluation. The remainder of this paper is organized as follows. Section II describes the background for the proposed approach. Section V presents the related work. The paper ends with our final remarks in Section VI.

Ii Background

Ii-a Embodied Agents

Embodied agents have a body and are physically situated, that is, they are physical agents interacting not only among themselves but also with the physical environment. They can communicate among themselves and also with human users. Robots, wireless devices and ubiquitous computing are examples of embodied agents [11].

Figure 1 depicts an embodied agent according to the description presented by the Laboratory of Artificial Life and Robotics [12]

about embodied agents. They define embodied agents as agents that have a body and are controlled by an analysis architecture, such as artificial neural networks. These agents use learning techniques, such as an evolutionary algorithm, to adapt to execute a specific task.

Fig. 1: Embodied agent model.

Ii-B IoT Embodied Agents

According to the description about embodied agents provided in this section, Figure 2

illustrates an IoT embodied agent in a scenario of autonomous cars. In this example, the body of the agent is a car with four wheels, GPS, headlights, etc. As described above, an embodied agent must have a local analysis architecture to sense the environment and behave accordingly. In such example, the autonomous car uses an artificial neural network. There is an input neuron for each one of the car’s sensors and an output neuron for each one of the motors and actuators. The neuron output values may determine the direction of the wheels and whether the car turn on the headlights.

Fig. 2: An example of an IoT embodied agent.

Ii-C Context-aware Approaches

Context is any information that can be used to characterize the state of different entities, such as persons, places or physical objects [13]. Context-aware computing is used to represent a system that understands the context and takes an action based on that particular context [14].

Nascimento et al. [15] investigate the advantages of considering the context in a machine learning-based approach. Accordingly, a system that is able to reconfigure its analysis model according to the context outperforms a system that uses an unique and versatile model.

Ii-D Reconfigurable Systems

To achieve variability in a system, the first step is to understand and represent variability in its application domain. Our approach incorporates feature-oriented domain analysis (FODA) [9]

to represent the software’s variability. According to the FODA notation, features can be classified as mandatory, optional and alternative. Alternative features are not to be used in the same instance.

Iii Approach

We aim to support the development of embodied agents to work in real or simulated scenarios related to the Internet of Things. Therefore, we designed a platform to support i) handling variability in IoT embodied agents, ii) selecting the physical components that will compose each agent, and iii) finding their appropriate behavior according to their bodies and the scenario where they will be applied.

Iii-a Variability in IoT Embodied Agents

In an Internet of Things application suite, there are several options for physical components and software behaviors for the design of a physical agent [10]. According to existing experiments [16] and our experience with the IoT domain [2, 17, 4], we introduce possible variants of an IoT embodied agent and we use a FODA notation to explore this variability.

Fig. 3: Feature model of an IoT embodied agent’s body.
Fig. 4: Feature model of an IoT embodied agent’s analysis architecture.

The physical devices may vary in terms of the types of sensors, such as temperature and humidity, and in terms of actuators, as depicted in the feature diagram presented in Figure 3 . Each sensor can also vary in terms of brands, changing such parameters as energy consumption and battery life. Note that, depending on the application domain, this feature diagram may contain different and more specific features. For example, to create smart street light agents, we can provide a different version of this feature model, discarding some options of sensors, such as heart, EEG and pressure sensors.

In addition, we also need to deal with variants in agent architecture that the agent uses to sense the environment and behave accordingly. For example, this architecture can be a decision tree, a state machine or a neural network. Many approaches

[5, 4] use neuroevolution

, which is “a learning algorithm which uses genetic algorithms to train neural networks”

[18]). This type of network determines the behavior of an agent automatically based on its physical characteristics and the environment being monitored. A neural network is a well-known approach to provide responses dynamically and automatically, and create a mapping of input-output relations, which may compactly represent a set of “if..then” conditions [4], such as: “if the temperature is below 10

C, then turn on the heat.” However, finding an appropriate neural network architecture based on the physical features that were selected for an agent, is not easy. To model the neural network, we also need to account for its architectural variability, such as the activation function, the number of layers and neurons and properties such as the use of winner-take-all (WTA) as a neural selection mechanisms

[19] and the inclusion of recurrent connections [5]. We explore these variabilities in Figure 4.

With respect to analysis variabilities, Marocco and Nolfi [5], performed two experiments with the same embodied agents, varying only the neural network architectures and neural activation functions. In the first experiment, they used a neural network without internal neurons, while in the second experiment, they used a neural network with internal neurons and recurrent connections. In addition, they also used different functions to compute the neurons’ outputs. Based only on the neural network characteristics, they classified the robots from the first experiment as reactive robots (i.e. “motor actions can only be determined on the basis of the current sensory state”), and non-reactive robots (i.e. “motor actions are also influenced by previous sensory and internal states”). Marocco and Nolfi [5] analyzed whether the type of neural architecture influenced the performance of a team of robots. They showed that the differences in performance between reactive and non-reactive robots vary according to the environmental conditions and how the robots have been evaluated.

Oliveira and Loula [7] investigated symbol representations in communication based on the neural architecture topology that is used to control an embodied agent. They found that the communication system varies according to how the hidden layers connect the visual inputs to the auditory inputs.

Jarraya et al. [20] propose a multiagent approach for pervasive computing that aims to identify human activities in smart homes. In their approach, the set of agents must observe sensor data and make local predictions. Jarraya et al. [20] state that “depending on the nature of sensor data, agents may hold different types of classifiers.”

These findings have helped us to conclude that to support the design of IoT embodied agents, we need to account for the variability of the physical body and the architecture that analyses the inputs. In addition, we also concluded that the analysis architecture cannot be considered as a black box in the system, since its structure must fit the characteristics that were selected to compose the body and behavior of the agent.

Iii-B Architecture

Fig. 5: High-level model of the self-configurable agent approach to generate embodied agents.

Based on the two main variation points we have identified, we propose a platform to design embodied agents applications based on the Internet of Things. Figure 5 depicts the high-level model of our proposed approach. Basically, this platform contains five modules: i) a reconfigurable system that contains the characteristics that can be used to compose the set of agents, according to the application domain; ii) a manual control that allows an IoT expert to select the first set of features manually; iii) the creation of a set of agents containing the selected characteristics that are also able to use a neural network to learn about the environment; iv) a module for evaluating feedback tasks, by investigating the performance of the group of agents in the application scenario during the learning execution. The evaluation process has to be implemented according to the application and the learning algorithm. For example, if an application for automobile traffic control has the goal of reducing urban traffic congestion, the evaluation may be performed based on the number of vehicles that had finished their routes in a specific period; and (v) a module to store and retrieve machine learning models based on the context, as described in [15]. It allows the set of agents to switch the analysis architecture according to the context at runtime. The IoT expert can also use this context information to reconfigure the set of agents.

Iii-C A Context-Aware Machine Learning Embodied Agent Model

Fig. 6: Class diagram of the context-aware machine learning embodied agent for IoT analytics.

Figure 6 illustrates the class diagram of the proposed model. We separated the classes into three modules: (i) a reconfiguration system, that contains the set of characteristics to compose each embodied agent; (ii) the environment, that is the union of a set of contexts; and (iii) the machine learning module, that is associated with the environment and reconfiguration system classes. Accordingly, the product line has some fixed and variant features that can be used to compose the embodied agents. An embodied agent has sensors and actuators to sense and act on the current context. While the agent interacts with the environment, the learning algorithm can adjust the agent’s analysis architecture in order to improve its performance to the current context.

Iv Illustrative Example: Smart Street Lights

We selected one example from the IoT domain: a smart street light application. Our goal is to show how this application will be executed by using our proposed model. In this application scenario, we consider a set of street lights distributed in a neighborhood. For more details concerning this application scenario, see [4].

Each street light represents an IoT agent, which needs to operate in an environment composed of different contexts. For example, sometimes the background light can be bright and at other times dark. With respect to the environment background light, the application scenario has some variants: (i) night (background light is equal to 0.0); (ii) late afternoon (background light is equal to 0.5); and (iii) morning (background light is equal to 1.0). Each street light contains a lighting sensor, but its local brightness also interferes on the sensor measurement.

Iv-a Selecting Physical and Neural Network Features

An IoT expert selected three physical inputs and two physical outputs to measure and operate each one of the street lights. In addition, the engineer selected a neural network with one hidden layer with ten units as the initial network for each agent with the sigmoid function as the activation function of this neural network.

Iv-B Learning about the environment

During the training process, the algorithm evaluates the options for weights of the network based on energy consumption, the number of people that finished their routes before the simulation ends, and the total time spent by people moving during their trip.

Iv-B1 Adjusting the agents to an initial context

During a first simulation, while the background light was always bright, the collection of street lights found a solution that provided a performance, say X+1, during the morning.

Iv-B2 Environmental context changes

After a time, a change in the environment occurred. Now, these agents are operating in an environment in which sometimes the background light can be bright and at other times dark. After the environment changed to the night, the lights’ solution was adjusted to deal with this change. However, this new generic solution decreased the performance, say X, during the morning, as shown in Figure 7.

Fig. 7: Reconfiguring the set of features.

In a traditional learning approach, the street light would be unable to return to its previous configuration, as the street light would not maintain different versions of its configuration. However, the configuration history that is supported by our approach could enable the street light to switch its analysis architecture to specialized solutions that were trained for each one of the background lighting variants.

Iv-C Reconfiguring the set of agents for the new context

As our approach enables the set of agent to use more than one analysis model at runtime, the expert can provide a new model to be trained for this new context. For instance, he maintained the number of sensor inputs, but selected different variants for the neural network, such as the number of neurons in the hidden layer. Then, the learning algorithm was re-executed and the agents were able to select a new model to cope with this environmental change, as depicted in Figure 8.

Fig. 8: Selecting the analysis architecture according to the context.

V Contributions and Ongoing Work

To handling variability in IoT embodied agents, we identified the main variation points of these kinds of applications, including the variants that can be involved in a neural network design. We also provided a feature-oriented variability model, which is an established software engineering module.

In addition, we proposed an approach that takes context into account to train and deploy machine learning-based models for IoT embodied agents. To demonstrate the use of this context-aware approach, we reproduced an experiment showing how this application operates by taking the main steps of our proposed approach into account.

Our next step is to select more state-of-the-art experiments and verify if our proposed approach can improve their results. In addition, we also want to consider to enable the use of a learning technique to reconfigure the set of features based on environmental changes automatically. As we proposed a hybrid architecture, we can use this learning technique only to reconfigure the variants related to one of the variation points, such as the neural network properties. In such an instance, we can have an expert responsible for handling the body and behavior variability of the IoT agents.


This work has been supported by the Laboratory of Software Engineering (LES) at PUC-Rio. It has been developed in cooperation with the University of Waterloo, Canada. Our thanks to CNPq, CAPES, FAPERJ and PUC-Rio for their support through scholarships and fellowships.


  • [1] L. Atzori, A. Iera, G. Morabito, and M. Nitti, “The social internet of things (siot)–when social networks meet the internet of things: Concept, architecture and network characterization,” Computer networks, vol. 56, no. 16, pp. 3594–3608, 2012.
  • [2] N. M. do Nascimento and C. J. P. de Lucena, “Fiot: An agent-based framework for self-adaptive and self-organizing applications based on the internet of things,” Information Sciences, vol. 378, pp. 161–176, 2017.
  • [3] D. Herrero-Perez and H. Martinez-Barbera, “Decentralized coordination of automated guided vehicles (short paper),” in AAMAS 2008, 2008.
  • [4] N. M. do Nascimento and C. J. P. de Lucena, “Engineering cooperative smart things based on embodied cognition,” in Adaptive Hardware and Systems (AHS), 2017 NASA/ESA Conference on.   IEEE, 2017, pp. 109–116.
  • [5] D. Marocco and S. Nolfi, “Emergence of communication in embodied agents evolved for the ability to solve a collective navigation problem,” Connection Science, vol. 19, no. 1, pp. 53–74, 2007.
  • [6] S. Nolfi, J. Bongard, P. Husbands, and D. Floreano, Evolutionary Robotics.   Cham: Springer International Publishing, 2016, ch. 76, pp. 2035–2068.
  • [7] E. Oliveira and A. Loula, “Symbol interpretation in neural networks: an investigation on representations in communication,” in Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 36, no. 36, 2014.
  • [8] M. Galster, D. Weyns, D. Tofan, B. Michalik, and P. Avgeriou, “Variability in software systems—a systematic literature review,” IEEE Transactions on Software Engineering, vol. 40, no. 3, pp. 282–306, 2014.
  • [9] K. Pohl, G. Böckle, and F. J. van Der Linden, Software product line engineering: foundations, principles and techniques.   Springer Science & Business Media, 2005.
  • [10] I. Ayala, M. Amor, L. Fuentes, and J. M. Troya, “A software product line process to develop agents for the iot,” Sensors, vol. 15, no. 7, pp. 15 640–15 660, 2015.
  • [11] L. Steels, “Ecagents: Embodied and communicating agents,” SONY, Tech. Rep., 2004.
  • [12] S. Nolfi, “Laboratory of autonomous robotics and artificial life,” LARAL,, Tech. Rep., March 1995.
  • [13] G. D. Abowd, A. K. Dey, P. J. Brown, N. Davies, M. Smith, and P. Steggles, “Towards a better understanding of context and context-awareness,” in International Symposium on Handheld and Ubiquitous Computing.   Springer, 1999, pp. 304–307.
  • [14] O. B. Sezer, E. Dogdu, and A. M. Ozbayoglu, “Context-aware computing, learning, and big data in internet of things: a survey,” IEEE Internet of Things Journal, vol. 5, no. 1, pp. 1–27, 2018.
  • [15] N. Nascimento, P. Alencar, C. Lucena, and D. Cowan, “A context-aware machine learning-based approach,” in Proceedings of the 28th Annual International Conference on Computer Science and Software Engineering.   IBM Corp., 2018, pp. 40–47.
  • [16] G. Soni and S. Kandasamy, “Smart garbage bin systems–a comprehensive survey,” in International Conference on Intelligent Information Technologies.   Springer, 2017, pp. 194–206.
  • [17] C. J. P. d. L. Nathalia Moraes do Nascimento, Marx Leles Viana, “An iot-based tool for human gas monitoring,” in Congresso Brasileiro de Informática em Saúde - CBIS 2016, vol. 15.   CBIS 2016 (ISSN 2178-2857), 2016, pp. 96–98.
  • [18] S. Whiteson, N. Kohl, R. Miikkulainen, and P. Stone, “Evolving soccer keepaway players through task decomposition,” Machine Learning, vol. 59, no. 1-2, pp. 5–30, 2005.
  • [19] T. Fukai and S. Tanaka, “A simple neural network exhibiting selective activation of neuronal ensembles: from winner-take-all to winners-share-all,” Neural computation, vol. 9, no. 1, pp. 77–97, 1997.
  • [20] A. Jarraya, A. Bouzeghoub, A. Borgi, and K. Arour, “Distributed collaborative reasoning for har in smart homes (extended abstract),” in Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS).   IFAAMAS, 2018, pp. 1971–1973.