ComFlux: External Composition and Adaptation of Pervasive Applications

10/18/2017 ∙ by Raluca Diaconu, et al. ∙ University of Cambridge Queen Mary University of London 0

Technology is becoming increasingly pervasive. At present, the system components working together to provide functionality, be they purely software or with a physical element, tend to operate within silos, bound to a particular application or usage. This is counter to the wider vision of pervasive computing, where a potentially limitless number of applications can be realised through the dynamic and seamless interactions of system components. We believe this application composition should be externally controlled, driven by policy and subject to access control. We present ComFlux, our open source middleware, and show through a number of designs and implementations, how it supports this functionality with acceptable overhead.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

This century has witnessed increasingly specialised technology becoming better at dealing with specific tasks such as sensing, automation, and mobility. Sensors, actuators, mobile devices and wearables, integrated and driven by a range of software services, increasingly become incorporated into our everyday activities, and commercial promotion of the Internet of Things (IoT) will increase these tendencies.

The goal of pervasive computing [1, 2] is transparent integration of technology components into the environment to create contextual functionality —“a cyber-physical continuum”. Seamlessness is key in users’ interacting with pervasive technology; components need to interact smoothly with users, with each other and with the environment to achieve the functionality required. This involves exchange of data, between system components, be they physical (sensors, actuators, mobile devices) or software (apps, databases, web services).

To date, functional requirements have typically been met by composing software and hardware components into “silos of things” [3], managed in isolation. We argue that such component functionality could potentially be used for many diverse purposes, outside their initial scope. To achieve this, means are needed to ‘instruct’ components how, when and with whom they should interact, taking into account usability, modularity and access control. This should happen dynamically at runtime, without requiring application-level intervention. This would pave the way for new functionality, where existing components can be leveraged across application boundaries, in ways and for reasons not envisaged by their designers, and increase the longevity of components through updates and new uses. Section II expands on these requirements and challenges.

To support this vision, we propose a system architecture that enables an external management regime, whereby component interactions, communication methods and security constraints can be dynamically defined, extended and updated at runtime —without requiring redeployment or changes to the application logic of the components themselves, and without imposing constraints on system design (Section III). We show that such capabilities can be built into a messaging infrastructure, by presenting ComFlux, an open-source, proof-of-concept middleware (Section III). The aim of ComFlux is to support reconfiguration relating to the exchange of data in pervasive environments. We use ComFlux to demonstrate the potential of an external management regime in facilitating a range of dynamic, pervasive-computing scenarios, including those where control is user-(individual) and environment-centric.

External control and communication management raise important security concerns, in that only authorised components should be able to perform reconfiguration actions. IoT and pervasive computing may deal with sensitive data and this may be subject to law if people are identified. Breaches in personal data exposure [4] and malicious behaviour [5] are potentially harmful, unintended outcomes of IoT. Approaches to external reconfiguration should address these vulnerabilities, enforcing access control over the exchange of data (Section II-C).

The components used for particular functionality may change over time as a result of updates or changes in context, e.g., as people move about, their surroundings evolve. To support this, ComFlux has a modular design and provides mechanisms to extend and adapt the capabilities of an application at runtime including functional evolution of applications and software lifecycle updates. Different modules can be plugged in and out to allow flexible functionality (Section III-B3).

Ii Requirements and challenges

In this section we explore a wider vision of ubiquitous computing and highlight the challenges introduced by heterogeneous and dynamic environments. We use smart cities and pervasive gaming to motivate our “command and control” approach to achieving novel functionality.

Ii-a Pervasive computing scenarios

We are moving towards an environment where virtual and physical are strongly intertwined, driven by advances in mobile technologies, cloud-based services and the emerging IoT. Pervasive technologies are having a great impact in areas ranging from consumer products to smart city projects, creating interest from users, industry and researchers.

Current cityscapes have increasingly many sensors e.g., temperature, presence, pollution, and more automated functionality gives the potential for innovative applications, provided citizens are well-informed and perceive benefit. Digitally enhancing a space can help to make it more attractive. This involves adding more sensors, more actuators, and more ways for the user to interact and engage with it.

Pervasive games might also extend the virtual gaming experience into the real world, and these are not confined to a space and time [6, 7]. Their deployment and gameplay can use the available and accessible infrastructure of a city including not only geographical, public transport, pollution and traffic information, but also information about energy and water distribution, development plans, events and accidents [8].

Pervasive games can offer an interactive approach to understanding the history of a city [9] or to inform people’s choices and responsibilities as consumers [10, 8]. An example is The Water Must Flow, a strategic game of resource management about using water in common and private areas of a neighborhood: it tasks players with managing plots of land in relation to rainwater, flooding, draught, and other similarly water-related [8] issues. Urban gaming could benefit greatly from seamless interaction via deployed infrastructure.

In emergency situations, maximum functionality should be acquired from existing systems and commonly used components by automatically adapting their functionality and overriding access control to data as appropriate, provided such actions are audited. We envisage that if a person falls ill, the surrounding devices should have their connections changed dynamically to inform medical staff, and then to allow rescue services to access data; for instance, a smart door lock may temporarily allow any doctor to pass through.

Ii-B The need for external command and control

By their nature, pervasive computing systems are dynamic. Technology is not confined to a single place or context; instead, system components are mobile and the contexts under which they operate may vary. Given their complexity, pervasive systems pose the challenge of adapting to new environments and usage contexts. While the broader vision is of plug-and-play modules, dynamic reconfiguration and seamless access control, current solutions rely on pre-programmed functionality, where the only capacity for management is in terms of what applications expose through their in-built application interfaces (API), see Section VI.

Pervasive systems are also heterogeneous and highly distributed. To accommodate this, off-the-shelf IoT products generally come with a complete solution, involving a connected device, a cloud service, and a suite of control and visualisation apps for web or smart-phone. Often they require an account registration and sometimes a periodic fee. Popular examples range from wearables (e.g., Fitbit [11]) to IoT home solutions (Philips Hue [12], Nest [13], Apple HomeKit [14], etc.) to platforms for specific (particularly industrial) application contexts (e.g., ThingWorx [15]). Current approaches target a restricted system and/or application. In some cases, access to some sort of API may extend their usability, but achieving new functionality relies on pre-programming the functionality that manages components’ capabilities: with whom and how it interacts.

To get the most out of existing devices and services it must be possible to reuse them in different circumstances, to adapt them to new purposes and to mediate the communication between them. We argue that the capabilities enabled by the API functionality need also to be available externally, at runtime, and show in Section III how to achieve this. By enabling runtime reconfiguration, application development is simplified: application logic is decoupled from the reconfiguration and existing infrastructure can be reused, e.g., by interfacing with open services or public sensors.

It is a challenge to manage and coordinate complex and large-scale systems, such as smart cities, as the answers to how, when and why things should interact, change over time. Thus, applications need to be aware of the context they operate in, that is, they need a mechanism that allows them to receive input and adapt to changes in the environment [16, 17, 18], as well as individual preferences which change over time. As systems become increasingly heterogeneous, dynamic and complex, the emerging range of applications cannot rely on pre-programmed functionality. Instead, new applications must be able to discover and interact within new contexts dynamically, at runtime, and be managed externally. The key aspect in enabling transparent integration with the environment is to leverage appropriate external command and control over components’ interactions, communication methods and security constraints. The functionality must be dynamic and the control and command interface must be available at runtime.

Ii-C External control: Requirements and challenges

The real value of pervasive computing lies beyond its myriad technologies; it emerges from their operation within a context. The power of a technology component may be revealed when reused and adapted to fit new contexts. Nowadays, driven by human dynamics, heterogeneous components and their mobility create increasingly complex environments in which dynamism and creativity can bring great value. We now articulate pivotal challenges for future applications and propose ways to address them.

Capabilities management and communication coordination

Pervasive technologies should offer control over their capabilities. They will allow external control, from users and other objects in the environment, over their functionality and their connections. Consider a museum application that detects and coordinates interaction between people and its installations and services. On arrival, people will be able to discover the types of installations and load mechanisms to communicate with them. Inside, the environment detects people’s presence, communicates with them to help coordinate groups, and may reconfigure its components on-the-fly to offer the participants contextual information.

Seamless system composition

At urban scale, smaller-scale systems (and their constituent components) should be composable to bring new functionality. Resources should be used when and where needed, but no more, e.g., at night, lights could be activated only when people are detected, and lighting could “follow them”.

Component reuse and repurposing

Large-scale systems could be enabled by reusing and adapting smaller-scale components beyond their initial purpose. Connected smart cities could bring awareness of their problems or draw attention to their landmarks, using real time sensor information and actuators. Individuals and groups could customise the use of the components in the environment to enhance particular activities, e.g., during a festival, as part of a game and so forth. Because of their scale and complexity such applications will have to discover and interact with components (devices and services), already part of the city’s infrastructure. For example, when high pollution is detected, alerts and displays can indicate the danger zones, and routes for pedestrians and cyclists that avoid the pollution can be highlighted.

Long-lived components

Updating the capabilities of potentially long-lived, deployed components is important in an IoT context, to add or change functionality as appropriate. Importantly, reuse and repurposing extends the life of the components themselves, as new uses are found.

Security

External control over a component’s connections and capabilities raises significant security concerns. Access control and authentication need to enforce that only authorised entities perform external reconfiguration actions.

Automatic context adaptation

The environment must sometimes be able to coordinate itself without human involvement. In an emergency, when human intervention may be too slow or not available, e.g., people are trapped under a collapsed building or are suffering cardiac arrest, the devices must collectively make decisions and reconfigure their connections and even their function; see further Section III-A. Emergency override of access control must always be audited.

In Section III we describe ComFlux, an open source infrastructure (middleware) to support externally-driven, dynamic reconfiguration under access control. In Section IV we show how ComFlux meets the above challenges.

Iii ComFlux

Iii-a The ComFlux model for external, dynamic reconfiguration

The ComFlux model allows components to be instructed externally and at runtime with whom they should interact, what are the means to communicate with new components, and when, under which circumstances they should interact.

A ComFlux component is a process, an application or a service. In a pervasive environment it can be the process running on an embedded device that interacts with other components by exchanging data, often wirelessly, or there might be a series of components, for instance, representing different applications on a mobile phone. A pervasive component’s functionality is defined by the data it exchanges. Components can reconfigure themselves, via their API, and also certain components can control others. Our model is based on the following key external runtime controls:

1) External communication control: an authorised component can instruct another component to connect or disconnect to/from some specified component(s) and transmit or receive data, see Fig. 1 and Sections III-B1, III-B5.

Fig. 1: Example of external reconfiguration

2) External component (re)configuration: an authorised component can reconfigure other components’ functionality. Fig. 4 shows reconfigurations to change the functionality of a specific application; see also Section III-B3.

The mechanism for dynamic reconfiguration allows components to manage how other components communicate and function. External reconfiguration capabilities can involve changing communication protocols in a plug-and-play manner and adding/removing access privileges over other components; e.g., a command component can instruct a component to load a new communication interface for UDP, instead of TCP connections, if its battery runs low, or instruct a sensor component to relax its data access permissions in an emergency.

External control might also include commands that specify what data to pass and/or when to interact, e.g., a component may add filters to its send/receive messages to conform to a pattern, such as only extreme temperatures. Temporal filters may specify when to receive messages, e.g., at what frequency.

Developers of applications need not include and foresee interactions in all contexts; instead, the functionality is dynamic. The ability to coordinate components is crucial, as is the ability to change and adapt their operation. By providing the means for runtime external configuration of key functionality –given permission– our model provides greater flexibility, independent from the component logic and adaptable to the particular context. This model is achievable with a modular architecture, where capabilities can be plugged in and out dynamically, at runtime.

Iii-B The ComFlux architecture

ComFlux is a proof-of-concept reconfigurable middleware to support the dynamic development of applications. It was originally designed to demonstrate the power of reconfiguration by way of ‘interactive experiences’ in public spaces; though its functionality is more widely applicable. The current implementation, examples and documentation are open source and available online at www.comflux.net.

Since pervasive computing functionality emerges from the interactions between components, in order to work together, systems need to discover, understand, and communicate with each other. In distributed environments, functionality is driven by data and message exchange. Thus we built ComFlux as a messaging middleware, in which external control and communication mediation occurs with respect to data transfer.

Iii-B1 A messaging architecture

ComFlux is built as a framework for typed message exchange between endpoints. Applications define endpoints by specifying a communication type and a message structure. Both the sender and the receiver check the message against a message structure (or schema) that allows the specification of simple and complex types. Communication occurs between endpoints that are mapped (connected) and can have one of the following types:

  • source–sink: unidirectional communication between a source (producer) and any number of sinks (consumers)

  • request–response: a client issues a request to a server which replies with one response; for mapping to be possible both request and response definitions must match

  • request–response+: as above, but the server can issue one or more responses

  • streaming source–sink: unidirectional communication source–sink of unstructured/streaming data.

ComFlux can thus conform to various communication models: event driven (publish-subscribe), client-server, and streaming. Endpoints’ definitions contain one of these types along with a message structure for sources and sinks, and two structures for request and response endpoints. Streaming endpoints are unstructured and offer direct, end-to-end communication.

To keep the discovery and endpoint messages generic, components maintain a (reconfigurable) manifest that describes and identifies the component. The manifest is the information shared externally to facilitate discovery and communication. It contains an updatable collection of information about itself, e.g., supported communication types, addresses, access methods, public keys or user-declared information, and its endpoints, e.g. their type, message structure and permissions.

The communication model in ComFlux is fully decentralised to suit heterogeneous, ad-hoc environments. While its functioning does not rely on a central or cloud-based system, ComFlux mediates the communication with such services in a uniform manner, by exposing an endpoint to the application layer. The manifest enables schema negotiation on mapping requests, and message type matching.

Iii-B2 The ComFlux middleware implementation

Space does not permit a full description of the implementation. Full details can be found at www.comflux.net. An overview is given here.

ComFlux is written in C and offers developer interfaces in C and JavaScript. Messages are encapsulated in JSON and the type is specified with JSON schemas.

ComFlux comprises the following layers, see Fig. 2.

The core

is the backbone of the middleware and includes command and control functionality. It carries out data flow management and has the necessary mechanisms for external reconfiguration. It decouples the management plane from the application logic by maintaining endpoints and orchestrating functionality.

Communication and access control modules

are functional units that plug into the core dynamically. They are reusable building blocks which are loaded and unloaded at runtime. Modules are built as dynamic libraries that implement an interface (a set of predefined functions) recognised by the core. ComFlux offers communication modules that carry the communication over a specific medium, and access control modules for authentication and endpoint access.

The ComFlux API

is available to the application programmer as a set of functions that provide internal endpoint and middleware configuration functionality. To use the middleware, the application imports the ComFlux library which automatically spawns the core process. The API provides the interfaces for sending/receiving data and configuring the core.

Fig. 2: ComFlux architectural layers in a component

Internally, the core has two logical layers: (1) the command and control layer and (2) the endpoint representation layer (see Fig. 2). This exposes layer (1) in a uniform manner internally to the application, and externally, for other entities. The application manages the endpoints and accesses the middleware’s functionality via the API. External control capabilities are enabled via control endpoints which are defined automatically within the core, independent of the application logic.

Fig. 3: Data flow involving a thermometer component with a temperature data source () and two consumers (), a HVAC system and a mobile phone. The HVAC system supports only MQTT while the mobile phone application uses TCP sockets.

Iii-B3 Modules

The power of ComFlux is in providing a mechanism to extend and adapt the functionality of an application at runtime. It achieves this with a module-based design, where functionality can be plugged into the middleware at runtime, internally, by the application and externally, through the control interface. The core specifies interfaces for communication and access control. Corresponding modules are built as dynamic libraries which are then loaded (linked) at runtime. This design choice allows them to be implemented as building blocks that can be reused with a new component and in a different application context.

Communication modules may be either Bridge modules that expect a middleware core/component on the other side of the communication channel (currently handling TCP, UDP and SSL); or Interfacing modules that interface with other services (currently MQTT brokers and REST components).

Access control modules implement an authentication mechanism (currently username/password, SSL certificate or Kerberos) and maintain access control lists (ACL) for each endpoint, describing the components that may interact with it.

Iii-B4 Resource Discovery

allows components to dynamically identify to whom and how to connect. To support ComFlux mappings we implemented Resource Discovery Components (RDCs) which maintain a catalog of active, registered components. A component can register and periodically update its manifest with one or more RDCs so that it is discoverable by others. The RDC provides a lookup service, returning the addresses of components whose manifest and endpoint descriptions match the criteria in the query, e.g., a facilities management component may want to communicate with all light components in a building. If all the lights are registered with an RDC, the switch only needs to query that RDC for the lights of interest, allowing it to map (probe) the relevant devices in the building. Resource discovery is particularly important in interacting with mobile and dynamic components, which may ‘come and go’ – we explore this in Section IV.

Iii-B5 Supporting external reconfiguration

is achieved by exposing control endpoints that access the functionality in the control layer (see Fig. 2). We make use of these capabilities with Swiss Knife, our external command line tool. Control endpoints behave and function like regular endpoints. Their messages are directly handled by the core functions that resolve API calls. In this way both configuration and data functionality can be discoverable and negotiated at runtime. Loading/unloading modules, mediating connections, etc., can be executed by third parties at runtime.

Iv Demonstration of applications

In this section we first describe how ComFlux could be used to realise a number of applications for smart cities, including the use of games to explore a city. We then show how externally controlled, dynamic reconfiguration leads to smooth integration and seamless interactions between people and the environment. As our work was in the context of infrastructure for enabling interactive public spaces (with some applications demonstrated in the Science Museum, London), the examples focus on experiential aspects, which we use to demonstrate functionality of wider, more general relevance.

Fig. 4: A series of light show reconfigurations

Iv-a Smart cities

We consider application scenarios within the cities of the future in which smaller scale systems —be they managed or ad-hoc architectures— are composed, reused, and adapted to bring new functionality at a larger scale. We show how our reconfiguration framework can be used for the scenarios introduced in Section II-B: seamless system composition that allows users to have services available wherever they go, and emerging pervasive gaming applications.

Seamless system composition

Current smart city and smart building applications are deployed as suites of technologies in discrete management domains, and do not currently work together. ComFlux would enable their connection to provide seamless integration within the whole smart city, where composition can lead to new ranges of public-sector, commercial and individually-customised applications.

Consider an application where at night, you exit public transport and walk away from other people towards home. Lights are activated, “following you” as you move towards your apartment building. As you arrive, the door opens and lights are again activated as you proceed to your own flat. The door opens as you arrive and enter your home, which is warm, due to prior notification of your arrival.

With ComFlux this scenario is enabled with external reconfiguration of access credentials (and possibly communication modules). On the streets and indoors, users will be equipped with their credentials according to their role. The access in a workplace can be similarly enabled. In more sensitive areas, such as computing facilities, access may be allowed only between 9AM and 5PM, and only to administrators. With ComFlux to restrict access, a command will reconfigure the allowed credentials for the smart door lock at 5PM and will release it at 9AM. Exceptions can be issued by some authorised users or automatically, by emergency systems.

Enabling crowd applications: Pervasive gaming

A way to discover a new city is through users’ engagement with games. With the pervasiveness of connected technologies, a world of mixed reality emerges [8, 10, 19, 6]. Consider the following mixed reality game in which a player walks in a new city and discovers its history and hidden gems by interacting with its smart infrastructure. A series of lights guide the player at night to the city’s most impressive streets, information is given about specific landmarks and visualisation tools present their history. The scenario can be framed as a treasure hunt or geocaching in which the player has to find objects or solve puzzles. Local sensor data and real time visualisation of information can yield clues while the gameplay may indicate less crowded areas, event attendance, etc. Interaction with the infrastructure may be by shaking or tilting phones to control actuators, lights, doors.

Richer games could allow teams of tourist players to cooperate or compete in a virtual game on the streets of the modern city. This scenario embodies the interaction between the environment and people at large scale. A mixed reality game called Ghosts in the city is presented in [10] and we envisage that ComFlux would enable this scenario by coordinating a city’s lights’ infrastructure, the user’s presence and her augmented reality glasses or mobile phone.

Due to scale and heterogeneity, this scenario is not possible with static control, pre-programmed into the applications. Instead it needs to dynamically support new applications as they emerge. With ComFlux, the above scenarios can be enabled via reconfiguration operation to meet high-level goals.

Iv-B Group coordination implementations

We consider two approaches to implementing group coordination: (1) where a component embedded in an environment manages the others in the space for some collaborative effect; (2) fully decentralised control and group coordination.

(a) Group-oriented light show:  Inspired by the London 2012 Olympic Games —where the main stadium was turned into a “light canvas”, displaying patterns through a centrally-driven (stage-managed) orchestration of the lights on custom devices held by the audience (attached to the seating)— our first demonstrator application coordinates displays across people’s mobile devices to create a landscape for visual effects.

Our demonstrator environment contained two components, each producing a unique stream of colour patterns. A device connected to one of these components would display the pattern it produced. To realise the desired ‘big picture’ visual effects, another component, the environmental coordinator, issues various reconfiguration commands to dynamically alter the devices connected to the streaming components. To give greater control and better illustrate the infrastructure’s management capabilities, the people/devices were split into two groups (defined by their metadata, set at runtime).

We illustrate some reconfigurations and the resulting effects in Fig.4. Though a simple example,111Note that the effect of light patterns in the dark can be dramatic, and such an approach might suit more complex and data-rich streams, e.g., sound (see below), video feeds, etc. we include it to illustrate the ease and flexibility of reconfiguration commands in targeting and diverting groups of components, individual components, and those based on their current connections.

(b) Silent Disco: A silent disco differs from a traditional club/disco since, rather than having a sound system (speakers), participants use headphones with an embedded channel selector. Two DJs each broadcast music on a separate channel, and each person with headphones uses the selector to decide the music they listen/dance to. A coloured light on each headphone set indicates the channel one hears. The fun and spectacle is in the disconnect between those listening to different channels, differing in their dance moves and in ‘singing along’.

Currently the experience is somewhat passive, in that one can only select the channel they are listening to. We consider a richer experience, by enabling technical interactions between participants in the space. For example, a common action in silent discos is to have your friend(s) change to the channel you’re listening to. This can be rather disruptive, e.g., involving removing your own headphones to talk or even forcefully adjusting their selector. Here we consider changing the channel of an individual or a specific group by means of an external reconfiguration command to their headphones/devices, to divert them to the desired stream. Similarly, the DJs might decide at some stage to switch the whole crowd to a particular music stream, for instance, in building up to a finale. With the reconfiguration capabilities of ComFlux we can incorporate these features seamlessly in an engaging manner.

We include this example to highlight the capacity to manage participants and groups: participants might only be able to change the channels of their friends (rather than annoy strangers), while the DJs can interact with everyone. In our example, we are able to dynamically manage groups through component metadata and credentials (for each listening device), which is reflected in the RDC. This in turn allows a dynamic selection of peers within groups they belong, while limiting the visibility of those who do not belong and therefore by which interactions are barred. In Fig. 5 we present a series of reconfiguration rules for these interaction patterns.

 Function Reconfiguration command(s) Resulting actions (automatic via the middleware)
 Anne joins Bob’s birthday group manifest_add(‘group:birthday’) credential_add(‘birthday’) acl_add(‘music_input’,
      ‘remap’, ‘birthday’) Anne’s device updates the metadata on groups. This update is communicated to the RDC. A credential for the group is added, to assist in managing group interactions. Anne’s access control list is also updated to authorise those in the ‘birthday’ group (holding the credential) to alter (remap) the connections for her ‘music_input’ endpoint.
 Bob changes Anne’s channel to DJ Mikey (DJ1) remap(‘person:Anne’,
      ‘music_input’, ‘DJ1’) Bob’s device queries the RDC to find the address of Anne’s device, based on name and group membership. Bob’s device sends a command to remap Anne’s ‘music_input’ endpoint to DJ1. Given Bob is also part of ‘birthday’, Anne’s device accepts and actions the command by disconnecting her endpoint from DJ2 and connecting it to the requisite endpoint (matched based on schema) on DJ1.
 Anne changes the whole group to DJ Sandra (DJ2) remap(‘group:birthday’,
      ‘music_input’, ‘DJ2’) Similar to the previous example, but here it is Anne’s device that discovers and sends ‘music_input’ remapping commands to everyone in the ‘birthday’ group.
 As DJ Sandra is leaving, she diverts all her listeners to DJ Mikey (DJ1) divert(‘*’, ‘music_output’,
      ‘DJ1’) Here DJ Sandra’s music streaming component sends a remap command to every component connected to her ‘music_output’ endpoint (i.e., all her listeners). As a DJ, she holds credentials to alter the ‘music_input’ connections of all devices, regardless of their groupings. The result is each device disconnects from her, and connects to DJ Mikey.
Fig. 5: Example silent disco reconfigurations

This section has demonstrated implementations of centrally managed and distributed group interactions. We also discussed how new applications could be composed and existing applications adapt to new contexts. Through these examples we have shown that with ComFlux, seamlessness and interaction do not rely on application design, instead they emerge from dynamic communication reconfiguration. The interaction is not limited to the set of initial devices and communication patterns.

V Performance evaluation

We implemented a suite of demonstrators using ComFlux for Science Museum Lates [20] We now evaluate the performance of ComFlux in this scenario, to indicate the feasibility and practicality of of embedding such a control infrastructure within a communications framework. We compare workloads between a ComFlux-enabled component and an application without the middleware, both using the same mode of communication. Our measurements address only the average communication overheads.

Key performance measurements for pervasive systems include resource usage and latency. In our use cases, throughput is important for real-time interaction. Therefore our measurements will primarily focus on measuring latency of message transmission between applications. In addition, in targeted use cases, by using single threaded machines, latency directly reflects CPU consumption. We also discuss memory usage, a generally limited resource in embedded and IoT systems.

Methodology

Event driven architectures have become the de facto solution for reactive and interoperable IoT. Components listen to events published in a desired context and react accordingly. One of the most common scenarios for disseminating information is through indirect asynchronous communication in a publish/subscribe interaction model.

It is often the case that a source regularly broadcasts data to the subscribers, for instance sensors regularly publishing their readings. As an illustration of this, we use the source-sink interaction model for our museum demonstrations, where components listen for changes in the environment. In a collaborative drawing application, participants use this interaction model on two endpoints: to broadcast their updates and to receive the group’s updates on the canvas. We thus chose to evaluate ComFlux using a message transmission scenario between two applications, a source A and a sink B.

The evaluation deployment runs as follows. A source application A sends to the sink application B a message signalling the start of the transmission. Following the initial message, B starts the time counter and A begins sending messages at regular time intervals, every 10ms. After receiving messages, B records the total time for receiving the entire load (excluding the sleeping time). In the test components, the middleware is configured with the chosen communication module. The control application sends messages directly via the corresponding protocol interface. We employed BSD TCP sockets and the MQTT library [21, 22] with the Mosquitto broker [23].

In our test deployment we vary the load; the number of messages ranges over and the size of the message . The small message corresponds to a GPS location and date-time value. The larger message contains a 3.5KB text message and the largest a 1MB message containing the encoding of a photograph.

Recall (Section III) that our middleware includes type checking of messages and the enforcement of particular interaction paradigms. We therefore include message verification against a schema in the test.

Setup

We deployed the case study applications on Amazon cloud virtual machines, so located in the same independent geographic area. Components were mapped across different machines in the same availability zone. We used virtual machines of type t2-micro in the same Amazon region, eu-west-1 Ireland but on different physical machines denoted eu-west-1b and eu-west-1c. Each virtual machine had 1GB of memory and a single virtual CPU implemented with 6 CPU credits on an Intel Xeon E5. The host OS was Ubuntu-trusty-14.04. Experiments were conducted between July 28th and August 5th, 2017. The applications create a JSON message. The small message is encapsulated as in Fig. 7.

Fig. 6: Performance impact on TCP communication
Results

The core occupies 176KB and the API libraries 120KB. As a messaging-passing infrastructure, the core does not generally store messages after they are pushed to the application/remote ComFlux instance, and the memory consumption remains constant through the execution. Fig. 7 shows a message containing the GPS and date-time information. Each message is encapsulated into a JSON structure to facilitate message parsing by the receiver. The message set by the application is stored under the msg_JSON attribute. Additionally the message contains an id (field msg_id is an incremental integer) and a message type or status (in the example the message from a source to a sink has code 9).

  {
   "msg_id": "13",
   "status": 9,
   "msg_JSON": {
     "position": "41.24’12.2\"N 2.10’26.5\"E",
     "date": "2012-04-23T18:25:43.511Z" }
  }
Fig. 7: GPS and date-time message encapsulated with ComFlux

The performance results measure the overall time per message once the connection has been set up. On average, the middleware adds 16% to performance overhead for a medium sized message. Fig. 6 illustrates the measurements for a 3.5 KB message using TCP communication. In our experiments the performance overhead for MQTT was also 16%. This value increases up to 25% for smaller size messages and messages larger than 1MB. In the first case, the overhead is due to greater impact of the message size, while in the second, the main factor is the JSON parsing, which is more costly for larger messages.

A proof-of-concept implementation

We built a fully-fledged middleware incorporating a rich range of functionality. The current implementation of ComFlux is meant to facilitate the deployment of broad functionality for interactive applications. The key contribution of this paper is to provide a practical demonstration of the potential for dynamic, external command and control capabilities. As such, our focus was not on performance considerations, but on flexibility, reliability, and usability. For instance, messages are type-checked at both the sender and the receiver. Additionally, the mapping process enforces negotiation of communication and access control mechanisms and compliance with a common interaction type. These measurements are therefore intended to demonstrate the feasibility of embedding a command and control infrastructure within a communications framework. Our implementation was developed to support interactive applications for museums, in a dynamic, ad-hoc, people-centric environment, and was designed accordingly.

Optimisations

We plan several ways to optimise the performance. Firstly, transmission-related optimisations will be added to leverage per-message performance. Message type-checking can be simplified and even eliminated for a wide range of applications. While such a capability is the default for streaming interactions –communication is point-to point with non structured messages– all other endpoint types require and check against a schema. For illustration, communication modules that interface with existing services (e.g., MQTT brokers or REST web services) do not need transmission to conform to a JSON message structure, as compliance can be handled at the module or application level. Furthermore, in performance-oriented application examples, the preference may be to avoid a message structure. Secondly, we envisage ComFlux to run as a common system-wide middleware for all applications. Currently it is spawned by each application individually. Thirdly, ComFlux can leverage the performance of constrained embedded devices by running on a separate machine, in the local network.

Vi Related Work

Our approach complements existing work in context awareness, service composition and industry trends. Our vision is to integrate with research in pervasive computing on context aware, composable and adaptable systems, to facilitate the development and deployment of IoT applications.

Commercial solutions

Consumer solutions attempt to integrate a large range of IoT devices into “smart home” platforms. Apple HomeKit [14], Amazon Alexa and Echo [24] and Google Home [25] are commercial solutions for managing multiple third-party devices under a single user interface, via a voice activated hub and a phone. While they integrate the most popular devices and services, they tend to focus on reinforcing their own technical ecosystem, often in a more centralised manager, rather than the system-wide control and coordination.

In terms of development platforms, many focus on exploiting event-driven architectures and involve the network stack, cloud services, and a suite of APIs for web or smartphone. They require an account registration and sometimes a periodic fee. There are commercial PaaS solutions (Apple HomeKit [14], ThingWorx [15], Kaa [26], Xively [27], etc.), development frameworks (IoTivity/AllJoin [28]) and embedded OSs (e.g., Contiki [29], mbed OS [30]). Typically, the frameworks rely on the cloud for device management and do not always integrate easily with existing solutions (vendor ‘lock-in’, reinforces a technical ecosystem). But even when they allow decentralised deployment, they still do not support external control — rather, the configuration and preferences tend to be specified by developers, embedded in application code. As we have argued, the wider pervasive vision requires more, including the flexibility to externally and at run-time, adapt system interactions across applications.

IFTTT [31], is a service allowing the creation of a ‘recipes’ (a series of conditional statements), that trigger specific API functions of various components and services in response to particular events. Though providing a useful and accessible way forward in enabling the coordination and interaction of disparate components, it functions as ‘API-glue’; the possible functionality is limited to that exposed by the specific APIs of the components, and there is no consistent management framework in any reconfiguration is limited to what each component developer decides to explicitly include in their API. Moreover, the often closed and opaque tendencies of many commercial offerings are harder to incorporate into research.

IoT Management: adaptation and composition

Management of components in heterogeneous environments remains a research challenge [1, 2, 32]. The vision is that functionality should be encapsulated in modules, then composed and reused in different circumstances [17, 33, 34], but has yet to be fully realised.

Aggregate programming [34] is a paradigm for IoT development to facilitate design, creation, and maintenance of IoT systems. A proposed model for the development of complex components is that basic, distributed components —such as sensing, communication, localisation— are encapsulated and composed with one another into building blocks that can be adapted and reused. ComFlux relies on a similar overarching principle: system composition of heterogeneous building blocks. Moreover in ComFlux, composition can be performed ad-hoc, by external control, in response to triggers or commands from the environment.

SBUS is a stream-management infrastructure that has been used as a testbed for city-wide transport monitoring, healthcare and lifestyle management [35]. Our work on ComFlux redefines SBUS, as is necessary for the broader vision of pervasive computing. Specifically, ComFlux introduces a modular architecture, which not only extends the reconfiguration capabilities and increases flexibility, but also assists the longevity of deployments by enabling reconfiguration when/where necessary.

Context awareness

has been understood as the capacity of a component to acquire information from and reason about the environment, then adapt its behaviour accordingly. Most of the proposed solutions collect data from a limited number of physical (hardware) and virtual (software) sources [16] and focus on retrieval of data, with data management in the middleware layer. A great challenge in pervasive computing has been the management of components in the environment [36, 2, 17, 16]. The early work in [36] describes a small-scale indoor application for sensor-driven context-aware computing that enables applications to follow mobile users as they move around a building.

Targeting the development of adaptive systems, [18] proposes RCSM, a middleware and interface for specifying context-aware interfaces on the same device. It is adaptive in the sense that it changes its management and resource discovery to those of the local applications, but it does not communicate externally.

For an Android smart-phone, the work in [37] proposes decoupling application logic from complex adaptation decisions. CAreDroid uses a set of context-sensitive functions that offer runtime support for multiple, concurrent, context-aware applications accessing sensors. In [38] the authors propose a framework that supports reconfiguration for both internal (application triggered, API calls) and external (application unaware, contextual) events. For both types of trigger, the reconfiguration needs to recognise external triggers and program them into the middleware logic. In our approach, the reconfiguration is direct and data-driven, by connecting and interacting with the component.

The authors in [39] address similar problems of pervasive environments: heterogeneity, wide distribution, dynamism, and mobility. Their work presents an architectural model for context awareness that supports contextual data acquisition, control over functionality within the environment, and processing of contextual information. However, this approach relies on central components (servers) to maintain and process contextual information, including handling actuation commands. Furthermore, the design does not allow runtime management of middleware modules.

In [33] the authors identify the following context-based changes: peer coordination, process-service adaptation and utility-service adaptation. In the proposed system, Context-Aware Pervasive Service Composition (CAPSC), a composition adaptor receives commands resulting from context reasoning, then chooses and performs the service composition. Composition and adaptation are performed indirectly and triggered internally as a result of reasoning about the environment.

In the above examples, applications react to context triggers in a programatic way. In contrast, ComFlux enables contextual adaptation, by allowing—given the requisite permissions—direct control, management and mediation over components through the communications infrastructure (messaging), rather than in code. This allows flexible, ad-hoc and dynamic adaptations, capable of supporting customisation, personalisation, and evolution over time with minimal intervention overheads.

Smart Cities and pervasive gaming

There is interest among researchers and creative industries’ professionals in the development of games [40, 10, 19], which we consider a direct application of ComFlux. In a smart city scenario, [19] demarcates what future games might be like: interacting seamlessly with the environment, using public infrastructure in new ways and engaging citizens. Our contribution is a paradigm shift from API-based applications, where our external command and control will help development and deployment of interactive connected worlds, offering far richer experiences by allowing the integration of existing (ComFlux dedicated) infrastructure.

Privacy and security

Pervasive computing —with an emphasis on consumer IoT— is envisaged to become open, dynamic, and involve many actors with different motivations and capabilities. Security and data privacy receive considerable attention [41, 42]. Access control regarding devices’ capabilities, and the exchange of data is fundamental, and we believe that our approach an open, modular access control regime, enabling permissions to be dynamically managed will be an important mechanism for assisting in managing the security concerns of a highly dynamic environment.

A related concern is the legal and compliance concerns of the IoT. This was considered in [43], which suggested that a legally-compliant IoT can be enabled through policy enforcement (control and audit) over data exchange. ComFlux provides some of the capabilities necessary for realising this, which we will explore in future work.

Finally, though pervasive computing is an active area of research, not always is there code available for the community to take forward. An explicit aim of our work is to encourage use, experimentation and extension by the research community, by making the ComFlux platform (source and documentation) openly available (www.comflux.net).

Vii Concluding remarks

The vision for pervasive computing involves enabling contextual functionality adapted to suit the individuals and integrated seamlessly with a dynamic environment. We have argued that key to realising this vision is a modular approach in which components are composed and coordinated across management “silos” by enabling runtime external management of component connections and capabilities.

We introduced in this paper ComFlux, a modular infrastructure that demonstrated the feasibility of an external management regime. We also illustrated the practical considerations and proposed an implementation which demonstrates the feasibility of our approach with acceptable performance overhead.

Our approach is intended as a tangible step forward in enabling a range of new functional possibilities as presented in Section II. We have already shown that a general implementation aimed to support interactive exhibits in cultural and event spaces, see Section IV. By making the ComFlux source available, and describing its conceptual underpinnings, our aim is to provide the community with a practical basis for an external command-and-control regime, to move towards the broader vision of pervasive computing.

References

  • [1]

    M. Satyanarayanan, “Pervasive computing: vision and challenges,”

    IEEE Personal Communications, vol. 8, no. 4, pp. 10–17, Aug 2001.
  • [2] R. Cáceres and A. Friday, “Ubicomp systems at 20: Progress, opportunities, and challenges,” IEEE Pervasive Computing, vol. 11, no. 1, pp. 14–21, 2012.
  • [3] G. Mulligan, “Foreword,” in From Machine-To-Machine to the Internet of Things, J. H ller, V. Tsiatsis, C. Mulligan, S. Karnouskos, S. Avesand, and D. Boyle, Eds.  Oxford Academic Press, 2014, pp. xiii – xiv.
  • [4] P. Stone, “Hacking unicorns with web bluetooth,” February 2017. [Online]. Available: https://www.contextis.com/resources/blog/hacking-unicorns-web-bluetooth/
  • [5] E. Ronen, A. Shamir, A. Weingarten, and C. O’Flynn, “Iot goes nuclear: Creating a zigbee chain reaction,” in 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017.  IEEE Computer Society, 2017, pp. 195–212.
  • [6] M. Montola, J. Stenros, and A. Waern,

    Pervasive Games: Theory and Design

    .  San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2009.
  • [7] S. Benford, C. Magerkurth, and P. Ljungstrand, “Bridging the physical and digital in pervasive gaming,” Commun. ACM, vol. 48, no. 3, pp. 54–57, 2005.
  • [8] B. Schouten, G. Ferri, M. de Lange, and K. Millenaar, Games as Strong Concepts for City-Making.  Singapore: Springer Singapore, 2017, pp. 23–45.
  • [9] G. P. e. Marinos Ioannides, Nadia Magnenat-Thalmann, Mixed Reality and Gamification for Cultural Heritage.  Springer, 2017.
  • [10] A. Ricci, M. Piunti, L. Tummolini, and C. Castelfranchi, “The mirror world: Preparing for mixed-reality living,” IEEE Pervasive Computing, vol. 14, no. 2, pp. 60–63, 2015.
  • [11] “Fitbit developer.” [Online]. Available: https://dev.fitbit.com/
  • [12] “Philips Hue API.” [Online]. Available: https://www.developers.meethue.com/
  • [13] “Nest developers.” [Online]. Available: https://developers.nest.com/
  • [14] “HomeKit - Apple.” [Online]. Available: https://www.apple.com/ios/home/
  • [15] “ThingWorx.” [Online]. Available: https://www.thingworx.com/
  • [16] C. Perera, A. B. Zaslavsky, P. Christen, and D. Georgakopoulos, “Context aware computing for the internet of things: A survey,” IEEE Communications Surveys and Tutorials, vol. 16, no. 1, pp. 414–454, 2014.
  • [17] P. Makris, D. N. Skoutas, and C. Skianis, “A survey on context-aware mobile and wireless networking: On networking and computing environments’ integration,” IEEE Communications Surveys and Tutorials, vol. 15, no. 1, pp. 362–386, 2013.
  • [18] S. S. Yau and F. Karim, “An adaptive middleware for context-sensitive communications for real-time applications in ubiquitous computing environments,” Real-Time Systems, vol. 26, no. 1, pp. 29–61, 2004.
  • [19] R. P. Spicer, S. M. Russell, and E. S. Rosenberg, “The mixed reality of things: emerging challenges for human-information interaction,” SPIE Proceedings, vol. 10207, pp. 10 207 – 10 207 – 12, 2017.
  • [20] “Science Museum Lates.” [Online]. Available: http://www.sciencemuseum.org.uk/visitmuseum/plan_your_visit/lates
  • [21] A. Banks and R. G. (Eds.), “MQTT Version 3.1.1,” OASIS Standard, Tech. Rep., October 2014. [Online]. Available: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html
  • [22] “libmosquitto – Mosquitto C library.” [Online]. Available: https://mosquitto.org/man/libmosquitto-3.html
  • [23] “Mosquitto broker.” [Online]. Available: https://mosquitto.org
  • [24] “Amazon Alexa for Developers.” [Online]. Available: https://developer.amazon.com/alexa/
  • [25] “Google Home.” [Online]. Available: https://madeby.google.com/home/
  • [26] “Kaa Open – Source IoT Platform.” [Online]. Available: https://www.kaaproject.org/
  • [27] “Xively – IoT Platform for Connected Devices.” [Online]. Available: https://www.xively.com/
  • [28] “IoTivity.” [Online]. Available: https://www.iotivity.org/
  • [29] “Contiki: The Open Source Operating System for the Internet of Things.” [Online]. Available: www.contiki-os.org/
  • [30] “mbed OS.” [Online]. Available: https://www.mbed.com/en/platform/mbed-os/
  • [31] “IFTTT – If This Than That.” [Online]. Available: https://ifttt.com/
  • [32] R. Alur, E. D. Berger, A. W. Drobnis, L. Fix, K. Fu, G. D. Hager, D. P. Lopresti, K. Nahrstedt, E. D. Mynatt, S. Patel, J. Rexford, J. A. Stankovic, and B. G. Zorn, “Systems computing challenges in the internet of things,” CoRR, vol. abs/1604.02980, 2016. [Online]. Available: http://arxiv.org/abs/1604.02980
  • [33] J. Zhou, E. Gilman, J. Palola, J. Riekki, M. Ylianttila, and J. Sun, “Context-aware pervasive service composition and its implementation,” Personal and Ubiquitous Computing, vol. 15, no. 3, pp. 291–303, 2011.
  • [34] J. Beal, D. Pianini, and M. Viroli, “Aggregate programming for the internet of things,” IEEE Computer, vol. 48, no. 9, pp. 22–30, 2015.
  • [35] J. Singh and J. Bacon, “On middleware for emerging health services,” J. Internet Services and Applications, vol. 5, no. 1, pp. 6:1–6:19, 2014.
  • [36] A. Harter, A. Hopper, P. Steggles, A. Ward, and P. Webster, “The anatomy of a context-aware application,” Wireless Networks, vol. 8, no. 2-3, pp. 187–197, 2002.
  • [37] S. Elmalaki, L. F. Wanner, and M. B. Srivastava, “Caredroid: Adaptation framework for android context-aware applications,” in Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, MobiCom ’15, Paris, France, September 7-11, 2015.  ACM, 2015, pp. 386–399.
  • [38] A. Agirre, J. Parra, A. Armentia, E. Estévez-Estévez, and M. Marcos, “Qos aware middleware support for dynamically reconfigurable component based iot applications,” International Journal of Distributed Sensor Networks, vol. 12, no. 4, pp. 2 702 789:1–2 702 789:17, 2016.
  • [39] J. L. Lopes, M. Z. Gusmão, R. S. de Souza, P. Davet, A. Souza, C. A. da Costa, J. L. V. Barbosa, A. M. Pernas, A. C. Yamin, and C. F. R. Geyer, “Towards a distributed architecture for context-aware mobile applications in ubicomp,” in 19th Brazilian Symposium on Multimedia and the Web, WebMedia ’13, Salvador, Brazil, November 5-8, 2013.  ACM, 2013, pp. 43–50.
  • [40] M. Cowling, J. Tanenbaum, J. Birt, and K. Tanenbaum, “Augmenting reality for augmented reality,” interactions, vol. 24, no. 1, pp. 42–45, Dec. 2016.
  • [41] C. Luo, A. Fylakis, J. Partala, S. Klakegg, J. Gonçalves, K. Liang, T. Seppänen, and V. Kostakos, “A data hiding approach for sensitive smartphone data,” in Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2016, Heidelberg, Germany, September 12-16, 2016.  ACM, 2016, pp. 557–468.
  • [42] J. Singh, J. Bacon, and D. M. Eyers, “Policy enforcement within emerging distributed, event-based systems,” in The 8th ACM International Conference on Distributed Event-Based Systems, DEBS ’14, Mumbai, India, May 26-29, 2014.  ACM, 2014, pp. 246–255.
  • [43] J. Singh, T. F. J. Pasquier, J. Bacon, J. E. Powles, R. Diaconu, and D. M. Eyers, “Big ideas paper: Policy-driven middleware for a legally-compliant internet of things,” in Proceedings of the 17th International Middleware Conference, Trento, Italy, December 12 - 16, 2016, 2016, pp. 13:1–13:15.