Crowd Sensing and Living Lab Outdoor Experimentation Made Easy

by   Evangelos Pournaras, et al.

Living lab outdoor experimentation using pervasive computing provides new opportunities: higher realism, external validity and socio-spatio-temporal observations in large scale. However, experimentation `in the wild' is complex and costly. Noise, biases, privacy concerns, compliance with standards of ethical review boards, remote moderation, control of experimental conditions and equipment perplex the collection of high-quality data for causal inference. This article introduces Smart Agora, a novel open-source software platform for rigorous systematic outdoor experimentation. Without writing a single line of code, highly complex experimental scenarios are visually designed and automatically deployed to smart phones. Novel geolocated survey and sensor data are collected subject of participants verifying desired experimental conditions, for instance, their localization at certain urban spots. This new approach drastically improves the quality and purposefulness of crowd sensing, tailored to conditions that confirm/reject hypotheses. The features that support this innovative functionality and the broad spectrum of its applicability are demonstrated.


page 1

page 3

page 5


Aveiro Tech City Living Lab: A Communication, Sensing and Computing Platform for City Environments

This article presents the deployment and experimentation architecture of...

Extended Reality (XR) Remote Research: a Survey of Drawbacks and Opportunities

Extended Reality (XR) technology - such as virtual and augmented reality...

The Lab vs The Crowd: An Investigation into Data Quality for Neural Dialogue Models

Challenges around collecting and processing quality data have hampered p...

Citizen Social Lab: A digital platform for human behaviour experimentation within a citizen science framework

Cooperation is one of the behavioral traits that define human beings, ho...

Using Camouflaged Cyber Simulations as a Model to Ensure Validity in Cybersecurity Experimentation

Experimental research methods describe standards to safeguard scientific...

PlanAlyzer: Assessing Threats to the Validity of Online Experiments

Online experiments are ubiquitous. As the scale of experiments has grown...

The Smart Agora Living Lab

The Smart Agora living lab is an open-source crowd sensing framework for visually designing and carrying out complex outdoor socio-spatio-temporal experiments. It consists of the Smart Agora Dashboard and the Smart Agora App. The Smart Agora platform is documented and can be easily tested in a provided virtual machine [15]. All these are listed in ‘Software Accessibility’, at the end of this article.

Experimental design: Smart Agora Dashboard

The Smart Agora Dashboard is a web platform designed to simplify the creation of complex outdoor experimental scenarios using pervasive devices such as smart phones. The designer of Smart Agora experiments creates geographical points of interest where participants share experimental data, e.g. geolocated survey and sensor data. This dashboard provides the option to reuse and modularize several complex data collection processes as well as deploy them at different time points with different participants. The design of experimental scenarios is visual and interactive without the need to write a single line of code. However, each experimental scenario can be fully encoded and parsed within JSON files that allow automated generation of complex experiments. For instance, consider experiments that require the collection of data at every bus or tram station in a city. The points of interests can be automatically loaded by including the location of the stations in the JSON files, along with the necessary data collection configurations. The following JSON 1 code listing illustrates an example of how data collection is encoded.

1  {
2    "Id": "AXeG00HIQMa8aD8nfimV",
3    "Name": "Simple_09022021_134527",
4    "Url": "",
5    "Metadata": {
6      "record": {
7        "StartAndDestinationModel": {
8          "StartLatitude": null,
9          "StartLongitude": null,
10          "DestinationLatitude": null,
11          "DestinationLongitude": null,
12          "Mode": "Simple",
13          "DefaultCredit": "3"
14        },
15        "SampleDataModel": [{
16          "id": 1,
17          "Question": "How dangerous for bikers was the last section? ",
18          "Type": "radio",
19          "Latitude": "47.3715915",
20          "Longitude": "8.538603799999999",
21          "Sensor": [{
22            "id": 1,
23            "Name": "Gyroscope"
24          },{
25            "id": 2,
26            "Name": "Location"
27          }],
28          "Time": "3",
29          "Frequency": "Medium",
30          "Sequence": "Disable",
31          "Visibility": "true",
32          "Mandatory": "true",
33          "Option": [{
34            "id": 1,
35            "Name": "Safe",
36            "NextQuestion": null,
37            "Credits": ""
38          },{
39            "id": 2,
40            "Name": "Dangerous",
41            "NextQuestion": null,
42            "Credits": ""
43          },],
44          "Combination": null,
45          "Vicinity": "25"
46        }]
47      }
48  }}
JSON 1: Example of a data collection asset. It contains configurations for a point of interest coming with a survey question with two options to choose from as well as two sensor data collections (gyroscope and GPS location).

Geolocated data collection: Smart Agora App

The Smart Agora app is an Android app that allows participants to subscribe and participate to experiments. These experiments are loaded in an augmented map that contains the points of interest that participants visit to answer questions on spot, while sharing smart phone sensor data (see Figure 1). Visiting the points of interests is verified when participants are in close proximity with a point of interest measured by a radar-like radius of a localization cycle. A similar method with an ellipse is also supported, where the shortest axis of the ellipse points to the direction of the next point of interest [8]. This elliptical method of localization could be suitable in urban environments with narrow streets. Smart Agora localization currently operates based on GPS technology. However, approaches based on distributed ledgers operating with LoRaWAN and adhoc networks can be more secure and reliable alternatives [17].

Figure 1: The augmented map of the Smart Agora App.

Innovations and special features

Smart Agora supports by design innovative experiments that improve data collection quality. Compared to continuous privacy-intrusive sensing approaches [18], Smart Agora collects minimal data, tailored to specific context and purpose and as such it is easier for the designers of the experiments to meet minimal data collection requirements of ethical review boards. This is made possible by making participants’ responses subject of proving witnessed presence [17]. Answering survey questions and sharing sensor data is performed after verification of the location or another criterion related to the situational awareness of participants using QR codes, challenge questions, puzzles and CAPTCHA-like tests. This is useful in experiments with outdoor group interactions in which group members require to prove their identity and membership to each other group member. Similarly, the observations of experiments involving physical objects, e.g. the effect of a political poster, the response of a traffic light for pedestrians or arrival delays in a bus station, could be verified with a QR code on the physical object. These verification processes result in automating the control of experimental conditions that is usually performed manually by lab moderators in indoor lab environments.

Fields of application

Proving witnessed presence has a significant impact on improving the quality of data collection. Data sharing is designed as responsible testimonies and citizens’ interventions [17]. This makes Smart Agora highly versatile and applicable in different living lab manifestations: (i) Science - scientists design experiments and participants are recruited to participate. Cycling risk assessments is an example here [4, 17]. (ii) Policy-making - policy makers design participatory decision-making and deliberation processes, while citizens participate directly in crowd-sourced decisions and bottom-up policy formation. The participatory budgeting field test in the city of Aarau is an example here [9]. (iii) Education

- teachers and lecturers design highly engaging outdoor learning activities and students learn out of the class. The 2018 MSc course “Data Science in Techno-socio-economic Systems” at ETH Zurich is an example here.

Smart Agora Crowd Sensing Modeling

Figure 2 introduces the modeling architecture of Smart Agora. It consists of four core elements: (i) project, (ii) asset, (iii) task and (iv) assignment. A crowd sensing project can contain several data collection assets that encode experimental scenarios. Each asset can be used in several unique data collection processes by creating a task and linking this task with the asset via an assignment to participants. A task results in data collected by the assigned participants according to an asset. All experimental processes can be designed visually without writing a single line of code. The Smart Agora architecture is implemented by the hive platform built in Go and Elasticsearch.

Figure 2: The Smart Agora modeling architecture.

Figure 3 illustrates the lifecycle of a Smart Agora experiment from design to deployment and completion of the data collection. The designer of the experiment creates an account to the Smart Agora Dashboard and logins to create a crowd sensing project. The participants of an experiment download the Smart Agora App and are granted access to the subscription of projects using an access code provided to them by the moderators of the experiment as part of the recruitment process.

Figure 3: The lifecycle of a Smart Agora experiment.

In the meantime, the designer of the experiment creates data collection assets that contain the points of interests with the survey questions and the desired mobile sensor data for collection. These assets are associated with tasks via the creation of assignments that control which participants can join the experiment. The participants can subscribe to the task assignments. This loads the points of interests and survey questions to their smart phone map, while it configures the sensor data collection, e.g. frequency. Upon completion of the task, the data are transferred to the Smart Agora database for storage and analysis. Smart Agora also provides the option to forward collected data to DIAS (, the Dynamic Intelligent Aggregation Service [16] for decentralized real-time data analytics [17].

The rest of this section illustrates the features of Smart Agora and how they support different outdoor experimental scenarios.

Crowd sensing project

A Smart Agora crowd sensing project is the top of the modeling hierarchy and facilitates a group of experimental processes and collected data. It is initially created by the designer of the experiment and contains multiple crowd sensing assets, tasks and assignments. By default, an auto-assignment option connects a created asset with the most recent task and is made available immediately to all available participants in the pool. A crowd sensing project can be used to group together a series of several experiments and collected data that serve a research project. In this way, the whole experimental design of a research project can be built incrementally and managed more efficiently.

Crowd sensing asset

The crowd sensing asset provides toolkits to design ubiquitous experiments and the collection of socio-spatio-temporal data. An asset is associated with a project. An experiment can be conducted in three navigational modalities:

  • Simple modality: Navigation to different points of interests occurs in any order. This experimental modality

    is provided for experiments with a large number of points of interests (e.g. all bus stations of a city), probably expanding to a large geographic area of several possible participants. Data collection is usually long-term and

    large in scale, making it suitable to compare experimental observations over different geographical areas and over time, i.e. evolving human behavior and environment.

  • Sequential modality: Navigation to different points of interests occurs in a predetermined order introducing in this way a well structured experimental process that has to be followed by participants (e.g. following given vehicle routing trips). The information into which a participant is exposed at a certain point of interest determines the behavior and survey responses at the next points of interests. This allows to observe and study systematically a socio-spatio-temporal phenomenon (e.g. mobility pattern) by tracing at check points (the points of interests) how it incrementally evolves. Compared to the simple navigation modality, this one is suitable for more controlled experimental scenarios with a lower number of points of interests and participants, with the geographical scale depending on the transport mode involved, e.g. walking pedestrians vs. cyclists, vs. vehicle drivers.

  • Dynamic modality: Navigation to different points of interests is determined by survey responses or by detected events in the collected sensor data. The next point of interest to visit is determined by the answer to a question at a previous point of interest. This feature allows the design of ‘what-if’ and gamification scenarios (‘serious games’) in which participants are involved in an outdoor living lab problem solving. An experiment conducted under the dynamic modality can be used to vary on-the-fly experimental conditions of participants based on their demographics or environmental variables they experience and detected within the sensor data.

Moreover, an asset consists of multiple geolocated survey questions. Each question consists of a location on a map (GPS coordinates), the question formulation, the question type (radio, checkbox, likert scale and text box) and an optional amount of rewards that will be provided to the participants for answering this question. These rewards can represent the recruitment compensation and incentives to preserve engagement or scale up data collection, while minimizing attrition effects [2]. Rewards can be monetary, discount coupons or capture value measured with a designed cryptocurrency (token). Smart Agora can couple with other systems to automatically manage transactions and the actual secure value exchange, for instance digital wallets that verify transactions on distributed ledgers (blockchain). Depending the question type, the designer may introduce the options to choose from. Additional rewards may be provided for each selected option to allow experimental economics research.

Each point of interest can be further augmented with the collection of smart phone sensor data including sensors such as: light, gyroscope, proximity, accelerometer, GPS location and noise. Sensor data are collected in a determined vicinity around the point of interest during a determined duration and at a low (every 2 sec), medium (every 250 ms) or high (every 200 ms) frequency. A large localization radius can provide unrestricted collection of sensor data in experimentation scenarios requiring a continuous and passive crowd sensing data. Sensor data and sensor fusion can be used for human activity recognition (traveling vs. walking vs. cycling) as well as social inferences (who speaks with whom on spot), including the inferences to prove witnessed presence [17].

The creation of an asset is a collection of multiple points of interests configured to collect any combination of survey question data and/or smart phone sensor data. They are created interactively via a visual wizard that guides the designer step by step to build a crowd sensing asset. At the end, this wizardl generates files, such as the JSON 1 code, that represent the experiment and the designer’s choices. Several assets can represent different control and treatment groups for comparisons.

Crowd sensing task

A crowd sensing task is determined within a project. It is associated with an asset via an assignment and as such it contains the collected experimental data. Tasks allow a degree of freedom in the data that can be collected by (re)using one or more assets. For instance, given a crowd sensing asset, the same data can be collected at different time points by the same or different groups of participants. And vice versa, the same data can be collected by different groups of participants at the same or different time points.

Crowd sensing assignment

An assignment connects an asset and a task to the participants of an experiment. The designer has control of which participants to choose from the ones recruited and available in the pool.

Smart Agora in Action: Use Cases

Four different use cases are outlined here to demonstrate the applicability spectrum of Smart Agora and its impact on outdoor experimentation.

Crowd sensing of cycling safety

The capability of Smart Agora to facilitate informed decisions by geolocating the decision-making process (proof of witnessed presence) is assessed in a study on cycling safety [17]. The perception of bike riders about the cycling safety of different urban spots in Zurich is compared to an empirical safety model [4]

built using official data of the Federal Roads Office collected from Swiss GeoAdmin. If the two safety estimations match, this is indication that witnessed presence in participatory crowd sensing can indeed provide information quality comparable to the official but costly data collection methods.

The empirical risk estimation of a route is derived by a continuous spatial accident risk model of Zurich [4]. This is the baseline that is compared to the perceived cycling risk estimated via 11 Smart Agora test users. A crowd sensing asset is designed using the sequential navigation modality with four points of interest. The cycling risk of each road section that connects two consecutive points of interests is assessed at the latter point. A likert scale question pops up in the Smart Agora app to evaluate cycling risk at a linear scale between 1. very safe to 5. very dangerous. All test users cycled at the same day and time to minimize biases originated from weather, light condition and the condition of different bikes.

A high matching is found between the empirical evidence from the cycling accident data and the cycling risk that citizens witness [17]. This motivates an alternative more legitimate policy-making approach that relies on high-quality crowd sensing data collected by verifying witnessed presence.

Geolocated voting for participatory budgeting

Along with collecting high-quality policy-making data, Smart Agora can also empower novel participatory processes that have earlier been too costly or complex for digital democracy. For instance, a participatory budgeting field test is planned to run in the city of Aarau in Switzerland, where citizens will decide themselves for a Smart City project to fund using a municipal budget of 10K CHF to improve their life [9]. Examples of such projects include transport decarbonization initiatives (e.g. bike sharing stations), digital cultural heritage, smart waste bins and other.

The use of Smart Agora for outdoor experimentation opens up several new research avenues that will be explored in this project. For instance, to what extent proving witnessed presence results in more legitimate voting outcomes? Letting citizens of a local district community to witness urban spots that will be transformed by Smart City projects has an added value at different levels: Citizens can physically come closer to each other to deliberate about the impact of the project on the community and urban surrounding. Citizens can improve their awareness by collecting evidence to support a more informed and responsible decision. This process reunites the digital and physical decision space of citizens, inspiring the revive of a new cyber-physical ‘ancient Agora of Athens’. Ultimately, this process shall result in a voting outcome accepted by the broader community and even by citizens that voted against the elected project [9].

Moreover, voting rules, e.g. majority voting vs. multi-option preferential voting, can influence voting outcomes as well as the capacity of local communities to reach legitimate decisions that promote consensus. Smart Agora can implement different voting rules applied to different treatment groups during the field test to make novel comparisons in the context of outdoor geolocated decision-making.

Decentralized sensing of transport sustainability

Beyond the persistence of collected data in the Smart Agora database for offline analysis, the integration of DIAS to Smart Agora provides capabilities for decentralized real-time data analytics [17]. In particular, collective measurements (summation, average, max, min, count, etc.) can be made and updated in real-time based on the most recent shared data of participants localized at points of interest. When participants depart from a point of interest, automated rollback operations are performed to preserve accurate estimations using exclusively localized participants. These novel calculations are performed in an interactive and cooperative fashion by participants’ devices and without the involvement of any centralized third party. DIAS implements the distributed communication protocol and a memory system for efficient localized data management, see earlier work for technical details [16, 17].

To demonstrate this progressive capability, a complex field test was conducted in Zurich with 6 test users to assess the sustainability of transport usage  [17]. The participants choose the preferred transport mean to reach two points of interest using a simplified linear model of sustainability: 0. Car, 1. Bus, 2. Train, 3. Tram, 4. Bike, 5. Walking. The mean sustainability is updated in real-time as participants arrive and depart from the points of interest, where they choose their preferred transport mean, see the video at ‘Software Accessibility’. The observed estimations of transport sustainability approximate well the actual sustainability values. This confirms the efficiency of Smart Agora to carry out decentralized real-time collective measurements.

COVID-19 and field experimentation

The COVID-19 pandemic creates several challenges for indoor experimentation. Participants and moderators are exposed to significant infection risks. Several university labs have been closed during lockdowns, while setting up a safe lab can be costly, complex and time-consuming, requiring institutional approvals along with new frameworks of safety and operation (ventilation, disinfection, COVID-19 testing, recruitment, ethical approvals, etc.).

In contrast, outdoor experimentation with Smart Agora entails lower infection risk for participants as experimental tasks are deployed automatically and remotely. Smart Agora can also provide a higher degree of freedom to scale up recruitment and ubiquitous data collection, lower moderation cost and higher realism. Moving experimentation outdoors provides new opportunities to limit sampling biases of indoor lab pools of participants, often overpopulated by young individuals with a student profile. Participants from older age groups can be engaged given the priority they have to vaccination as well as the opportunity to eliminate isolation of elderly during lockdown and to contribute to public good.

Recruitment for experimentation can be heavily affected due to COVID-19 governmental constraints on mobility. These can restrict people to move only in close proximity for essential purposes such as going to work, shopping or exercising. Other activities may be prohibited and even fined. Smart Agora provides new opportunities to seamlessly integrate experimental processes within permitted mobility ranges. For instance, instead of an indoor driving simulator, driving behavior can be studied during pandemics when citizens drive to work or for shopping by implementing a travel diary using Smart Agora. Similarly, Smart Agora can also serve COVID-19 research [13, 14], for instance understanding infection risk and epidemic spread by crowd sensing collective human behavior and actions.

Incentives to participate in experimentation may deviate from the norm during pandemics. Lab recruitment can be a flexible income support given the loss of jobs. Non-monetary incentives have a potential to strengthen the sense of community belonging using participatory and gamification elements [20] such as the ones of Smart Agora. These incentives may capture new intrinsic human values during lockdowns, for instance, higher interest in contributing to science, i.e. citizen science, outdoor exercising or well being.

Broader and Future Perspective

As Smart Cities co-evolve to complex ecosystems of compelling socio-technical complexity, living lab outdoor experimentation will remain a timely and significant challenge to tackle. Collecting crowd sensing data is not only the end goal but also the mean to reason about desired experimental conditions to carry out rigorous studies involving causal inferences. Verifying such conditions in an automated way opens up new avenues to carry out large-scale outdoor experimentation with lower biases, noise and moderation level. Smart Agora is an endeavor along these lines, made to support both experiment designers and participants in different manifestations of science, policy-making or education.

Several challenges though remain such as how to incentivize participation and minimize dropouts as well as identify and minimize unknown biases that will emerge in complex urban environments. Outdoor experimental participation requires new intrinsic values for incentivizing citizens’ contributions to science. Distributed ledgers and cryptoeconomic principles such as token curated registries are promising means to capture and securely exchange such values. This is also expected to scale up the number of participants in future use cases.

Despite the minimal and targeted data collection approach of Smart Agora, solutions for privacy protection and data sharing control will be implemented in future, for instance, differential privacy and homomorphic encryption to privately aggregate sensitive citizens’ data. Participants may require self-determined access control rights to sensor data to deliver certain crowd sensing tasks [16]. Hence, data collectors need to match with data suppliers based on shared values that go beyond monetary ones and resolve social dilemmas related to privacy, ethics, quality of life, comfort and other. For instance, citizens’ assemblies may play a role of ethical review boards determining experimental participation and as such paving the way towards new standards and norms on how to conduct experimental research. This will allow citizens to ‘shape’ and ethically align experiments to their own common values and in a bottom-up way, pushing towards a more socially responsible science.

Software Accessibility

The Smart Agora platfrom and supporting material are open-source and accessible via the links below:


This research is funded by the SNF NRP77 ‘Digital Transformation’ project “Digital Democracy: Innovations in Decision-making Processes”, #407740 187249: The authors would also like to thank Prof. Dirk Helbing and the Chair of Computational Social Science for supporting this research. Edward Gaere supported the integration of Smart Agora with DIAS [16], while Dr. Alexey Gokhberg provided the Hive interfaces to Smart Agora. The authors would also like to thank the 2018 students of the course “Data Science in Techno-socio-economic Systems” at ETH Zurich who used Smart Agora and provided invaluable feedback. Many thanks go to other Empower Polis team members as well as the Institute of Science Technology and Policy (ISTP) of ETH Zurich for running the ETH Policy Challenge and providing a venue to cultivate ideas for outdoor living lab experimentation.