DeepAI AI Chat
Log In Sign Up

A Component-Based Approach to Traffic Data Wrangling

by   Hendrik Mölder, et al.

We produce an increasing amount of data. This is positive as it allows us to make better informed decisions if we can base them on a lot of data. However, in many domains the `raw' data that is produced, is not usable for analysis due to unreadable format, errors, noise, inconsistencies or other factors. An example of such domain is traffic – traffic data can be used for impactful decision-making from short-term problems to large-scale infrastructure projects. We call the process of preparing data for consumption Data Wrangling. Several data wrangling tools exist that are easy to use and provide general functionality. However, no one tool is capable of performing complex domain-specific data wrangling operations. The author of this project has chosen two popular programming languages for data science – R and Python – for implementing traffic data wrangling operators as web services. These web services expose HTTP (Hypertext Transfer Protocol) REST (Representational State Transfer) APIs (Application Programming Interfaces), which can be used for integrating the services into another system. As traffic data analysts often lack the necessary programming skills required for working with complex services, an abstraction layer was designed by the author. In the abstraction layer, the author wrapped the data wrangling services inside Taverna components – this made the services usable via an easy-to-use GUI (Graphical User Interface) provided by Taverna Workbench, which is a program suitable for carrying out data wrangling tasks. This also enables reuse of the components in other workflows. The data wrangling components were tested and validated by using them for two common traffic data wrangling requests. Datasets from Transport for Greater Manchester (TfGM) and the Met Office were used to carry out the experiments.


DSL-driven Integration of HTTP Services in DIME

As the integration of web services into web applications becomes more an...

FastWARC: Optimizing Large-Scale Web Archive Analytics

Web search and other large-scale web data analytics rely on processing a...

JepREST: Functional tests for distributed REST applications

Application services often support mobile and web applications with REST...

Challenges and Governance Solutions for Data Science Services based on Open Data and APIs

Increasingly common open data and open application programming interface...

PyXNAT: XNAT in Python

As neuroimaging databases grow in size and complexity, the time research...

BioSimulators: a central registry of simulation engines and services for recommending specific tools

Computational models have great potential to accelerate bioscience, bioe...