Annotation of Car Trajectories based on Driving Patterns

05/15/2017 ∙ by Sobhan Moosavi, et al. ∙ Nationwide Mutual Insurance Company The Ohio State University 0

Nowadays, the ubiquity of various sensors enables the collection of voluminous datasets of car trajectories. Such datasets enable analysts to make sense of driving patterns and behaviors: in order to understand the behavior of drivers, one approach is to break a trajectory into its underlying patterns and then analyze that trajectory in terms of derived patterns. The process of trajectory segmentation is a function of various resources including a set of ground truth trajectories with their driving patterns. To the best of our knowledge, no such ground-truth dataset exists in the literature. In this paper, we describe a trajectory annotation framework and report our results to annotate a dataset of personal car trajectories. Our annotation methodology consists of a crowd-sourcing task followed by a precise process of aggregation. Our annotation process consists of two granularity levels, one to specify the annotation (segment border) and the other one to describe the type of the segment (e.g. speed-up, turn, merge, etc.). The output of our project, Dataset of Annotated Car Trajectories (DACT), is available online at .



There are no comments yet.


page 3

page 5

page 7

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Nowadays, the ubiquity of various sensors (accelerometer, barometer, GPS, etc.) enables the collection of voluminous datasets of car trajectories. Examples of such datasets are the New York taxi cab [1], Porto taxi cab [8], and GeoLife [9]. Such datasets enable analysts to make sense of driving patterns and behaviors. By exploring driver behaviors and comparing different drivers, an analyst provides solutions to increase drivers safety and decrease the risk.111The risk can be interpreted from an insurance perspective.

We consider a dataset of car trajectories where each of trajectory is a time-stamped sequence of data points and each data point is described using attributes such as speed, bearing, location, etc. In order to understand driver behavior, one approach is to break each trajectory into its underlying patterns, i.e., segmentation, and then describe the trajectory in terms of derived patterns. One example of a trajectory and its underlying patterns is depicted in Figure 1.

Figure 1: A sample trajectory with several underlying driving patterns specified by ovals. Also, red arrows show the points of transition between patterns, where new patterns commence.

Trajectory segmentation has been thoroughly studied in the literature. Buchin et al. propose to exploit spatiotemporal criteria (like changes in speed or direction of moving vehicle) for segmentation [5]. Explorations of multiple settings for stable criteria in terms of monotonic and non-monotonic attributes is another approach which is introduced by Alewijnse et al. [2]. Moreover, there are several other research tracks which aim to solve the segmentation problem by statistical modelings and other algorithmic approaches. For instance Alewijnse et al. [3] propose to use Brownian Bridge Model to model the movement and leverage a dynamic programming approach to discover segments. The aforementioned approaches are mostly focused on animal movement trajectories and do not discuss human behavior. However, there exists few generic approaches such as the seminal work by Anagnostopoulos et al. which leverages a geometric based approach for segmenting human generated trajectories [4]. Also in a recent study, we proposed an approach to transform a trajectory based on a statistical modeling and then applied a dynamic programming-based segmentation approach and identified the best number of segments by using Minimum Description Length (MDL) [7].

One of the main shortcomings of all aforementioned efforts is that the applicability and performance of the approach is often exhibited through formal statements or a few real-world use cases. This does not provide the evaluation of the underlying algorithms in practice. For such evaluation, there should be a set of labeled trajectories as ground truth, where the true segment borders are specified in trajectories by experts222Such task can be seen as a domain-specific task, where each domain requires its own expert for annotation.. However, to the best of our knowledge, there is no such dataset available providing this information.

In this paper we describe a novel trajectory annotation framework and present the methodology of our annotation process comprising two main steps: Expert-Annotation and Annotation-Aggregation. Moreover, we report the result of using the proposed framework to annotate a dataset of personal car trajectories. For annotation purpose, we ran a crowd-sourcing task followed by a precise process of aggregation.

The rest of this paper is organized as follows: First, we formally introduce the problem annotating car trajectories in Section 2. Next, trajectory annotation framework is discussed in details in Section 3, followed by a description of the trajectory dataset in Section 4. Section 5 provides annotation results. Finally, we conclude in Section 7.

2 Problem Statement

We are given a trajectory dataset of size , where each trajectory , , is a sequence of data points . Also, each data point , , is a tuple of the form . In this tuple, is the time stamp, shows the vehicle’s speed (mph), shows the change in direction of moving vehicle based on the direction in previous time stamp, and and represent the exact location of vehicle at the current time stamp.

Definition 1

Segmentation of a trajectory is the task of breaking to a set of non-overlapping sequences of sub-trajectories.

In Definition 1, the segmentation of a trajectory into segments, is about finding a set of cutting indexes that mark the end points of the segments in . Thus, we can define a set of cutting data points for the segmented trajectory as . Note that . All data points between indexes and , excluding point and including point , belong to the segment. An example of a segmented trajectory with all segments and cutting points is shown in Figure 1.

Definition 2

Annotation of a trajectory is the task of identification of segment borders (cut points) within using human expertise. Moreover, an expert can assign one or more labels (segment types) to each segment.

A label can be a description of the pattern which occurs in a segment. For example, segment in Figure 1 can be labeled as slow-down and segment as loop.

3 Annotation Framework

We propose a novel framework for trajectory annotation. This framework has two important components: Expert Annotation, and Annotation Aggregation. The former one refers to the main annotation task, where a set of trajectories are assigned to human experts for annotation. The second component is to finalize the annotations for each trajectory by aggregating existing annotations provided by different human experts and land on a consensus.

3.1 Expert Annotation

This component consists of three principled sub-procedures: Annotation Portal Preparation, Human Expert Preparation, and Annotation Process. In the following, we describe each sub-procedure in details.

Figure 2: Trajectory Annotation Portal. The trajectory under investigation is represented on a map (right). The speed and heading change profiles are also reported (left). An annotator can select a point on any of these three diagrams and annotate it by using controls in Box . The first step is to choose if the point represents the end of a segment or not (Segment, Maybe-Segment, or Non-Segment). The next step is to specify the type of the pattern (e.g. turn, speed-up, exit, etc.).

3.1.1 Preparing Annotation Portal

The user interface for our annotation process is a web-based portal which represents a set of information for each trajectory to enable experts make decisions on annotations. Our interactive portal is implemented in PHP and JavasScript. A screenshot of the portal is illustrated in Figure 2. Different components of this portal are as follows.

  • [leftmargin=*]

  • Annotator Data Box (Box in Figure 2) provides basic information such as annotator’s name and also the list of trajectories (trips) assigned to the annotator.

  • Annotation Box (Box in Figure 2) provides functionalities to enter annotations. For this aim, the annotator should first opt the annotation type (Segment, Maybe-Segment, or Non-Segment) and then select a type (e.g., speed-up, turn, exit, merge, etc.).

  • Trajectory Time Box (Box in Figure 2) provides information about the start and end time of the trajectory.

  • Speed Profile is the time series of speed values throughout the trajectory measured in miles per hour (mph). The X and Y axes provide time and speed data, respectively.

  • Heading Change Profile is the time series of direction change in direction of the moving vehicle. The change of heading for timestamp is calculated by , where shows the heading of vehicle for timestamp and gives the absolute value. The X and Y axes represent the time and change in heading values, respectively. We employ Plot.Ly API333 to construct both speed and head change diagrams.

  • Trajectory On Map: We illustrate location-oriented information of the trajectory on a geographical map. We employ Google Maps API444 to visualize maps.

The annotation process can be described as follows.

  • [leftmargin=*]

  • The annotator choose a trip ID and hit the load button.

  • Following the trajectory on any of three diagrams, the annotator may decide to choose a point as the end point of segment (pattern) by clicking on that.

  • To annotate the selected point (current segment), the annotator should choose the annotation type which is one of “Segment”, “Maybe-Segment”, or “Non-Segment”. The “Segment” choice is picked when the expert is completely sure about a point to be the end point of segment. For example, in Figure 2 we observe that the selected point on map is the end point of a segment which is a merge into a highway. However, in the case of uncertainty about the segment, an annotator can choose the “Maybe-Segment” option. For instance, there may be a pattern which is like a turn, but the annotator is not sure about it. Another case which may lead to choose the “Maybe-Segment” is where we have a sub-pattern within a main pattern. For example, we may have a slow-down within a turn (see Figure 3(a)). In such a case, the annotator can annotate the slow-down as “Maybe-Segment” and the turn as “Segment”. Finally, if the annotator wants to undo her previously assigned annotations, she can choose the same point and assign the “Non-Segment”.

  • Once the annotation type is chosen, the next step is to decide about the type of segment (pattern). The list of our observed patterns are described in Table 1.

  • The annotator proceeds the annotation process by selecting another point of the trajectory and iterating back to step . Once the annotator reached the end of the trajectory, she should submit her annotations.

We leverage some functionalities of Google Maps API to synchronize all three diagrams and enable coordinated views. This makes the annotation task more straightforward for human annotators. When an annotator moves the mouse on the map, she notices a red marker on Speed and Heading diagrams designating the same point. Moreover, the annotator can click on Show Annotation button anytime during the annotation process to track her previously marked annotations. Annotators can also benefit from the functionalities of Plot.Ly API to zoom-in into speed or heading diagrams in order to make wiser annotations.

Pattern Description Visual Demonstration
Exit Car exists a highway/road
Merge Car merges into a highway/road, in this case we have changes in speed value.
Exit-Merge Car exists a highway/road and merges into another one.
Loop Car takes the ramp and makes a complete loop to merge into a highway/road.
Car makes a turn which has significant effect
in change of direction of moving vehicle.
Car makes a smooth turn which has some effects
in change of direction of moving vehicle.
Left-Turn Car makes a left turn in an intersection.
Right-Turn Car makes a right turn in an intersection.
Car keeps moving to left and right which has continues effect in change of
direction of vehicle, as it is observable from heading change values.
Speed-Up Car speeds up to merge a highway, after traffic light, after stop sign, etc.
Car slows down because of traffic congestion, traffic light,
stop sign, intersection, etc.
Car stops behind a traffic light for a while, in such case,
we can see the speed is zero or has some minor changes.
Table 1: Description of different types of segments (patterns), which can be used as labels for identified segments.

3.1.2 Preparing Human Experts

In order to prepare human experts for annotation, we perform following steps:

  • Preparing a Tutorial: As tutorial, we prepared two set of resources: Instructive document which contains the description of the task and the annotation portal, and A short video clip to describe the task and annotate a sample trajectory by using the portal.

  • Running a Pilot Phase: We ran a pilot phase of annotation where we assigned two trajectories to each annotator. The goal of running this phase was to give this opportunity to the annotators to work with the portal, comprehend the task, digest the process, and reduce the potential errors for the expert-annotation phase.

3.1.3 Main Annotation

In this step, we assign a set of trajectories to each annotator. We randomly distribute trajectories among annotators, such that each trajectory is assigned to exactly two different annotators. The number of assigned trajectories was not the same for all annotators, and it was determined based on a combination of few constraints such as time, availability, and expertise. Among annotators, we had four students, two researchers and a car insurance expert.

3.2 Annotation Aggregation

The annotation aggregation is to finalize subjective decisions and generate the final dataset of annotated car trajectories. The aggregation task can be regarded in terms of three following important decision items: to accept, refine or reject an annotation.

The most important challenge in aggregation is the subjectivity

of annotation, where there may not be some strong agreements among annotators for some cases. In order to deal with such subjectivity, we perform the aggregation in two different phases: 1) Strict aggregation, and 2) Easy aggregation. We describe both later in this section. Moreover, we leverage a set of heuristic-based generated annotations along with human expert annotations to be used as input for aggregation process, as it will be described later in this section. Also, to successfully accomplish the aggregation task, we designed an

Aggregation Portal which we describe next.

3.2.1 Aggregation Portal

The desiderata for the aggregation task are as follows.

  • [leftmargin=*]

  • Trajectory data: We need the trajectory data in order to finalize the annotations. Such data includes speed profile, changes of heading, and also map representation.

  • Annotations by experts: This is the main input for aggregation. We need to have the annotations by different experts to be appropriately represented for analysis and decision making.

  • Decision making features in order to land on an annotation aggregation.

Figure 3: Annotation Aggregation Portal.. The trajectory is illustrated on map (right). The Speed and Change in Heading profiles are illustrated (left). The aggregation expert employs controls on the map to show or hide the annotations. To finalize annotations, the aggregator may decide to keep, refine, or ignore annotations. In order to make a decision about acceptance/refinement, the aggregator may choose a point on either of three diagrams, and then make a decision by using controls in Box .

To reach the above desiderata, we designed a new portal akin to the one for the expert annotation phase (Section 3.1.1) with extra features. A snapshot of this portal is depicted in Figure 3. The aggregation portal is composed of following components:

  • [leftmargin=*]

  • Trajectory Selection Box (Box in Figure 3) enables the aggregator expert to pick a trajectory for annotation.

  • Annotation Box (Box in Figure 3) is employed to finalize the annotations for a trajectory. The controls in this box are almost identical to ones in the annotation portal (Figure 2). However, the aggregator expert is able to choose multiple types to specify the segment type. This feature is considered to deal with subjectivity of the task.

  • Trajectory Time Box (Box in Figure 3) is used to show the real start and end time of the selected trajectory.

  • Speed Profile represents the time series of speed values for the selected trajectory.

  • Heading Change Profile represents the profile of the changes in heading values for the selected trajectory.

  • Trajectory On Map: The trajectory is represented on a geographical map and is enriched with some controls for the annotation task. Additional control for the aggregator is “Show/Hide Users Annots” and “Annotator” buttons. The first button is to show or hide the annotations which are made by experts during the expert-annotation phase (Section 3.1.3), and the second button is to choose the annotators who want to show or hide their annotations, instead of showing or hiding all annotators data by using the first button.

3.2.2 Strict Aggregation

During the strict aggregation phase, we focus on the subjectivity of the aggregation task and opt to maximize the usage of users’ input by considering following principles:

  • [leftmargin=*]

  • Catch any Existing Segment: Different annotators may identify different sets of segments in a trajectory. By strict aggregation, we try to catch any existing segment. In other words, we aim to annotate both main and intermediate segments (patterns). Main segments are those patterns which are more obvious, like an entire loop (Table 1), and intermediate segments are those patterns which occur inside a main segment, like the slow-down within a loop. Figure 3(a) shows an example of strict aggregation where an intermediate segment (S2) is annotated along with two main segments (S1, and S3). Main segments will be annotated as SEGMENT and intermediate segments as MAYBE-SEGMENT.

  • Use Strict Thresholds: When an annotator is looking for patterns like speed-up, slow-down, smooth-turn, or jiggling, the question is how to identify them? What are the requirements to catch a segment and label that with either of the mentioned types? The answer to such question is where we define the concept of strictness in contrast with easiness. During the strict aggregation, any change of the speed by at-least 5 mph is a significant change and the related segment will be annotated by speed-up or slow-down. Moreover, any continues change in heading values for five consecutive seconds555As we may have GPS drifts in data leading to erroneous heading values, we catch significant changes in heading values based on a continuous pattern, instead of a single change. is a significant change in heading and related pattern will be annotated as smooth-turn or jiggling. We found these numbers in trial-and-error experiments and based on the expertise of some of annotators.

  • Assign all Relevant Segment Types: Unlike the expert-annotation phase, the aggregator expert in aggregation phase can assign as many segment types as she thinks are relevant. As a common example, when a car exits a highway, a significant slow-down shall often happen. In such a case, the aggregator expert can assign both EXIT and SLOW-DOWN to the related segment.

(a) Strict Aggregation
(b) Easy Aggregation
Figure 4: Difference between strict (a) and easy (b) aggregation: In the first case, we consider both intermediate and main segments (patterns), while, for the second one, we just consider the main segments. Intermediate segments will be annotated as MAYBE-SEGMENT and the main segments as SEGMENT. Examples of main segments are S1 and S3 in (a) and an example of intermediate segment is S2 in (a).

3.2.3 Easy Aggregation

The easy aggregation is defined in contrast with strict aggregation, which is based on the following important principles.

  • [leftmargin=*]

  • Annotate Main Segments: Unlike the strict aggregation, the aggregator expert will just annotate the main segments and will ignore the intermediate ones during the easy-aggregation. An example of easy aggregation is shown in Figure 3(b), where both S1 and S2 are main segments (annotated as SEGMENT) and the slow down pattern (S2 in Figure 3(a)) is not selected by the aggregator.

  • Use Relaxed Thresholds: The second important difference between strict and easy aggregation is the selection of thresholds which will be used to catch patterns like speed-up, slow-down, smooth-turn, and jiggling. In case of easy aggregation, we use some relaxed version of the previously defined thresholds which are as follows: for speed, a change by at-least 10 mph is a significant change. Moreover, for heading, we look for a consecutive series of changes which last for at-least 10 seconds. By using these relaxed thresholds, we expect to see less patterns of above mentioned types.

  • Assign all Relevant Segment Types: Similar to strict aggregation, the aggregator expert will assign all relevant segment types to an identified segment.

3.2.4 Heuristic-Based Annotation

In order to deal with subjectivity and also to make sure to have a comprehensive set of potential annotations prior to aggregation, we leverage a set of heuristically generated annotation, in addition to human expert annotations (Section 3.1), to be used as main input for aggregation process. Being provided alongside human annotations, it enables experts to get to a consensus among annotators. The heuristic annotations are generated by an algorithm called AutoAnn. The results of the algorithm is a second-order guide to the aggregator to decide on the best segments. AutoAnn heuristics are based on obvious patterns we observed during initial investigations. AutoAnn’s logic is based on speed-based, position-based and heading-based heuristics. We stress that employed heuristics are very simplistic and cannot replace the human power of decision making. Nonetheless, the benefit of an automated annotator is two-folded.

AutoAnn scans each single point in a trajectory (in the ascending order of timestamps) and makes comparisons with neighbor points. The parameter depends on the temporal precision of records in the dataset. Based on our data observations, we experimentally set . Given the temporal precision of our dataset, i.e., seconds, implies that we analyze behaviors in a -second time window. For a given point and its neighbor points, AutoAnn preforms following heuristics.

  • [leftmargin=*]

  • Speed-wise, if falls into a local maxima among it’s neighbors (i.e., previous and subsequent points), then marks the end of a “speed-up” segment.

  • Similarly, if falls into a local minima, marks the end of a “slow-down” segment.

  • If holds as the end of a “slowdown” and the speed at is lower than a threshold, then marks the end of a “traffic-jam” segment. Based on our data observation, we experimentally set the low speed threshed to miles per hour. The intuition is that a serious slow-down should be due to a traffic jam. Normal slowdowns may occur due a slight turn or sun rays where the speed does not get lower than miles per hour.

  • If the speed of all previous points of are lower than the low speed threshold and the speed of the immediate point after is larger then the low speed threshed, the marks the end of a “traffic-light” segment.

  • We define heading-change as the first-order derivative of ’s and ’s heading values where is the immediate point after . To simplify computation, we replace derivatives with the mathematical subtraction. The point marks the end of a “turn” segment iff it satisfies following conditions: The heading-change of neighbors of are zero and The heading-change at is larger than a threshold. We experimentally set the high heading change threshold to .

  • If the position of the point is merely identical with the point which happened before , then marks the beginning and marks the end of a “loop” segment. The intuition is that when a driver performs a loop (for instance through a highway exit), the first and last locations of the loop are identical with a tiny error. However, this position-based heuristic cannot be applied on our data because the same situation is reported when the car is still. A part of our future work is to make this heuristic more sophisticated. One potential extension is to consider speed, the heuristic which seems to be valid is ’s speed is larger than a minimum threshold.

On our data, AutoAnn generates 2418 segments among which 10% are speed-ups, 59% are slow-downs, 6% are traffic-jams, 20% are traffic-lights and 5% are turns. Of course AutoAnn introduces many false positives and false negatives.

4 Trajectory Dataset

We employ a subset of a large-scale, real-world dataset of personal car trajectories, collected using highly accurate devices connected to On Board Diagnostic (OBD-II) port of vehicles. The data is collected with a consistent sampling rate of one second, whereas most of the existing public datasets provide inconsistent or lower rates for data collection process [1, 9, 8]. For instance, driving data is collected every other 15 seconds in [8] and data collection rate is varying between 1 and 5 seconds in [9].

The set of data items which were collected every second includes speed, acceleration, heading, GPS coordinates, and time stamp. We consider following criteria to obtain a subset of the dataset for annotation.

  • [leftmargin=*]

  • Remove trajectories with missing point(s).

  • Keep trajectories with diverse set of drivers.

  • Keep trajectories with a fair diversity of traffic condition.

  • Keep trajectories with a fair diversity of highway and metropolitan coverage.

Based on the above criteria, we obtained a sample of the trajectory dataset which is summarized in Table 2. The sampled dataset covers a fair amount of driving data covering about 13 hours of drive. Moreover, the diversity of data is another strength factor of this sampled dataset which is measured as the number of drivers who are involved in generation of the driving data. It is also worth mentioning that most of the trajectories in the sampled dataset occurred between 3pm and 7pm, thus they cover a fair amount of both rush hour and normal traffic condition.

Number of
Number of
Driving Time
Total Driving
in Highway
Total Driving
in City
Trajectory Length
50 19 13.3 Hours 12.3 Hours 1 Hours 16 Minutes
Table 2: Some statistical facts about the sampled Trajectory Dataset for annotation

5 Annotation Results

In this section we first provide some detailed description of inter-annotator agreement analysis. Then, we present the results for different annotation phases in terms of Statistical Facts, Distribution of Annotation Types, Distribution of Segment Types, and Agreement Analysis.

5.1 Inter-Annotator Agreement Analysis

One of the crucial experiments based on the results of annotation is to see how much agreement exist between the annotators. For this aim, we conduct an inter-annotator agreement analysis. We formulate this task as follows: suppose for a trajectory , we have two annotators A and B who provided two different sets of annotations and , where each is an annotation tuple specifying the exact point of annotation in as well as types of annotation and segment. Lets assume each annotation tuple is like a label Yes for corresponding data point of . Similarly, we can assume each data point of which we don’t have any annotation for that is labeled as No. In this way, Table 3 simplifies the agreement between A and B. In this table, the non-negative integer shows the intersection between and . Moreover, the non-negative integer shows for how many data points of trajectory , both annotators A and B decided to not assign any annotation. Similarly, () shows the number of data points of which for them we have annotations in (), but have no annotation in ().

We use two different measures to obtain correlation between and , Cohen’s Kappa and Overlap. Cohen’s Kappa is formulated by Equation 1, regarding its original formulation as proposed by [6] and information in Table 3:


where , , , and . Note that by using this measure, we consider the possibility of the agreement occurring by chance. Also, based on Table 3, we define the Overlap function by Equation 2:


As one can see, the Overlap is, in some sense, the fraction of intersection between and . Now, we describe how to obtain numbers , , , and in Table 3. Given annotation tuples and , we calculate their Haversine distance666See using the latitude and longitude of each annotation tuple. Then, if their distance is less than a pre-defined threshold , we say they are correlated and will increase , otherwise, they are uncorrelated and will increase either (if failed to find a match for ) or (if failed to find a match for ). Finally, after finalizing the values of , , and , we set , where is the number of data points in trajectory .

Yes No
A Yes a b
No c d
Table 3: An example of annotation by two annotators. The annotation task is about labeling of the inputs as Yes or No. Numbers , , , and show the agreements/disagreements between annotators A and B.

5.2 Results and Comparison

5.2.1 Statistical Facts

In this section, we discuss some statistical facts about the annotation phases. Table 4 provides some statistics about all three phases of annotation. We employed seven annotators for expert annotation phase and one aggregator annotator for each of the the strict and easy aggregation phases.

We observe in Table 4 that the most time-consuming phase was the strict aggregation and the number of extracted segments during this phase was also the largest among all phases. Another observation is on the average number of extracted segments for each trajectory, where this number is similar for expert-annotation and easy-aggregation phases, but almost half of the extracted segments for strict-aggregation. Moreover, the annotators and the aggregator of strict-aggregation spent the same amount of time to annotate a new segment (31 seconds). However, the aggregator provided more segments for each trajectory on average. This justifies the reason of spending about 25 minutes to segment a single trajectory by the aggregator, where annotator just spent about 10 minutes on average.

Annotation Phase
Num of Segments
Specified By Expert(s)
Avg Num of Segments
for each Trajectory
Total Active
Annotation Time
Avg Annotation Time
for a Single Trajectory
Avg Time to Specify
a Single Segment
Expert Annotation 1,997 20 17 Hours 10.3 Minutes 31 Seconds
Strict Aggregation 2,465 49 21 Hours 25.2 Minutes 31 Seconds
Easy Aggregation 1,372 27 6.3 Hours 7.5 Minutes 15 Seconds
Table 4: Some statistical facts about different phases of annotation

5.2.2 Distribution of Annotation Types

Figure 5 shows the frequency distribution of annotation types for three phases of annotation. Recall that the possible annotation types are SEGMENT and MAYBE-SEGMENT, where we mostly refer to former one as main and the later one as intermediate segment ( Section 3.2.2).

Figure 5: Frequency distribution of Annotation Types (SEGMENT and MAYBE-SEGMENT).

In Figure 5, we observe the same pattern of (relative) frequency for annotation types, where the majority of annotations for each phase is related to type SEGMENT. However, an interesting observation is that for expert-annotation and strict-aggregation phases, we have about 80% of annotations as SEGMENT, while this number is about 90% for easy-aggregation. This is interpreted as an outcome of using different principles for easy aggregation (as discussed in Section 3.2.3).

5.2.3 Distribution of Segment Types

In this section, we discuss frequency distribution of different segment types (Figure 6). We observe that the most common segment types are speed-up, slow-down, and smooth-turn which cover about 55%, 70%, and 66% of annotations for expert-annotation, strict-aggregation, and easy-aggregation, respectively. Moreover, there is a tendency for the aggregator to assign less segments of the generic type Other, in comparison to annotators. We also observe a balance between the frequency of correlated segment types. For example, there is some frequency-balance between Exit and Merge, Left-Turn and Right-Turn, or Speed-Up and Slow-Down, based on all three phases of annotation.

Figure 6: Frequency distribution of Segment Types, as are listed in Table 1, for different annotation phases.

5.2.4 Agreement Analysis

We report the agreement for each phase of the annotation as follows:

  • [leftmargin=*]

  • Expert-Annotation phase: given a distance threshold , we first calculate the agreement for each pair of annotators, who annotated the same trajectory in the dataset (recall that each trajectory was assigned to two experts during the expert-annotation phase). Then, we obtain the average agreement between all pairs of annotators and report it as the agreement based on the distance threshold .

  • Strict-Aggregation phase: given a distance threshold and a trajectory , we first calculate the agreement between the aggregator of strict-aggregation phase and all annotators of . Then, we obtain the average agreements for all pairs of , and report it as the agreement based on distance threshold .

  • Easy-Aggregation phase: the process for easy-aggregation is the same as for strict-aggregation phase, unless we calculate the agreement between annotations made by the aggregator of easy-aggregation phase and all other annotators of the same trajectory.

Figure 6(a) shows the inter-annotator agreement analysis based on Cohen’s Kappa, and Figure 6(b) shows the same analysis based on Overlap, for different distance thresholds. We observe that the largest agreement values based on both metrics are obtained for easy-aggregation phase. Also, the agreement for strict-aggregation is still larger than the agreement for expert-annotation. Based on the agreement analysis results, we can conclude following points.

  • [leftmargin=*]

  • Subjectivity. We observe that the disagreement between different annotators is large, even by employing a relaxed distance threshold like 200 meters or more. This justifies our claim on the subjectivity of trajectory annotation.

  • Aggregation to Tackle Subjectivity. We observe in Figures 6(a) and 6(b) that the aggregator utilizes the annotations of annotation phase, during both strict and easy aggregation phases. This enables a larger agreement for this two phases.

  • The closeness of Easy-aggregation to Expert-annotation. Based upon Figures 6(a) and 6(b), we conclude that there is larger agreement between results of easy-aggregation and expert-annotation phases, and this is due to employing some relaxed version of constraints during the easy aggregation, in comparison to strict aggregation phase.

  • Larger Agreement Values by Cohen’s Kappa. The other observation is that we obtain larger agreement values by using Cohen’s Kappa. This is potentially related to using both labels Yes and No to obtain the correlation (agreement), while the Overlap exploits only the label Yes (Section 5.1).

(a) Cohen’s Kappa
(b) Overlap
Figure 7: The result of intern-annotator agreement analysis based on (a) Cohen’s Kappa and (b) Overlap metrics.

6 Dataset of Annotated Car Trajectories (DACT)

We describe the main outcome of this study which is a Dataset of Annotated Car Trajectories, abbreviated as DACT. In this dataset, we have a collection of trajectories, where each trajectory is a time-ordered sequence of tuples. Each tuple consists of the following attributes:

  • [leftmargin=*]

  • Trip ID: This is a unique identifier which is assigned to each trajectory.

  • TimeStep: This is a positive integer which shows the index of current tuple in a sequence of tuples which constitute a trajectory.

  • TimeStamp: This is the sampling time of the current tuple which is reported in EDT (i.e. Eastern Daylight Time).

  • Speed: This is the speed of vehicle in current moment, which is reported in

    mph (miles per hour).

  • Acceleration: This is the acceleration of vehicle in current moment, which is reported in , that is, meter per second squared.

  • Heading: This shows the direction of moving vehicle at current moment which is a number between 0 and 359, where 0 means north and 180 means south.

  • HeadingChange: This shows the change of heading of vehicle in comparison with the observed heading in the last timestep. By definition, the HeadingChange of the first timestep of a trajectory is set to .

  • Latitude: This shows the (GPS) latitude coordinate of the location of the vehicle in current moment.

  • Longitude: This shows the (GPS) longitude coordinate of the location of the vehicle in current moment.

  • Annotation: If there is any annotation for a tuple, it will be listed here. The possible values for this attribute are Segment, Maybe-Segment, and NULL (i.e., no annotation for a tuple).

  • SegmentType: If there is any annotation for a tuple, the type(s) of segment will be listed here. The possible values for this attribute are NULL, Other, and any combination of segment types as are listed in Table 1.

As we had two phases of aggregation, i.e. strict and easy, the final dataset is provided in terms of two CSV files, one for each type of the aggregation. The columns of the CSV files are the same as attributes which we described above. The DACT dataset is available online at

7 Conclusion

In this paper, we introduced a new trajectory annotation framework in order to annotate a dataset of personal car trajectories. Among annotation tasks, the trajectory annotation is challenging and needs more expertise and time to be accomplished appropriately. Consequently, the design of a framework for such task is complicated as well. Besides, one of the important challenges which we addressed during this study is the subjectivity of trajectory annotation. In order to deal with this challenge, we applied two separate phases of aggregation by using strict and relaxed (easy) constrains to finalize the annotations, and also leveraged the AutoAnn, an annotator robot, to diminish the subjectivity. Our inter-annotator agreement analysis shows that we increased the agreement from 44% to about 60% by aggregation based on Cohen’s Kappa coefficient. Finally, the main output of this study which is a dataset of annotated car trajectories is made available and is publicly available for research projects.

8 Acknowledgement

The authors would like to thank Dr. Placido A. Souza Neto and his research team at the Federal Institute of Rio Grande do Norte, and also the students of the Ohio State University for their priceless help and participation in the annotation project. We also want to thank the Nationwide Mutual Insurance Company for providing the valuable sources of data, which made this project possible.


  • [1] New York taxi dataset. Accessed: 2017-04-14.
  • [2] S. Alewijnse, K. Buchin, M. Buchin, A. Kölzsch, H. Kruckenberg, and M. A. Westenberg. A framework for trajectory segmentation by stable criteria. In Proceedings of the 22nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pages 351–360. ACM, 2014.
  • [3] S. P. Alewijnse, K. Buchin, M. Buchin, S. Sijben, and M. A. Westenberg. Model-based segmentation and classification of trajectories. In Dead Sea, Israel: Proceedings of the 30th European Workshop on Computational Geometry March, pages 3–5, 2014.
  • [4] A. Anagnostopoulos, M. Vlachos, M. Hadjieleftheriou, E. Keogh, and P. S. Yu. Global distance-based segmentation of trajectories. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 34–43. ACM, 2006.
  • [5] M. Buchin, A. Driemel, M. van Kreveld, and V. Sacristán. Segmenting trajectories: A framework and algorithms using spatiotemporal criteria. Journal of Spatial Information Science, 2011(3):33–63, 2011.
  • [6] J. Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46, 1960.
  • [7] S. Moosavi, R. Ramnath, and A. Nandi. Discovery of driving patterns by trajectory segmentation. In Proceedings of the 3rd ACM SIGSPATIAL PhD Symposium, page 4. ACM, 2016.
  • [8] L. Moreira-Matias, J. Gama, M. Ferreira, J. Mendes-Moreira, and L. Damas. Predicting taxi–passenger demand using streaming data. IEEE Transactions on Intelligent Transportation Systems, 14(3):1393–1402, 2013.
  • [9] Y. Zheng, H. Fu, X. Xie, W.-Y. Ma, and Q. Li. Geolife GPS trajectory dataset - User Guide, July 2011.