Affordance Learning In Direct Perception for Autonomous Driving

by   Chen Sun, et al.
University of Waterloo

Recent development in autonomous driving involves high-level computer vision and detailed road scene understanding. Today, most autonomous vehicles are using mediated perception approach for path planning and control, which highly rely on high-definition 3D maps and real time sensors. Recent research efforts aim to substitute the massive HD maps with coarse road attributes. In this paper, we follow the direct perception based method to train a deep neural network for affordance learning in autonomous driving. Our goal in this work is to develop the affordance learning model based on freely available Google Street View panoramas and Open Street Map road vector attributes. Driving scene understanding can be achieved by learning affordances from the images captured by car-mounted cameras. Such scene understanding by learning affordances may be useful for corroborating base maps such as HD maps so that the required data storage space is minimized and available for processing in real time. We compare capability in road attribute identification between human volunteers and our model by experimental evaluation. Our results indicate that this method could act as a cheaper way for training data collection in autonomous driving. The cross validation results also indicate the effectiveness of our model.



There are no comments yet.


page 1

page 3

page 4

page 5

page 7


Learning from Maps: Visual Common Sense for Autonomous Driving

Today's autonomous vehicles rely extensively on high-definition 3D maps ...

DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving

Today, there are two major paradigms for vision-based autonomous driving...

Computing Systems for Autonomous Driving: State-of-the-Art and Challenges

The recent proliferation of computing technologies, e.g., sensors, compu...

Exploring OpenStreetMap Availability for Driving Environment Understanding

With the great achievement of artificial intelligence, vehicle technolog...

Detecting Driveable Area for Autonomous Vehicles

Autonomous driving is a challenging problem where there is currently an ...

Model Guided Road Intersection Classification

Understanding complex scenarios from in-vehicle cameras is essential for...

Street-Map Based Validation of Semantic Segmentation in Autonomous Driving

Artificial intelligence for autonomous driving must meet strict requirem...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

IN recent years, autonomous driving technology has become closer to fully being realized. There are many driving forces to this realization, key among them is the advance in perception techniques such as CNN(Convolution Neural Networks). The perception techniques allow a self-driving vehicle to understand the driving environment, which is one of the most important steps for vehicle path planning and control. In many applications

[1], the autonomous vehicle must be equipped with ubiquitous and robust state-of-the-art vision-based system for it to be able to sense and understand a great deal of driving scene.

Recent review [2] in autonomous driving concluded three main regimes for vehicle planning and control. The traditional mediated perception approach sets up the understanding of the road scene by fusing the pre-stored 3-D high definition map and high quality sensor input including LiDAR and GNSS signal [3]

. The other approach tries to mimic and learn from human expert driving using imitation learning, also called behavior reflex

[4]. It’s relatively cheap and easier to collect human driving demonstrations at scale. Agents can use the collected data to train a model that maps the sensory and video directly to control outputs without reconstructing the environment. Dates back to late 1980s, ALVINN[5] used neural network to construct a direct mapping between camera images and laser measurements to steering output. Modern works [6, 7] deploy DNN control policies using batch imitation learning approach showed the effectiveness in executing simple driving tasks such as lane following. Recent research proposed [8] the direct perception approach trying to combine the advantages in mediated perception and behavior reflex approaches. The direct perception approach uses neural networks to extract coarse affordances from input video or images and then define mapping from affordances to explicit control. As suggested in [8], the mediated perception may add unnecessary complexity to perception problem by detecting redundant objects that may not be useful in driving control decisions. On the other hand, behavior reflex may not be robust enough to adapt to all traffic and driving scenarios. This is due to varying environments and human driving skills. In this regard, we agree that direct perception is more robust and efficient and acts as a middle ground between mediated perception and behavior reflex.

Fig. 1: A general pipeline framework for data collection, processing, model training and testing for autonomous driving.

There are several challenges in the direct perception approach. Since the low level control is decided based on a given set of road attributes, the affordances to be learned must be pre-defined by human. Selecting the suitable affordances usually requires feature engineering and driving scene based analysis [9]. After deciding on the coarse road inference layout, we need to collect and label road data in order to develop a relatively good and robust model. Ari Seff et al. proposed a method that leverage the google street view images and OpenStreetMap (OSM) for automatic-labelling and model training [10]. In this paper, we follow the same line with [10] for automatic labeling procedure to collect training and testing data. Furthermore, we trained a Convolution Neural Network (CNN) to detect traffic scene affordances from a single street view image. We have tested effectiveness of the method and accuracy of our model in experiments. The key contributions of this paper are: (1) Efficient CNN Training Model: Instead of using pre-trained AlexNet [11] CNN model on Places database [12] to get good weight as [10], we created a customized CNN architecture based on VGG11 and AlexNet. Using non-initialized model and training on third the number of training images used, we were able to obtain results comparable to [10]. (2) Validation on Automatic labelling: We collect data near Waterloo area in Canada while Ari Seff’s data are collected in San Francisco, Bay area. We verified their automatic labelling methodology on the different geographical location. Further, in addition to testing our model on the San Francisco GSV images, we examined our network on KITTI [3] tracking data set which is collected in Europe to demonstrate the generalization ability for our model. (3) Refining the affordances by driving scene: We refine the definition of selected road attributes from [10]. The affordances set may change according to different driving scenarios.

The rest of the paper is arranged as follows: Section II discusses related work in driving context understanding, affordances learning and techniques leveraging open sourced data. We outline the data collection, affordance selection and auto-labelling methodology in Section III. In Section IV, we demonstrate the network design and training methodology compared with recent research works. Experimental results and discussion are shown in Section V, along with the conclusion and future work discussion in Section VI.

Ii Related Work

Traffic scene and driving context understanding are an ongoing challenges in autonomous driving. Over the past few years, more of more resources and focus have been put towards scene understanding as a primary challenge in autonomous driving, especially since DARPA Urban Challenge [13]. One type of strategy for the static driving context understanding is simultaneous localization and mapping (SLAM) [14]. A virtual representation of the road, traffic and surrounding buildings can be constructed based on the reconciliation of real-time sensor data and pre-stored HD maps. With the detailed driving context representation, the detailed path planning and driving policy can be further derived. However, the main bottlenecks for this type of approaches are the high requirements on computing power and data transmission [15].

Vision based methods tries to mimic the human driver using camera recorded images as major sensory input. German Ros et. al presented an Offline-Online perception framework in [16] where the 3D semantic maps are pre-stored offline and online semantic segmentation can be achieved by performing SVM based classification on video-sequences. While re-localization process in this framework can be achieved real-time, the online retrieval of semantics does not necessarily adapt to environment change. Authors in [17] proposed a unified multi-net structure that performs the joint classification, detection and semantic segmentation in real time. Such driving context understanding methods like semantic segmentation with camera images are eventually aim to assist the control design for the ego vehicle. The research group in Princeton University demonstrated the idea of directly learning the affordances from an image using the direct perception approach [8]. They train images (from a car racing game TORCS) using a ConvNet to predict affordances such as host vehicle distance to front vehicle or left/right lanes for driving action. They tested their approach both on virtual and real environments and reported good performance in close range to the state-of-the-art deformable parts model car detector [18]. Based on the determined affordances, they are able to build simple rule based controller for the vehicle control in TORCS. This idea proves that meaningful driving affordances can be incorporated in autonomous driving decision making. Similar approach was adopted by [19] where they further advanced the driving image affordance learning and control with conditional neural network and more photo-realistic simulation platform CARLA [20].

Recent research [21, 22, 23] demonstrated that data-driven perception model often surpass the hard-coded reasoning in context prediction leveraging the large-scale data since much more expert driving experience can be exploited. However, the process of data-set collecting and labelling often requires huge amount of effort. Typical pipeline framework for data-driven approach can be concluded as Fig. 1. The aforementioned research works [8, 20, 19] use simulation data since the ground truth information is programmed and can be exported through provided APIs. However, there is still a gap between the simulated environment and the real world data [24]. Many open-sourced driving data sets received increasing attention in the recent years. The Caltech lane data set [25] focused on lane marking wheares KITTI [3]

provide detailed marking with multiple sensors including LiDar. With high-end expensive sensors the vehicle dynamic state estimation could be achieved by methods mentioned in recent review paper

[26]. Recenly, Xu et. al published BDD100K data set [23] where the diverse driving data is collected in a distributed way by Uber drivers across California and New York and annotated by human labour. These data sets were collected and labelled with a deliberate designed system but are not automated and still quite expensive. OpenStreetMap (OSM) [27] is an open-source mapping project started since 2004, where over 21 million miles of road geographical information is available for public use. In [10], the authors trained a CNN model to predict road attributes using images from Google Street View (GSV). They presented automatic labeling method based on location matching with attributes from OSM. We follow the similar idea with [10] where we use ’cheap’ data with automatic labelling to teach a model predicting cues given image inputs. In our work, we refine road attributes and how it influences driving actions, using not only intersection but also affordances on a straight road.

Iii Dataset and Labelling

As we have discussed in previous section, deep ConvNet frameworks was used to train data and determine affordances such as host vehicle to road angle, number of lanes and driveable areas in many recent research works. However, this requires huge number of labeled images to be able to train a reliable model. There are a number of real world street scene labeled data sources such as KITTI [3] and synthetic data (from games and movies) such as Virtual KITTI [28]. For tasks such as determining bike lanes, wrong-way vs. right-way, the available data in [3, 28] and most other open source autonomous driving benchmark data sets [23] may not be enough or labeled for these specific affordances. Consequently, as proposed in [10], we take advantage of huge free and open source imagery and corresponding attributes repository available on GSV and OSM, to train a ConvNet model to predict the road attributes.


Fig. 2: A standard OSM map covering area near University of Waterloo in Ontario Canada is shown in (a). The same area highlighting high density of attributes available in OSM such as road polylines, parcels near the road and building polygons are shown in (b). Each node in the map contains coordinate information that can be used to associate it with other location based features from sources such Google Street View panoramas.

Iii-a Data Collection

OpenStreetMap is a community driven and local knowledge based open data platform. The contributed data is tied together using location information in a world geodetic coordinate System (WGS84). Similarly, Google has a huge deposit of street view imagery with each panorama encoded with vehicle true heading at the time of image capture and location information in WGS84 coordinate system. Using location neighborhood constraint, it becomes possible to associate each panorama from GSV with nearby road attributes such as number of lanes or if an intersection is likely in view [27]. The accuracy of feature association is directly affected by the location accuracy in both GSV and OSM and whether information in both sources were updated in same time frame. As will be highlighted later, we found some mislabeled affordances due to time latency and unresolved location differences especially at bridges or close road networks where a small location deviation would associate features of one road to an image showing a different road.

The following sections give more details about the process.

Iii-A1 Google Street View Panoramas

We have downloaded over one hundred thousands panorama images starting with a seed panorama image ID at University of Waterloo (shown in Fig. 3). These panoramas were then cropped and warped into sized images with a field of view (FOV) of 100 degrees. The image size and FOV were kept similar to what [10] used after finding them sufficient for driving scene. Each image is encoded with coordinates in WGS84 reference system. This is later used to query and overlay with data from OSM.

Fig. 3: A seed panorama used as a starting point for downloading GSV panoramas. The pedestrian bridge in view connects Engineering buildings 3 and 5 (E3 to E5) at University of Waterloo.

Iii-A2 OpenStreetMaps Vectorized Data

OSM [27] is a vectorized map with attributes contributed by volunteers. Attributes include but not limited to poly-lines such as those defining extents of road networks, bike lanes and traffic markings. It also includes point features such as stop signs, traffic lights and speed signs. Its data availability may be lacking in rural areas or small towns, since volunteers tend to contribute to maps around where they reside.

Iii-B Affordance Labeling

Here we discuss the affordance set selection. It is still an open question nowadays for the optimized road attributes selection for the driving context understanding. Chen et al. proposed 13 affordance indicators in [8] for multi-lane tracks in TORCS. It is rather simplified since there is neither intersection, pedestrian nor traffic light in TORCS. Authors of [19] advanced the affordance learning in the single lane urban scenario simulated in CARLA where 6 affordances were selected. Both works utilized the global information embedded in the simulation engine such as the global information for all the agents in the map and the distance for the vehicle between the road center line. It is rather difficult for us to obtain these global information in real data, hence we choose the target road attributes based on the available OSM and GSV data. The OSM data set and GSV panoramas are encoded with location coordinates in WGS84 reference frame. It was possible to query and overlap an image cropped from GSV panoramas with corresponding attributes in the OSM data. We conclude the list of automatically labeled affordances in TABLE. I.

Labeled Affordances Data Type Range
Heading_Angle Continuous
Drivable_Heading? Boolean {True, False}
Intersection_Ahead? Boolean {True, False}
Distance_to_Intersection Continuous [0, 30] m
Number_of_lanes Discrete {1, 2, 3}
Wrong_Way? Boolean {True, False}
Bike_Lane? Boolean {True, False}
TABLE I: Road Attributes labeled for waterloo region GSV images

Iii-B1 Lane Following

The vehicle heading angle is rather important for lane following task. We label the affordances Heading_Angle to represent the current ego vehicle heading angle corresponding to the driving lane. Fig. 4 gives examples of the angle prediction using our model. We further extend the angle prediction to a classification problem in order to compare human capability of identify the heading angle from single image. The detailed comparison will be discussed in Section V.

The Drivable_Heading is to detect whether there is a continual path to execute the lane following task. In [10], this attribute is used to predict whether the current heading of the panorama image align with the road disregarding the driving lanes. In our work, we refine the definition of Drivable_Heading where not only the Heading_Angle is within , there should not be any static object in front of the panorama (road edges, endings). Some examples of our model prediction and calculated true labelling are demonstrated in Fig. 4. The labelled vehicle heading angles are calculated based on OSM lane attributes and GSV panorama applied rotations. Note that the true labels are not necessarily accurate since the calculation is based on one assumption that the vehicle that collected the GSV panoramas are driving heading straight in most of the lanes. Nevertheless, we found that our model predictions on heading angle trained with these generated labels showed good level of effectiveness. We will discuss more on this issue later in Section V.

Fig. 4: Vehicle heading angle () prediction result and drivable classification.

Iii-B2 Intersection Handling

Various intersections is the most common scenario in the urban driving setting where human driver need to decide the high level command in order to follow the planned driving path. We denote the the following two road attributes for the intersection handling: Intersection_Ahead? and Distance_to_Intersection. We follow the similar procedure with [10] to mapping the intersections where first locate them on OSM by finding shared vertices and then mapping with GSV panoramas. The nearest intersection distance from the corresponding ego vehicle heading is used as our ground truth labelling. We use the GSV images and the computed ground truth images to train our model for the distance prediction as we can see in Fig. 5. The Intersection_Ahead? is a binary classification problem. After the manually comparing and assessing with the sampled images, we agreed with the methodology in [10] where we choose given distance 30m and below as the exist intersection ahead. Similarly, we choose those intersection distance greater than 100m as false prediction in order to provide clear distinction for the classification task.

Two set of sample results are demonstrated in Fig. 5. It is an obvious task to identify or measure the distance between intersection when you approaching one, as we can see from the top three samples in Fig. 5. However, the estimation error of our model grows when predicting on a view at the intersection. The visual inputs at intersections usually are not as structured as general road segments. The open view of a unstructured terrain usually confuse CNN based model since only one shot of the image is given. We believe the prediction results can be improved by using memory based model such as LSTM [29].

Fig. 5: Comparison between prediction and true labels on distance to intersection. The ‘true’ distance label is calculated by measuring the distance between the GSV referring point and the center of the intersection in OSM.

Iii-B3 Multi-lane Handling

It is rather important for autonomous driving can identify the multi-lane driving context especially in urban or highway driving in order to proceed to path planning and driving maneuver control. The attribute Number_of_Lanes identify how many lanes in the current driving road. It is addressed in [10] where they only include one-way roads in training data due to the inconsistency for two-way roads when considering the driving direction. In our work, we further included the images of two-way roads in our training set. We find that our model prediction was consistent with the labels for the most part, except when there was occlusion, lanes were not visible or the label was incorrect. In Fig. 6, top three images demonstrated the effectiveness of our model prediction. Despite the curved road shown in top right image, the model was able to generalize well and made correct prediction. Yet the task for predicting number of lanes from single shot of image input is still challenging as pointed out by [10] due to the lack of lane markings. We list three typical false prediction in the bottom of Fig. 6 as a complement of [10]. The road constructions or other dynamic changes may result in consistency between expected truth of the driving context and the static road attributes labels. The static OSM data cannot adapt to recognize the traffic cones as demonstrated in the bottom left of Fig. 6. Furthermore, the GPS location accuracy may raise the label error issue especially near intersections or highway ramps (bottom right in Fig. 6). In general, our model can predict well on the number of lanes given enough lane markings or other vehicles. Unfortunately, the dynamic change of the road segments and obstruction of the view may result in false predictions.

Fig. 6: Multi-lane prediction using our model trained on labels provided by OSM. In some cases when there is a dynamic change (construction, road change etc), static labels from OSM cannot represent well of the road segment (Bottom left & right). The narrow view and obstruct by dynamic objects (Bottom middle) may also result in false prediction.

One side problem for counting the number of lanes is to identify the possible road side bike lanes. The road attribute Bike_Lane? is true if there exist bike lane based on the given panorama. Our trained model to predict bike lanes performed 3% worse than Seff et. Al’s in [10]. However, It should be pointed out that the validation accuracy was affected by the mislabeled images. As explained in the previous results, in some cases, the models made correct predictions despite of incorrect labels. For bike lanes, this is still the case. As can be seen in Fig. 7 on left image, the model predicted the road to have no bike lanes and this is correct from visual inspection. However, the image label indicated that there is a bike lane. It is likely that the previous views of the road had bike lanes but ended before intersection. After examine on the false prediction samples we find that the sampled bike lanes tend to end before major intersections. Bike lane may confused with the highway emergency lane, due to the CNN model only take single shot image as input. One false prediction example is given in Fig. 7 on the right where the model may treat rural road with a bike lane as the highway ramp or emergency lane.

Fig. 7: Exist bike lane prediction using our trained CNN model.

We also interested to identify whether the ego-vehicle is on the right way of driving given the panorama image. Wrong_Way? classification is based on the driving rules that you must drive on the right side of the road. By carefully examining left and middle images in Fig. 8

, one can verify that indeed the model makes correct predictions. For image on the left, it is correctly predicted to be the right-way. This is informed by the driver view largely being on the right side of the road. The middle image is classified as wrong way of driving, which is correct since the driver view mostly falls on the left side of the road. Image to the right of Fig.

8 is less ambiguous to classify and correctly predicted as wrong way.

Fig. 8: Right or wrong way classification using our trained CNN model. The right of way corresponding to label 1 and wrong way is labelled as 0.

Iv Model Design and Training

Initially, we followed the same path as [10] using AlexNet [11] architecture for our model. However, we tuned hyper-parameters to find the architecture that is most efficient and produced best validation accuracy for driving affordance training. As shown in Fig. 9, we ended up with a network comprised of five Convolution layers as AlexNet and three fully connected (FC) layers each with 4096 channels. We found that using receptive field for each convolution layer as proposed [30]

, produced better validation accuracy for all trained affordances. For all convolution and FC layers we used rectified linear (ReLu) as activation function. We also applied padding and max pooling to preserve input size and spatial resolution, respectively, through convolution layers. We chose not to use dropouts but instead employed batch normalization as a substitute, after the first two convolution layers and each FC layer. This increased robustness of the weights and reduced over fitting while also preserving the learned features. Higher weights at each layer (except output layer) were penalized by applying an L2 regularization of 0.0001. This helped us keep an eye on learned weights and drastically reduced over-fitting. The output layer structure depends on whether the model is for regression or binary classification. For regression, we used an output layer with one kernel and no activation function, i.e. the outputs were not scaled into probability output. The model was compiled using RMSprop optimizer with a learning rate of 0.0001 and mean squared error (MSE) as the loss function. The accuracy of our regression model was reported in mean absolute error (MAE). The output layer for binary classification model has one kernel and uses sigmoid as the activation function to scale the predictions into values between 0 and 1. Similar to regression, the compiling was done using RMSprop optimizer with learning rate of 0.0001. However, the loss function was changed to binary crossentropy with accuracy reported as a percentage.

Fig. 9: The architecture of the proposed CNN. The inputs is the crop of GSV panorama image and the output layer consists of selected features and affordance indicators. Note that we perform batch normalization after convolution layers 1, 2 and each fully connected (FC) layers instead of dropouts to reduce over fitting issue.

Our model was built in Keras


running on top of TensorFlow framework. As shown in Fig.

9, the first convolution layer input accepts an RGB image of size and passes it through 96 filters of size

with ReLu as activation function, strides of 1 pixel and padding set to ’same’ i.e., it outputs same dimensions as input. Kernel regularizer is set to L-2 norm. This is followed by a scaled and centered batch normalization layer with 0.99 momentum and 0.001 epsilon. The moving variance initializer is set to one. A

max pooling layer with strides of two pixels and padding set to ’valid’ (no padding), comes next. The second convolution layer has 256 filters of size . The activation function, stride, padding and regularizer are set similar to first convolution layer. The batch normalization and max pooling layers follow (with similar set up as previous described). The 3rd and 4th convolution layers have 384 filters of size

and are separated by a max pooling layer. Another max pooling layer is inserted before the 5th convolution layer with 256 filters of size 3x3 (all convolution layers maintain similar structure to first layer. they only differ in number of filters). A flattening layer is implemented before the first FC layer. All the fully connected layers have 4096 neurons with ReLu as the activation function and L-2 norm (0.0001) regularizer. Each of them are separated by a batch normalization layer.

Before training the images were normalized by changing the pixel values to float and diving with 255. random images were augmented by applying a rotation of

, width and height shift of 0.2, shear of 0.2 and zoomed by a factor of 0.2. The images were trained in batches with a batch size of 32 and 50 epochs. The steps per epoch for both training and validation depended on total number of images as in Eq. (

1). Images and corresponding labels were also randomly shuffled during the training phase.


As a comparison with [10], their model was pretrained on Places Database while ours used random initialization without pre-trained weights. Their model tend to converge after a “few thousand iterations”. Instead, our model used only 50 epochs and less than two hundred steps per epoch since we use much smaller data set. The detailed model performance comparison and cross validation will be given in next section.

V Results and Discussion

In this section, we present the quantitative evaluation result of the proposed model. We also discuss the improvement on our model and current data collection pipeline for autonomous driving.

V-a Accuracy Evaluation

We validate our CNN models performance across three different geographical regions, namely data collected from Waterloo (abbreviated as W), data used in [10] collected in San Francisco, Bay area (abbreviated as SF) and KITTI tracking data collected from Europe. We also provide the comparison between human baseline and model prediction on classification tasks.

V-A1 Our CNN models vs. Human

We asked five human volunteers to label 1000 images for each affordance. We evaluate our models on the same images and compare results which are presented in Fig. 10. We focused on classification tasks as we found it difficult for humans to meaningfully measure angles or distances from a low quality images. Consequently, we did not consider distance to an intersection and heading angles were deduced to binary classification by asking humans to predict whether the image showed a negative rotation (left rotation with respect to road) or positive rotation (right rotation with respect to road).

Each human volunteer was first shown ten example images and corresponding labels for each affordance under consideration, in order to train them and highlight the image to affordance association in the context of driving. We then let each volunteer label provided images per affordance without access to OSM derived true labels. Consequently, for each affordance, we generated a single set of human labels by combining five individual labels using consensus model [32].

Fig. 10: Comparison between human baseline and our trained CNN model on the classification prediction accuracy (higher is better). The tasks investigated here are drivable (D), heading angle (HA), number of lanes (NL), bicycle lanes (BL) and right way or wrong way prediction(W vs R).

As evident in Fig. 10, our CNN model predictions and human labels were within of each other. Our model performed better than human for driveable space (D), bikelane (BL) and wrong-way vs. right-way (W vs R) affordances. Also, it had comparable results for number of lane (NL) and driving heading angle (HA) affordances.

V-A2 Model Generalization Test

To find out how well our model would generalize on data collected in different geographical locations, we took advantage of GSV panoramas from San Francisco Bay Area available for download on [10] data page. We did cross validation by comparing the prediction on San Francisco Bay area GSV images by model trained on Waterloo dataset, prediction on Waterloo GSV images by model trained on both Waterloo and San Francisco datasets and prediction on San Francisco Bay area GSV images by Model trained on Waterloo and San Francisco data sets. We then used the CNN models for heading angle (HA), intersection distance (ID) and number of lanes (NL) affordances that were trained on Waterloo data set (from henceforth referred to as model set 1 to predict on San Francisco Bay Area images. We recognized that data augmentation applied to training data might not be robust enough for driving scene training since it is relatively hard to augment buildings and other features (such as trees, grass and curbs) proximity to road. Hence, we trained new models for HA, ID, and NL using a data set with half of the images from San Francisco and another half from Waterloo (referred as model set 2. This model was tested on a Waterloo data set and San Francisco data set, independently. We were careful to make sure that the test images were never used during training and validation of the models. However, the same testing data set from SF was kept consistently the same in model set 1 and model set 2 testing. The MAE between the predictions and true labels were computed and plotted in Fig. 11. The MAE plot shows that model set 2 is more accurate and generalizes better than model 1. Model set 2 performed best in all affordances. We should also point out that the difference in MAE for both models should be examined independently for each affordance. For instance, number of lanes in Waterloo range from 1 to 4 lanes. Therefore a MAE of 1.04 in number of lane would be considered too big since it means that a model would likely be predicting wrong number of lanes most of the time. However a MAE of 4.8 meters for intersection distance may be tolerable given that the intersection distance in consideration range from 0 to 30 metres.

Fig. 11: Comparison of mean absolute error between our CNN model trained and tested on data sets across different geological regions (lower is better). The tasks investigated here are heading angle (HA), intersection distance (ID) and number of lanes (NL).

In Table. II, we compare the the performance of the proposed architecture and trained models by making prediction on San Francisco testing data. Three set of model are compared. We use the model proposed in [10] trained in SF data as an baseline. The other two model are aforementioned model 1 and 2 using the same architecture. We compare the accuracy relative to the training data size used in HA, ID and NL affordance training. Our models trained on both Waterloo and San Francisco datasets performed better than models in [10] for HA and NL affordances. This is despite only using of the data size. We used 4K images while [10] used over 12K images for training. Although our second model trained on just Waterloo (W) datasets while it reports worst accuracy, it is still comparable to Seff et Al. results for all the three affordances. Moreover, the model trained using only 4K images on the combined data outperforms the other models in most of the regression tasks. This agrees with most of the findings in data augmentation where it is best to use data collected in various geographical locations and in different conditions in order to train a perception model that could generalize well.

Model in [10] Ours on W Ours on W & SF
Train Samples >12K 6K 4K
ID (MAE) 4.3 6.01 4.77
HA (MAE) 9.2 13.4 5.89
NL (MAE) 0.9 1.04 0.76
TABLE II: The comparison across models proposed in [10] trained based on SF data, our architecture trained on W data as well as the same architecture trained on combined SF and W data. All the models compared here are tested on same SF data. Our model trained on W and SF data reports best results.

V-A3 Driving Heading Angle prediction on KITTI dataset

Our driving heading angle model was used to predict on KITTI tracking data set and results are plotted in Fig. 12 with the angular rate around Z axis for each image as included in the image metadata from KITTI website. This plot clearly shows similar trend between our CNN model predicted heading angle with reported angular rate at the time of image capture. Note that the size of the KITTI tracking images are in width and height, however our trained model takes input. It is hard to resizing the KITTI images in scale to fit the input size of our model without cutting out any road features. Unfortunately, we lose spatial resolution and apply distortion by shrinking the image horizontally. We can observe the magnitude difference between the red line (raw CNN model prediction) and the blue line (KITTI ground truth). In order to demonstrate this issue, we applied the resizing factor (RF) plotted the new heading angle magnitudes. As evident in Fig. 12, the heading angles with resizing factor applied are very close to the KITTI tracking angular rates at the image capture.

Fig. 12: Comparison on regression task of vehicle heading angle prediction. We feed the resized KITTI images (collected in Europe) into our CNN model trained on data collected in Waterloo area in Canada. The blue line corresponds to the ground truth measurement from KITTI, and the red line corresponds to the raw prediction result from our CNN model without considering the distortion of the input image. The CNN prediction with applying the resizing factor is plotted in yellow.

V-B Automatic Labeling for driving data sets

The automatic labelling for driving data by leveraging existing OSM and GSV data is a complement way or cheap substitution of generalizing training data workflow for autonomous driving. The growing use of OSM data for training may contribute to more accurate static labelling in return. Furthermore, these road attributes can be stored as a light reference vector map with very fast accessing speed for autonomous driving as a complement of heavy and expensive high-definition maps.

It is rather important to increase the accuracy of automatic labelling as we have demonstrated in previous sections. The correctness of automatic labeling was defined by a number of factors. First, positioning in both GSV and OSM data carries a degree of error. GSV panoramas are collected using Google Street cars equipped with navigation system (GPS/INS) whose accuracy depends on environment. Its positioning accuracy in these areas can range in meters. When GPS and IMU are combined to create a fused solution, a centimeter-level accuracy (after post-processing or using real-time kinematics) can be achieved. However, this is true in open sky areas as GPS signal is easily obstructed in areas with a lot of buildings or trees causing deterioration of accuracy to decimetre-level accuracy even with a high-end IMU [33]. These positioning challenges are inherited by the collected panoramas and contribute to mismatching with OSM data. Another factor is that OSM data is contributed by volunteers and hence integrity of its data varies and may not be up-to-date. This is highlighted in Fig. 7 where the left image is labeled as having a bike lane but in reality it is the road segment prior to current location. Moreover, the OSM road attributes data and GSV images are static which means that they cannot represent well when there is dynamic change of the road segments such as road construction, change of weather etc. The road construction may affect the correctness of the labelling more as shown in the bottom left image of Fig. 6. The GSV panoramas are collected by google street car mostly in sunny days with clear view with almost no variation on weather. The driving data from different weather such as raining and snowing are necessary in order to train a robust perception model that could generalize well. The prediction error may also inherited from the downside of the CNN architecture where only single shot of front view image is used as input, which is presented in the bottom middle image of Fig. 6 when there is dynamic obstruct blocking the view. Designing a distributed way of collecting visual driving data across various countries and under various weathers and applying network structure with memory block and using images sequence as input for prediction are both potential future improvement for this line of work.

Vi Conclusion

In this paper we proposed an efficient CNN model for driving affordances learning by leveraging online static databases. The open resourced road data is collected in a distributed way across the world which enable us automate the data collection and labelling pipeline. We have examine our trained model based on different data set across geographical regions. The quantitative results indicated the effectiveness our CNN model for affordance prediction across driving data collected in Waterloo area in Canada, California area in US and Europe respectively. The aim of this work was to extend the automated pipeline approach for training affordance learning. Exploring more advanced neural network structures and refining the static OSM labels with dynamic observations is currently under investigation.


  • [1] J. Janai, F. Güney, A. Behl, and A. Geiger, “Computer vision for autonomous vehicles: Problems, datasets and state-of-the-art,” arXiv preprint arXiv:1704.05519, 2017.
  • [2] W. Schwarting, J. Alonso-Mora, and D. Rus, “Planning and decision-making for autonomous vehicles,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 1, pp. 187–210, 2018.
  • [3] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1231–1237, 2013.
  • [4] P. Gaussier, S. Moga, M. Quoy, and J.-P. Banquet, “From perception-action loops to imitation processes: A bottom-up approach of learning by imitation,”

    Applied Artificial Intelligence

    , vol. 12, no. 7-8, pp. 701–727, 1998.
  • [5] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in Advances in neural information processing systems, 1989, pp. 305–313.
  • [6] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang et al., “End to end learning for self-driving cars,” arXiv preprint arXiv:1604.07316, 2016.
  • [7] V. Rausch, A. Hansen, E. Solowjow, C. Liu, E. Kreuzer, and J. K. Hedrick, “Learning a deep neural net policy for end-to-end control of autonomous vehicles,” in American Control Conference (ACC), 2017.   IEEE, 2017, pp. 4914–4919.
  • [8] C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in 2015 IEEE International Conference on Computer Vision (ICCV).   IEEE, 2015, pp. 2722–2730.
  • [9] D. Silver, J. A. Bagnell, and A. Stentz, “Learning from demonstration for autonomous navigation in complex unstructured terrain,” The International Journal of Robotics Research, vol. 29, no. 12, pp. 1565–1592, 2010.
  • [10] A. Seff and J. Xiao, “Learning from maps: Visual common sense for autonomous driving,” arXiv preprint arXiv:1611.08583, 2016.
  • [11]

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in

    Advances in neural information processing systems, 2012, pp. 1097–1105.
  • [12] B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, and A. Oliva, “Places: An image database for deep scene understanding,” arXiv preprint arXiv:1610.02055, 2016.
  • [13] M. Buehler, K. Iagnemma, and S. Singh, The DARPA urban challenge: autonomous vehicles in city traffic.   springer, 2009, vol. 56.
  • [14] S. Choudhary, A. J. Trevor, H. I. Christensen, and F. Dellaert, “Slam with object discovery, modeling and mapping,” in Intelligent Robots and Systems (IROS 2014), 2014 IEEE/RSJ International Conference on.   IEEE, 2014, pp. 1018–1025.
  • [15] H. G. Seif and X. Hu, “Autonomous driving in the icity—hd maps as a key challenge of the automotive industry,” Engineering, vol. 2, no. 2, pp. 159–162, 2016.
  • [16] G. Ros, S. Ramos, M. Granados, A. Bakhtiary, D. Vazquez, and A. M. Lopez, “Vision-based offline-online perception paradigm for autonomous driving,” in Applications of Computer Vision (WACV), 2015 IEEE Winter Conference on.   IEEE, 2015, pp. 231–238.
  • [17] M. Teichmann, M. Weber, M. Zoellner, R. Cipolla, and R. Urtasun, “Multinet: Real-time joint semantic reasoning for autonomous driving,” in 2018 IEEE Intelligent Vehicles Symposium (IV).   IEEE, 2018, pp. 1013–1020.
  • [18] A. Geiger, M. Lauer, C. Wojek, C. Stiller, and R. Urtasun, “3d traffic scene understanding from movable platforms,” IEEE transactions on pattern analysis and machine intelligence, vol. 36, no. 5, pp. 1012–1025, 2014.
  • [19] A. Sauer, N. Savinov, and A. Geiger, “Conditional affordance learning for driving in urban environments,” arXiv preprint arXiv:1806.06498, 2018.
  • [20] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning, 2017, pp. 1–16.
  • [21] L. Caltagirone, M. Bellone, L. Svensson, and M. Wahde, “Simultaneous perception and path generation using fully convolutional neural networks,” arXiv preprint arXiv:1703.08987, 2017.
  • [22] Y. Xing, C. Lv, L. Chen, H. Wang, H. Wang, D. Cao, E. Velenis, and F.-Y. Wang, “Advances in vision-based lane detection: algorithms, integration, assessment, and perspectives on acp-based parallel vision,” IEEE/CAA Journal of Automatica Sinica, vol. 5, no. 3, pp. 645–661, 2018.
  • [23] H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end learning of driving models from large-scale video datasets,” arXiv preprint, 2017.
  • [24] F.-Y. Wang, N.-N. Zheng, D. Cao, C. M. Martinez, L. Li, and T. Liu, “Parallel driving in cpss: A unified approach for transport automation and vehicle intelligence,” IEEE/CAA Journal of Automatica Sinica, vol. 4, no. 4, pp. 577–587, 2017.
  • [25] M. Aly, “Real time detection of lane markers in urban streets,” in Intelligent Vehicles Symposium, 2008 IEEE.   IEEE, 2008, pp. 7–12.
  • [26] H. Guo, D. Cao, H. Chen, C. Lv, H. Wang, and S. Yang, “Vehicle dynamic state estimation: state of the art schemes and perspectives,” IEEE/CAA Journal of Automatica Sinica, vol. 5, no. 2, pp. 418–431, 2018.
  • [27] M. Haklay and P. Weber, “Openstreetmap: User-generated street maps,” Ieee Pervas Comput, vol. 7, no. 4, pp. 12–18, 2008.
  • [28] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig, “Virtual worlds as proxy for multi-object tracking analysis,” in

    Proceedings of the IEEE conference on computer vision and pattern recognition

    , 2016, pp. 4340–4349.
  • [29] F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with lstm,” 1999.
  • [30] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  • [31] F. Chollet et al., “Keras,” 2015.
  • [32] E. Herrera-Viedma, F. Herrera, and F. Chiclana, “A consensus model for multiperson decision making with different preference structures,” IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, vol. 32, no. 3, pp. 394–402, 2002.
  • [33] J. Zhou, J. Traugott, B. Scherzinger, C. Miranda, and A. Kipka, “A new integration method for mems based gnss/ins multi-sensor systems,” in Proceedings of the International Technical Meeting of the Ion Satellite Division (ION GNSS), Tampa, FL, USA, 2015, pp. 14–18.