Assessing out-of-domain generalization for robust building damage detection

11/20/2020
by   Vitus Benson, et al.
0

An important step for limiting the negative impact of natural disasters is rapid damage assessment after a disaster occurred. For instance, building damage detection can be automated by applying computer vision techniques to satellite imagery. Such models operate in a multi-domain setting: every disaster is inherently different (new geolocation, unique circumstances), and models must be robust to a shift in distribution between disaster imagery available for training and the images of the new event. Accordingly, estimating real-world performance requires an out-of-domain (OOD) test set. However, building damage detection models have so far been evaluated mostly in the simpler yet unrealistic in-distribution (IID) test setting. Here we argue that future work should focus on the OOD regime instead. We assess OOD performance of two competitive damage detection models and find that existing state-of-the-art models show a substantial generalization gap: their performance drops when evaluated OOD on new disasters not used during training. Moreover, IID performance is not predictive of OOD performance, rendering current benchmarks uninformative about real-world performance. Code and model weights are available at https://github.com/ecker-lab/robust-bdd.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

11/21/2019

xBD: A Dataset for Assessing Building Damage from Satellite Imagery

We present xBD, a new, large-scale dataset for the advancement of change...
04/12/2020

Building Disaster Damage Assessment in Satellite Imagery with Multi-Temporal Fusion

Automatic change detection and disaster damage assessment are currently ...
04/15/2020

RescueNet: Joint Building Segmentation and Damage Assessment from Satellite Imagery

Accurate and fine-grained information about the extent of damage to buil...
11/04/2021

Characterizing Human Explanation Strategies to Inform the Design of Explainable AI for Building Damage Assessment

Explainable AI (XAI) is a promising means of supporting human-AI collabo...
10/15/2021

Automated Quality Control of Vacuum Insulated Glazing by Convolutional Neural Network Image Classification

Vacuum Insulated Glazing (VIG) is a highly thermally insulating window t...
12/15/2020

Post-Hurricane Damage Assessment Using Satellite Imagery and Geolocation Features

Gaining timely and reliable situation awareness after hazard events such...
12/22/2019

A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks

Evaluation of Bayesian deep learning (BDL) methods is challenging. We of...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.