Automating the assessment of biofouling in images using expert agreement as a gold standard

08/21/2020
by   Nathaniel J. Bloomfield, et al.
0

Biofouling is the accumulation of organisms on surfaces immersed in water. It is of particular concern to the international shipping industry because fouling increases the drag on vessels as they move through the water, resulting in higher fuel costs, and presents a biosecurity risk by providing a pathway for marine non-indigenous species (NIS) to establish in new areas. There is growing interest within jurisdictions to strengthen biofouling risk-management regulations, but it is expensive to conduct in-water inspections and assess the collected data to determine the biofouling state of vessel hulls. Machine learning is well suited to tackle the latter challenge, and here we apply so-called deep learning to automate the classification of images from in-water inspections for the presence and severity of biofouling. We combined images collected from in-water surveys conducted by the Australian Department of Agriculture, Water and the Environment, the New Zealand Ministry for Primary Industries and the California State Lands Commission, and annotated them using the Amazon Mechanical Turk (MTurk) crowdsourcing platform. We compared the annotations from three biofouling experts on a 120-sample subset of these images, and found that for two tasks, identifying images containing fouling, and identifying images containing heavy fouling, they showed 89 CI: 87-92 agreement with experts, which we defined as performing at most 5 experts (p=0.004-0.020). Our deep learning model trained with the MTurk annotations also showed reasonable performance in comparison to expert agreement, although at a lower significance level (p=0.071-0.093). We also demonstrate that significantly better performance than expert agreement can be achieved if a classifier with high recall or precision was required.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 3

page 7

07/17/2019

AquaSight: Automatic Water Impurity Detection Utilizing Convolutional Neural Networks

According to the United Nations World Water Assessment Programme, every ...
12/19/2019

Understanding Human Judgments of Causality

Discriminating between causality and correlation is a major problem in m...
05/05/2020

CODA-19: Reliably Annotating Research Aspects on 10,000+ CORD-19 Abstracts Using a Non-Expert Crowd

This paper introduces CODA-19, a human-annotated dataset that codes the ...
08/10/2021

U-Net-and-a-half: Convolutional network for biomedical image segmentation using multiple expert-driven annotations

Development of deep learning systems for biomedical segmentation often r...
09/08/2021

AgreementLearning: An End-to-End Framework for Learning with Multiple Annotators without Groundtruth

The annotation of domain experts is important for some medical applicati...
11/05/2021

Agreement Implies Accuracy for Substitutable Signals

Inspired by Aumann's agreement theorem, Scott Aaronson studied the amoun...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.