High Accurate and Explainable Multi-Pill Detection Framework with Graph Neural Network-Assisted Multimodal Data Fusion

03/17/2023
by   Anh Duy Nguyen, et al.
0

Due to the significant resemblance in visual appearance, pill misuse is prevalent and has become a critical issue, responsible for one-third of all deaths worldwide. Pill identification, thus, is a crucial concern needed to be investigated thoroughly. Recently, several attempts have been made to exploit deep learning to tackle the pill identification problem. However, most published works consider only single-pill identification and fail to distinguish hard samples with identical appearances. Also, most existing pill image datasets only feature single pill images captured in carefully controlled environments under ideal lighting conditions and clean backgrounds. In this work, we are the first to tackle the multi-pill detection problem in real-world settings, aiming at localizing and identifying pills captured by users in a pill intake. Moreover, we also introduce a multi-pill image dataset taken in unconstrained conditions. To handle hard samples, we propose a novel method for constructing heterogeneous a priori graphs incorporating three forms of inter-pill relationships, including co-occurrence likelihood, relative size, and visual semantic correlation. We then offer a framework for integrating a priori with pills' visual features to enhance detection accuracy. Our experimental results have proved the robustness, reliability, and explainability of the proposed framework. Experimentally, it outperforms all detection benchmarks in terms of all evaluation metrics. Specifically, our proposed framework improves COCO mAP metrics by 9.4 12.0 protecting patients from medication errors using an AI-based pill identification solution.

READ FULL TEXT

page 2

page 4

page 14

page 15

page 21

page 22

page 23

research
12/20/2021

Product Re-identification System in Fully Automated Defect Detection

In this work, we introduce a method and present an improved neural work ...
research
01/26/2022

Causality and Correlation Graph Modeling for Effective and Explainable Session-based Recommendation

Session-based recommendation which has been witnessed a booming interest...
research
08/04/2022

Image-based Contextual Pill Recognition with Medical Knowledge Graph Assistance

Identifying pills given their captured images under various conditions a...
research
03/16/2020

AVR: Attention based Salient Visual Relationship Detection

Visual relationship detection aims to locate objects in images and recog...
research
07/19/2022

XG-BoT: An Explainable Deep Graph Neural Network for Botnet Detection and Forensics

In this paper, we proposed XG-BoT, an explainable deep graph neural netw...
research
04/10/2019

An In-Depth Comparison of s-t Reliability Algorithms over Uncertain Graphs

Uncertain, or probabilistic, graphs have been increasingly used to repre...
research
01/24/2019

Glass+Skin: An Empirical Evaluation of the Added Value of Finger Identification to Basic Single-Touch Interaction on Touch Screens

The usability of small devices such as smartphones or interactive watche...

Please sign up or login with your details

Forgot password? Click here to reset