Image-based Contextual Pill Recognition with Medical Knowledge Graph Assistance

08/04/2022
by   Anh Duy Nguyen, et al.
11

Identifying pills given their captured images under various conditions and backgrounds has been becoming more and more essential. Several efforts have been devoted to utilizing the deep learning-based approach to tackle the pill recognition problem in the literature. However, due to the high similarity between pills' appearance, misrecognition often occurs, leaving pill recognition a challenge. To this end, in this paper, we introduce a novel approach named PIKA that leverages external knowledge to enhance pill recognition accuracy. Specifically, we address a practical scenario (which we call contextual pill recognition), aiming to identify pills in a picture of a patient's pill intake. Firstly, we propose a novel method for modeling the implicit association between pills in the presence of an external data source, in this case, prescriptions. Secondly, we present a walk-based graph embedding model that transforms from the graph space to vector space and extracts condensed relational features of the pills. Thirdly, a final framework is provided that leverages both image-based visual and graph-based relational features to accomplish the pill identification task. Within this framework, the visual representation of each pill is mapped to the graph embedding space, which is then used to execute attention over the graph representation, resulting in a semantically-rich context vector that aids in the final classification. To our knowledge, this is the first study to use external prescription data to establish associations between medicines and to classify them using this aiding information. The architecture of PIKA is lightweight and has the flexibility to incorporate into any recognition backbones. The experimental results show that by leveraging the external knowledge graph, PIKA can improve the recognition accuracy from 4.8 compared to baselines.

READ FULL TEXT

page 15

page 16

research
04/11/2021

Edge: Enriching Knowledge Graph Embeddings with External Text

Knowledge graphs suffer from sparsity which degrades the quality of repr...
research
09/22/2016

Image-embodied Knowledge Representation Learning

Entity images could provide significant visual information for knowledge...
research
10/20/2022

Context-driven Visual Object Recognition based on Knowledge Graphs

Current deep learning methods for object recognition are purely data-dri...
research
03/17/2023

High Accurate and Explainable Multi-Pill Detection Framework with Graph Neural Network-Assisted Multimodal Data Fusion

Due to the significant resemblance in visual appearance, pill misuse is ...
research
07/13/2019

Relational Memory-based Knowledge Graph Embedding

Knowledge graph embedding models often suffer from a limitation of remem...
research
02/17/2021

ConTraKG: Contrastive-based Transfer Learning for Visual Object Recognition using Knowledge Graphs

Deep learning techniques achieve high accuracy in computer vision tasks....
research
03/03/2020

Image-based OoD-Detector Principles on Graph-based Input Data in Human Action Recognition

Living in a complex world like ours makes it unacceptable that a practic...

Please sign up or login with your details

Forgot password? Click here to reset