Topological Data Analysis Guided Segment Anything Model Prompt Optimization for Zero-Shot Segmentation in Biological Imaging

by   Ruben Glatt, et al.

Emerging foundation models in machine learning are models trained on vast amounts of data that have been shown to generalize well to new tasks. Often these models can be prompted with multi-modal inputs that range from natural language descriptions over images to point clouds. In this paper, we propose topological data analysis (TDA) guided prompt optimization for the Segment Anything Model (SAM) and show preliminary results in the biological image segmentation domain. Our approach replaces the standard grid search approach that is used in the original implementation and finds point locations based on their topological significance. Our results show that the TDA optimized point cloud is much better suited for finding small objects and massively reduces computational complexity despite the extra step in scenarios which require many segmentations.


page 2

page 4

page 8


SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model

Foundation models have taken over natural language processing and image ...

See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data

Zero-shot point cloud segmentation aims to make deep models capable of r...

"Zero Shot" Point Cloud Upsampling

Point cloud upsampling using deep learning has been paid various efforts...

Inferring Quality in Point Cloud-based 3D Printed Objects using Topological Data Analysis

Assessing the quality of 3D printed models before they are printed remai...

Prompt-guided Scene Generation for 3D Zero-Shot Learning

Zero-shot learning on 3D point cloud data is a related underexplored pro...

Topological Data Analysis with ε-net Induced Lazy Witness Complex

Topological data analysis computes and analyses topological features of ...

Please sign up or login with your details

Forgot password? Click here to reset