DeepAI AI Chat
Log In Sign Up

SAIBench: Benchmarking AI for Science

by   Yatao Li, et al.
Institute of Computing Technology, Chinese Academy of Sciences

Scientific research communities are embracing AI-based solutions to target tractable scientific tasks and improve research workflows. However, the development and evaluation of such solutions are scattered across multiple disciplines. We formalize the problem of scientific AI benchmarking, and propose a system called SAIBench in the hope of unifying the efforts and enabling low-friction on-boarding of new disciplines. The system approaches this goal with SAIL, a domain-specific language to decouple research problems, AI models, ranking criteria, and software/hardware configuration into reusable modules. We show that this approach is flexible and can adapt to problems, AI models, and evaluation methods defined in different perspectives. The project homepage is


page 1

page 2

page 3

page 4


On the importance of AI research beyond disciplines

As the impact of AI on various scientific fields is increasing, it is cr...

Quantifying the Benefit of Artificial Intelligence for Scientific Research

The ongoing artificial intelligence (AI) revolution has the potential to...

AI for Science: An Emerging Agenda

This report documents the programme and the outcomes of Dagstuhl Seminar...

Measuring science: irresistible temptations, easy shortcuts and dangerous consequences

In benchmarking international research, although publication and citatio...

Scientists' Perspectives on the Potential for Generative AI in their Fields

Generative AI models, including large language models and multimodal mod...

Rearrangement: A Challenge for Embodied AI

We describe a framework for research and evaluation in Embodied AI. Our ...

Ten Simple Rules for Attending Your First Conference

Conferences are a mainstay of most scientific disciplines, where scienti...