DeepAI AI Chat
Log In Sign Up

SAIBench: Benchmarking AI for Science

06/11/2022
by   Yatao Li, et al.
Microsoft
Institute of Computing Technology, Chinese Academy of Sciences
0

Scientific research communities are embracing AI-based solutions to target tractable scientific tasks and improve research workflows. However, the development and evaluation of such solutions are scattered across multiple disciplines. We formalize the problem of scientific AI benchmarking, and propose a system called SAIBench in the hope of unifying the efforts and enabling low-friction on-boarding of new disciplines. The system approaches this goal with SAIL, a domain-specific language to decouple research problems, AI models, ranking criteria, and software/hardware configuration into reusable modules. We show that this approach is flexible and can adapt to problems, AI models, and evaluation methods defined in different perspectives. The project homepage is https://www.computercouncil.org/SAIBench

READ FULL TEXT

page 1

page 2

page 3

page 4

02/13/2023

On the importance of AI research beyond disciplines

As the impact of AI on various scientific fields is increasing, it is cr...
04/17/2023

Quantifying the Benefit of Artificial Intelligence for Scientific Research

The ongoing artificial intelligence (AI) revolution has the potential to...
03/07/2023

AI for Science: An Emerging Agenda

This report documents the programme and the outcomes of Dagstuhl Seminar...
12/21/2018

Measuring science: irresistible temptations, easy shortcuts and dangerous consequences

In benchmarking international research, although publication and citatio...
04/04/2023

Scientists' Perspectives on the Potential for Generative AI in their Fields

Generative AI models, including large language models and multimodal mod...
11/03/2020

Rearrangement: A Challenge for Embodied AI

We describe a framework for research and evaluation in Embodied AI. Our ...
01/21/2021

Ten Simple Rules for Attending Your First Conference

Conferences are a mainstay of most scientific disciplines, where scienti...