How Deep is your Learning: the DL-HARD Annotated Deep Learning Dataset

05/17/2021
by   Iain Mackie, et al.
0

Deep Learning Hard (DL-HARD) is a new annotated dataset designed to more effectively evaluate neural ranking models on complex topics. It builds on TREC Deep Learning (DL) topics by extensively annotating them with question intent categories, answer types, wikified entities, topic categories, and result type metadata from a commercial web search engine. Based on this data, we introduce a framework for identifying challenging queries. DL-HARD contains fifty topics from the official DL 2019/2020 evaluation benchmark, half of which are newly and independently assessed. We perform experiments using the official submitted runs to DL on DL-HARD and find substantial differences in metrics and the ranking of participating systems. Overall, DL-HARD is a new resource that promotes research on neural ranking methods by focusing on challenging and complex topics.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/09/2022

CODEC: Complex Document and Entity Collection

CODEC is a document and entity ranking benchmark that focuses on complex...
research
10/09/2019

Explaining Deep Learning-Based Networked Systems

While deep learning (DL)-based networked systems have shown great potent...
research
02/26/2020

DLSpec: A Deep Learning Task Exchange Specification

Deep Learning (DL) innovations are being introduced at a rapid pace. How...
research
12/03/2021

Characterizing Performance Bugs in Deep Learning Systems

Deep learning (DL) has been increasingly applied to a variety of domains...
research
03/02/2022

Code Smells in Machine Learning Systems

As Deep learning (DL) systems continuously evolve and grow, assuring the...
research
05/02/2022

Studying Retrievability of Publications and Datasets in an Integrated Retrieval System

In this paper, we investigate the retrievability of datasets and publica...
research
09/26/2019

Explainable Deep Learning for Augmentation of sRNA Expression Profiles

The lack of well-structured metadata annotations complicates there-usabi...

Please sign up or login with your details

Forgot password? Click here to reset