Mostafa Dehghani

research

∙ 04/24/2023

End-to-End Spatio-Temporal Action Localisation with Video Transformers

The most performant spatio-temporal action localisation models use exter...

0 Alexey Gritsenko, et al. ∙

research

∙ 12/19/2022

DSI++: Updating Transformer Memory with New Documents

Differentiable Search Indices (DSIs) encode a corpus of documents in the...

0 Sanket Vaibhav Mehta, et al. ∙

research

∙ 12/09/2022

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

Training large, deep neural networks to convergence can be prohibitively...

0 Aran Komatsuzaki, et al. ∙

research

∙ 10/20/2022

Transcending Scaling Laws with 0.1

Scaling language models improves performance but comes with significant ...

2 Yi Tay, et al. ∙

research

∙ 10/14/2022

Λ-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells

Differentiable neural architecture search (DARTS) is a popular method fo...

0 Sajad Movahedi, et al. ∙

research

∙ 08/19/2022

Intersection of Parallels as an Early Stopping Criterion

A common way to avoid overfitting in supervised learning is early stoppi...

10 Ali Vardasbi, et al. ∙

research

∙ 07/21/2022

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

There have been a lot of interest in the scaling properties of Transform...

0 Yi Tay, et al. ∙

research

∙ 07/08/2022

Beyond Transfer Learning: Co-finetuning for Action Localisation

Transfer learning is the predominant paradigm for training deep networks...

9 Anurag Arnab, et al. ∙

research

∙ 05/02/2022

Retrieval-Enhanced Machine Learning

Although information access systems have long supported people in accomp...

0 Hamed Zamani, et al. ∙

research

∙ 11/25/2021

PolyViT: Co-training Vision Transformers on Images, Videos and Audio

Can we train a single transformer model capable of processing multiple m...

5 Valerii Likhosherstov, et al. ∙

research

∙ 11/20/2021

Discrete Representations Strengthen Vision Transformer Robustness

Vision Transformer (ViT) is emerging as the state-of-the-art architectur...

10 Chengzhi Mao, et al. ∙

research

∙ 10/25/2021

The Efficiency Misnomer

Model efficiency is a critical aspect of developing and deploying machin...

5 Mostafa Dehghani, et al. ∙

research

∙ 10/18/2021

SCENIC: A JAX Library for Computer Vision Research and Beyond

Scenic is an open-source JAX library with a focus on Transformer-based m...

31 Mostafa Dehghani, et al. ∙

research

∙ 10/05/2021

Exploring the Limits of Large Scale Pre-training

Recent developments in large-scale machine learning suggest that by scal...

5 Samira Abnar, et al. ∙

research

∙ 09/22/2021

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

There remain many open questions pertaining to the scaling behaviour of ...

3 Yi Tay, et al. ∙

research

∙ 07/14/2021

The Benchmark Lottery

The world of empirical machine learning (ML) strongly relies on benchmar...

8 Mostafa Dehghani, et al. ∙

research

∙ 06/21/2021

TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?

In this paper, we introduce a novel visual representation learning which...

0 Michael S. Ryoo, et al. ∙

research

∙ 06/10/2021

Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent

We focus on the problem of domain adaptation when the goal is shifting t...

7 Samira Abnar, et al. ∙

research

∙ 06/08/2021

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

State-of-the-art parameter-efficient fine-tuning methods rely on introdu...

0 Rabeeh Karimi Mahabadi, et al. ∙

research

∙ 05/07/2021

Are Pre-trained Convolutions Better than Pre-trained Transformers?

In the era of pre-trained language models, Transformers are the de facto...

11 Yi Tay, et al. ∙

research

∙ 03/29/2021

ViViT: A Video Vision Transformer

We present pure-transformer based models for video classification, drawi...

25 Anurag Arnab, et al. ∙

research

∙ 03/01/2021

OmniNet: Omnidirectional Representations from Transformers

This paper proposes Omnidirectional Representations from Transformers (O...

19 Yi Tay, et al. ∙

research

∙ 11/08/2020

Long Range Arena: A Benchmark for Efficient Transformers

Transformers do not scale very well to long sequence lengths largely bec...

5 Yi Tay, et al. ∙

research

∙ 10/22/2020

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

While the Transformer architecture has become the de-facto standard for ...

6 Alexey Dosovitskiy, et al. ∙

research

∙ 09/14/2020

Efficient Transformers: A Survey

Transformer model architectures have garnered immense interest lately du...

33 Yi Tay, et al. ∙

research

∙ 06/22/2020

IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression

In this paper we analyse and improve integer discrete flows for lossless...

12 Rianne van den Berg, et al. ∙

research

∙ 05/31/2020

Transferring Inductive Biases through Knowledge Distillation

Having the right inductive biases can be crucial in many tasks or scenar...

70 Samira Abnar, et al. ∙

research

∙ 03/24/2020

MetNet: A Neural Weather Model for Precipitation Forecasting

Weather forecasting is a long standing scientific challenge with direct ...

0 Casper Kaae Sønderby, et al. ∙

research

∙ 10/12/2018

HiTR: Hierarchical Topic Model Re-estimation for Measuring Topical Diversity of Documents

A high degree of topical diversity is often considered to be an importan...

0 Hosein Azarbonyad, et al. ∙

research

∙ 07/10/2018

Universal Transformers

Self-attentive feed-forward sequence models have been shown to achieve i...

6 Mostafa Dehghani, et al. ∙

research

∙ 06/21/2018

Learning to Rank from Samples of Variable Quality

Training deep neural networks requires many training samples, but in pra...

16 Mostafa Dehghani, et al. ∙

research

∙ 11/30/2017

Learning to Learn from Weak Supervision by Full Supervision

In this paper, we propose a method for training neural networks when we ...

0 Mostafa Dehghani, et al. ∙

research

∙ 11/15/2017

Words are Malleable: Computing Semantic Shifts in Political and Media Discourse

Recently, researchers started to pay attention to the detection of tempo...

0 Hosein Azarbonyad, et al. ∙

research

∙ 11/08/2017

Fidelity-Weighted Learning

Training deep neural networks requires many training samples, but in pra...

0 Mostafa Dehghani, et al. ∙

research

∙ 11/01/2017

Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision

Training deep neural networks requires massive amounts of training data,...

0 Mostafa Dehghani, et al. ∙

research

∙ 11/01/2017

On Search Powered Navigation

Query-based searching and browsing-based navigation are the two main com...

0 Mostafa Dehghani, et al. ∙

research

∙ 08/11/2017

Learning to Attend, Copy, and Generate for Session-Based Query Suggestion

Users try to articulate their complex information needs during search se...

0 Mostafa Dehghani, et al. ∙

research

∙ 07/24/2017

Share your Model instead of your Data: Privacy Preserving Mimic Learning for Ranking

Deep neural networks have become a primary tool for solving problems in ...

0 Mostafa Dehghani, et al. ∙

research

∙ 07/13/2017

Neural Networks for Information Retrieval

Machine learning plays a role in many aspects of modern IR systems, and ...

0 Tom Kenter, et al. ∙

research

∙ 04/28/2017

Neural Ranking Models with Weak Supervision

Despite the impressive improvements achieved by unsupervised deep neural...

0 Mostafa Dehghani, et al. ∙

research

∙ 09/02/2016

On Horizontal and Vertical Separation in Hierarchical Text Classification

Hierarchy is a common and effective way of organizing data and represent...

0 Mostafa Dehghani, et al. ∙

Mostafa Dehghani

Featured Co-authors