b'Di He'

Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding

The attention module, which is a crucial component in Transformer, canno...
06/23/2021 ∙ by Shengjie Luo, et al. ∙ 9 ∙ share

read it
First Place Solution of KDD Cup 2021 OGB Large-Scale Challenge Graph-Level Track

In this technical report, we present our solution of KDD Cup 2021 OGB La...
06/15/2021 ∙ by Chengxuan Ying, et al. ∙ 7 ∙ share

read it
Do Transformers Really Perform Bad for Graph Representation?

The Transformer architecture has become a dominant choice in many domain...
06/09/2021 ∙ by Chengxuan Ying, et al. ∙ 16 ∙ share

read it
Adversarial Training with Rectified Rejection

Adversarial training (AT) is one of the most effective strategies for pr...
05/31/2021 ∙ by Tianyu Pang, et al. ∙ 0 ∙ share

read it
How could Neural Networks understand Programs?

Semantic understanding of programs is a fundamental problem for programm...
05/10/2021 ∙ by Dinglan Peng, et al. ∙ 0 ∙ share

read it
Wav2vec-C: A Self-supervised Model for Speech Representation Learning

Wav2vec-C introduces a novel representation learning technique combining...
03/09/2021 ∙ by Samik Sadhu, et al. ∙ 0 ∙ share

read it
Transformers with Competitive Ensembles of Independent Mechanisms

An important development in deep learning from the earliest MLPs has bee...
02/27/2021 ∙ by Alex Lamb, et al. ∙ 8 ∙ share

read it
LazyFormer: Self Attention with Lazy Update

Improving the efficiency of Transformer-based language pre-training is a...
02/25/2021 ∙ by Chengxuan Ying, et al. ∙ 0 ∙ share

read it
Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder

Many real-world applications use Siamese networks to efficiently match t...
02/18/2021 ∙ by Shuqi Lu, et al. ∙ 0 ∙ share

read it
Revisiting Language Encoding in Learning Multilingual Representations

Transformer has demonstrated its great power to learn contextual word re...
02/16/2021 ∙ by Shengjie Luo, et al. ∙ 0 ∙ share

read it
Towards Certifying ℓ_∞ Robustness using Neural Networks with ℓ_∞-dist Neurons

It is well-known that standard neural networks, even with a high classif...
02/10/2021 ∙ by Bohang Zhang, et al. ∙ 2 ∙ share

read it
CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics

Accurate and robust prediction of patient's response to drug treatments ...
01/31/2021 ∙ by Di He, et al. ∙ 0 ∙ share

read it
A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine

An unsolved fundamental problem in biology and ecology is to predict obs...
10/09/2020 ∙ by Di He, et al. ∙ 0 ∙ share

read it
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

Normalization plays an important role in the optimization of deep neural...
09/07/2020 ∙ by Tianle Cai, et al. ∙ 1 ∙ share

read it
Taking Notes on the Fly Helps BERT Pre-training

How to make unsupervised language pre-training more efficient and less r...
08/04/2020 ∙ by Qiyu Wu, et al. ∙ 0 ∙ share

read it
Transferred Discrepancy: Quantifying the Difference Between Representations

Understanding what information neural networks capture is an essential p...
07/24/2020 ∙ by Yunzhen Feng, et al. ∙ 0 ∙ share

read it
Rethinking Positional Encoding in Language Pre-training

How to explicitly encode positional information into neural networks is ...
06/28/2020 ∙ by Guolin Ke, et al. ∙ 0 ∙ share

read it
Rethinking the Positional Encoding in Language Pre-training

How to explicitly encode positional information into neural networks is ...
06/28/2020 ∙ by Guolin Ke, et al. ∙ 0 ∙ share

read it
MC-BERT: Efficient Language Pre-Training via a Meta Controller

Pre-trained contextual representations (e.g., BERT) have become the foun...
06/10/2020 ∙ by Zhenhui Xu, et al. ∙ 0 ∙ share

read it
Invertible Image Rescaling

High-resolution digital images are usually downscaled to fit various dis...
05/12/2020 ∙ by Mingqing Xiao, et al. ∙ 8 ∙ share

read it
Incorporating BERT into Neural Machine Translation

The recently proposed BERT has shown great power on a variety of natural...
02/17/2020 ∙ by Jinhua Zhu, et al. ∙ 0 ∙ share

read it
On Layer Normalization in the Transformer Architecture

The Transformer is widely used in natural language processing tasks. To ...
02/12/2020 ∙ by Ruibin Xiong, et al. ∙ 0 ∙ share

read it
MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius

Adversarial training is one of the most popular ways to learn robust mod...
01/08/2020 ∙ by Runtian Zhai, et al. ∙ 4 ∙ share

read it
Defective Convolutional Layers Learn Robust CNNs

Robustness of convolutional neural networks has recently been highlighte...
11/19/2019 ∙ by Tiange Luo, et al. ∙ 20 ∙ share

read it
Fast Structured Decoding for Sequence Models

Autoregressive sequence models achieve state-of-the-art performance in d...
10/25/2019 ∙ by Zhiqing Sun, et al. ∙ 0 ∙ share

read it
On the Anomalous Generalization of GANs

Generative models, especially Generative Adversarial Networks (GANs), ha...
09/27/2019 ∙ by Jinchen Xuan, et al. ∙ 46 ∙ share

read it
Hint-Based Training for Non-Autoregressive Machine Translation

Due to the unparallelizable nature of the autoregressive factorization, ...
09/15/2019 ∙ by Zhuohan Li, et al. ∙ 0 ∙ share

read it
Multilingual Neural Machine Translation with Language Clustering

Multilingual neural machine translation (NMT), which translates multiple...
08/25/2019 ∙ by Xu Tan, et al. ∙ 0 ∙ share

read it
Representation Degeneration Problem in Training Natural Language Generation Models

We study an interesting problem in training neural network-based models ...
07/28/2019 ∙ by Jun Gao, et al. ∙ 0 ∙ share

read it
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View

The Transformer architecture is widely used in natural language processi...
06/06/2019 ∙ by Yiping Lu, et al. ∙ 0 ∙ share

read it
Adversarially Robust Generalization Just Requires More Unlabeled Data

Neural network robustness has recently been highlighted by the existence...
06/03/2019 ∙ by Runtian Zhai, et al. ∙ 0 ∙ share

read it
A Gram-Gauss-Newton Method Learning Overparameterized Deep Neural Networks for Regression Problems

First-order methods such as stochastic gradient descent (SGD) are curren...
05/28/2019 ∙ by Tianle Cai, et al. ∙ 0 ∙ share

read it
Multilingual Neural Machine Translation with Knowledge Distillation

Multilingual machine translation, which translates multiple languages wi...
02/27/2019 ∙ by Xu Tan, et al. ∙ 0 ∙ share

read it
Non-Autoregressive Machine Translation with Auxiliary Regularization

As a new neural machine translation approach, Non-Autoregressive machine...
02/22/2019 ∙ by Yiren Wang, et al. ∙ 0 ∙ share

read it
Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input

Non-autoregressive translation (NAT) models, which remove the dependence...
12/23/2018 ∙ by Junliang Guo, et al. ∙ 0 ∙ share

read it
Sentence-wise Smooth Regularization for Sequence to Sequence Learning

Maximum-likelihood estimation (MLE) is widely used in sequence to sequen...
12/12/2018 ∙ by Chengyue Gong, et al. ∙ 0 ∙ share

read it
When CTC Training Meets Acoustic Landmarks

Connectionist temporal classification (CTC) training criterion provides ...
11/05/2018 ∙ by Di He, et al. ∙ 8 ∙ share

read it
Augmenting Input Method Language Model with user Location Type Information

Geo-tags from micro-blog posts have been shown to be useful in many data...
09/21/2018 ∙ by Di He, et al. ∙ 0 ∙ share

read it
FRAGE: Frequency-Agnostic Word Representation

Continuous word representation (aka word embedding) is a basic building ...
09/18/2018 ∙ by Chengyue Gong, et al. ∙ 0 ∙ share

read it
Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter

Neural machine translation usually adopts autoregressive models and suff...
09/01/2018 ∙ by Lijun Wu, et al. ∙ 0 ∙ share

read it
Double Path Networks for Sequence to Sequence Learning

Encoder-decoder based Sequence to Sequence learning (S2S) has made remar...
06/13/2018 ∙ by Kaitao Song, et al. ∙ 0 ∙ share

read it
Towards Binary-Valued Gates for Robust LSTM Training

Long Short-Term Memory (LSTM) is one of the most widely used recurrent s...
06/08/2018 ∙ by Zhuohan Li, et al. ∙ 0 ∙ share

read it
Dense Information Flow for Neural Machine Translation

Recently, neural machine translation has achieved remarkable progress by...
06/03/2018 ∙ by Yanyao Shen, et al. ∙ 0 ∙ share

read it
Improved ASR for Under-Resourced Languages Through Multi-Task Learning with Acoustic Landmarks

Furui first demonstrated that the identity of both consonant and vowel c...
05/15/2018 ∙ by Di He, et al. ∙ 0 ∙ share

read it
Acoustic Landmarks Contain More Information About the Phone String than Other Frames for Automatic Speech Recognition with Deep Neural Network Acoustic Model

Most mainstream Automatic Speech Recognition (ASR) systems consider all ...
10/27/2017 ∙ by Di He, et al. ∙ 0 ∙ share

read it
Acoustic Landmarks Contain More Information About the Phone String than Other Frames

Most mainstream Automatic Speech Recognition (ASR) systems consider all ...
10/27/2017 ∙ by Di He, et al. ∙ 0 ∙ share

read it
Dual Learning for Machine Translation

While neural machine translation (NMT) is making good progress in the pa...
11/01/2016 ∙ by Yingce Xia, et al. ∙ 0 ∙ share

read it
Sentence Level Recurrent Topic Model: Letting Topics Speak for Themselves

We propose Sentence Level Recurrent Topic Model (SLRTM), a new topic mod...
04/07/2016 ∙ by Fei Tian, et al. ∙ 0 ∙ share

read it
A Game-theoretic Machine Learning Approach for Revenue Maximization in Sponsored Search

Sponsored search is an important monetization channel for search engines...
06/03/2014 ∙ by Di He, et al. ∙ 0 ∙ share

read it
Generalized Second Price Auction with Probabilistic Broad Match

Generalized Second Price (GSP) auctions are widely used by search engine...
04/15/2014 ∙ by Wei Chen, et al. ∙ 0 ∙ share

read it

Di He

Featured Co-authors