b'Di He'

research

∙ 06/09/2022

Adversarial Noises Are Linearly Separable for (Nearly) Random Neural Networks

Adversarial examples, which are usually generated for specific inputs wi...

Huishuai Zhang, et al. ∙

research

∙ 06/04/2022

Is L^2 Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?

The Physics-Informed Neural Network (PINN) approach is a new and promisi...

Chuwei Wang, et al. ∙

research

∙ 05/26/2022

Your Transformer May Not be as Powerful as You Expect

Relative Positional Encoding (RPE), which encodes the relative distance ...

Shengjie Luo, et al. ∙

research

∙ 04/13/2022

METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

We present an efficient method of pretraining large-scale autoencoding l...

Payal Bajaj, et al. ∙

research

∙ 03/09/2022

Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets

This technical note describes the recent updates of Graphormer, includin...

Yu Shi, et al. ∙

research

∙ 02/28/2022

An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets

This technical note describes the recent updates of Graphormer, includin...

Yu Shi, et al. ∙

research

∙ 02/22/2022

VADOI:Voice-Activity-Detection Overlapping Inference For End-to-end Long-form Speech Recognition

While end-to-end models have shown great success on the Automatic Speech...

Jinhan Wang, et al. ∙

research

∙ 02/16/2022

HousE: Knowledge Graph Embedding with Householder Parameterization

The effectiveness of knowledge graph embedding (KGE) largely depends on ...

Rui Li, et al. ∙

research

∙ 11/02/2021

Can Vision Transformers Perform Convolution?

Several recent studies have demonstrated that attention-based networks, ...

Shanda Li, et al. ∙

research

∙ 10/13/2021

Boosting the Certified Robustness of L-infinity Distance Nets

Recently, Zhang et al. (2021) developed a new neural network architectur...

Bohang Zhang, et al. ∙

research

∙ 06/23/2021

Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding

The attention module, which is a crucial component in Transformer, canno...

Shengjie Luo, et al. ∙

research

∙ 06/15/2021

First Place Solution of KDD Cup 2021 OGB Large-Scale Challenge Graph-Level Track

In this technical report, we present our solution of KDD Cup 2021 OGB La...

Chengxuan Ying, et al. ∙

research

∙ 06/09/2021

Do Transformers Really Perform Bad for Graph Representation?

The Transformer architecture has become a dominant choice in many domain...

Chengxuan Ying, et al. ∙

research

∙ 05/31/2021

Adversarial Training with Rectified Rejection

Adversarial training (AT) is one of the most effective strategies for pr...

Tianyu Pang, et al. ∙

research

∙ 05/10/2021

How could Neural Networks understand Programs?

Semantic understanding of programs is a fundamental problem for programm...

Dinglan Peng, et al. ∙

research

∙ 03/09/2021

Wav2vec-C: A Self-supervised Model for Speech Representation Learning

Wav2vec-C introduces a novel representation learning technique combining...

Samik Sadhu, et al. ∙

research

∙ 02/27/2021

Transformers with Competitive Ensembles of Independent Mechanisms

An important development in deep learning from the earliest MLPs has bee...

Alex Lamb, et al. ∙

research

∙ 02/25/2021

LazyFormer: Self Attention with Lazy Update

Improving the efficiency of Transformer-based language pre-training is a...

Chengxuan Ying, et al. ∙

research

∙ 02/18/2021

Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder

Many real-world applications use Siamese networks to efficiently match t...

Shuqi Lu, et al. ∙

research

∙ 02/16/2021

Revisiting Language Encoding in Learning Multilingual Representations

Transformer has demonstrated its great power to learn contextual word re...

Shengjie Luo, et al. ∙

research

∙ 02/10/2021

Towards Certifying ℓ_∞ Robustness using Neural Networks with ℓ_∞-dist Neurons

It is well-known that standard neural networks, even with a high classif...

Bohang Zhang, et al. ∙

research

∙ 01/31/2021

CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics

Accurate and robust prediction of patient's response to drug treatments ...

Di He, et al. ∙

research

∙ 10/09/2020

A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine

An unsolved fundamental problem in biology and ecology is to predict obs...

Di He, et al. ∙

research

∙ 09/07/2020

GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

Normalization plays an important role in the optimization of deep neural...

Tianle Cai, et al. ∙

research

∙ 08/04/2020

Taking Notes on the Fly Helps BERT Pre-training

How to make unsupervised language pre-training more efficient and less r...

Qiyu Wu, et al. ∙

research

∙ 07/24/2020

Transferred Discrepancy: Quantifying the Difference Between Representations

Understanding what information neural networks capture is an essential p...

Yunzhen Feng, et al. ∙

research

∙ 06/28/2020

Rethinking Positional Encoding in Language Pre-training

How to explicitly encode positional information into neural networks is ...

Guolin Ke, et al. ∙

research

∙ 06/28/2020

Rethinking the Positional Encoding in Language Pre-training

How to explicitly encode positional information into neural networks is ...

Guolin Ke, et al. ∙

research

∙ 06/10/2020

MC-BERT: Efficient Language Pre-Training via a Meta Controller

Pre-trained contextual representations (e.g., BERT) have become the foun...

Zhenhui Xu, et al. ∙

research

∙ 05/12/2020

Invertible Image Rescaling

High-resolution digital images are usually downscaled to fit various dis...

Mingqing Xiao, et al. ∙

research

∙ 02/17/2020

Incorporating BERT into Neural Machine Translation

The recently proposed BERT has shown great power on a variety of natural...

Jinhua Zhu, et al. ∙

research

∙ 02/12/2020

On Layer Normalization in the Transformer Architecture

The Transformer is widely used in natural language processing tasks. To ...

Ruibin Xiong, et al. ∙

research

∙ 01/08/2020

MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius

Adversarial training is one of the most popular ways to learn robust mod...

Runtian Zhai, et al. ∙

research

∙ 11/19/2019

Defective Convolutional Layers Learn Robust CNNs

Robustness of convolutional neural networks has recently been highlighte...

Tiange Luo, et al. ∙

research

∙ 10/25/2019

Fast Structured Decoding for Sequence Models

Autoregressive sequence models achieve state-of-the-art performance in d...

Zhiqing Sun, et al. ∙

research

∙ 09/27/2019

On the Anomalous Generalization of GANs

Generative models, especially Generative Adversarial Networks (GANs), ha...

Jinchen Xuan, et al. ∙

research

∙ 09/15/2019

Hint-Based Training for Non-Autoregressive Machine Translation

Due to the unparallelizable nature of the autoregressive factorization, ...

Zhuohan Li, et al. ∙

research

∙ 08/25/2019

Multilingual Neural Machine Translation with Language Clustering

Multilingual neural machine translation (NMT), which translates multiple...

Xu Tan, et al. ∙

research

∙ 07/28/2019

Representation Degeneration Problem in Training Natural Language Generation Models

We study an interesting problem in training neural network-based models ...

Jun Gao, et al. ∙

research

∙ 06/06/2019

Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View

The Transformer architecture is widely used in natural language processi...

Yiping Lu, et al. ∙

research

∙ 06/03/2019

Adversarially Robust Generalization Just Requires More Unlabeled Data

Neural network robustness has recently been highlighted by the existence...

Runtian Zhai, et al. ∙

research

∙ 05/28/2019

A Gram-Gauss-Newton Method Learning Overparameterized Deep Neural Networks for Regression Problems

First-order methods such as stochastic gradient descent (SGD) are curren...

Tianle Cai, et al. ∙

research

∙ 02/27/2019

Multilingual Neural Machine Translation with Knowledge Distillation

Multilingual machine translation, which translates multiple languages wi...

Xu Tan, et al. ∙

research

∙ 02/22/2019

Non-Autoregressive Machine Translation with Auxiliary Regularization

As a new neural machine translation approach, Non-Autoregressive machine...

Yiren Wang, et al. ∙

research

∙ 12/23/2018

Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input

Non-autoregressive translation (NAT) models, which remove the dependence...

Junliang Guo, et al. ∙

research

∙ 12/12/2018

Sentence-wise Smooth Regularization for Sequence to Sequence Learning

Maximum-likelihood estimation (MLE) is widely used in sequence to sequen...

Chengyue Gong, et al. ∙

research

∙ 11/05/2018

When CTC Training Meets Acoustic Landmarks

Connectionist temporal classification (CTC) training criterion provides ...

Di He, et al. ∙

research

∙ 09/21/2018

Augmenting Input Method Language Model with user Location Type Information

Geo-tags from micro-blog posts have been shown to be useful in many data...

Di He, et al. ∙

research

∙ 09/18/2018

FRAGE: Frequency-Agnostic Word Representation

Continuous word representation (aka word embedding) is a basic building ...

Chengyue Gong, et al. ∙

research

∙ 09/01/2018

Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter

Neural machine translation usually adopts autoregressive models and suff...

Lijun Wu, et al. ∙

Di He

Featured Co-authors