SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency

by   Yan Wang, et al.

Extensive studies have shown that deep learning models are vulnerable to adversarial and natural noises, yet little is known about model robustness on noises caused by different system implementations. In this paper, we for the first time introduce SysNoise, a frequently occurred but often overlooked noise in the deep learning training-deployment cycle. In particular, SysNoise happens when the source training system switches to a disparate target system in deployments, where various tiny system mismatch adds up to a non-negligible difference. We first identify and classify SysNoise into three categories based on the inference stage; we then build a holistic benchmark to quantitatively measure the impact of SysNoise on 20+ models, comprehending image classification, object detection, instance segmentation and natural language processing tasks. Our extensive experiments revealed that SysNoise could bring certain impacts on model robustness across different tasks and common mitigations like data augmentation and adversarial training show limited effects on it. Together, our findings open a new research topic and we hope this work will raise research attention to deep learning deployment systems accounting for model performance. We have open-sourced the benchmark and framework at


page 2

page 4

page 8


RobustART: Benchmarking Robustness on Architecture Design and Training Techniques

Deep neural networks (DNNs) are vulnerable to adversarial noises, which ...

Holistic Adversarial Robustness of Deep Learning Models

Adversarial robustness studies the worst-case performance of a machine l...

Benchmarking the Robustness of Quantized Models

Quantization has emerged as an essential technique for deploying deep ne...

Adaptive Adversarial Training to Improve Adversarial Robustness of DNNs for Medical Image Segmentation and Detection

Recent methods based on Deep Neural Networks (DNNs) have reached high ac...

Deep Learning-based Multi-Organ CT Segmentation with Adversarial Data Augmentation

In this work, we propose an adversarial attack-based data augmentation m...

READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises

For many real-world applications, the user-generated inputs usually cont...

Please sign up or login with your details

Forgot password? Click here to reset