Deep Learning Training Procedure Augmentations

by   Cristian Simionescu, et al.

Recent advances in Deep Learning have greatly improved performance on various tasks such as object detection, image segmentation, sentiment analysis. The focus of most research directions up until very recently has been on beating state-of-the-art results. This has materialized in the utilization of bigger and bigger models and techniques which help the training procedure to extract more predictive power out of a given dataset. While this has lead to great results, many of which with real-world applications, other relevant aspects of deep learning have remained neglected and unknown. In this work, we will present several novel deep learning training techniques which, while capable of offering significant performance gains they also reveal several interesting analysis results regarding convergence speed, optimization landscape smoothness, and adversarial robustness. The methods presented in this work are the following: ∙ Perfect Ordering Approximation; a generalized model agnostic curriculum learning approach. The results show the effectiveness of the technique for improving training time as well as offer some new insight into the training process of deep networks. ∙ Cascading Sum Augmentation; an extension of mixup capable of utilizing more data points for linear interpolation by leveraging a smoother optimization landscape. This can be used for computer vision tasks in order to improve both prediction performance as well as improve passive model robustness.


page 1

page 19

page 37


Deep learning research landscape roadmap in a nutshell: past, present and future – Towards deep cortical learning

The past, present and future of deep learning is presented in this work....

Understanding Deep Learning Techniques for Image Segmentation

The machine learning community has been overwhelmed by a plethora of dee...

Offloading Deep Learning Powered Vision Tasks from UAV to 5G Edge Server with Denoising

Offloading computationally heavy tasks from an unmanned aerial vehicle (...

Computation-efficient Deep Learning for Computer Vision: A Survey

Over the past decade, deep learning models have exhibited considerable a...

DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation

While deep learning models have greatly improved the performance of most...

An Overview of the Data-Loader Landscape: Comparative Performance Analysis

Dataloaders, in charge of moving data from storage into GPUs while train...

Playing to distraction: towards a robust training of CNN classifiers through visual explanation techniques

The field of deep learning is evolving in different directions, with sti...

Please sign up or login with your details

Forgot password? Click here to reset