Revealing Secrets From Pre-trained Models

07/19/2022
by   Mujahid Al Rafi, et al.
0

With the growing burden of training deep learning models with large data sets, transfer-learning has been widely adopted in many emerging deep learning algorithms. Transformer models such as BERT are the main player in natural language processing and use transfer-learning as a de facto standard training method. A few big data companies release pre-trained models that are trained with a few popular datasets with which end users and researchers fine-tune the model with their own datasets. Transfer-learning significantly reduces the time and effort of training models. However, it comes at the cost of security concerns. In this paper, we show a new observation that pre-trained models and fine-tuned models have significantly high similarities in weight values. Also, we demonstrate that there exist vendor-specific computing patterns even for the same models. With these new findings, we propose a new model extraction attack that reveals the model architecture and the pre-trained model used by the black-box victim model with vendor-specific computing patterns and then estimates the entire model weights based on the weight value similarities between the fine-tuned model and pre-trained model. We also show that the weight similarity can be leveraged for increasing the model extraction feasibility through a novel weight extraction pruning.

READ FULL TEXT

page 1

page 4

research
03/02/2021

TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning

The increasing of pre-trained models has significantly facilitated the p...
research
05/13/2022

Revisiting the Updates of a Pre-trained Model for Few-shot Learning

Most of the recent few-shot learning algorithms are based on transfer le...
research
02/15/2018

Cost-Effective Training of Deep CNNs with Active Model Adaptation

Deep convolutional neural networks have achieved great success in variou...
research
10/14/2021

A Comprehensive Study on Torchvision Pre-trained Models for Fine-grained Inter-species Classification

This study aims to explore different pre-trained models offered in the T...
research
09/29/2022

FastPacket: Towards Pre-trained Packets Embedding based on FastText for next-generation NIDS

New Attacks are increasingly used by attackers everyday but many of them...
research
02/24/2020

Using wavelets to analyze similarities in image datasets

Deep learning image classifiers usually rely on huge training sets and t...
research
06/03/2022

Uncertainty Estimation in Machine Learning

Most machine learning techniques are based upon statistical learning the...

Please sign up or login with your details

Forgot password? Click here to reset