Despite the successful application in computer vision tasks such as image classification[35, 25], Deep Neural Networks (DNNs) have been found vulnerable to adversarial attacks. In particular, the DNN’s prediction can be arbitrarily changed by just applying an imperceptible perturbation to the input image [75, 21]. Moreover, such adversarial attacks can effectively compromise the recent state-of-the-art DNNs such as Inception [73, 74] and ResNet 
. This poses a serious security risk on many DNN-based applications such as face recognition, where recognition evasion or impersonation can be easily achieved[15, 70, 34, 78].
also suffer from similar vulnerability. Taking the image-based product search as an example, a fair ranking system should rank the database products according to their visual similarity to the query, as shown in Fig.1 (row 1). Nevertheless, a malicious seller may attempt to raise the rank of his/her own product by adding perturbation to the image (CA+, row 2), or lower the rank of his competitor’s product (CA-, row 3); Besides, a “man-in-the-middle” attacker (e.g., a malicious advertising company) could hijack and imperceptibly perturb the query image in order to promote (QA+, row 4) or impede (QA-, row 5) the sales of specific products.
Unlike classification tasks where images are predicted independently, the rank of one candidate is related to the query as well as other candidates for image ranking. The relative relations among candidates and queries determine the final ranking order. Therefore, we argue that the existing adversarial classification attacks are incompatible with the ranking scenario. Thus, we need to thoroughly study the adversarial ranking attack.
In this paper, adversarial ranking attack aims to raise or lower the ranks of some chosen candidates with respect to a specific query set . This can be achieved by either Candidate Attack (CA) or Query Attack (QA). In particular, CA is defined as to raise (abbr. CA+) or lower (abbr. CA-) the rank of a single candidate with respect to the query set by perturbing itself; while QA is defined as to raise (abbr. QA+) or lower (abbr. QA-) the ranks of a candidate set with respect to a single query by perturbing . Thus, adversarial ranking attack can be achieved by performing CA on each , or QA on each . In practice, the choice of CA or QA depends on the accessibility to the candidate or query respectively, i.e., CA is feasible for modifiable candidate, while QA is feasible for modifiable query.
An effective implementation of these attacks is proposed in this paper. As we know, a typical DNN-based ranking model maps objects (i.e., queries and candidates) to a common embedding space, where the distances among them determine the final ranking order. Predictably, the object’s position in the embedding space will be changed by adding a perturbation to it. Therefore, the essential of adversarial ranking attack is to find a proper perturbation, which could push the object to a desired position that leads to the expected ranking order. Specifically, we first represent the expected ranking order as a set of inequalities. Subsequently, a triplet-like objective function is designed according to those inequalities, and combined with Projected Gradient Descent (PGD) to efficiently obtain the desired adversarial perturbation.
Opposed to the proposed attacks, adversarial ranking defense is worth being investigated especially for security-sensitive deep ranking applications. Until now, the Madry defense  is regarded as the most effective method for classification defense. However, we empirically discovered a primary challenge of diverging training loss while directly adapting such mechanism for ranking defense, possibly due to the generated adversarial examples being too “strong”. In addition, such defense mechanism needs to defend against distinct ranking attacks individually, but a generic defense method against all CA+, CA-, QA+ and QA- attacks is preferred.
To this end, a shift-distance based ranking defense is proposed, which could simultaneously defend against all attacks. Note that the position shift of objects in the embedding space is the key for all ranking attacks. Although different attacks prefer distinct shift directions (e.g., CA+ and CA- often prefer opposed shifting directions), a large shift distance is their common preference. If we could reduce the shift distance of embeddings incurred by adversarial perturbation, all attacks can be simultaneously defensed. Specifically, we first propose a shift-distance based ranking attack, which aims to push the objects as far from their original positions as possible. And then, the adversarial examples generated from such attack is involved in the adversarial training. Experimental results manifest that our ranking defense can converge and moderately improve model robustness.
In addition, our ranking attacks have some good properties for realistic applications. First, our adversary is transferable, i.e., the adversary obtained from a known DNN ranker can be directly used to attack an unknown DNN ranker (i.e., the network architecture and parameters are unknown). Second, our attacks can be extended to universal ranking attacks with slight performance drop, i.e., we could learn a universal perturbation to all candidates for CA, or a universal perturbation to all queries for QA. Such properties illustrate the possibility of practical black-box attack.
To the best of our knowledge, this is the first work that thoroughly studies the adversarial ranking attack and defense. In brief, our contributions are:
The adversarial ranking attack is defined and implemented, which can intentionally change the ranking results by perturbing the candidates or queries.
An adversarial ranking defense method is proposed to improve the ranking model robustness, and mitigate all the proposed attacks simultaneously.
2 Related Works
Adversarial Attacks. Szegedy et al.  claimed that DNN is susceptible to imperceptible adversarial perturbations added to inputs, due to the intriguing “blind spot” property, which was later ascribed to the local linearity  of neural networks. Following these findings, many white-box (model architecture and parameters are known to the adversary) attacking methods [54, 61, 36, 7, 10, 13, 66, 72, 50, 80, 9, 20] are proposed to effectively compromise the state-of-the-art DNN classifiers. Among them, PGD  is regarded as one of the most powerful attacks . Notably, adversarial examples are discovered to be transferable [60, 59] among different neural network classifiers, which inspired a series of black-box attacks [71, 79, 83, 45, 14, 28]. On the other hand, universal (i.e., image-agnostic) adversarial perturbations are also discovered [53, 41]. The existence of adversarial examples stimulated research interests in areas such as object detection [48, 11, 85], semantic segmentation 65], etc. It is even possible to create physical adversarial examples [70, 4, 18, 78].
Deep Ranking. Different from the traditional “learning to rank” [42, 31] methods, DNN-based ranking methods often embed data samples (including both queries and candidates) of all modalities into a common embedding space, and subsequently determine the ranking order based on distance. Such workflow has been adopted in distance metric learning [8, 76, 57, 30], image retrieval , cross-modal retrieval [56, 19, 39, 33], and face recognition .
Adversarial Attacks in Deep Ranking. For information retrieval and ranking systems, the risk of malicious users manipulating the ranking always exists [23, 27]. However, only a few research efforts have been made in adversarial attacks in deep ranking. Liu et al.  proposed adversarial queries leading to incorrect retrieval results; while Li et al.  staged similar attack with universal perturbation that corrupts listwise ranking results. None of the aforementioned research efforts explore the adversarial ranking attack.
, evidential deep learning, etc. However, defense in deep ranking systems remains mostly uncharted.
3 Adversarial Ranking
Generally, a DNN-based ranking task could be formulated as a metric learning problem. Given the query and candidate set , deep ranking is to learn a mapping
(usually implemented as a DNN) which maps all candidates and query into a common embedding space, such that the relative distances among the embedding vectors could satisfy the expected ranking order. For instance, if candidateis more similar to the query than candidate , it is encouraged for the mapping to satisfy the inequality 111Sometimes cosine distance is used instead., where denotes norm. For brevity, we denote as in following text.
Therefore, adversarial ranking attack is to find a proper adversarial perturbation which leads the ranking order to be changed as expected. For example, if a less relevant is expected to be ranked ahead of a relevant , it is desired to find a proper perturbation to perturb , i.e. , such that the inequality could be changed into . In the next, we will describe Candidate Attack and Query Attack in detail.
3.1 Candidate Attack
Candidate Attack (CA) aims to raise (abbr. CA+) or lower (abbr. CA-) the rank of a single candidate with respect to a set of queries by adding perturbation to the candidate itself, i.e. .
Let denote the rank of the candidate with respect to the query , where indicates the set of all candidates, and a smaller rank value represents a higher ranking. Thus, the CA+ that raises the rank of with respect to every query by perturbation could be formulated as the following problem,
where is a -bounded -neighbor of , is a predefined small positive constant, the constraint limits the perturbation to be “visually imperceptible”, and ensures the adversarial example remains a valid input image. Although alternative “imperceptible” constraints exist (e.g., [72, 12],  and [7, 54] variants), we simply follow [21, 36, 50] and use the constraint.
In metric learning, given two candidates where is ranked ahead of , i.e. , the ranking order is represented as an inequality and formulated in triplet loss:
Similarly, the attacking goal of CA+ in Eq. (1) can be readily converted into a series of inequalities, and subsequently turned into a sum of triplet losses,
To solve the optimization problem, Projected Gradient Descent (PGD) method [50, 36] (a.k.a the iterative version of FGSM ) can be used. Note that PGD is one of the most effective first-order gradient-based algorithms , popular among related works about adversarial attack.
Specifically, in order to find an adversarial perturbation to create a desired adversarial candidate , the PGD algorithm alternates two steps at every iteration . Step one updates according to the gradient of Eq. (4); while step two clips the result of step one to fit in the -neighboring region :
where is a constant hyper-parameter indicating the PGD step size, and is initialized as . After iterations, the desired adversarial candidate is obtained as , which is optimized to satisfy as many inequalities as possible. Each inequality represents a pairwise ranking sub-problem, hence the adversarial candidate will be ranked ahead of other candidates with respect to every specified query .
Likewise, the CA- that lowers the rank of a candidate with respect to a set of queries can be obtained in similar way:
3.2 Query Attack
Query Attack (QA) is supposed to raise (abbr. QA+) or lower (abbr. QA-) the rank of a set of candidates with respect to the query , by adding adversarial perturbation to the query . Thus, QA and CA are two “symmetric” attacks. The QA- for lowering the rank could be formulated as follows:
where is the -neighbor of . Likewise, this attacking objective can also be transformed into the following constrained optimization problem:
and it can be solved with the PGD algorithm. Similarly, the QA+loss function for raising the rank of is as follows:
Unlike CA, QA perturbs the query image, and hence may drastically change its semantics, resulting in abnormal retrieval results. For instance, after perturbing a “lamp” query image, some unrelated candidates (e.g., “shelf”, “toaster”, etc.) may appear in the top return list. Thus, an ideal query attack should preserve the query semantics, i.e., the candidates in 222The complement of the set . should retain their original ranks if possible. Thus, we propose the Semantics-Preserving Query Attack (SP-QA) by adding the SP term to mitigate the semantic changes , e.g.,
where , i.e., contains the top- most-relevant candidates corresponding to , and the term helps preserve the query semantics by retaining some candidates in the retrieved ranking list. Constant is a predefined integer; and constant is a hyper-parameter for balancing the attack effect and semantics preservation. Unless mentioned, in the following text QA means SP-QA by default.
3.3 Robustness & Defense
Adversarial defense for classification has been extensively explored, and many of them follows the adversarial training mechanism [29, 37, 50]. In particular, the adversarial counterparts of the original training samples are used to replace or augment the training samples. Until now, Madry defense  is regarded as the most effective [77, 3] adversarial training method. However, when directly adapting such classification defense to improve ranking robustness, we empirically discovered a primary challenge of diverging training loss, possibly due to the generated adversarial examples being too “strong”. Moreover, such defense mechanism needs to defend against distinct attacks individually. Therefore, a generic defense against all the proposed attacks is preferred.
Note that the underlying principle of adversarial ranking attack is to shift the embeddings of candidates/queries to a proper place, and a successful attack depends on a large shift distance as well as a correct shift direction. A large shift distance is an indispensable objective for all the CA+, CA-, QA+ and QA- attacks. Predictably, a reduction in shift distance could improve model robustness against all attacks simultaneously.
To this end, we propose a “maximum-shift-distance” attack that pushes an embedding vector as far from its original position as possible (resembles Feature Adversary  for classification), . Then we use adversarial examples obtained from this attack to replace original training samples for adversarial training, hence reduce the shift distance incurred by adversarial perturbations.
A ranking model can be normally trained with the defensive version of the triplet loss:
Unlike the direct adaptation of Madry defense, the training loss does converge in our experiments.
|(CT) Cosine Distance, Triplet Loss (R@1=99.1%)|
. We respectively train typical vanilla models on these datasets with PyTorch, and conduct attacks on their corresponding validation datasets (used as ).
Evaluation Metric. Adversarial ranking attack aims to change the ranks of candidates. For each candidate , its normalized rank is calculated as where , and is the length of full ranking list. Thus, , and a top ranked will have a small . The attack effectiveness can be measured by the magnitude of change in .
Performance of Attack. To measure the performance of a single CA attack, we average the rank of candidate across every query , i.e., . Similarly, the performance of a single QA attack can be measured by the average rank across every candidate , i.e., . For the overall performance of an attack, we conduct times of independent attacks and report the mean of or , accordingly.
CA+ & QA+. For CA+, the query set is randomly sampled from . Likewise, for QA+, the candidate set is from . Without attack, both the and will approximate to , and the attacks should significantly decrease the value.
CA- & QA-. In practice, the for CA- and the for QA- cannot be randomly sampled, because the two attacks are often to lower some top ranked candidates. Thus, the two sets should be selected from the top ranked samples (top- in our experiments) in . Formally, given the candidate for CA-, we randomly sample the queries from as . Given the query for QA-, candidates are randomly sampled from as . Without attack, both the and will be close to , and the attacks should significantly increase the value.
Hyper-Parameters. We conduct CA with queries, and QA with candidates, respectively. In QA, we let . The SP balancing parameter is set to for QA+ , and for QA-. In addition, We investigate attacks of different strength , e.g. , where is the strongest attack. The PGD step size is empirically set to , and the number of PGD iterations to . We perform times of attack to obtain the reported performance.
4.1 MNIST Dataset
|(CTD) Cosine Distance, Triplet Loss, Defensive (R@1=98.3%)|
Following conventional settings with the MNIST  dataset, we train a CNN ranking model comprising convolutional layers and fully-connected layer. This CNN architecture (denoted as C2F1) is identical to the one used in  except for the removal of the last fully-connected layer. Specifically, the ranking model is trained with cosine distance and triplet loss. The retrieval performance of the model is Recall@1= (R@), as shown in Tab. 1 in grey highlight.
Attacking results against this vanilla model (i.e., the ranking model which is not enhanced with our defense method) are presented in Tab. 1. For example, a strong CA+ attack (i.e., ) for can raise the rank from to . Likewise, the rank of can be raised to , , for chosen queries, respectively.
On the other hand, a strong CA- attack for can lower the rank from to . The results of strong CA- attacks for are similar to the case.
The results of QA+ and QA- are also shown in Tab. 1. the rank changes with QA attacks are less dramatic (but still significant) than CA. This is due to the additional difficulty introduced by SP term in Eq. (12), and the QA attack effectiveness is inversely correlated with . For instance, a strong QA- for can only lower the rank from to , but the attacking effect can be further boosted by decreasing . More experimental results are presented in following discussion. In brief, our proposed attacks against the vanilla ranking model is effective.
Next, we evaluate the performance of our defense method. Our defense should be able to enhance the robustness of a ranking model, which can be measured by the difference between the attack effectiveness with our defense and the attack effectiveness without our defense. As a common phenomenon of adversarial training, our defense mechanism leads to a slight retrieval performance degradation for unperturbed input (highlighted in blue in Tab. 2), but the attacking effectiveness is clearly mitigated by our defense. For instance, the same strong CA+ attack for on the defensive model (i.e., the ranking model which is enhanced by our defense method) can only raise the rank from to , compared to its vanilla counterpart raising to . Further analysis suggests that the weights in the first convolution layer of the defensive model are closer to21].
To visualize the effect of our attacks and defense, we track the attacking effect with varying from to on the vanilla and defensive models, as shown in Fig. 2. It is noted that our defense could significantly suppress the maximum embedding shift distance incurred by adversarial perturbation to nearly , but the defensive model is still not completely immune to attacks. We speculate the defensive model still has “blind spots”  in some local areas that could be exploited by the attacks.
In summary, these results and further experiments (see supplementary material) suggest that: (1) deep ranking models are vulnerable to adversarial ranking attacks, no matter what loss function or distance metric is selected; (2) vanilla models trained with contrastive loss are more robust than those trained with triplet loss. This is possibly due to contrastive loss explicitly reducing the intra-class embedding variation. Additionally, our defense method could consistently improve the robustness of all these models; (3) different distance metrics have almost negligible contribution on robustness. Specifically, Euclidean distance-based models are slightly more susceptible to weak (e.g., ) attacks; (4) Euclidean distance-based models are harder to defend than cosine distance-based ones. Beyond these experiments, we also find that the margin hyper-parameter of triplet loss and the dimensionality of the embedding space have marginal influences on model robustness.
4.2 Fashion-MNIST Dataset
|(CT) Cosine Distance, Triplet Loss (R@1=88.8%)|
|(CTD) Cosine Distance, Triplet Loss, Defensive (R@1=79.6%)|
Fashion-MNIST  is an MNIST-like but more difficult dataset, comprising training examples and test samples. The samples are greyscale images covering different fashion product classes, including “T-shirt” and “dress”, etc. We train the vanilla and defensive models based on the cosine distance and triplet loss and conduct attack experiments.
The attack and defense results are available in Tab. 3. From the table, we note that our attacks could achieve better effect compared to experiments on MNIST. For example, in a strong CA+ for , the rank can be raised to . On the other hand, despite the moderate improvement in robustness, the defensive model performs worse in unperturbed sample retrieval, as expected. The performance degradation is more pronounced on this dataset compared to MNIST. We speculate the differences are related to the increased dataset difficulty.
4.3 Stanford Online Products Dataset
Stanford Online Products (SOP) dataset  contains k images of k classes of real online products from eBay for metric learning. We use the same dataset split as used in the original work . We also train the same vanilla ranking model using the same triplet ranking loss function with Euclidean distance, except that the GoogLeNet  is replaced with ResNet-18 . The ResNet-18 achieves better retrieval performance.
Attack and defense results on SOP are present in Tab. 4. It is noted that our attacks are quite effective on this difficult large-scale dataset, as merely perturbation () to any candidate image could make it ranked ahead or behind of nearly all the rest candidates (as shown by the CA+ and CA- results with ). The QA on this dataset is significantly effective as well. On the other hand, our defense method leads to drastically decreased retrieval performance, i.e. R@1 from to , which is expected on such a difficult dataset. Meanwhile, our defense could moderately improve the model robustness against relatively weaker adversarial examples (e.g. ), but improving model robustness on this dataset is more difficult, compared to experiments on other datasets.
By comparing the results among all the three datasets, we find that ranking models trained on simpler datasets are less prone to be attacked, and are easier to defend.On the contrary, models trained on harder datasets are more susceptible to adversarial ranking attack, and are more difficult to defend. Therefore, we speculate that models used in realistic applications could be easier to attack, because they are usually trained on larger-scale and more difficult datasets.
|(ET) Euclidean Distance, Triplet Loss (R@1=63.1%)|
|(ETD) Euclidean Distance, Triplet Loss, Defensive (R@1=40.2%)|
In this section, we study the transferability of our adversarial ranking examples, and universal adversarial perturbation for ranking. Both of them illustrate the possibility of practical black-box attack. Additionally, we also perform parameter search on the balancing parameter for QA.
5.1 Adversarial Example Transferability
As demonstrated in previous experiments, deep ranking models can be compromised by our white-box attacks. In realistic scenarios, the white-box attacks are not practical enough because the model to be attacked is often unknown (i.e., the architecture and parameters are unknown).
On the other hand, adversarial examples for classification have been found transferable [60, 59] (i.e. model-agnostic) between different models with different network architectures. And the transferability has become the foundation of a class of existing black-box attacks. Specifically, for such a typical attack, adversarial examples are generated from a replacement model  using a white-box attack, and are directly used to attack the black-box model.
Adversarial ranking attack could be more practical if the adversarial ranking examples have the similar transferability. Besides the C2F1 model, we train two vanilla models on the MNIST dataset: (1) LeNet , which has lower model capacity compared to C2F1; (2) ResNet-18  (denoted as Res18), which has a better network architecture and higher model capacity.
|CA+ Transfer (Black Box),|
|CA- Transfer (Black Box),|
|QA+ Transfer (Black Box),|
|QA- Transfer (Black Box),|
The results are present in Tab. 5. For example, in the CA+ transfer attack, we generate adversarial candidates from the C2F1 model and directly use them to attack the Res18 model (row 2, column 3, top-left table), and the ranks of the adversarial candidates with respect to the same query is still raised to . We also find the CA- transfer attack is effective, where the ranks of our adversarial candidates are lowered, e.g. from to (row 2, column 3, bottom-left table). Similar results can be observed on the QA transfer experiments, and they show weaker effect due to the SP term.
From these results, we find that: (1) CNN with better architecture and higher model capacity (i.e., Res18), is less susceptible to adversarial ranking attack. This conclusion is consistent with one of Madry’s , which claims that higher model capacity could help improve model robustness; (2) adversarial examples generated from the Res18 have the most significant effectiveness in transfer attack; (3) CNN of low model capacity (i.e., LeNet), performs moderately in terms of both adversarial example transferability and model robustness. We speculate its robustness stems from a forced regularization effect due low model capacity. Beyond these, we also noted adversarial ranking examples are transferable disregarding the difference in loss function or distance metric.
Apart from transferability across different architectures, we also investigated the transferability between the C2F1 models with different network parameters. Results suggest similar transferability between these models. Notably, when transferring adversarial examples to a defensive C2F1 model, the attacking effect is significantly mitigated. The result further demonstrates the effectiveness of our defense.
5.2 Universal Perturbation for Ranking
Recently, universal (i.e. image-agnostic) adversarial perturbation  for classification has been found possible, where a single perturbation may lead to misclassification when added to any image. Thus, we also investigate the existence of universal adversarial perturbation for ranking.
To this end, we follow  and formulate the image-agnostic CA+ (abbr. I-CA+). Given a set of candidates and a set of queries , I-CA+ is to find a single universal adversarial perturbation , so that the rank of every perturbed candidate with respect to can be raised. The corresponding optimization problem of I-CA+ is:
When applied with such universal perturbation, the rank of any candidate w.r.t is expected to be raised. The objective functions of I-CA-, I-QA+ and I-QA- can be obtained in similar way. Note, unlike  which aims to find universal perturbation that can make image retrieval system return irrelevant results, our universal perturbations have distinct purposes.
We conduct experiment on the MNIST dataset. For I-CA+ attack, we randomly sample of for generating the universal perturbation. Following , another non-overlapping examples are randomly sampled from to test whether the generated perturbation is generalizable on “unseen” (i.e., not used for generating the perturbation) images. Experiments for the other image-agnostic attacks are conducted similarly. Note, we only report the I-CA- and I-QA- effectiveness on the top ranked samples, similar to CA- and QA-.
|50 2.1||2.1 93.9||50 0.2||0.5 94.1|
|50 18.1||0.6 9.5||50 20.5||2.1 7.6|
|I-CA+ (unseen)||I-CA- (unseen)||I-QA+ (unseen)||I-QA- (unseen)|
|50 18.5||0.7 9.4||50 21.0||2.2 7.4|
As shown in Tab. 6, our I-CA can raise the ranks of to , or lower them to . When added to “unseen“ candidate images, our universal perturbation could retain nearly the same effectiveness. This may due to low intra-class variance of the MNIST dataset.
5.3 Semantics Preserving for QA
As discussed previously, the Query Attack (QA) may drastically change the semantics of the query . To alleviate this problem, the Semantics-Preserving (SP) term is added to the naive QA to help preserve the query semantics. Predictably, it is more difficult to perform QA with a large , as the ranks of are almost not allowed to be changed.
To investigate the actual influence of the balancing parameter , we provide parameter search on it with MNIST dataset. In particular, We set to , and compare their results. Note that when , the QA becomes naive QA as the SP term is eliminated. With a strong SP constant, e.g. , the semantics of the chosen query is almost not allowed to be changed, hence result in extreme difficulty of attack.
As shown in Tab. 7, setting to could greatly boost the attacking effect, but consequently the ranks of will be drastically changed. In contrast, when is set to the excessive value for a perfectly stealth QA, the attack can still raise the rank of chosen candidate from to in QA+ with , or lower the rank of chosen candidate from to in QA- with . During these attacks, the ranks of are kept within despite of the extreme difficulty. It means the query semantics can be preserved. In practice, we empirically set the parameter as for QA+, or as for QA- for the balance between attack effectiveness and preserving query semantics.
|(CT) Cosine distance, Triplet loss|
|0||0.2, 33.6||6.3, 23.7||18.5, 26.5||29.6, 25.7||94.1, 89.4||93.2, 90.3||92.6, 90.9||92.3, 91.2|
|6.3, 3.6||11.2, 5.7||22.5, 7.7||32.1, 7.7||55.5, 35.6||52.4, 37.6||50.2, 39.3||49.4, 40.0|
|14.1, 0.6||20.8, 0.7||31.2, 0.7||38.1, 0.7||8.6, 1.6||6.6, 1.6||5.3, 1.5||4.8, 1.5|
|37.9, 0.1||42.6, 0.1||46.3, 0.1||47.8, 0.1||1.9, 0.1||1.4, 0.1||1.2, 0.1||1.1, 0.1|
Deep ranking models are vulnerable to adversarial perturbations that could intentionally change the ranking result. In this paper, we define and implement adversarial ranking attack that can compromise deep ranking models. We also propose an adversarial ranking defense that can significantly suppress embedding shift distance and moderately improve the ranking model robustness. Moreover, the transferability of our adversarial examples and the existence of universal adversarial perturbations for ranking attack illustrate the possibility of practical black-box attack and potential risk of realistic ranking applications.
In the potential of future work, we may explore (1) better ranking loss functions and defenses; (2) better black-box attacks and more transferable adversarial examples.
-  Anurag Arnab, Ondrej Miksik, and Philip HS Torr. On the robustness of semantic segmentation models to adversarial attacks. In CVPR, pages 888–897, 2018.
-  Anish Athalye and Nicholas Carlini. On the robustness of the cvpr 2018 white-box adversarial example defenses. arXiv preprint arXiv:1804.03286, 2018.
-  Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420, 2018.
-  Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. arXiv preprint arXiv:1707.07397, 2017.
Tu Bui, L Ribeiro, Moacir Ponti, and John Collomosse.
Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network.CVIU, 164:27–37, 2017.
-  Nicholas Carlini and David Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint arXiv:1607.04311, 2016.
-  Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017.
-  Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. JMLR, 11(Mar):1109–1135, 2010.
-  Jianbo Chen and Michael I Jordan. Boundary attack++: Query-efficient decision-based adversarial attack. arXiv preprint arXiv:1904.02144, 2019.
-  Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Ead: elastic-net attacks to deep neural networks via adversarial examples. In AAAI, 2018.
Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Polo Chau.
Shapeshifter: Robust physical adversarial attack on faster r-cnn
Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 52–68. Springer, 2018.
-  Francesco Croce and Matthias Hein. Sparse and imperceivable adversarial attacks. In ICCV, pages 4724–4732, 2019.
-  Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR, June 2018.
-  Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In CVPR, pages 4312–4321, 2019.
-  Yinpeng Dong, Hang Su, Baoyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu. Efficient decision-based black-box adversarial attacks on face recognition. In CVPR, pages 7714–7722, 2019.
-  Yinpeng Dong, Hang Su, Jun Zhu, and Fan Bao. Towards interpretable deep neural networks by leveraging adversarial examples. arXiv preprint arXiv:1708.05493, 2017.
-  Abhimanyu Dubey, Laurens van der Maaten, Zeki Yalniz, Yixuan Li, and Dhruv Mahajan. Defense against adversarial images using web-scale nearest-neighbor search. In CVPR, pages 8767–8776, 2019.
-  Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning models. arXiv preprint arXiv:1707.08945, 2017.
-  Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improved visual-semantic embeddings. arXiv preprint arXiv:1707.05612, 2(7):8, 2017.
-  Aditya Ganeshan and R Venkatesh Babu. Fda: Feature disruptive attack. In ICCV, pages 8069–8079, 2019.
-  Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
-  Divya Gopinath, Guy Katz, Corina S Pasareanu, and Clark Barrett. Deepsafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv preprint arXiv:1710.00486, 2017.
-  Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. Ranking robustness under adversarial document manipulations. In ACM SIGIR, pages 395–404. ACM, 2018.
-  Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00117, 2017.
-  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, June 2016.
-  Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defense: Ensembles of weak defenses are not strong. In 11th USENIX Workshop on Offensive Technologies (WOOT 17), 2017.
-  Xiangnan He, Zhankui He, Xiaoyu Du, and Tat-Seng Chua. Adversarial personalized ranking for recommendation. In ACM SIGIR, pages 355–364. ACM, 2018.
-  Qian Huang, Zeqi Gu, Isay Katsman, Horace He, Pian Pawakapan, Zhiqiu Lin, Serge Belongie, and Ser-Nam Lim. Intermediate level adversarial attack for enhanced transferability. arXiv preprint arXiv:1811.08458, 2018.
-  Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with a strong adversary. CoRR, abs/1511.03034, 2015.
-  Pierre Jacob, David Picard, Aymeric Histace, and Edouard Klein. Metric learning with horde: High-order regularizer for deep embeddings. In ICCV, pages 6539–6548, 2019.
-  Thorsten Joachims. Optimizing search engines using clickthrough data. In ACM SIGKDD, pages 133–142. ACM, 2002.
-  Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pages 97–117. Springer, 2017.
-  Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014.
-  Stepan Komkov and Aleksandr Petiushko. Advhat: Real-world adversarial attack on arcface face id system. arXiv preprint arXiv:1908.08705, 2019.
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages 1097–1105, 2012.
-  Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.
-  Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236, 2016.
-  Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
-  Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, pages 201–216, 2018.
-  Jie Li, Rongrong Ji, Hong Liu, Xiaopeng Hong, Yue Gao, and Qi Tian. Universal perturbation attack against image retrieval. In ICCV, pages 4899–4908, 2019.
-  Hong Liu, Rongrong Ji, Jie Li, Baochang Zhang, Yue Gao, Yongjian Wu, and Feiyue Huang. Universal adversarial perturbation via prior driven uncertainty approximation. In ICCV, pages 2941–2949, 2019.
-  Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval, 3(3):225–331, 2009.
-  Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In ECCV, pages 369–385, 2018.
-  Xuanqing Liu, Yao Li, Chongruo Wu, and Cho-Jui Hsieh. Adv-bnn: Improved adversarial defense through robust bayesian neural network. arXiv preprint arXiv:1810.01279, 2018.
-  Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:1611.02770, 2016.
-  Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Who’s afraid of adversarial queries?: The impact of image modifications on content-based image retrieval. In ICMR, pages 306–314. ACM, 2019.
-  Jiajun Lu, Theerasit Issaranon, and David Forsyth. Safetynet: Detecting and rejecting adversarial examples robustly. In ICCV, pages 446–454, 2017.
-  Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. No need to worry about adversarial examples in object detection in autonomous vehicles. arXiv preprint arXiv:1707.03501, 2017.
-  Yan Luo, Xavier Boix, Gemma Roig, Tomaso Poggio, and Qi Zhao. Foveation-based mechanisms alleviate adversarial examples. arXiv preprint arXiv:1511.06292, 2015.
-  Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
-  Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In ACM SIGSAC, pages 135–147. ACM, 2017.
-  Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint arXiv:1702.04267, 2017.
-  Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In CVPR, pages 1765–1773, 2017.
-  Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In CVPR, pages 2574–2582, 2016.
-  Chaithanya Kumar Mummadi, Thomas Brox, and Jan Hendrik Metzen. Defending against universal perturbations with shared adversarial training. In ICCV, pages 4928–4937, 2019.
-  Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Hierarchical multimodal lstm for dense visual-semantic embedding. In ICCV, pages 1881–1889, 2017.
-  Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, pages 4004–4012, 2016.
-  Nicolas Papernot and Patrick McDaniel. On the effectiveness of defensive distillation. arXiv preprint arXiv:1607.05113, 2016.
-  Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016.
-  Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506–519. ACM, 2017.
-  Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372–387. IEEE, 2016.
-  Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582–597. IEEE, 2016.
-  Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. None, 2017.
-  Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer. Deflecting adversarial attacks with pixel deflection. In CVPR, pages 8571–8580, 2018.
-  Yao Qin, Nicholas Carlini, Ian Goodfellow, Garrison Cottrell, and Colin Raffel. Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. arXiv preprint arXiv:1903.10346, 2019.
-  Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J Fleet. Adversarial manipulation of deep representations. arXiv preprint arXiv:1511.05122, 2015.
-  Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, pages 815–823, 2015.
-  Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In NeurIPS, pages 3179–3189, 2018.
-  Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust optimization. Neurocomputing, 307:195–204, 2018.
-  Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In ACM SIGSAC, pages 1528–1540. ACM, 2016.
-  Yucheng Shi, Siyu Wang, and Yahong Han. Curls & whey: Boosting black-box adversarial attacks. arXiv preprint arXiv:1904.01160, 2019.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai.
One pixel attack for fooling deep neural networks.
IEEE Transactions on Evolutionary Computation, 2019.
-  Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1–9, 2015.
-  Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818–2826, 2016.
-  Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
-  Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. Learning fine-grained image similarity with deep ranking. In CVPR, pages 1386–1393, 2014.
-  Jianyu Wang and Haichao Zhang. Bilateral adversarial training: Towards fast training of more robust models against adversarial attacks. In ICCV, pages 6629–6638, 2019.
-  Zhibo Wang, Siyan Zheng, Mengkai Song, Qian Wang, Alireza Rahimpour, and Hairong Qi. advpattern: Physical-world attacks on deep person re-identification via adversarially transformable patterns. In ICCV, pages 8341–8350, 2019.
-  Lei Wu, Zhanxing Zhu, Cheng Tai, et al. Understanding and enhancing the transferability of adversarial examples. arXiv preprint arXiv:1802.09707, 2018.
-  Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. arXiv preprint arXiv:1801.02612, 2018.
-  Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
-  Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In CVPR, pages 501–509, 2019.
-  Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In CVPR, pages 2730–2739, 2019.
-  Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE TNNLS, 2019.
-  Haichao Zhang and Jianyu Wang. Towards adversarially robust object detection. In ICCV, pages 421–430, 2019.
-  Yaoyao Zhong and Weihong Deng. Adversarial learning with margin-based triplet embedding regularization. In ICCV, pages 6549–6558, 2019.