Thanks to the widespread applications of deep neural networks[26, 21] in the learning-to-rank tasks [49, 39], deep ranking algorithms have witnessed significant progress, but unfortunately they have also inherited the long-standing adversarial vulnerabilities  of neural networks. Considering the “search by image” application for example, an imperceptible adversarial perturbation to the query image is sufficient to intentionally alter the ranking results of candidate images. Typically, such adversarial examples can be designed to cause the ranking model to “misrank” [32, 29] (, rank items incorrectly), or purposefully raise or lower the ranks of selected candidates .
Since “misranking” can be interpreted as deliberately lowering the ranks of well-matching
candidates, previous attacks on ranking models unanimously focus on
changing the absolute ranks of a set of candidates, while neglecting the manipulation of relative order among them.
However, an altered relative order can be disruptive in some applications,
such as impacting sales on e-commerce platforms powered by content-based
can be disruptive in some applications, such as impacting sales on e-commerce platforms powered by content-based image retrieval, where potential customers attempt to find merchandise via image search.
As shown in Fig. 1, an attacker may want to adversarially perturb the query image and thus change the relative order among products A, B, C, D, and E into in the search-by-image result. The sales of a product closely correlates to its Click-Through Rate (CTR), while the CTR can be significantly influenced by its ranking position [10, 37] (which also decides the product pagination on the client side). Hence, subtle changes in the relative order of searches can be sufficient to alter CTR and impact the actual and relative sales among A to E.
Such vulnerability in a commercial platform may be exploited in a malfeasant business competition among the top-ranked products, , via promoting fraudulent web pages containing adversarial example product images generated in advance. Specifically, the attack of changing relative order does not aim to incur significant changes in the absolute ranks of the selected candidates (, moving from list bottom to top), but it intentionally changes the relative order of them subtly without introducing conspicuous abnormality. Whereas such goal cannot be implemented by absolute rank attacks such as , the relative order vulnerability may justify and motivate a more robust and fair ranking model.
Specifically, we propose the Order Attack (OA), a new adversarial attack problem in deep ranking. Given a query image , a set of selected candidates
, and a predefined permutation vector, Order Attack aims to find an imperceptible perturbation ( and ), so that as the adversarial query can convert the relative order of the selected candidates into . For example, a successful OA with will result in , as shown in Fig. 1.
To implement OA, we first assume the white-box threat model (, the ranking model details, incl. the gradient are accessible to the attacker). Recall that a conventional deep ranking model [49, 56, 39, 25]
maps the query and candidates onto a common embedding space, and determines the ranking list according to the pairwise similarity between the query and these candidates. Thus, OA can be formulated as the optimization of a triplet-style loss function based on the inequality chain representing the desiredrelative order, which simultaneously adjusts the similarity scores between the query and the selected candidates. Additionally, a semantics-preserving penalty term  is also included to limit conspicuous changes in ranking positions. Finally, the overall loss function can be optimized with gradient methods such as PGD  to find the adversarial example.
However, in a real-world black-box attack scenario, practical limitations (, gradient inaccessibility) invalidate the proposed method. In accommodate them and make OA practical, we propose a “Short-range Ranking Correlation” (SRC) metric to measure the alignment between a desired permutation and the actual ranking result returned to clients by counting concordant and discordant pairs, as a practical approximation of the proposed triplet-style white-box loss. Though non-differentiable, SRC can be used as a surrogate objective for black-box OA and optimized by an appropriate black-box optimizer, to achieve similar effect as the white-box OA. SRC can also be used as a performance metric for the white-box method, as it gracefully degenerates into Kendall’s ranking correlation  in white-box scenario.
To validate the white-box and black-box OA, we conduct comprehensive experiments on Fashion-MNIST and Stanford-Online-Product datasets. To illustrate the viability of the black-box OA in practice, we also showcase successful attacks against the “JD SnapShop” , a major retailing e-commerce platform based on content-based image retrieval. Extensive quantitative and qualitative evaluations illustrate the effectiveness of the proposed OA, and reveals a new type of ranking model vulnerability.
To the best of our knowledge, this is the first work that tampers the relative order in deep ranking. We believe our contributions include, (1) the formulation of Order Attack (OA), a new adversarial attack that covertly alters the relative order among selected candidates; (2) a triplet-style loss for ideal-case white-box OA; (3) a Short-range Ranking Correlation (SRC) metric as a surrogate objective approximating the triplet-style loss for practical black-box OA; (4) extensive evaluations of OA including a successful demonstration on a major online retailing e-commerce platform.
2 Related Works
Adversarial Attack. Szegedy 
find the DNN classifiers susceptible to imperceptible adversarial perturbations, which leads to misclassification. This attracted research interest among the community, as shown by subsequent works on adversarial attacks and defenses[14, 13, 46]. In particular, the attacks can be categorized into several groups: (1) White-box attack, which assumes the model details including the gradient are fully accessible [19, 27, 33, 34, 7, 2, 3, 12]. Of these methods, PGD  is the most popular one; (2) Transfer-based attack, which is based on the transferability of adversarial examples [15, 52, 16]
. Such attack typically transfers adversarial examples found from a locally trained substitute model onto another model. (3) Score-based attack, which only depends on the soft classification labels, , the logit values[22, 47, 31, 9, 1]. Notably,  proposes a black-box threat model for classification that is similar to our black-box ranking threat model; (4) Decision-based attack, a type of attack that requires the least amount of information from the model, , the hard label (one-hot vector) [6, 8, 11, 17, 41, 28]. All these extensive adversarial attacks unanimously focus on classification, which means they are not directly suitable for ranking scenarios.
Adversarial Ranking. In applications such as web retrieval, documents may be promoted in rankings by intentional manipulation . Likewise, the existence of aforementioned works inspired attacks against deep ranking, but it is still insufficiently explored. In light of distinct purposes, ranking attacks can be divided into absolute rank attacks and relative order attacks. Most absolute rank attacks attempt to induce random “misranking” [45, 29, 32, 53, 54, 48, 55, 5, 18, 30]. Some other attacks against ranking model aim to incur purposeful changes in absolute rank, , to raise or lower the ranks of specific candidates [4, 57]. On the contrary, the relative order attacks remains under-explored. And this is the first work that tampers the relative order in deep ranking.
As shown in Fig. 2, relative order is “orthogonal” to absolute rank. Let “” denote any uninterested candidate. Suppose the selected candidates and permutation are [A, B, C] and , respectively. The exemplar absolute rank loss  is ignorant to the difference in relative order comparing I and II, or IV and III. The relative order loss proposed in this paper is ignorant to the difference in absolute rank comparing I and IV, or II and III. Although focusing on different aspects of ranking, the two types of loss functions can be combined and jointly optimized.
3 Adversarial Order Attack
Typically, a deep ranking model is built upon deep metric learning [49, 39, 56, 25, 38]. Given a query and a set of candidates selected from database (), a deep ranking model evaluates the distance between every pair of query and candidate, , where . Thus, by comparing all the pairwise distances , the whole candidate set can be ranked with respect to the given query. For instance, the model outputs the ranking list if it determines .
Based on these, Order Attack (OA) aims to find an imperceptible perturbation ( and ), so that as the adversarial query can convert the relative order of the selected candidates into , where is a permutation vector predefined by the attacker. In particular, we assume that the attacker is inclined to select the candidate set from the top- ranked candidates , as the ranking lists returned to the clients are usually “truncated” (, only the top-ranked candidates will be shown). We call the length () of the “truncated” list as a “visible range”. The white-box and black-box OA will be discussed in Sec.3.1 and Sec.3.2 respectively. For sake of brevity, we let .
3.1 Triplet-style Loss Function for White-Box OA
During the training process, a typical deep ranking model involves a triplet (anchor , positive , negative ) in each iteration. In order to rank ahead of , the model is penalized when does not hold. This inequality can be reformulated exploiting the form of a hinge loss, resulting in the triplet ranking loss function  , where , and denotes the margin hyper-parameter.
Inspired by this, to implement the OA, we decompose the inequality chain prescribed by the predefined permutation vector , namely into inequalities, , , . Reformulation of these inequalities into the hinge loss form leads to the relative order loss,
Subsequently, given this loss function, the OA can be cast as a constrained optimization problem,
which can be solved with first-order-gradient-based methods such as Projected Gradient Descent (PGD) , ,
where is the PGD step size, and is initialized as a zero vector. PGD stops at a predefined maximum iteration , and the final is the desired adversarial perturbation.
It is worth noting that query image semantics can be drastically changed even with a very slight perturbation . As a result, candidates are prone to be excluded from the topmost part of ranking, and become invisible when the ranking result is “truncated”. To mitigate such side effect, we follow  and introduce a semantics-preserving term to keep within the topmost part of the ranking by raising their absolute ranks, , to keep ranked ahead of other candidates. Finally, the relative order loss term and the absolute rank loss term are combined to form the complete white-box OA loss ,
where is a positive constant balancing factor between the relative order and absolute rank goals.
Despite the formulation of Eq. (4), an ideal that fully satisfy the desired relative order of does not necessarily exist. Consider a Euclidean embedding space, where candidates lie consecutively on a straight line. It is impossible to find a query embedding that leads to . That indicates the compatibility between the specified relative order and the factual geometric relations of the candidate embeddings affects the performance upper-bound of OA. In cases like this, our algorithm can still find an inexact solution that satisfies as many inequalities as possible to approximate the specified relative order. In light of this, Kendall’s ranking correlation  between the specified relative order and the real ranking order appears to be a more reasonable performance metric than the success rate for OA.
3.2 Short-range Ranking Correlation
A concrete triplet-style implementation of OA is present in Sec. 3.1, but it is infeasible in a real-world attack scenario. In particular, multiple challenges are present for black-box OA, including (1) Gradient inaccessibility. The gradient of the loss w.r.t the input is inaccessible, as the network architecture and parameters are unknown; (2) Lack of similarity (or distance) scores. Exact similarity scores rarely appear in the truncated ranking results; (3) Truncated ranking results. In practice, a ranking system only presents the top- ranking results to the clients; (4) Limited query budget. Repeated, intensive queries within a short time frame may be identified as threats, , Denial of Service (DoS) attack. Therefore, it is preferable to construct adversarial examples within a reasonable amount of queries. These restrictions collectively invalidate the triplet-style method.
To address these challenges, we present the “Short-range Ranking Correlation” (SRC; denoted as ) metric, a practical approximation of the (Eq. 4) as a surrogate loss function for black-box OA.
Specifically, to calculate given , and the top- retrieved candidates w.r.t query , we first initialize a
-shaped zero matrix, and permute into . Assuming () exist in , we define as a concordant pair as long as and are simultaneously greater or smaller than and , respectively, where denotes the integer rank value of in , , . Otherwise, is defined as a discordant pair. Namely a concordant matches a part of the specified permutation, and could result in a zero loss term in Eq. 1, while a discordant pair does not match, and could result in a positive loss term in Eq. 1. Thus, in order to approximate the relative order loss (Eq. 1), a concordant pair and a discordant pair will be assigned a score of (as reward) and (as penalty), respectively. Apart from that, when or does not exist in , will be directly assigned with an “out-of-range” penalty , which approximates the semantics-preserving term in Eq. 4. Finally, after comparing the ordinal relationships of every pair of candidates and assigning penalty values in , the average score of the lower triangular of excluding the diagonal is the , as summarized in Algo. 1.
The value of reflects the real ranking order’s alignment to the order specified by , where a semantics-preserving penalty is spontaneously incorporated. When the specified order is fully satisfied, , any pair of and is concordant, and none of the elements in disappear from , will be . In contrast, when every candidate pair is discordant or absent from the top- result , will be . Overall, percent of the candidate pairs are concordant, and the rest are discordant or “out-of-range”.
Maximization of leads to the best alignment to the specified permutation, as discordant pairs will be turned into concordant pairs, while maintaining the presence of within the top- visible range. Thus, although non-differentiable, the metric can be used as a practical surrogate objective for black-box OA, , , which achieves a very similar effect to the white-box OA.
Particularly, when exists in , degenerates into Kendall’s  between and the permutation of in . Namely, also degenerates gracefully to in the white-box scenario because the whole ranking is visible. However, is inapplicable on truncated ranking results.
does not rely on any gradient or any similarity score, and can adapt to truncated ranking results. When optimized with an efficient black-box optimizer (, NES ), the limited query budget can also be efficiently leveraged. Since all the black-box challenges listed at the beginning of this section are handled, it is practical to perform black-box OA by optimizing in real-world applications.
|Stanford Online Products|
|Fashion-MNIST , ,|
|Stanford Online Products , ,|
To evaluate the white-box and black-box OA, we conduct experiments on the Fashion-MNIST  and the Stanford-Online-Products (SOP)  datasets which comprise images of retail commodity. Firstly, we train a CNN with -convolution--fully-connected network on Fashion-MNIST, and a ResNet-18  without the last fully-connected layer on SOP following  that focuses on the absolute rank attack. Then we perform OA with the corresponding test set as the candidate database . Additionally, we also qualitatively evaluate black-box OA on “JD SnapShop”  to further illustrate its efficacy. In our experiments, the value of rank function starts from , , the -th ranked candidate has the rank value of .
Selection of and . As discussed, we assume that the attacker is inclined to select the candidate set from the top- ranked candidates given the visible range limit. For simplicity, we only investigate the -OA, , OA with the top--within-top- () candidates selected as as . It is representative because an OA problem with some candidates randomly selected from the top- results as is a sub-problem of -OA. Namely, our attack will be effective for any selection of as long as the -OA is effective. For white-box OA, we conduct experiments with , and . For black-box attack, we conduct experiments with , and . Besides, a random permutation vector is specified for each individual query.
Evaluation Metric. Since is equivalent to when , we use as the performance metric for both white-box and black-box OA. Specifically, in each experiment, we conduct times of OA attack. In each attack, we randomly draw a sample from as the query . In the end, we report the average over the trials. Also, when , we additionally calculate the mean rank of the candidate set (demoted as “mR”, which equals ), and report the average mean rank over the attacks. Larger value and smaller mR value are preferable.
Parameter Settings. We set the perturbation budget as following  for both white-box and black-box attacks. The query budget is set to . For white-box OA, the PGD step size is set to , the PGD step number to . The balancing parameter is set as and for Fashion-MNIST and SOP respectively. The learning rates of black-box optimizer NES  and SPSA  are both set to . See supplementary for more details of the black-box optimizers.
Search Space Dimension Reduction. As a widely adopted trick, dimension reduction of the adversarial perturbation search space has been reported effective in [14, 9, 41, 28]. Likewise, we empirically reduce the space to for black-box OA on Stanford-Online-Products dataset and “JD SnapShop”. In fact, a significant performance drop in can be observed without this trick.
4.1 White-Box Order Attack Experiments
The first batch of the experiments is carried out on the Fashion-MNIST dataset, as shown in the upper part of Tab. 1. With the original query image (), the expected performance of -OA is , and the retains their original ranks as the mR equals . With a adversarial perturbation budget, our OA achieves , which means on average 222Solution of system ; . of the inequalities reflecting the specified permutations are satisfied by the adversarial examples. Meanwhile, the mR changes from to , due to adversarial perturbation can move the query embedding off its original position  while seeking for a higher . Nevertheless, the mR value of indicates that the are still kept visible in the topmost part of the ranking result by the loss term . With larger perturbation budget , the metric increases accordingly, , reaches when , which means nearly of the inequalities are satisfied. Likewise, the experimental results on SOP are available in the lower part of Tab. 1, which also demonstrate the effectiveness of our method under different settings.
Besides, we note that different balancing parameter for leads to distinct results, as shown in Tab. 2. We conduct -OA with with different values ranging from to on both datasets. Evidently, a larger leads to a better (smaller) mR value, but meanwhile a worse as the weighted term dominates the total loss. There is a trade-off between the and mR, which is effectively controlled by the tunable constant parameter . Hence, we empirically set as and for Fashion-MNIST and SOP respectively, in order to keep the mR of most experiments in Tab. 1 below a sensible value, , .
Additionally, Tab. 1 reveals that the mR trends w.r.t. on the two datasets differ. To investigate this counter-intuitive phenomenon, we plot loss curves in Fig. 3. In the , case for Fashion-MNIST, the total loss decreases but the surges at the beginning and then plateaus. After increasing to , the rises more smoothly. The curve eventually decreases at , along with a small mR and a notable penalty on as a result. Besides, the “sawtooth-shaped” curves also indicate that the term is optimized while sacrificing the mR as a side-effect at the even steps, while the optimizer turns to optimize
at the odd steps due to the semantics-preserving penalty, causing a slight increase in. These figures indicate that optimizing without sacrificing is difficult on Fashion-MNIST. Moreover, perturbation budget is irrelevant. Comparing the first and the fourth sub-figures, we find a larger budget () unhelpful in reducing optimization difficulty as the curve still soars and plateaus. Based on these cues, we speculate that the different curve patterns of mR stem from the optimization difficulty due to a fixed PGD step size that cannot be smaller333 Every element of perturbation should be an integral multiple of ., and different dataset properties.
The intra-class variance of the simple Fashion-MNIST dataset is smaller than that of SOP, which means sample embeddings of the same class are densely clustered. As each update can change thedrastically, it is difficult to adjust the query embedding position in a dense area with a fixed PGD step for a higher without significantly disorganizing the ranking list (hence a lower mR). In contrast, a larger intra-class variance of the SOP dataset makes easier to be maintained, as shown in the 2nd row of Fig. 3.
|None||0.0, 2.0||0.0, 2.0||0.0, 2.0||0.0, 2.0||0.0, 4.5||0.0, 4.5||0.0, 4.5||0.0, 4.5||0.0, 12.0||0.0, 12.0||0.0, 12.0||0.0, 12.0|
|Rand||0.211, 2.1||0.309, 2.3||0.425, 3.0||0.508, 7.7||0.172, 4.6||0.242, 5.0||0.322, 6.4||0.392, 12.7||0.084, 12.3||0.123, 13.1||0.173, 15.8||0.218, 25.8|
|Beta||0.241, 2.1||0.360, 2.6||0.478, 4.6||0.580, 19.3||0.210, 4.8||0.323, 5.7||0.430, 9.6||0.510, 30.3||0.102, 12.4||0.163, 13.8||0.237, 19.7||0.291, 42.7|
|PSO||0.265, 2.1||0.381, 2.3||0.477, 4.4||0.580, 21.1||0.239, 4.8||0.337, 5.7||0.424, 9.7||0.484, 34.0||0.131, 12.7||0.190, 14.6||0.248, 21.7||0.286, 54.2|
|NES||0.297, 2.3||0.416, 3.1||0.520, 8.7||0.630, 46.3||0.261, 5.0||0.377, 6.6||0.473, 14.3||0.518, 55.6||0.142, 13.0||0.217, 15.9||0.286, 28.3||0.312, 74.3|
|SPSA||0.300, 2.3||0.407, 3.2||0.465, 7.1||0.492, 16.3||0.249, 5.0||0.400, 6.6||0.507, 12.8||0.558, 27.5||0.135, 12.9||0.236, 16.3||0.319, 27.1||0.363, 46.4|
4.2 Black-Box Order Attack Experiments
To simulate a real-world attack scenario, we convert the ranking models trained for Sec. 4.1 into black-box versions, which are subject to limitations discussed in Sec. 3.2. Black-box OA experiments are conducted on these models.
To optimize the surrogate loss , we adopt and compare several black-box
(1) Random Search (Rand), which independently samples
every dimension of the perturbation from uniform distribution
, a classic meta-heuristic
black-box optimizer with an extra step that clips the adversarial example to using estimated gradient;
, we adopt and compare several black-box optimizers: (1) Random Search (Rand), which independently samples every dimension of the perturbation from uniform distribution, then clips adversarial example to ; (2) Beta-Attack (Beta), a modification of -Attack 40]
, a classic meta-heuristic black-box optimizer with an extra step that clips the adversarial example to; (4) NES [22, 50], which performs PGD 
using estimated gradient; (5) SPSA[47, 43], which can be interpreted as NES using a different sampling distribution.
We first investigate the black-box -OA, as shown in the upper part () of Tab. 3 and Tab. 5. In these cases, does not pose any semantics-preserving penalty since , which is similar to white-box attack with . With the Rand optimizer, can be optimized to with on Fashion-MNIST. As increases, we obtain better results, and larger mR values as an expected side-effect.
|None||0.0, 2.0||0.0, 2.0||0.0, 2.0||0.0, 2.0||0.0, 4.5||0.0, 4.5||0.0, 4.5||0.0, 4.5||0.0, 12.0||0.0, 12.0||0.0, 12.0||0.0, 12.0|
|Stanford Online Product|
|Rand||0.187, 2.6||0.229, 8.5||0.253, 85.8||0.291, 649.7||0.167, 5.6||0.197, 13.2||0.208, 92.6||0.222, 716.4||0.093, 14.1||0.110, 27.6||0.125, 146.7||0.134, 903.7|
|Beta||0.192, 3.3||0.239, 15.3||0.265, 176.7||0.300, 1257.7||0.158, 6.2||0.186, 19.9||0.207, 139.0||0.219, 992.5||0.099, 15.5||0.119, 37.1||0.119, 206.5||0.132, 1208.5|
|PSO||0.122, 2.1||0.170, 3.0||0.208, 13.3||0.259, 121.4||0.135, 4.8||0.177, 6.5||0.206, 22.8||0.222, 166.5||0.104, 12.7||0.122, 16.7||0.137, 49.5||0.140, 264.2|
|NES||0.254, 3.4||0.283, 15.6||0.325, 163.0||0.368, 1278.7||0.312, 7.2||0.351, 26.3||0.339, 227.1||0.332, 1486.7||0.242, 18.0||0.259, 51.5||0.250, 324.1||0.225, 1790.8|
|SPSA||0.237, 3.5||0.284, 11.9||0.293, 75.2||0.318, 245.1||0.241, 7.8||0.325, 22.2||0.362, 112.7||0.383, 389.0||0.155, 18.1||0.229, 41.9||0.286, 185.6||0.306, 557.8|
|Stanford Online Product|
|Stanford Online Product|
Since all the queries of Rand are independent, one intuitive way
to achieve better performance is to leverage the historical query results to
adjust the search distribution.
Following this idea, we modify -Attack  into
Beta-Attack, replacing the Gaussian distributions into Beta distributions.
It generates perturbations from a Beta distribution per dimension independently.
And all the distribution parameters are initialized as ,
where Beta distribution degenerates into Uniform distribution (Rand).
During optimization, it modifies its probability density functions
into Beta-Attack, replacing the Gaussian distributions into Beta distributions. It generates perturbations from a Beta distribution per dimension independently. And all the distribution parameters are initialized as
, where Beta distribution degenerates into Uniform distribution (Rand). During optimization, it modifies its probability density functions according toresults, in order to increase the expectation of the next adversarial perturbation drawn from these distributions. Results in Tab. 3 suggest an evident advantage of Beta against Rand, but it also shows that the Beta distributions are insufficient for modeling the adversarial perturbations for OA.
According to all the -OA results in Tab. 3 and Tab. 5, NES and SPSA outperform Rand, Beta and PSO. This means black-box optimizers based on estimated gradient are still the most effective and efficient ones for -OA. The metrics of all algorithms unanimously increase with a larger , but the side effect of worse (larger) mR is also notable. Predictably, when or , the algorithms may confront a great penalty due to the absence of the selected candidates from the top- visible range.
Further results of -OA and -OA confirm our speculation, as shown in the middle () and bottom () parts of Tab. 3 and Tab. 5. With and a fixed , algorithms that result in a small mR (especially for those with ) also perform comparably as in -OA. Conversely, algorithms that lead to a large mR in -OA are greatly penalized in -OA. The results also manifest a special characteristic of OA that differs from adversarial attacks against classification, that peaks at a certain small , and does not positively correlate with . This is rather apparent in difficult settings such as -OA on the SOP dataset. In brief, the optimizers based on estimated gradients still perform the best in black-box -OA and -OA, and a very large perturbation budget is not necessary.
All these experiments demonstrate the effectiveness of optimizing the surrogate objective to conduct black-box OA. As far as we know, optimizers based on gradient estimation, such as NES and SPSA are the most reliable choices for practical OA. Next, we will adopt SPSA and perform practical OA experiments in a real-world ranking application.
Time Complexity. Although the complexity of Algo. 1 is , the actual run time of our Rust444Rust Programming language. See https://www.rust-lang.org/.-based SRC implementation is short, as measured with Python cProfile with a Xeon 5118 CPU and a GTX1080Ti GPU.
As shown in Tab. 4, for instance, SRC calculation merely consumes seconds on average across the five algorithms for generating an adversarial example on SOP with
. The overall time consumption is dominated by sorting and PyTorch model inference. Predictably, in real-world attack scenarios, It is highly likely that the time complexity bottleneck stems from other factors irrelevant to our method, such as network delays and bandwidth, or the query-per-second limit set by the service provider.
4.3 Practical Black-Box Order Attack Experiments
The “JD SnapShop”  is an e-commerce platform based on content-based image retrieval . Clients can upload query merchandise images via an HTTP-protocol-based API, then the API returns the top- similar products. This black-box model exactly matches the setting of -OA. As the API specifies a file size limit ( 3.75MB), and a minimum image resolution (), we use the standard size for the query. We merely provide limited evaluations of black-box OA because the API poses an hard limit of queries per day per user.
As noted in Sec. 4.2, the peaks with a certain small . Likewise, an empirical search for suggests that and are the most suitable choices for OA against “JD SnapShop”. Any larger value easily leads to the disappearance of from . This is meanwhile a preferable characteristic, as smaller perturbations are less perceptible.
As shown in Fig. 1, we select the top- candidates as , and specify . Namely, the expected relative order among is . By maximizing the with SPSA and , we successfully convert the relative order to the specified one using times of queries. This shows that performing OA by optimizing our proposed surrogate loss with a black-box optimizer is practical. Some limited quantitative results are presented in Tab. 6, where is further limited to in order to gather more data, while a random permutation and a random query image from SOP is used for each of the times of attacks.
In practice, the adversarial example may be slightly changed by transformations such as the lossy JPEG/PNG compression, and resizing (as a pre-processing step), which eventually leads to changes on the surface. But a black-box optimizer should be robust to such influences.
Deep ranking systems inherited the adversarial vulnerabilities of deep neural networks. In this paper, the Order Attack is proposed to tamper the relative order among selected candidates. Multiple experimental evaluations of the white-box and black-box Order Attack illustrate their effectiveness, as well as the deep ranking systems’ vulnerability in practice. Ranking robustness and fairness with respect to Order Attack may be the next valuable direction to explore.
-  (2020) Square attack: a query-efficient black-box adversarial attack via random search. In ECCV, pp. 484–501. Cited by: §2.
Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples.
International Conference on Machine Learning, pp. 274–283. Cited by: §2.
-  (2018) Synthesizing robust adversarial examples. In ICML, pp. 284–293. Cited by: §2.
-  (2019) Metric attack and defense for person re-identification. arXiv preprint arXiv:1901.10650. Cited by: §2.
-  (2020-06) Vulnerability of person re-identification models to metric adversarial attacks. In CVPR workshop, Cited by: §2.
-  (2018) Decision-based adversarial attacks: reliable attacks against black-box machine learning models. In ICLR, External Links: Cited by: §2.
-  (2017) Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57. Cited by: §2.
-  (2020) HopSkipJumpAttack: a query-efficient decision-based adversarial attack. In 2020 IEEE Symposium on Security and Privacy (SP), Cited by: §2.
Zoo: zeroth order optimization based black-box attacks to deep neural networks without training substitute models.
Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15–26. Cited by: §2, §4.
-  (2012) Position-normalized click prediction in search advertising. In SIGKDD, pp. 795–803. Cited by: §1.
-  (2019) Query-efficient hard-label black-box attack: an optimization-based approach. In ICLR, Cited by: §2.
-  (2020) Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, pp. 2206–2216. Cited by: §2.
-  (2020) Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, Cited by: §2.
-  (2020-06) Benchmarking adversarial robustness on image classification. In CVPR, Cited by: §2, §4.
-  (2018-06) Boosting adversarial attacks with momentum. In CVPR, Cited by: §2.
-  (2019) Evading defenses to transferable adversarial examples by translation-invariant attacks. In CVPR, pp. 4312–4321. Cited by: §2.
Efficient decision-based black-box adversarial attacks on face recognition. CVPR, pp. 7706–7714. Cited by: §2.
-  (2020) Adversarial attack on deep product quantization network for image retrieval. In AAAI, Vol. 34, pp. 10786–10793. Cited by: §2.
-  (2015) Explaining and harnessing adversarial examples. ICLR. Cited by: §2.
-  (2018) Ranking robustness under adversarial document manipulations. In ACM SIGIR, pp. 395–404. Cited by: §2.
-  (2016-06) Deep residual learning for image recognition. In CVPR, Cited by: §1, §4.
-  (2018) Black-box adversarial attacks with limited queries and information. In ICML, pp. 2137–2146. Cited by: §2, §3.2, §4.2, §4.
-  (Website) External Links: Cited by: §A.1, §1, §4.3, §4.
-  (1945) The treatment of ties in ranking problems. Biometrika 33 (3), pp. 239–251. Cited by: §1, §3.1, §3.2.
-  (2019) Deep metric learning beyond binary supervision. In CVPR, pp. 2288–2297. Cited by: §1, §3.
-  (2012) Imagenet classification with deep convolutional neural networks. In NeurIPS, pp. 1097–1105. Cited by: §1.
-  (2017) Adversarial examples in the physical world. ICLR workshop. Cited by: §A.2, §2, §4.
-  (2020) QEBA: query-efficient boundary-based blackbox attack. In CVPR, pp. 1221–1230. Cited by: §2, §4.
-  (2019) Universal perturbation attack against image retrieval. In ICCV, pp. 4899–4908. Cited by: §1, §2.
-  (2021) QAIR: practical query-efficient black-box attacks for image retrieval. CVPR. Cited by: §2.
-  (2019) Nattack: learning the distributions of adversarial examples for an improved black-box attack on deep neural networks. In ICML, pp. 3866–3876. Cited by: §2, §4.2, §4.2.
-  (2019) Who’s afraid of adversarial queries?: the impact of image modifications on content-based image retrieval. In ICMR, pp. 306–314. Cited by: §1, §2.
Towards deep learning models resistant to adversarial attacks. ICLR. Cited by: §1, §2, §3.1, §4.2.
-  (2016) Deepfool: a simple and accurate method to fool deep neural networks. In CVPR, pp. 2574–2582. Cited by: §2.
-  (2016) Deep metric learning via lifted structured feature embedding. In CVPR, pp. 4004–4012. Cited by: §4.
-  (2017) Automatic differentiation in pytorch. None. Cited by: §4.2.
-  (2011) Stochastic variability in sponsored search auctions: observations and models. In the 12th ACM conference on Electronic commerce, pp. 61–70. Cited by: §1.
-  (2020) Revisiting training strategies and generalization performance in deep metric learning. In ICML, pp. 8242–8252. Cited by: §3.
-  (2015) Facenet: a unified embedding for face recognition and clustering. In CVPR, pp. 815–823. Cited by: §1, §1, §3.1, §3.
A modified particle swarm optimizer.
1998 IEEE international conference on evolutionary computation proceedings. IEEE world congress on computational intelligence (Cat. No. 98TH8360), pp. 69–73. Cited by: §4.2.
-  (2020) Hard label black-box adversarial attacks in low query budget regimes. arXiv preprint arXiv:2007.07210. Cited by: §2, §4.
-  (2000) Content-based image retrieval at the end of the early years. IEEE TPAMI 22 (12), pp. 1349–1380. Cited by: §1, §4.3.
-  (1992) Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control 37 (3), pp. 332–341. Cited by: §4.2.
-  (2014) Intriguing properties of neural networks. ICLR. Cited by: §1, §2.
-  (2019) Targeted mismatch adversarial attack: query with a flower to retrieve the tower. In ICCV, pp. 5037–5046. Cited by: §2.
-  (2020) On adaptive attacks to adversarial example defenses. In NeurIPS, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33, pp. 1633–1645. External Links: Cited by: §2.
-  (2018) Adversarial risk and the dangers of evaluating against weak attacks. In ICML, pp. 5025–5034. Cited by: §2, §4.2, §4.
-  (2020-06) Transferable, controllable, and inconspicuous adversarial attacks on person re-identification with deep mis-ranking. In CVPR, Cited by: §2.
-  (2014) Learning fine-grained image similarity with deep ranking. In CVPR, pp. 1386–1393. Cited by: §1, §1, §3.
-  (2008) Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 3381–3387. Cited by: §4.2.
-  (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747. Cited by: §4.
-  (2019) Improving transferability of adversarial examples with input diversity. In CVPR, pp. 2730–2739. Cited by: §2.
-  (2018) Adversarial examples for hamming space search. IEEE transactions on cybernetics. Cited by: §2.
Unsupervised adversarial attacks on deep feature-based retrieval with gan. arXiv preprint arXiv:1907.05793. Cited by: §2.
-  (2018) Open set adversarial examples. arXiv preprint arXiv:1809.02681. Cited by: §2.
-  (2020) Ladder loss for coherent visual-semantic embedding. In AAAI, Vol. 34, pp. 13050–13057. Cited by: §1, §3.
-  (2020) Adversarial ranking attack and defense. In ECCV 2020, pp. 781–799. External Links: Cited by: §C.2, §1, §1, §1, §2, §2, §3.1, §4.1, §4.
Appendix A Order Attack against “JD SnapShop” API
a.1 More Technical Details
Since the selected candidates are [A, B, C, D, E], and the permutation vector is , the expected relative order among is , , A E D C B.
Each product corresponds to multiple images. Only the default product images specified by the service provider are displayed in the figure.
The API in fact provides a similarity score for every candidate, which indeed can be leveraged for, , better gradient estimation. However, the other practical ranking applications may not necessarily provide these similarity scores. Hence, we simply ignore such discriminative information to deliberately increase the difficulty of attack. The concrete ways to take advantage of known similarity scores are left for future works.
In Fig. 1, the original similarity scores of candidates from A to E are [ , , , , ]. With our adversarial query, the scores of A to E are changed into [ , , , , ].
The API supports a “topK” argument, which enables the clients to change the visible range . We leave it as the recommended default value .
From Fig. 1, we note some visually duplicated images among the candidates. For instance, there are many other candidates similar to candidate E, due to reasons such as different sellers reusing the same product image. These images are not adjacent to each other in the ranking list, since the platform assigns them with different similarity scores. For instance, the -th and -th candidates in the first row of Fig. 1 are assigned with similarity scores , while the -nd, -th, and -th items in the second row are assigned with similarity scores . Whether the calculation of similarity scores involves multiple cues (, by aggregating the similarity scores of multiple product images, or using information from other modalities such as text), or even engineering tricks are unknown and beyond the scope of discussion.
Users (with the free plan) are merely allowed to perform times of queries per day as a hard limit.
The API documentation can be found at
The SKU ID atop every candidate image can be used to browse the real product webpages on the “JingDong” shopping platform. The URL format is
For example, the webpage for the product with SKU ID 72210617020 is located at
Note, due to irresistible reasons such as sellers withdrawing their product webpage and the ranking algorithm/database updates, some of the links may become invalid during the review process, and the ranking result for the same query may change.
It consumes 200 times of queries to find the adversarial example presented in Fig. 1. This process takes around 170 seconds, mainly due our limited network condition.
a.2 Empirical Search for on the API
According to the white-box and black-box OA experiments in the manuscript, we learn that the perturbation budget affects the OA performance. And it meanwhile controls the adversarial perturbation imperceptibility. Thus, we search for a proper for OA against “JD SnapShop”. Due to the limitation that only times of queries per day are allowed for each user, we merely present some empirical and qualitative observation for different settings.
As shown in Tab. 7, we test the “JD SnapShop” API with adversarial query images using the Rand algorithm with different values. We conduct times of attack per value. Our qualitative observation is summarized in the table.
|Empirical Qualitative Observation|
|Almost any disappear from the top- candidates, resulting in .|
|In most cases only selected candidate remains within the top- result.|
|In most cases only selected candidates remain within the top- result.|
|Nearly all remain in top- with significant order change. (suitable for with )|
|Top- seldom moves. The rest part is slightly changed. (suitable for with )|
From the table, we find that and are the most suitable choices for the -OA against “JD SnapShop”. This is meanwhile very preferable since such slight perturbations are imperceptible to human. As shown in Fig. 4, the adversarial perturbation can hardly be perceived, but the largest perturbation (, ) used by  is relatively visible.
a.3 More Showcases of OA against the API
In showcase #2 (Fig.LABEL:fig:advorder2), the original similarity scores of the top- candidates are [ , , , , ]. They are changed into [ , , , , ] with the perturbation.
In showcase #3 (Fig.LABEL:fig:advorder3), the original top- candidate similarity [ , , , , ] is changed into [ , , , , ].
Fig. LABEL:fig:miss1 shows two examples where one selected candidate disappear from top- result with our adversarial query. In the “white shoes” case, the top- candidate similarity scores are changed from [ , , , , ] to [ N/A, , , , ]. In the “vase” case, the top- candidate similarity scores are changed from [ , , , , ] to [ N/A, , , , ].
Fig. LABEL:fig:longtail shows some long-tail queries on which our OA will not take effect, because a large portion of the top-ranked candidates have completely the same similarity scores. In the 1st row, , results for a “Machine Learning” textbook query, the similarity scores of the -th to -th candidates are . The score of the -th to -th candidates are . That of the -th to -th are the same . In the “Deep Learning” textbook case (2nd row), the similarity scores of the -th to -th candidates are . In the “RTX 3090 GPU” case (3rd row), the similarity scores of the -th to -th candidate are unexceptionally . OA cannot change the relative order among those candidates with the same similarity scores.
Appendix B Visualizing Black-Box OA on Fashion-MNIST & Stanford Online Products
We present some visualizations of the black-box OA on the Fashion-MNIST dataset and the SOP dataset, as shown in (Fig. LABEL:fig:fb1, Fig. LABEL:fig:fb2, Fig. LABEL:fig:fb3) and (Fig. LABEL:fig:sb1, Fig. LABEL:fig:sb2, Fig. LABEL:fig:sb3), respectively. All the adversarial perturbations are found under and .
All these figures are picture matrices of size . In particular, pictures at location , and are the original query, the perturbation and the perturbed query image, respectively. The 2nd row in each figure is the original query and the corresponding ranking list (truncated to the top- results). The 3rd row in each figure is the permuted top- candidates. The 4th row in each figure is the adversarial query and the corresponding ranking list (also truncated to the top- results). Every picture is annotated with its ID in the dataset and its label for classification.
Appendix C Additional Information for White-Box OA
c.1 More Results on Ablation of
|Stanford Online Products|
In order to make sure the selected candidates will not disappear from the top- results during the OA process, a semantics-preserving term is introduced to maintain the absolute ranks of . Tab. 2 in the manuscript presents two ablation experimental results of for the white-box -OA with . In this subsection, we provide the full ablation experiments of in all parameter settings, as shown in Tab. 8. After removing the term from the loss function (, setting ), the white-box OA can achieve a better , meanwhile a worse mR. When comparing it with Tab. 1 in the manuscript, we find that (1) the semantics-preserving term is effective for keeping the selected within top- results; (2) will increase the optimization difficulty, so there will be a trade off between and . These results support our analysis and discussion in Sec. 4.1.
Some adversarial examples targeted at ranking models have been found transferable . Following , we train Lenet and ResNet18 models on Fashion-MNIST besides the C2F1 model used in the manuscript. Each of them is trained with two different parameter initializations (annotated with #1 and #2). Then we conduct transferability-based attack using a white-box surrogate model with and , as shown in Tab. 9.
|From \ To||Lenet #1||Lenet #2||C2F1 #1||C2F1 #2||ResNet18 #1||ResNet18 #2|
According to the table, OA adversarial example does not exhibit transferability over different architectures or different parameter initializations. We believe these models learned distinct embedding spaces, across which enforcing a specific fixed ordering is particularly difficult without prior knowledge of the model being attacked. As transferability is not a key point of the manuscript, we only discuss it in this appendix.
c.3 Intra-Class Variance & OA Difficulty
In the last paragraph of Sec. 4.1 (White-Box Order Attack Experiments), it is claimed that,
The intra-class variance of the relatively simple Fashion-MNIST dataset is smaller than that of SOP, which means samples of the same class are densely clustered in the embedding space. As each update could change the more drastically, it is more difficult to adjust the query embedding position with a fixed PGD step for a higher without sacrificing the mR value.
To better illustrate the idea, a diagram is present in Fig. 5. Given a trained model, and a fixed PGD step size as , the query image is projected near a dense embedding cluster in case I, while near a less dense embedding cluster in case II. Then the query image is updated with a fixed step size in both case I and case II, resulting in similar position change in the embedding space. At the same time the ranking list for the updated query will change as well. However, the ranking list in case I changes much more dramatically than that in case II. As a result, there is a higher chance for the ranks of the selected to significantly change in case I, leading to a high value of mR. Besides, is already the smallest appropriate choice for the PGD update step size, as in practice the adversarial examples will be quantized into the range . Namely, the position of the query embedding cannot be modified in a finer granularity given such limitation. Since a dataset with small intra-class variance tend to be densely clustered in the embedding space, we speculate that optimizing on a dataset with small intra-class variance (, Fashion-MNIST) is difficult, especially in terms of maintaining a low mR.