Overview. Extreme multi-label classification (XC) involves tagging a data point with the subset of labels most relevant to it, from an extremely large set of labels. XC finds applications in several domains including product recommendation (Medini et al., 2019), related searches (Jain et al., 2019), related products (Mittal et al., 2021), etc. This paper demonstrates that XC methods stand to benefit significantly from utilizing label correlation data, by presenting ECLARE, an XC method that utilizes textual label descriptions and label correlation graphs over millions of labels to offer predictions that can be 2–14% more accurate than those offered by state-of-the-art XC methods, including those that utilize label metadata such as label text.
Rare Labels. XC applications with millions of labels typically find that most labels are rare, with very few training data points tagged with those labels. Fig 1 exemplifies this on two benchmark datasets where 60–80% labels have
training points. The reasons behind rare labels are manifold. In several XC applications, there may exist an inherent skew in the popularity of labels,e.g, it is natural for certain products to be more popular among users on an e-commerce platform. XC applications also face missing labels (Jain et al., 2016; Yu et al., 2014) where training points are not tagged with all the labels relevant to them. Reasons for this include the inability of human users to exhaustively mark all products of interest to them, and biases in the recommendation platform (e.g. website, app) itself which may present or impress upon its users, certain products more often than others.
Need for Label Metadata. Rare labels are of critical importance in XC applications. They allow highly personalized yet relevant recommendations that may delight and surprise a user, or else allow precise and descriptive tags to be assigned to a document, etc. However, the paucity of training data for rare labels makes it challenging to predict them accurately. Incorporating label metadata such as textual label descriptions (Mittal et al., 2021), label taxonomies (Kanagal et al., 2012; Menon et al., 2011; Sachdeva et al., 2018) and label co-occurrence into the classification process are possible ways to augment the available information for rare labels.
Key Contributions of ECLARE. This paper presents ECLARE, an XC method that performs collaborative learning that benefits rare labels. This is done by incorporating multiple forms of label metadata such as label text as well as dynamically inferred label correlation graphs. Critical to ECLARE are augmentations to the architecture and learning pipeline that scale to millions of labels:
Introduce a framework that allows collaborative extreme learning using label-label correlation graphs that are dynamically generated using asymmetric random walks. This is in contrast to existing approaches that often perform collaborative learning on static user-user or document-document graphs (He et al., 2020; Hamilton et al., 2017; Ying et al., 2018).
Introduce the use of multiple representations for each label: one learnt from label text alone (LTE), one learnt collaboratively from label correlation graphs (GALE
), and a label-specific refinement vector.ECLARE
proposes a robust yet inexpensive attention mechanism to fuse these multiple representations to generate a single one-vs-all classifier per label.
Propose critical augmentations to well-established XC training steps, such as label clustering, negative sampling, classifier initialization, shortlist creation (GAME), etc, in order to incorporate label correlations in a systematic and scalable manner.
Offer an end-to-end training pipeline incorporating the above components in an efficient manner which can be scaled to tasks with millions of labels and offer up to 14% performance boost on standard XC prediction metrics.
Comparison to State-of-the-art. Experiments indicate that apart from significant boosts on standard XC metrics (see Tab 2), ECLARE offers two distinct advantages over existing XC algorithms, including those that do use label metadata such as label text (see Tab 6)
Superior Rare Label Prediction: In the first example in Tab 6, for the document “Tibetan Terrier”, ECLARE correctly predicts the rare label “Dog of Osu” that appeared just twice in the training set. All other methods failed to predict this rare label. It is notable that this label has no common tokens (words) with the document text or other labels which indicates that relying solely on label text is insufficient. ECLARE offers far superior performance on propensity scored XC metrics which place more emphasis on predicting rare labels correctly (see Tabs 2 and 3).
Superior Intent Disambiguation: The second and third examples in Tab 6 further illustrate pitfalls of relying on label text alone as metadata. For the document “85th Academy Awards”, all other methods are incapable of predicting other award ceremonies held in the same year and make poor predictions. On the other hand, ECLARE was better than other methods at picking up subtle cues and associations present in the training data to correctly identify associated articles. ECLARE offers higher precision@1 and recall@10 (see Tabs 2 and 3).
2. Related Work
Summary. XC algorithms proposed in literature employ a variety of label prediction approaches like tree, embedding, hashing and one-vs-all-based approaches (Babbar and Schölkopf, 2017; Prabhu and Varma, 2014; Prabhu et al., 2018a, b; Jain et al., 2016, 2017, 2019; Babbar and Schölkopf, 2019; Yen et al., 2016, 2017, 2018; Khandagale et al., 2019; Jasinska et al., 2016; Siblini et al., 2018; Jalan and Kar, 2019; Tagami, 2017; Liu et al., 2017; You et al., 2018; Chang et al., 2020; Wydmuch et al., 2018; Guo et al., 2019; Bhatia et al., 2015; Gupta et al., 2019; Dahiya et al., 2021; Mittal et al., 2021). Earlier works learnt label classifiers using fixed representations for documents (typically bag-of-words) whereas contemporary approaches learn a document embedding architecture (typically using deep networks) jointly with the label classifiers. In order to operate with millions of labels, XC methods frequently have to rely on sub-linear time data structures for operations such as shortlisting labels, sampling hard negatives, etc. Choices include hashing (Medini et al., 2019), clustering (Prabhu et al., 2018b; You et al., 2018; Chang et al., 2020), negative sampling (Mikolov et al., 2013), etc. Notably, most XC methods except DECAF (Mittal et al., 2021), GLaS (Guo et al., 2019), and X-Transformer (Chang et al., 2020) do not incorporate any form of label metadata, instead treating labels as black-box identifiers.
Fixed Representation. Much of the early work in XC used fixed bag-of-words (BoW) features to represent documents. One-vs-all methods such as DiSMEC (Babbar and Schölkopf, 2017), PPDSparse (Yen et al., 2017), ProXML (Babbar and Schölkopf, 2019) decompose the XC problem into several binary classification problems, one per label. Although these offered state-of-the-art performance until recently, they could not scale beyond a few million labels. To address this, several approaches were suggested to speed up training (Khandagale et al., 2019; Jain et al., 2019; Prabhu et al., 2018b; Yen et al., 2018), and prediction (Jasinska et al., 2016; Niculescu-Mizil and Abbasnejad, 2017) using tree-based classifiers and negative sampling. These offered high performance as well as could scale to several millions of labels. However, these architectures were suited for fixed features and did not support jointly learning document representations. Attempts, such as (Jain et al., 2019), to use pre-trained features such as FastText (Joulin et al., 2017) were also not very successful if the features were trained on an entirely unrelated task.
Representation Learning. Recent works such as X-Transformer (Chang et al., 2020), ASTEC (Dahiya et al., 2021), XML-CNN (Liu et al., 2017), DECAF (Mittal et al., 2021) and AttentionXML (You et al., 2018) propose architectures that jointly learn representations for the documents as well as label classifiers. For the most part, these methods outperform their counterparts that operate on fixed document representations which illustrates the superiority of task-specific document representations over generic pre-trained features. However, some of these methods utilize involved architectures such as attention (You et al., 2018; Chang et al., 2020) or convolutions (Liu et al., 2017). It has been observed (Dahiya et al., 2021; Mittal et al., 2021) that in addition to being more expensive to train, these architectures also suffer on XC tasks where the documents are short texts, such as user queries, or product titles.
XC with Label Metadata. Utilizing label metadata such as label text, label correlations, etc. can be critical for accurate prediction of rare labels, especially on short-text applications where documents have textual descriptions containing only 5-10 tokens which are not very descriptive. Among existing works, GLaS (Guo et al., 2019) uses label correlations to design a regularizer that improved performance over rare labels, while X-Transformer (Chang et al., 2020) and DECAF (Mittal et al., 2021) use label text as label metadata instead. X-Transformer utilizes label text to perform semantic label indexing (essentially a shortlisting step) along with a pre-trained-then-fine-tuned RoBERTa (Liu et al., 2019) architecture. On the other hand, DECAF uses a simpler architecture to learn both label and document representations in an end-to-end manner.
Collaborative Learning for XC. Given the paucity of data for rare labels, the use of label text alone can be insufficient to ensure accurate prediction, especially in short-text applications such as related products and related queries search, where the amount of label text is also quite limited. This suggests using label correlations to perform collaborative learning on the label side. User-user or document-document graphs (Ying et al., 2018; He et al., 2020; Kipf and Welling, 2017; Tang et al., 2020; Zhang et al., 2020; Hamilton et al., 2017; Velićković et al., 2018; Pal et al., 2020) have become popular, with numerous methods such as GCN (Kipf and Welling, 2017), LightGCN (He et al., 2020), GraphSAGE (Hamilton et al., 2017), PinSage (Ying et al., 2018), etc
. utilizing graph neural networks to augment user/document representations. However, XC techniques that directly enable label collaboration with millions of labels have not been explored. One of the major barriers for this seems to be that label correlation graphs in XC applications turn out to be extremely sparse,e.g, for the label correlation graph ECLARE constructed for the LF-WikiSeeAlsoTitles-320K dataset, nearly 18% of labels had no edges to any other label. This precludes the use of techniques such as Personalised Page Rank (PPR) (Ying et al., 2018; Klicpera et al., 2018) over the ground-truth to generate a set of shortlisted labels for negative sampling. ECLARE solves this problem by first mining hard-negatives for each label using a separate technique, and subsequently augmenting this list by adding highly correlated labels.
3. Eclare: Extreme Classification with Label Graph Correlations
Summary. ECLARE consists of four components 1) a text embedding architecture adapted to short-text applications, 2) one-vs-all classifiers, one per label that incorporate label text as well as label correlations, 3) a shortlister that offers high-recall label shortlists for data points, allowing ECLARE to offer sub-millisecond prediction times even with millions of labels, and 4) a label correlation graph that is used to train both the one-vs-all classifiers as well as the shortlister. This section details these components as well as a technique to infer label correlation graphs from training data itself.
Notation. Let denote the number of labels and the dictionary size. All training points are presented as . is a bag-of-tokens representation for the document i.e. is the TF-IDF weight of token in the document. is the ground truth label vector with if label is relevant to the document and otherwise. For each label , its label text is similarly represented as .
3.1. Document Embedding Architecture
ECLARE learns -dimensional embeddings for each vocabulary token and uses a light-weight embedding block (see Fig 2) implementing a residual layer. The embedding block contains two trainable parameters, a weight matrix and a scalar weight (see Fig 2). Given a document as a sparse bag-of-words vector, ECLARE performs a rapid embedding (see Fig 3) by first using the token embeddings to obtain an initial representation , and then passing this through an instantiation
of the text embedding block, and a ReLU non-linearity, to obtain the final representation. All documents (train/test) share the same embedding block . Similar architectures have been shown to be well-suited to short-text applications (Dahiya et al., 2021; Mittal et al., 2021).
3.2. Label Correlation Graph
XC applications often fail to provide label correlation graphs directly as an input. Moreover, since these applications also face extreme label sparsity, using label co-occurrence alone yields fractured correlations as discussed in Sec 2. For example, label correlations gleaned from products purchased together in the same session, or else queries on which advertisers bid together, may be very sparse. To remedy this, ECLARE infers a label correlation graph using the ground-truth label vectors i.e. themselves. This ensures that ECLARE is able to operate even in situations where the application is unable to provide a correlation graph itself. ECLARE adopts a scalable strategy based on random walks with restarts (see Algorithm 1) to obtain a label correlation graph that augments the often meager label co-occurrence links (see Fig 4) present in the ground truth. Non-rare labels (the so-called head and torso labels) pose a challenge to this step since they are often correlated with several labels and can overwhelm the rare labels. ECLARE takes two precautions to avoid this:
Partition: Head labels (those with training points) are disconnected from the graph by setting and for all all head labels and (see Fig 5).
Normalization: is normalized to favor edges to/from rare labels as , where are diagonal matrices with the row and column-sums of respectively.
is used with a restart probability of 80% and a random walk length of 400. Thus, it is overwhelmingly likely that several dozens of restarts would occur for each label. A high restart probability does not let the random walk wander too far thus preventing tenuous correlations among labels from getting captured.
3.3. Label Representation and Classifiers
As examples in Tab 6 discussed in Sec 4 show, label text alone may not sufficiently inform classifiers for rare labels. ECLARE remedies this by learning high-capacity one-vs-all classifiers with 3 distinct components described below.
Label Text Embedding (LTE). The first component incorporates label text metadata. A separate instance of the embedding block is used to embed label text. Given a bag-of-words representation of a label, the LTE representation is obtained as where as before, we have the “initial” representation . The embedding block is shared by all labels. We note that the DECAF method (Mittal et al., 2021) also uses a similar architecture to embed label text.
Graph Augmented Label Embedding (GALE). ECLARE augments the LTE representation using the label correlation graph constructed earlier and a graph convolution network (GCN) (Kipf and Welling, 2017). This presents a departure from previous XC approaches. A typical graph convolution operation consists of two steps which ECLARE effectively implements at extreme scales as shown below
[leftmargin = *]
Convolution: initial label representations are convolved in a scalable manner using as . Note that due to random restarts used by Algorithm 1, we have for all and thus contains a component from itself.
Transformation: Whereas traditional GCNs often use a simple non-linearity as the transformation, ECLARE instead uses a separate instance of the embedding block to obtain the GALE representation of the label as .
ECLARE uses a single convolution and transformation operation which allowed it to scale to applications with millions of nodes.
Recent works such as LightGCN (He et al., 2020) propose to accelerate GCNs by removing all non-linearities. Despite being scalable, using LightGCN itself was found to offer imprecise results in experiments. This may be because ECLARE also uses LTE representations. ECLARE can be seen as improving upon existing GCN architectures such as LightGCN by performing higher order label-text augmentations for as where encodes the hop neighborhood, and is a separate embedding block for each order . Thus, ECLARE’s architecture allows parallelizing high-order convolutions. Whereas ECLARE could be used with larger orders , using was found to already outperform all competing methods, as well be scalable.
Refinement Vector and Final Classifier. ECLARE combines the LTE and GALE representations for a label with a high-capacity per-label refinement vector (for a general value of , is used) to obtain a one-vs-all classifier (see Fig 3). To combine , ECLARE uses a parameterized attention block (see Fig 2) to learn label-specific attention weights for the three components. This is distinct from previous works such as DECAF which use weights that are shared across labels. Fig 9 shows that ECLARE benefits from this flexibility, with the refinement vector being more dominant for popular labels that have lots of training data whereas the label metadata based vectors being more important for rare labels with less training data. The attention block is explained below (see also Fig 2). Recall that ECLARE uses .
The attention block is parameterized by the matrices and . concatenates the transformed components before applying the attention layer . The above attention mechanism can be seen as a scalable paramaterized option (requiring only additional parameters) instead of a more expensive label-specific attention scheme which would have required learning parameters.
3.4. Meta-labels and the Shortlister
Despite being accurate, if used naively, one-vs-all classifiers require time at prediction and time to train. This is infeasible with millions of labels. As discussed in Sec 2, sub-linear structures are a common remedy in XC methods to perform label shortlisting during prediction (Khandagale et al., 2019; Prabhu et al., 2018b; Yen et al., 2017; Chang et al., 2020; Jain et al., 2019; Bhatia et al., 2015; Dahiya et al., 2021). These shortlisting techniques take a data point and return a shortlist of labels that is expected to contain most of the positive labels for that data point. However, such shortlists also help during training since the negative labels that get shortlisted for a data point are arguably the most challenging and likely to get confused as being positive for that data point. Thus, one-vs-all classifiers are trained only on positive and shortlisted negative labels, bringing training time down to . Similar to previous works (Prabhu et al., 2018a; Mittal et al., 2021), ECLARE uses a clustering-based shortlister where is a balanced partitioning of the labels into clusters. We refer to each cluster as a meta label. is a set of one-vs-all classifiers, learnt one per meta label.
Graph Assisted Multi-label Expansion (GAME). ECLARE incorporates graph correlations to further to improve its shortlister. Let denote the cluster assignment matrix i.e. if label is in cluster . We normalize so that each column sums to unity. Given a data point and a beam-size , its embedding (see Fig 3) is used to shortlist labels as follows
[leftmargin = *]
Find the top clusters, say according to the meta-label scores . Let be a vector containing scores for the top clusters passed through a sigmoid i.e. if else .
Use the induced cluster-cluster correlation matrix to calculate the “GAME-ified” scores .
Find the top clusters according to , say , retain their scores and set scores of other clusters in to . Return the shortlist and the modified score vector .
Note that this cluster re-ranking step uses an induced cluster correlation graph and can bring in clusters with rare labels missed by the one-vs-all models . This is distinct from previous works which do not use label correlations for re-ranking. Since the clusters are balanced, the shortlisted clusters always contain a total of labels. ECLARE uses clusters and a beam size of (see Sec 4
for a discussion on hyperparameters).
3.5. Prediction with Game
The prediction pipeline for ECLARE is depicted in Fig 6 and involves 3 steps that repeatedly utilize label correlations via GAME.
Given a document , use the shortlister to get a set of meta labels and their corresponding scores .
For shortlisted labels, apply the one-vs-all classifiers to calculate the ranking score vector with if for some else .
GAME-ify the one-vs-all scores to get (note that is used this time and not ). Make final predictions using the joint scores if else .
3.6. Efficient Training: the DeepXML Pipeline
Summary. ECLARE adopts the scalable DeepXML pipeline (Dahiya et al., 2021) that splits training into 4 modules. In summary, Module I jointly learns the token embeddings , the embedding block and the shortlister . remains frozen hereon. Module II refines and uses it to retrieve label shortlists for all data points. After performing initialization in Module III, Module IV uses the shortlists generated in Module II to jointly fine-tune and learn the one-vs-all classifiers , implicitly learning the embedding blocks and the attention block in the process.
Module I. Token embeddings are randomly initialized using (He et al., 2015), the residual block within is initialized to identity. After creating label clusters (see below), each cluster is treated as a meta-label yielding a meta-XC problem on the same training points, but with meta-labels instead of the original labels. Meta-label text is created for each as . Meta labels are also endowed with the meta-label correlation graph where is the cluster assignment matrix. One-vs-all meta-classifiers are now learnt to solve this meta XC problem. These classifiers have the same form as those for the original problem with 3 components, LTE, GALE, (with corresponding blocks ) and refinement vector, with an attention block supervising their combination (parameters within are initialized randomly). However, in Module-I, refinement vectors are turned off for to force good token embeddings to be learnt without support from refinement vectors. Module I solves the meta XC problem while training , implicitly learning .
Meta-label Creation with Graph Augmented Label Centroids. Existing approaches such as Parabel or DECAF cluster labels by creating a label centroid for each label by aggregating features for training documents associated with that label as . However, ECLARE recognizes that missing labels often lead to an incomplete ground truth, and thus, poor label centroids for rare labels in XC settings (Jain et al., 2017; Babbar and Schölkopf, 2019). Fig 8 confirms this suspicion. ECLARE addresses this by augmenting the label centroids using the label co-occurrence graph to redefine the centroids as . Balanced hierarchical binary clustering (Prabhu et al., 2018b) is now done on these label centroids for 17 levels to generate label clusters. Note that since token embeddings have not been learnt yet, the raw TF-IDF documents vectors are used instead of .
Module II. The shortlister is fine-tuned in this module. Label centroids are recomputed as where this time, using learnt in Module I. The meta XC problem is recreated and solved again. However, this time, ECLARE allows the meta-classifiers to also include refinement vectors to better solve the meta-XC problem. In the process of re-learning , the model parameters are fine-tuned ( is frozen after Module I). The shortlister thus obtained is thereafter used to retrieve shortlists for each data point . However, distinct from previous work such as Slice (Jain et al., 2019), X-Transformer (Chang et al., 2020), DECAF (Mittal et al., 2021), ECLARE uses the GAME strategy to obtain negative samples that take label correlations into account.
Module III. Residual blocks within are initialized to identity, parameters within are initialized randomly, and the shortlister and token embeddings are kept frozen. Refinement vectors for all labels are initialized to . We find this initialization to be both crucial (see Sec 4) as well as distinct from previous works such as DECAF which initialized its counterpart of refinement vectors using simply . Such correlation-agnostic initialization was found to offer worse results than ECLARE’s graph augmented initialization.
Module IV. In this module, the embedding blocks are learnt jointly with the per-label refinement vectors and attention block , thus learning the one-vs-all classifiers in the process, However, training is done in time by restricting training to positives and shortlisted negatives for each data point.
Loss Function and Regularization. ECLARE
uses the binary cross entropy loss function for training in Modules I, II and IV using the Adam(Kingma and Ba, 2014) optimizer. The residual weights in the various embedding blocks as well as the weights in the attention block were all subjected to spectral regularization (Miyato et al., 2018). All ReLU layers in the architecture also included a dropout layer with 20% rate.
Key Contributions in Training. ECLARE markedly departs from existing techniques by incorporating label correlation information in a scalable manner at every step of the learning process. Right from Module I, label correlations are incorporated while creating the label centroids leading to higher quality clusters (see Fig 8 and Table 7). The architecture itself incorporates label correlation information using the GALE representations. It is crucial to initialize the refinement vectors properly for which ECLARE uses a graph-augmented initialization. ECLARE continues infusing label correlation during negative sampling using the GAME step. Finally, GAME is used multiple times in the prediction pipeline as well. As the discussion in Sec 4 will indicate, these augmentations are crucial to the performance boosts offered by ECLARE.
|Short text dataset statistics|
|Original |||With GAME|
|Original |||With GAME|
Datasets and Features. ECLARE was evaluated on 4 publicly available111Extreme Classification Repository (Bhatia et al., 2016) benchmark datasets, LF-AmazonTitles-131K, LF-WikiSeeAlso-Titles-320K, LF-WikiTitles-500K and LF-AmazonTitles-1.3M. These datasets were derived from existing datasets e.g. Amazon-670K, by taking those labels for which label text was available and performing other sanitization steps such as reciprocal pair removal (see (Mittal et al., 2021) for details). ECLARE
was also evaluated on proprietary datasets P2P-2M and P2P-10M, both mined from click logs of the Bing search engine, where a pair of products were considered similar if the Jaccard index of the set of queries which led to a click on them was found to be more than a certain threshold.ECLARE used the word piece tokenizer (Schuster and Nakajima, 2012) to create a shared vocabulary for documents and labels. Please refer to Tab 1 for dataset statistics.
Baseline algorithms. ECLARE’s performance was compared to state-of-the-art deep extreme classifiers which jointly learn document and label representations such as DECAF (Mittal et al., 2021), AttentionXML (You et al., 2018), Astec (Dahiya et al., 2021), X-Transformer (Chang et al., 2020), and MACH (Medini et al., 2019). DECAF and X-Transformer are the only methods that also use label text and are therefore the most relevant for comparison with ECLARE. For the sake of completeness, ECLARE was also compared to classifiers which use fixed document representations like DiSMEC (Babbar and Schölkopf, 2017), Parabel (Prabhu et al., 2018b), Bonsai (Khandagale et al., 2019), and Slice (Jain et al., 2019). All these fixed-representation methods use BoW features, except Slice which used pre-trained FastText (Bojanowski et al., 2017) features. The GLaS (Guo et al., 2019) method could not be included in our analysis as its code was not publicly available.
Evaluation. Methods were compared using standard XC metrics, namely Precision (P@) and Propensity-scored precision (PSP@) (Jain et al., 2017). Recall (R@) was also included since XC methods are typically used in the shortlisting pipeline of recommendation systems. Thus, having high recall is equally important as having high precision. For evaluation, guidelines provided on the XML repository (Bhatia et al., 2016) were followed. To be consistent, all models were run on a 6-core Intel Skylake 2.4 GHz machine with one Nvidia V100 GPU.
Hyper-parameters. A beam size of was used for the dataset LF-AmazonTitles-131K and for all other datasets. Embedding dimension was set to 300 for datasets with K labels and 512 otherwise. The number of meta-labels was fixed to for all other datasets except for LF-AmazonTitles-131K where was chosen since the dataset itself has around
labels. The default PyTorch implementation of the Adam optimizer was used. Dropout with probability 0.2 was used for all datasets. Learning rate was decayed by a decay factor of 0.5 after an interval ofepoch length. Batch size was taken to be 255 for all datasets. ECLARE Module-I used 20 epochs with an initial learning rate of 0.01. In Modules-II and IV, 10 epochs were used for all datasets with an initial learning rate of 0.008. While constructing the label correlation graph using random walks (see Algorithm 1), a walk length of and restart probability of were used for all datasets.
Results on benchmark datasets. Tab 2 demonstrates that ECLARE can be significantly more accurate than existing XC methods. In particular, ECLARE could be upto 3% and 10% more accurate as compared to DECAF and X-Transformer respectively in terms of P@. It should be noted that DECAF and X-Transformer are the only XC methods which use label meta-data. Furthermore, ECLARE could outperform Astec which is specifically designed for rare labels, by up to 7% in terms of PSP@, indicating that ECLARE offers state-of-the-art accuracies without compromising on rare labels. To further understand the gains of ECLARE, the labels were divided into five bins such that each bin contained an equal number of positive training points (Fig 7). This ensured that each bin had an equal opportunity to contribute to the overall accuracy. Fig 7 indicates that the major gains of ECLARE come from predicting rare labels correctly. ECLARE could outperform all other deep-learning-based XC methods, as well as fixed-feature-based XC methods by a significant margin of at least 7%. Moreover, ECLARE’s recall could be up to 2% higher as compared to all other XC methods.
Results on Propreitary Bing datasets. ECLARE’s performance was also compared on the proprietary datasets LF-P2PTitles-2M and LF-P2PTitles-10M. Note that ECLARE was only compared with those XC methods that were able to scale to 10 million labels on a single GPU within a timeout of one week. ECLARE could be up to 14%, 15%, and 15% more accurate as compared to state-of-the-art methods in terms of P@, PSP@ and R@ respectively. Please refer to Tab 3 for more details.
Ablation experiments. To scale accurately to millions of labels, ECLARE makes several meticulous design choices. To validate their importance, Tab 5 compares different variants of ECLARE:
Label-graph augmentations: The label correlation graph is used by ECLARE in several steps, such as in creating meaningful meta labels, negative sampling, shortlisting of meta-labels, etc. To evaluate the importance of these augmentations, in ECLARE-NoGraph, the graph convolution component was removed from all training steps such as GAME, except while generating the label classifiers (GALE). ECLARE-NoGraph was found to be up to 3% less accurate than ECLARE. Furthermore, in a variant ECLARE-PPR, inspired by state-of-the-art graph algorithms (Ying et al., 2018; Klicpera et al., 2018), negative sampling was performed via Personalized Page Rank. ECLARE could be up to 25% more accurate than ECLARE-PPR. This could be attributed to the highly sparse label correlation graph and justifies the importance of ECLARE’s label correlation graph as well as it’s careful negative sampling.
Label components: Much work has been done to augment (document) text representations using graph convolution networks (GCNs) (Tang et al., 2020; Kipf and Welling, 2017; Zhang et al., 2020; Hamilton et al., 2017; He et al., 2020; Velićković et al., 2018; Pal et al., 2020). These methods could also be adapted to convolve label text for ECLARE’s GALE component. ECLARE’s GALE component was replaced by a LightGCN (He et al., 2020) and GCN (Kipf and Welling, 2017) (with the refinement vectors in place). Results indicate that ECLARE could be upto 2% and 10% more accurate as compared to LightGCN and GCN. In another variant of ECLARE, the refinement vectors were removed (ECLARE-NoRefine). Results indicate that ECLARE could be up to 10% more accurate as compared to ECLARE-NoRefine which indicates that the per-label (extreme) refinement vectors are essential for accuracy.
Graph construction: To evaluate the efficacy of ECLARE’s graph construction, we compare it to ECLARE-Cooc where we consider the label co-occurrence graph generated by instead of the random-walk based graph used by ECLARE. ECLARE-Cooc could be up to 2% less accurate in terms of PSP@1 than ECLARE. This shows that ECLARE’s random-walk graph indeed captures long-term dependencies thereby resulting in improved performance on rare labels. Normalizing a directed graph, such as the graph obtained by Algorithm 1 is a non-trivial problem. In ECLARE-PN, we apply the popular Perron normalization (Chung, 2005) to the random-walk graph . Unfortunately, ECLARE-PN leads to a significant loss in propensity-scored metrics for rare labels. This validates the choice of ECLARE’s normalization strategy.
Combining label representations: The LTE and GALE components of ECLARE could potentially be combined using strategies different from the attention mechanism used by ECLARE. A simple average/sum of the components, (ECLARE-SUM) could be up to 2% worse, which corroborates the need for the attention mechanism for combining heterogeneous components.
Higher-order Convolutions: Since the ECLARE framework could handle higher order convolutions efficiently, we validated the effect of increasing the order. Using was found to hurt precision by upto 2%. Higher orders e.g. etc. were intractable at XC scales as the graphs got too dense. The drop in performance when using could be due to two reasons: (a) at XC scales, exploring higher order neighborhoods would add more noise than information unless proper graph pruning is done afterward and (b) ECLARE’s random-walk graph creation procedure already encapsulates some form of higher order proximity which negates the potential for massive benefits when using higher order convolutions.
Meta-labels and GAME: ECLARE uses a massive fanout of meta-labels in its shortlister. Ablations with (ECLARE-8K) show that using a smaller fanout can lead to upto 8% loss in precision. Additionally Tab 4 shows that incorporating GAME with other XC methods can also improve their accuracies (although ECLARE continues to lead by a wide margin). In particular incorporating GAME in Parabel and AttentionXML led to up to 1% increase in accuracy. This validates the utility of GAME as a general XC tool.
|ECLARE||Tibetan Spaniel, Tibetan kyi apso, Lhasa Apso, Dog of Osu, Tibetan Mastiff|
|DECAF||Fox Terrier, Tibetan Spaniel, Terrier, Bull Terrier, Bulldog|
|Astec||Standard Tibetan, List of domesticated Scottish breeds, List of organizations of Tibetans in exile, Tibet, Riwoche horse|
|Parabel||Tibet, List of organizations of Tibetans in exile, List of domesticated Scottish breeds, History of Tibet, Languages of Bhutan|
|AttentionXML||List of organizations of Tibetans in exile, List of domesticated Scottish breeds, Dog, Bull Terrier, Dog crossbreed|
|Document||85th Academy Awards|
|ECLARE||List of submissions to the 85th Academy Awards for Best Foreign Language Film, 33rd Golden Raspberry Awards, 19th Screen Actors Guild Awards, 67th Tony Awards, 70th Golden Globe Awards|
|DECAF||List of American films of 1956, 87th Academy Awards, List of American films of 1957, 1963 in film, 13th Primetime Emmy Awards|
|Astec||65th Tony Awards, 29th Primetime Emmy Awards, 32nd Golden Raspberry Awards, 64th Primetime Emmy Awards, 18th Screen Actors Guild Awards|
|Parabel||1928 in film, 1931 in film, 1930 in film, 48th Academy Awards, 26th Primetime Emmy Awards, 31st European Film Awards|
|AttentionXML||29th Primetime Emmy Awards, 62nd British Academy Film Awards, 60th Primetime Emmy Awards, 65th Tony Awards, 29th Golden Raspberry Awards|
|Document||Draper’s & Damon’s Women’s Chevron Faux Fur Coat Tan L|
|ECLARE||Grey Wolf Faux Fur Coat XXL / Grey, Big on Dots Faux-Fur Coat by LUXE, Avec Les Filles Bonded Faux-Fur Long Coat Size Large Black, Roaman’s Women’s Short Faux-Fur Coat (Black) 1X, Dennis Basso Faux Fur Jacket with Stand Collar Size XX-Small Cappuccino|
|DECAF||Draper’s & Damon’s Women’s Petite Cabana Big Shirt Blue P-L, Draper’s & Damon’s Women’s Petite Top It Off Stripe Jacket Blue P-L, Draper’s & Damon’s Women’s Petite Standing Ovation Jacket Black P-L, Draper & Damon Jackets & Coats | Draper & Damons Size L Colorful Coat Wpockets | Color: Black/Green | Size: L, Draper’s & Damon’s Women’s Impressionist Textured Jacket Multi L|
|Astec||Draper’s & Damon’s Women’s Petite Cabana Big Shirt Blue P-L, Draper’s & Damon’s Women’s Petite Top It Off Stripe Jacket Blue P-L, Draper’s & Damon’s Women’s Impressionist Textured Jacket Multi L, Draper’s & Damon’s Women’s Embroidered Tulle Jacket Dress Blue 14, Draper’s & Damon’s Women’s Petite Standing Ovation Jacket Black P-L|
|Parabel||Draper’s & Damon’s Women’s Impressionist Textured Jacket Multi L, Draper’s & Damon’s Women’s Over The Rainbow Jacket Multi P-L, Draper’s & Damon’s Women’s Petite Painted Desert Jacket White P-M, Draper Women’s Drapers & Damons Pants Suit - Pant Suit | Color: Black | Size: L, Draper’s & Damon’s Women’s Petite Floral & Stripe Knit Mesh Jacket Scarlet Multi P-L|
Analysis. This section critically analyzes ECLARE’s performance gains by scrutinizing the following components:
Clustering: As is evident from Fig 8, ECLARE’s offers significantly better clustering quality. Other methods such as DECAF use label centroids over an incomplete ground truth, resulting in clusters of seemingly unrelated labels. For e.g. the label “Bulldog” was clustered with “Great house at Sonning” by DECAF and the label “Dog of Osu” was clustered with “Ferdinand II of Aragon” which never co-occur in training. However ECLARE clusters much more relevant labels together, possibly since it was able to (partly) complete the ground truth using it’s label correlation graph . This was also verified quantitatively by evaluating the clustering quality using the standard Loss of Mutual Information metric (LMI) (S. et al., 2003). Tab 7 shows that ECLARE has the least LMI compared to other methods such as DECAF and those such as MACH that use random hashes to cluster labels.
Component Contribution: ECLARE chooses to dynamically attend on multiple (and heterogeneous) label representations in its classifier components, which allows it to capture nuanced variations in the semantics of each label. To investigate the contribution of the label text and refinement classifiers, Fig. 9 plots the average product of the attention weight and norm of each component. It was observed that the label text components LTE and GALE are crucial for rare labels whereas the (extreme) refinement vector is more important for data-abundant popular labels.
Label Text Augmentation: For the document “Tibetan Terrier”, ECLARE could make correct rare label predictions like “Dog of Osu” even when the label text exhibits no token similarity with the document text or other co-occurring labels. Other methods such as DECAF failed to understand the semantics of the label and mis-predicted the label “Fox Terrier” wrongly relying on the token “Terrier”. We attribute this gain to ECLARE’s label correlation graph as “Dog of Osu” correlated well with the labels “Tibetan Spaniel”, “Tibetan kyi apso” and “Tibetan Mastiff” in . Several such examples exist in the datasets. In another example from the P2P-2M dataset, for the product “Draper’s & Damon’s Women’s Chevron Fauz Fur Coat Tan L”, ECLARE could deduce the intent of purchasing “Fur Coat” while other XC methods incorrectly fixated on the brand “Draper’s & Damon’s”. Please refer to Tab 6 for detailed examples.
This paper presents the architecture and accompanying training and prediction techniques for the ECLARE method to perform extreme multi-label classification at the scale of millions of labels. The specific contributions of ECLARE include a framework for incorporating label graph information at massive scales, as well as critical design and algorithmic choices that enable collaborative learning using label correlation graphs with millions of labels. This includes systematic augmentations to standard XC algorithmic operations such as label-clustering, negative sampling, shortlisting, and re-ranking, to incorporate label correlations in a manner that scales to tasks with millions of labels, all of which were found to be essential to the performance benefits offered by ECLARE. The creation of label correlation graphs from ground truth data alone and its use in a GCN-style architecture to obtain multiple label representations is critical to ECLARE’s performance benefits. The proposed approach greatly outperforms state-of-the-art XC methods on multiple datasets while still offering millisecond level prediction times even on the largest datasets. Thus, ECLARE establishes a standard for incorporating label metadata into XC techniques. These findings suggest promising directions for further study including effective graph pruning for heavy tailed datasets, using higher order convolutions () in a scalable manner, and performing collaborative learning with heterogeneous and even multi-modal label sets. This has the potential to enable generalisation to settings where labels include textual objects such as (related) webpages and documents, but also videos, songs, etc.
Acknowledgements.The authors thank the reviewers for helpful comments that improved the presentation of the paper. They also thank the IIT Delhi HPC facility for computational resources. AM is supported by a Google PhD Fellowship.
- DiSMEC: distributed sparse machines for extreme multi-label classification. In WSDM, Cited by: §2, §2, §4.
- Data scarcity, robustness and extreme multi-label classification. ML. Cited by: §2, §2, §3.6.
- The extreme classification repository: multi-label datasets and code. External Links: Cited by: §4, footnote 1.
- Sparse local embeddings for extreme multi-label classification. In NIPS, Cited by: §2, §3.4.
- Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics. Cited by: §4.
- Taming pretrained transformers for extreme multi-label text classification. In KDD, Cited by: §2, §2, §2, §3.4, §3.6, §4.
- Laplacians and the cheeger inequality for directed graphs. Annals of Combinatorics 9 (1), pp. 1–19. Cited by: item 3.
- DeepXML: a deep extreme multi-label learning framework applied to short text documents. In WSDM, Cited by: §2, §2, §3.1, §3.4, §3.6, §4.
- Breaking the glass ceiling for embedding-based classifiers for large output spaces. In Neurips, Cited by: §2, §2, §4.
- Distributional semantics meets multi-label learning. In AAAI, Cited by: §2.
- Inductive representation learning on large graphs. In NIPS, pp. 1024–1034. Cited by: item 1, §2, item 2.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In
Proceedings of the IEEE international conference on computer vision, pp. 1026–1034. Cited by: §3.6.
- LightGCN: simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20. Cited by: item 1, §2, §3.3, item 2.
- Slice: scalable linear extreme classifiers trained on 100 million labels for related searches. In WSDM, Cited by: §1, §2, §2, §3.4, §3.6, §4.
- Extreme multi-label loss functions for recommendation, tagging, ranking and other missing label applications. In KDD, Cited by: §1, §2.
- Scalable generative models for multi-label learning with missing labels. In ICML, Cited by: §2, §3.6, §4.
- Accelerating extreme classification via adaptive feature agglomeration. IJCAI. Cited by: §2.
Extreme f-measure maximization using sparse probability estimates. In ICML, Cited by: §2, §2.
- Bag of tricks for efficient text classification. In Proceedings of the European Chapter of the Association for Computational Linguistics, Cited by: §2.
- Supercharging recommender systems using taxonomies for learning user purchase behavior. VLDB. Cited by: §1.
- Bonsai - diverse and shallow trees for extreme multi-label classification. CoRR. Cited by: §2, §2, §3.4, §4.
- Adam: A method for stochastic optimization. CoRR. Cited by: §3.6.
- Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, Cited by: §2, §3.3, item 2.
- Predict then propagate: graph neural networks meet personalized pagerank. In International Conference on Learning Representations, Cited by: §2, item 1.
- Deep learning for extreme multi-label text classification. In SIGIR, Cited by: §2, §2.
- RoBERTa: A Robustly Optimized BERT Pretraining Approach. Note: arXiv:1907.11692 Cited by: §2.
- Extreme classification in log memory using count-min sketch: a case study of amazon search with 50m products. In Neurips, Cited by: §1, §2, §4.
- Response prediction using collaborative filtering with hierarchies and side-information. In KDD, Cited by: §1.
- Distributed representations of words and phrases and their compositionality. In NIPS, Cited by: §2.
- DECAF: Deep Extreme Classification with Label Features. In WSDM, Cited by: §1, §1, §2, §2, §2, §3.1, §3.3, §3.4, §3.6, §4, §4.
Spectral normalization for generative adversarial networks. In ICLR, Cited by: §3.6.
- Label filters for large scale multilabel classification. In AISTATS, Cited by: §2.
- PinnerSage: multi-modal user embedding framework for recommendations at pinterest. In KDD ’20, KDD ’20, New York, NY, USA, pp. 2311–2320. External Links: Cited by: §2, item 2.
- Extreme multi-label learning with label features for warm-start tagging, ranking and recommendation. In WSDM, Cited by: §2, §3.4.
- Parabel: partitioned label trees for extreme classification with application to dynamic search advertising. In WWW, Cited by: §2, §2, §3.4, §3.6, §4.
- FastXML: a fast, accurate and stable tree-classifier for extreme multi-label learning. In KDD, Cited by: §2.
- A Divisive Information-Theoretic Feature Clustering Algorithm for Text Classification. JMLR 3, pp. 1265–1287. Cited by: item 1.
- Attentive neural architecture incorporating song features for music recommendation. In RecSys, Cited by: §1.
- Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5149–5152. Cited by: §4.
CRAFTML, an efficient clustering-based random forest for extreme multi-label learning. In ICML, Cited by: §2.
- AnnexML: approximate nearest neighbor search for extreme multi-label classification. In KDD, Cited by: §2.
- Multi-label patent categorization with non-local attention-based graph convolutional network. In AAAI, 2020, Cited by: §2, item 2.
- Graph Attention Networks. ICLR. Cited by: §2, item 2.
- A no-regret generalization of hierarchical softmax to extreme multi-label classification. In NIPS, Cited by: §2.
- PPDSparse: a parallel primal-dual sparse method for extreme classification. In KDD, Cited by: §2, §2, §3.4.
- PD-sparse: a primal and dual sparse approach to extreme multiclass and multilabel classification. In ICML, Cited by: §2.
- Loss decomposition for fast learning in large output spaces. In ICML, Cited by: §2, §2.
Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974–983. Cited by: item 1, §2, item 1.
AttentionXML: extreme multi-label text classification with multi-label attention based recurrent neural networks. CoRR. Cited by: §2, §2, §4.
- Large-scale multi-label learning with missing labels. In ICML, Cited by: §1.
- Deep learning on graphs: a survey. IEEE Transactions on Knowledge and Data Engineering. Cited by: §2, item 2.