1 Introduction
Various widely used probability mapping functions such as sumnormalization, softmax, and spherical softmax enable mapping of vectors from the euclidean space to probability distributions. The need for such functions arises in multiple problem settings like multiclass classification aly2005survey ; bridle90
Sutton:1998:IRL:551283 ; gao17 and more recently in attention mechanism bahdanau2014neural ; conf/icml/XuBKCCSZB15 ; choattn15 ; rushchopraweston:2015:EMNLP ; diverseattn17in deep neural networks, amongst others. Even though softmax is the most prevalent approach amongst them, it has a shortcoming in that its outputs are composed of only nonzeroes and is therefore illsuited for producing sparse probability distributions as output. The need for sparsity is motivated by parsimonious representations
Bach:2011:OSP:2208224investigated in the context of variable or feature selection. Sparsity in the input space offers benefits of model interpretability as well as computational benefits whereas on the output side, it helps in filtering large output spaces, for example in large scale multilabel classification settings
Sorower2010ALS . While there have been several such mapping functions proposed in literature such as softmax gao17 , spherical softmax sphericalnips15 ; DBLP:journals/corr/BrebissonV15 and sparsemax martins2016 ; Duchi:2008:EPL:1390156.1390191 , very little is understood in terms of how they relate to each other and their theoretical underpinnings. Further, for sparse formulations, often there is a need to tradeoff interpretability for accuracy, yet none of these formulations offer an explicit control over the desired degree of sparsity.Motivated by these shortcomings, in this paper, we introduce a general formulation encompassing all such probability mapping functions which serves as a unifying framework to understand individual formulations such as hardmax, softmax, sumnormalization, spherical softmax and sparsemax as special cases, while at the same time helps in providing explicit control over degree of sparsity. With the aim of controlling sparsity, we propose two new formulations: sparsegenlin and sparsehourglass
. Our framework also ensures simple closedform solutions and existence of subgradients similar to softmax. This enables them to be employed as activation functions in neural networks which require gradients for backpropagation and are suitable for tasks that require sparse attention mechanism
martins2016 . We also show that the sparsehourglass formulation can extend from translation invariance to scale invariance with an explicit control, thus helping to achieve an adaptive tradeoff between these invariance properties as may be required in a problem domain.We further propose new convex loss functions which can help induce the behaviour of the above proposed formulations in a multilabel classification setting. These loss functions are derived from a violation of constraints required to be satisfied by the corresponding mapping functions. This way of defining losses leads to an alternative loss definition for even the sparsemax function martins2016 . Through experiments we are able to achieve improved results in terms of sparsity and prediction accuracies for multilabel classification.
Lastly, the existence of subgradients for our proposed formulations enable us to employ them to compute attention weights bahdanau2014neural ; choattn15
in natural language generation tasks. The explicit controls provided by sparsegenlin and sparsehourglass help to achieve higher interpretability while providing better or comparable accuracy scores. A recent work
DBLP:journals/corr/NiculaeB17 had also proposed a framework for attention; however, they had not explored the effect of explicit sparsity controls. To summarize, our contributions are the following:
A general framework of formulations producing probability distributions with connections to hardmax, softmax, sparsemax, spherical softmax and sumnormalization (Sec.3).

A formulation sparsehourglass which enables us to adaptively tradeoff between the translation and scale invariance properties through explicit control (Sec.3.5).

Convex multilabel loss functions correponding to all the above formulations proposed by us. This enable us to achieve improvements in the multilabel classification problem (Sec.4).

Experiments for sparse attention on natural language generation tasks showing comparable or better accuracy scores while achieving higher interpretability (Sec.5).
2 Preliminaries and Problem Setup
Notations: For , we denote . Let be a real vector denoted as . 1 and 0 denote vector of ones and zeros resp. Let be the dimensional simplex and be denoted as . We use . Let be the set of maximal elements of .
Definition: A probability mapping function is a map which transforms a score vector to a categorical distribution (denoted as ). The support of is . Such mapping functions can be used as activation function for machine learning models. Some known probability mapping functions are listed below:

Softmax function is defined as: Softmax is easy to evaluate and differentiate and its logarithm is the negative loglikelihood loss martins2016 .

Spherical softmax  Another function which is simple to compute and derivativefriendly: Spherical softmax is not defined for .

Sumnormalization : It is not used in practice much as the mapping is not defined if for any and for .
The above mapping functions are limited to producing distributions with full support. Consider there is a single value of significantly higher than the rest, its desired probability should be exactly 1, while the rest should be grounded to zero (hardmax mapping). Unfortunately, that does not happen unless the rest of the values tend to (in case of softmax) or are equal to (in case of spherical softmax and sumnormalization).

Sparsemax recently introduced by martins2016 circumvents this issue by projecting the score vector onto a simplex Duchi:2008:EPL:1390156.1390191 : . This offers an intermediate solution between softmax (no zeroes) and hardmax (zeroes except for the highest value).
The contour plots for softmax, sparsemax and sumnormalization in twodimensions () are shown in Fig.1. The contours of sparsemax are concentrated over a narrow region, while the remaining region corresponds to sparse solutions. For softmax, the contour plots are spread over the whole real plane, confirming the absence of sparse solutions. Sumnormalization is not defined outside the first quadrant, and yet, the contours cover the whole quadrant, denying sparse solutions.
3 Sparsegen Activation Framework
Definition: We propose a generic probability mapping function inspired from the sparsemax formulation (in Sec.2) which we call sparsegen:
(1) 
where is a componentwise transformation function applied on . Here denotes the th component of . The coefficient controls the regularization strength. For , the second term becomes negative 2 norm of . In addition to minimizing the error on projection of , Eq.1 tries to maximize the norm, which encourages larger probability values for some indices, hence moving the rest to zero. The above formulation has a closedform solution (see App. A.1 for solution details), which can be computed in time using the modified randomized median finding algorithm as followed in Duchi:2008:EPL:1390156.1390191 while solving the projection onto simplex problem.
The choices of both and can help control the cardinality of the support set , thus influencing the sparsity of . can help produce distributions with support ranging from full (uniform distribution when ) to minimum (hardmax when ). Let denote the support of sparsegen for a particular coefficient . It is easy to show: if , then there exists for an such that . In other words, if a sparser solution exists, it can be obtained by changing . The following result has an alternate interpretation for :
Result: The sparsegen formulation (Eq.1) is equivalent to the following, when (where ): .
The above result says that scaling by is equivalent to applying the negative 2 norm with coefficient when considering projection of onto the simplex. Thus, we can write:
(2) 
This equivalence helps us borrow results from sparsemax to establish various properties for sparsegen.
Jacobian of sparsegen: To train a model with sparsegen as an activation function, it is essential to compute its Jacobian matrix denoted by for using gradientbased optimization techniques. We use Eq.2 and results from martins2016 (Sec.
2.5) to derive the Jacobian for sparsegen by applying chain rule of derivatives:
(3) 
where is Jacobian of and . Here is an indicator vector whose entry is 1 if . is a matrix created using as its diagonal entries.
3.1 Special cases of Sparsegen: sparsemax, softmax and spherical softmax
Apart from , one can control the sparsity of sparsegen through as well. Moreover, certain choices of and help us establish connections with existing activation functions (see Sec.2). The following cases illustrate these connections (more details in App.A.2):
Example 1: (sparsegenexp): denotes elementwise exponentiation of , that is . Sparsegenexp reduces to softmax when , as it results in as per Eq.13 in App.A.2.
Example 2: (sparsegensq): denotes elementwise square of . As observed for sparsegenexp, when , sparsegensq reduces to spherical softmax.
Example 3 : : This case is equivalent to the projection onto the simplex objective adopted by sparsemax. Setting leads the regularized extension of sparsemax as seen next.
3.2 Sparsegenlin: Extension of sparsemax
3.3 Desirable properties for probability mapping functions
Let us enumerate below some properties a probability mapping function should possess:
1. Monotonicity: If , then . This does not always hold true for sumnormalization and spherical softmax when one or both of is less than zero. For sparsegen, both and should be monotonic increasing, which implies needs to be less than 1.
2. Full domain: The domain of should include negatives as well as positives, i.e. . Sumnormalization does not satisfy this as it is not defined if some dimensions of are negative.
3. Existence of Jacobian: This enables usage in any training algorithm where gradientbased optimization is used. For sparsegen, the Jacobian of should be easily computable (Eq.3).
4. Lipschitz continuity: The derivative of the function should be upper bounded. This is important for the stability of optimization technique used in training. Softmax and sparsemax are 1Lipschitz whereas spherical softmax and sumnormalization are not Lipschitz continuous. Eq.3 shows the Lipschitz constant for sparsegen is upper bounded by times the Lipschitz constant for .
5. Translation invariance: Adding a constant to every element in should not change the output distribution : . Sparsemax and softmax are translation invariant whereas sumnormalization and spherical softmax are not. Sparsegen is translation invariant iff for all there exist a such that . This follows from Eq.2.
6. Scale invariance: Multiplying every element in by a constant should not change the output distribution : . Sumnormalization and spherical softmax satisfy this property whereas sparsemax and softmax are not scale invariant. Sparsegen is scale invariant iff for all there exist a such that . This also follows from Eq.2.
7. Permutation invariance: If there is a permutation matrix , then . For sparsegen, the precondition is that should be a permutation invariant function.
8. Idempotence: , . This is true for sparsemax and sumnormalization. For sparsegen, it is true if and only if and .
In the next section, we discuss in detail about the scale invariance and translation invariance properties and propose a new formulation achieving a tradeoff between these properties.
3.4 Trading off Translation and Scale Invariances
As mentioned in Sec.1, scale invariance is a desirable property to have for probability mapping functions. Consider applying sparsemax on two vectors , . Both would result in as the output. However, ideally should have mapped to a distribution nearer to instead. Scale invariant functions will not have such a problem. Among the existing functions, only sumnormalization and spherical softmax satisfy scale invariance. While sumnormalization is only defined for positive values of , spherical softmax is not monotonic or Lipschitz continuous. In addition, both of these methods are also not defined for , thus making them unusable for practical purposes. It can be shown that any probability mapping function with the scale invariance property will not be Lipschitz continuous and will be undefined for .
A recent workDBLP:journals/corr/BrebissonV15 had pointed out the lack of clarity over whether scale invariance is more desired than the translation invariance property of softmax and sparsemax. We take this into account to achieve tradeoff between the two invariances. In the usual scale invariance property, scaling vector essentially results in another vector along the line connecting and the origin. That resultant vector also has the same output probability distribution as the original vector (See Sec. 3.3). We propose to scale the vector along the line connecting it with a point (we call it anchor point henceforth) other than the origin, yet achieving the same output. Interestingly, the choice of this anchor point can act as a control to help achieve a tradeoff between scale invariance and translation invariance.
Let a vector be projected onto the simplex along the line connecting it with an anchor point , for (See Fig.3a for ). We choose
as the point where this line intersects with the affine hyperplane
containing the probability simplex. Thus, is set equal to , where (we denote it as as is a function of ). From the translation invariance property of sparsemax, the resultant mapping function can be shown equivalent to considering in Eq.1. We refer to this variant of sparsegen assuming and as sparsecone.Interestingly, when the parameter , sparsecone reduces to sumnormalization (scale invariant) and when , it is equivalent to sparsemax (translation invariant). Thus the parameter acts as a control taking sparsecone from scale invariance to translation invariance. At intermediate values (that is, for ), sparsecone is approximate scale invariant with respect to the anchor point . However, it is undefined for where (beyond the black dashed line shown in Fig.3a). In this case the denominator term of (that is, ) becomes negative destroying the monotonicity of . Also note that sparsecone is not Lipschitz continuous.
3.5 Proposed Solution: Sparsehourglass
To alleviate the issue of monotonicity when , we choose to restrict applying sparsecone only for the positive halfspace . For the remaining negative halfspace , we define a mirror point function to transform to a point in , on which sparsecone can be applied. Thus the solution for a point in the negative halfspace is given by the solution of its corresponding mirror point in the positive halfspace. This mirror point function has some necessary properties (see App. A.4 for details), which can be satisfied by defining : . Interestingly, this can alternatively be achieved by choosing , where is a slight modification of given by . This leads to the definition of a new probability mapping function (which we call sparsehourglass):
(5) 
Like sparsecone, sparsehourglass also reduces to sparsemax when . Similarly, for sparsehourglass leads to a corrected version of sumnormalization (we call it sum normalization++ ), which works for the negative domain as well unlike the original version defined in Sec.2. Another advantage of sparsehourglass is that it is Lipschitz continuous with Lipschitz constant equal to (proof details in App.A.5). Table.1 summarizes all the formulations seen in this paper and compares them against the various important properties mentioned in Sec.3.3. Note that sparsehourglass is the only probability mapping function which satisfies all the properties. Even though it does not satisfy both scale invariance and translation invariance simultaneously, it is possible to achieve these separately through different values of parameter, which can be decided independent of .
Function  Idempotence  Monotonic  Translation Inv  Scale Inv  Full Domain  Lipschitz 
Sum Normalization  ✔  ✗  ✗  ✔  ✗  
Spherical Softmax  ✗  ✗  ✗  ✔  ✗  
Softmax  ✗  ✔  ✔  ✗  ✔  1 
Sparsemax  ✔  ✔  ✔  ✗  ✔  1 
Sparsegenlin  ✓  ✔  ✔  ✗  ✔  
Sparsegenexp  ✗  ✔  ✗  ✗  ✔  
Sparsegensq  ✗  ✗  ✗  ✗  ✔  
Sparsecone  ✔  ✗  ✓  ✓  ✗  
Sparsehourglass  ✔  ✔  ✓  ✓  ✔  
Sum Normalization++  ✔  ✔  ✗  ✔  ✔ 
4 Sparsity Inducing Loss Functions for Multilabel Classification
An important usage of such sparse probability mapping functions is in the output mapping of multilabel classification models. Typical multilabel problems have hundreds of possible labels or tags, but any single instance has only a few tags Kapoor+12 . Thus, a function which takes in a vector in and outputs a sparse version of the vector is of great value.
Given training instances , we need to find a model function that produces score vector over the label space, which on application of (the sparse probability mapping function in question) leads to correct prediction of label vector . Define , which is a probability distribution over the labels. Considering a loss function and representing , a natural way for training using is to find a function that minimises the error below over a hypothesis class :
In the prediction phase, for a test instance , one can simply predict the nonzero elements in the vector where is the minimizer of the above training objective .
For all cases where produces sparse probability distributions, one can show that the training objective above is highly nonconvex in , even for the case of a linear hypothesis class . However, when we remove the requirement of being there in training part, and use a loss function which works with and , we can obtain a system which is convex in . We, thus, design a loss function such that only if . To derive such a loss function, we proceed by enumerating a list of constraints which will be satisfied by the zeroloss region in the dimensional space of the vector . For sparsehourglass, the closedform solution is given by (see App.A.1). This enables us to list down the following constraints for zero loss: (1) , and (2) . The value of the loss when any such constraints is violated is simply determined by piecewise linear functions, which lead to the following loss function for sparsehourglass:
(6) 
It can be easily proved that the above loss function is convex in using the properties that both sum of convex functions and maximum of convex functions result in convex functions. The above strategy can also be applied to derive a multilabel loss function for sparsegenlin:
(7) 
The above loss function for sparsegenlin can be used to derive a multilabel loss for sparsemax by setting (which we use in our experiments for “sparsemax+hinge”) . The piecewiselinear losses proposed in this section based on violation of constraints are similar to the wellknown hinge loss, whereas the sparsemax loss proposed by martins2016 (which we use in our experiments for “sparsemax+huber”) has connections with Huber loss. We have shown through our experiments in next section, that hinge loss variants for multilabel classification work better than Huber loss variants.
5 Experiments and Results
Here we present two sets of evaluations for the proposed probability mapping functions and loss functions. First, we apply them on the multilabel classification task studying the effect of varying label density in synthetic dataset, followed by evaluation on real multilabel datasets. Next, we report results of sparse attention on NLP tasks of machine translation and abstractive summarization.
5.1 Multilabel Classification
We compare the proposed activations and loss functions for multilabel classification with both synthetic and real datasets. We use a linear prediction model followed by a loss function during training. During test time, the corresponding activation is directly applied to the output of the linear model. We consider the following activationloss pairs: (1) softmax+log: KLdivergence loss applied on top of softmax outputs, (2) sparsemax+huber: multilabel classification method from martins2016 , (3) sparsemax+hinge: hinge loss as in Eq.7 with is used during training compared to Huber loss in (2), and (4) sparsehg+hinge: for sparsehourglass (in short sparsehg), loss in Eq.6 is used during training. Please note as we have a convex system of equations due to an underlying linear prediction model, applying Eq.7 in training and applying sparsegenlin activation during test time produces the same result as sparsemax+hinge. For softmax+log, we used a threshold , above which a label is predicted to be “on”. For others, a label is predicted “on” if its predicted probability is nonzero. We tune hyperparams for sparsehg+hinge and for softmax+log using validation set.
5.1.1 Synthetic dataset with varied label density
We use scikitlearn for generating synthetic datasets (details in App.A.6). We conducted experiments in three settings: (1) varying mean number of labels per instance, (2) varying range of number of labels and, (3) varying document length. In the first setting, we study the ability to model varying label sparsity. We draw number of labels uniformly at random from set where 29 is mean number of labels. For the second setting we study how these models perform when label density varies across instances. We draw uniformly at random from set . Parameter controls variation of label density among instances. In the third setting we experiment with different document lengths, we draw from Poisson with mean and vary document length from 200 to 2000. In first two settings document length was fixed to .
We report Fscore
^{1}^{1}1Microaveraged score. and JensenShannon divergence (JSD) on test set in our results.Fig.5 shows Fscore on test sets in the three experimental settings. We can observe that sparsemax+hinge and sparsehg+hinge consistently perform better than sparsemax+huber in all three cases, especially the label distributions are sparser. Note that sparsehg+hinge performs better than sparsemax+hinge in most cases. From empirical comparison between sparsemax+hinge and sparsemax+huber, we can conclude that the proposed hinge loss variants are better in producing sparser and and more accurate predictions. This observation is also supported in our analysis of sparsity in outputs (see Fig.4  lower the curve the sparser it is  this is analysis is done corresponding to the setting in Fig.4(a)), where we find that hinge loss variants encourage more sparsity. We also find the hinge loss variants are doing better than softmax+log in terms of the JSD metric (details in App.A.7.1).
5.1.2 Real Multilabel datasets
We further experiment with three real datasets^{2}^{2}2Available at http://mulan.sourceforge.net/datasetsmlc.html for multilabel classification: Birds, Scene and Emotions. The experimental setup and baselines are same as that for synthetic dataset described in Sec.5.1.1. For each of the datasets, we consider only those examples with atleast one label. Results are shown in Table 3 in App.A.7.2. All methods give comparable results on these benchmark datasets.
5.2 Sparse Attention for Natural Language Generation
Here we demonstrate the effectiveness of our formulations experimentally on two natural language generation tasks: neural machine translation and abstractive sentence summarization. The purpose of these experiments are two fold: firstly, effectiveness of our proposed formulations sparsegenlin and sparsehourglass in attention framework on these tasks, and secondly, control over sparsity leads to enhanced interpretability. We borrow the encoderdecoder architecture with attention (see Fig.7 in App.A.8). We replace the softmax function in attention by our proposed functions as well as sparsemax as baseline. In addition we use another baseline where we tune for the temperature in softmax function. More details are provided in App.A.8.
Experimental Setup: In our experiments we adopt the same experimental setup followed by DBLP:journals/corr/NiculaeB17 on top of the OpenNMT framework opennmt17
. We varied only the control parameters required by our formulations. The models for the different control parameters were trained for 13 epochs and the epoch with the best validation accuracy is chosen as the best model for that setting. The best control parameter for a formulation is again selected based on validation accuracy. For all our formulations, we report the test scores corresponding to the best control parameter in Table
2.Neural Machine Translation: We consider the FREN language pair from the NMTBenchmark project and perform experiments both ways. We see (refer Table.2) that sparsegenlin surpasses BLEU scores of softmax and sparsemax for FREN translation, whereas sparsehg formulations yield comparable performance. Quantitatively, these metrics show that adding explicit controls do not come at the cost of accuracy. In addition, it is encouraging to see (refer Fig.8 in App.A.8) that increasing for sparsegenlin leads to crisper and hence more interpretable attention heatmaps (the lesser number of activated columns per row the better it is). We have also analyzed the average sparsity of heatmaps over the whole test dataset and have indeed observed that larger leads to sparser attention. Abstractive Summarization: We next perform our experiments on abstractive summarization datasets like Gigaword, DUC2003 & DUC2004 and report ROUGE metrics. The results in Table.2 show that sparsegenlin stands out in performance with other formulations closely following and comparable to softmax and sparsemax. It is also encouraging to see that all the models trained on Gigaword generalizes well on other datasets DUC2003 and DUC2004. Here again the control leads to more interpretable attention heatmaps as shown in Fig.9 in App.A.8 and we have also observed the same with average sparsity of heatmaps over the test set.
Attention  Translation  Summarization  

FREN  ENFR  Gigaword  DUC 2003  DUC 2004  
BLEU  BLEU  R1  R2  RL  R1  R2  RL  R1  R2  RL  
softmax  36.38  36.00  34.80  16.64  32.15  27.95  9.22  24.54  30.68  12.24  28.12 
softmax (with temp.)  36.63  36.08  35.00  17.15  32.57  27.78  8.91  24.53  31.64  12.89  28.51 
sparsemax  36.73  35.78  34.89  16.88  32.20  27.29  8.48  24.04  30.80  12.01  28.04 
sparsegenlin  37.27  35.78  35.90  17.57  33.37  28.13  9.00  24.89  31.85  12.28  29.13 
sparsehg  36.63  35.69  35.14  16.91  32.66  27.39  9.11  24.53  30.64  12.05  28.18 
6 Conclusions and Future Work
In this paper, we investigated a family of sparse probability mapping functions, unifying them under a general framework. This framework helped us to understand connections to existing formulations in the literature like softmax, spherical softmax and sparsemax. Our proposed probability mapping functions enabled us to provide explicit control over sparsity to achieve higher interpretability. These functions have closedform solutions and subgradients can be computed easily. We have also proposed convex loss functions, which helped us to achieve better accuracies in the multilabel classification setting. Application of these formulations to compute sparse attention weights for NLP tasks also yielded improvements in addition to providing control to produce enhanced interpretability. As future work, we intend to apply these sparse attention formulations for efficient read and write operations of memory networks NIPS2015_5846 . In addition, we would like to investigate application of these proposed sparse formulations in knowledge distillation and reinforcement learning settings.
Acknowledgements
We thank our colleagues in IBM, Abhijit Mishra, Disha Shrivastava, and Parag Jain for the numerous discussions and suggestions which helped in shaping this paper.
References
 (1) M. Aly. Survey on multiclass classification methods. Neural networks, pages 1–9, 2005.

(2)
John S. Bridle.
Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition.
In Françoise Fogelman Soulié and Jeanny Hérault, editors, Neurocomputing, pages 227–236, Berlin, Heidelberg, 1990. Springer Berlin Heidelberg.  (3) Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.

(4)
B. Gao and L. Pavel.
On the properties of the softmax function with application in game theory and reinforcement learning.
ArXiv eprints, 2017.  (5) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, 2015.
 (6) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Francis R. Bach and David M. Blei, editors, ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 2048–2057. JMLR.org, 2015.
 (7) K. Cho, A. Courville, and Y. Bengio. Describing multimedia content using attentionbased encoderdecoder networks. IEEE Transactions on Multimedia, 17(11):1875–1886, Nov 2015.

(8)
Alexander M. Rush, Sumit Chopra, and Jason Weston.
A neural attention model for abstractive sentence summarization.
InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing
, pages 379–389, Lisbon, Portugal, September 2015. Association for Computational Linguistics.  (9) Preksha Nema, Mitesh Khapra, Anirban Laha, and Balaraman Ravindran. Diversity driven attention model for querybased abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, August 2017. Association for Computational Linguistics.
 (10) Francis Bach, Rodolphe Jenatton, and Julien Mairal. Optimization with SparsityInducing Penalties (Foundations and Trends(R) in Machine Learning). Now Publishers Inc., Hanover, MA, USA, 2011.
 (11) Mohammad S. Sorower. A literature survey on algorithms for multilabel learning. 2010.
 (12) Pascal Vincent, Alexandre de Brébisson, and Xavier Bouthillier. Efficient exact gradient update for training deep networks with very large sparse targets. In Proceedings of the 28th International Conference on Neural Information Processing Systems  Volume 1, NIPS’15, pages 1108–1116, Cambridge, MA, USA, 2015. MIT Press.
 (13) Alexandre de Brébisson and Pascal Vincent. An exploration of softmax alternatives belonging to the spherical loss family. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
 (14) André F. T. Martins and Ramón F. Astudillo. From softmax to sparsemax: A sparse model of attention and multilabel classification. In Proceedings of the 33rd International Conference on International Conference on Machine Learning  Volume 48, ICML’16, pages 1614–1623. JMLR.org, 2016.
 (15) John Duchi, Shai ShalevShwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the l1ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pages 272–279, New York, NY, USA, 2008. ACM.
 (16) Vlad Niculae and Mathieu Blondel. A regularized framework for sparse and structured neural attention. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3340–3350. Curran Associates, Inc., 2017.
 (17) Ashish Kapoor, Raajay Viswanathan, and Prateek Jain. Multilabel classification using bayesian compressed sensing. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 2645–2653. Curran Associates, Inc., 2012.
 (18) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. Opennmt: Opensource toolkit for neural machine translation. CoRR, abs/1701.02810, 2017.
 (19) Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. Endtoend memory networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2440–2448. Curran Associates, Inc., 2015.
Appendix A Supplementary Material
a.1 ClosedForm Solution of Sparsegen
Formulation is given by:
(8) 
The Lagrangian of the above formulation is:
(9) 
Defining the KarushKuhnTucker conditions with respect to optimal solutions ():
(10) 
(11) 
(12) 
From Eq.12, if we want for certain , then we must have . This implies (according to Eq. 10). This leads to . For , where , we have (as ). From Eq. 11 we obtain , which leads to .
Proposition 0.1
The closedform solution of sparsegen is as follows ():
(13) 
where is the threshold which makes . Let be the sorted coordinates of . The cardinality of the support set is given by . Then can be obtained as,
a.2 Special cases of Sparsegen
Example 1: (sparsegenlin):
When , sparsegen reduces to a regularized extension of sparsemax. This is translation invariant, has monotonicity and can work for full domain. It is also Lipschitz continuous with the Lipschitz constant being . The visualization of this variant is shown in Fig.2.
Example 2: (sparsegenexp):
here means elementwise exponentiation of , that is . For where all , sparsegenexp leads to sparser solutions than sparsemax.
Checking with properties from Sec.3.3, sparsegenexp satisfies monotonicity as is a monotonically increasing function. However, as is not Lipschitz continuous, sparsegenexp is not Lipschitz continuous.
It is interesting to note that sparsegenexp reduces to softmax when is dependent on and equals , as it results in and according to Eq.13.
Example 3: (sparsegensq):
Unlike sparsegenlin and sparsegenexp, sparsegensq does not satisfy the monotonicity property as is not monotonic. Also sparsegensq has neither translation invariance nor scale invariance. Moreover, as is not Lipschitz continuous, sparsegensq is not Lipschitz continuous. Additionally, when depends on and , sparsegensq reduces to spherical softmax.
Example 4: (sparsegenlog):
here means elementwise natural logarithm of , that is . This is scale invariant but not translation invariant. However, it is not defined for negative values or zero values of .
a.3 Derivation of Sparsecone
Let us project a point onto the simplex along the line connecting with . The equation of this line is . If the point of intersection of the line with the probability simplex (denoted by ) is represented as , then it should have the property: , which leads to . Also, condition must be satisfied, as it should lie on the probability simplex. Thus, the required point can be obtained by solving the following:
(14)  
(15)  
(16) 
a.4 Derivation of Sparsehourglass
The above formulation fails for such that . For example, it fails for , where (the solution will give greater probability mass to 2 compared to 1), that is . To make it work for such a , we can find a mirror point satisfying the following properties : (1) , and (2) , and (3) . Let be given by (as seen earlier) and be defined by . Can we find which satisfies the mirror point properties above? Property (3) is true iff . Using property (2) we get, . From definition of sparsecone, . Using property (1), we get . As is a function of , let it be represented as . Thus, sparsehourglass is defined as follows:
(17)  
(18)  
(19) 
a.5 Proof for Lipschitz constant of Sparsehourglass
Lipschitz constant of composition of functions is bounded above by the product of Lipschitz constants of functions in the composition. Applying this principle on Eq.18 we find an upper bound for the Lipschitz constant of sparsehg (denoted as ) as the product of Lipschitz constant of sparsemax (denoted as ) and Lipschitz constant of (which we denote as ).
(20) 
We use the property that Lipschitz constant of a function is the largest matrix norm of Jacobian of that function. For sparsemax that would be,
where, is the Jacobian of sparsemax at given value . From the term given in Sec.3, whose matrix norm is
. We use the property that, for any symmetric matrix, matrix norm is the largest absolute eigenvalue, and the eigenvalues of
are and . Hence, .For , let us look at the Jacobian of , which is given as
(21) 
where
denotes identity matrix. Eigen values of
are and , and the largest among them is clearly . As , all the eigen values of are greater than 0, making a positive definite matrix. We now use the property that for any positive definite matrix, matrix norm is the largest eigenvalue. From Eq.19, we see that the largest eigen value of is which assumes the largest value when . The highest value of is ; therefore, . Thus,It turns out this is also a tight bound; hence, .
a.6 Synthetic dataset creation
We use scikitlearn for generating synthetic datasets with 5000 instances split across train (0.5), val (0.2) and test (0.3). Each instance is a multilabel document with a sequence of words. Vocabulary size and number of labels are fixed to . Each instance is generated as follows: We draw number of true labels from either a discrete uniform distribution. Then we sample labels from without replacement and sample number of words (document length) from the mixture of sampled labelspecific distribution over words.
a.7 More results on multilabel classification experiments
a.7.1 Synthetic Dataset
Fig.6 shows JSD between normalized true label and predicted distribution. Though sparsemax+huber is performing overall the best, in the case when number of labels is small we could see sparsehg+hinge and sparsemax+hinge performing as good as sparsemax+huber. In general we could see that sparsehg+hinge is performing better than sparsemax+hinge, which in turn is better than softmax+log.
a.7.2 Real Datasets
Activation + Loss  SCENE  EMOTIONS  BIRDS 

softmax+log  0.71  0.65  0.43 
sparsemax+huber  0.70  0.63  0.42 
sparsemax+hinge  0.68  0.65  0.41 
sparsehg+hinge  0.69  0.65  0.41 
a.8 Controlled Sparse Attention
Here we discuss attention mechanism with respect to encoderdecoder architecture as illustrated in Fig. 7. Lets say we have a sequence of inputs (possibly sequence of words) and their corresponding word vectors sequence be . The task will be to produce a sequence of words as output. Here an encoder RNN encodes the input sequence into a sequence of hidden state vectors . A decoder RNN generates its sequence of hidden states by considering different while generating different . This is known as hard attention. This approach is known to be nondifferentiable as it is based on a sampling based approach [6]. As an alternative solution, a softer version (called soft attention) is more popularly used. It involves generating a score vector based on relevance of encoder hidden states with respect to current decoder state vector using Eq.22. The score vector is then transformed to a probability distribution using the Eq.23. Considering the attention model parameters [, , ], we define the following:
(22)  
(23) 
Following the approach by [14], we replace Eq.23 with our formulations, namely, sparsegenlin and sparsehourglass. This enables us to apply explicit control over the attention weights (with respect to sparsity and tradeoff between scale and translation invariance) produced for the sequence to sequence architecture. This falls in the sparse attention paradigm as proposed in the earlier work. Unlike hard attention, these formulations lead to easy computation of subgradients, thus enabling backpropagation in this encoderdecoder architecture composed RNNs. This approach can also be extended to other forms of nonsequential input (like images) as long as we can compute the score vector from them, possibly using a different encoder like CNN instead of RNN.
a.8.1 Neural Machine Translation
The first task we consider for our experiments is neural machine translation. Our aim is to see how our techniques compare with softmax and sparsemax with respect BLEU metric and interpretability. We consider the frenchenglish language pair from the NMTBenchmark project^{3}^{3}3http://scorer.nmtbenchmark.net/ and perform experiments both ways using controlled sparse attention. The dataset has around 1M parallel training instances along with equal validation and test sets of 1K parallel instances.
We find from our experiments (refer Table.2) that sparsegenlin surpasses the BLEU scores of softmax and sparsemax for FREN translation, whereas the other formulations yield comparable performance. On the other hand, for ENFR translation, softmax is still better than others. Quantitatively, these metrics show that adding explicit controls do not come at the cost of accuracy. In addition, it is encouraging to see (refer Fig.8) that increasing for sparsegenlin leads to crisper and hence more interpretable attention heatmaps (the lesser number of activated columns per row the better it is).
a.8.2 Abstractive Summarization
We next perform our experiments on abstractive summarization datasets like Gigaword, DUC2003 & DUC2004^{4}^{4}4https://github.com/harvardnlp/sentsummary. This dataset consists of pairs of sentences where the task is to generate the second sentence (news headline) as a summary of the larger first sentence. We trained our models on nearly 4M training pairs of Gigaword and validated on 190K pairs. Then we reported the test scores according to ROUGE metrics on 2K test instances of Gigaword, 624 instances of DUC2003 and 500 instances of DUC2004.
The results in Table.2 show that sparsegenlin stands out in performance amongst other formulations with the other formulations closely following and comparable to softmax and sparsemax. It is also encouraging to see that all the models trained on Gigaword generalizes well on other datasets DUC2003 and DUC2004. Here again the control leads to more interpretable attention heatmaps as shown in Fig.9.