1 Introduction
A vast array of Deep Learning (DL) approaches have been proven successful in many different learning tasks in recent years. One of the key strength of DL models is their ability to automatically learn a representation of the features composing a data set. Deep Neural Networks (DNNs) represent the foremost and widely spread class of DL models. Broadly speaking, DNNs are subsymbolic ML approaches that are very good at extracting the useful information contained in large data sets. One of the advantages of DL techniques is that, in general, they do not rely on stringent assumptions on the distribution of the underlying data and on the function to be learned or approximated. This allows them to be applied in many different areas with very good results, without significant changes to the DNNs’ structure and training algorithm.
However, there are contexts where purely datadriven models are not an ideal fit, for example when scarce data of good quality and very difficult learning tasks. In such situations, a great boost in the performance of neural networks (and ML models in general) can be obtained through the exploitation of domain knowledge, e.g. problemspecific information that can be used to improve the DL model and/or simplify the training process (for instance, structured data, knowledge about the data generation process, domain experts experience, etc). Hence, it makes sense to take advantage of domain information to improve the performance of DL models, so that they do not have to start from scratch while dealing with difficult learning tasks. In other words, why learn again something that you already know?.
In general, the integration of domain knowledge in DL models is a multifaceted topic that has been extensively explored from multiple angles, with a wide range of approaches dealing with different types of domain knowledge and a very large number of different target DL models to be boosted. In this paper we do not claim to provide a comprehensive and exhaustive overview of all the methodologies proposed in the literature (for a broad overview [32] provide a great taxonomy), but we want to focus on a particular class of techniques for injecting prior information in DL models. Specifically, we consider prior knowledge that can be expressed in the form of constraints and as target models to be improved with this information we restrict our interest to DNNs. We take into account constraints of different nature, ranging from firstorder and propositional logic predicates to linear and nonlinear equations. In the scope of this paper the constraints can represent relationships between the input features, relationships between input and output features, bounds on the output variables. We consider both hard constraints (which set conditions that must be satisfied) and soft constraints (which set conditions with an associated penalty, in case they are not satisfied).
The paper is structured as follows. Section 2 is the core of the survey and presents research items from the literature for the injection of domain knowledge (expressed as constraints) in DNNs, grouping the related works in five macroareas. Section 3 highlights analogies between the surveyed papers and other, closely related, fields within the DL area. Section 4 discusses common trends among the injection approaches and provides observations and insights. Finally, Section 5 summarizes and concludes the paper.
2 Domain Knowledge Injection Approaches
In this section we discuss recent methods for injecting prior information in DNNs. We assume to have a learning task where the learner is trying to approximate a function that maps an input to an output ; the training set is composed by examples . We consider domain knowledge that can be expressed as a set of constraints, or predicates, . Domain knowledge can be represented by algebraic equations, such as linear and nonlinear equations, equality and inequality constraints, logic formulas. These predicates can be applied to different groups of variables: 1) they can involve only the input features , e.g. it is known that the input features of wellformed data instances share particular properties, or are linked by a set of precise relations; 2) the constraints can concern only the output variables , e.g. the output of the network must fall within a determined range; 3) the condition can involve both input and output , for instance, the real function to be approximated could be monotonic (
). As a particular case of constraintbased information we also include domain knowledge expressed in the form of graphs (e.g. knowledge graphs), as graphs can be decomposed as collections of simpler constraints encapsulating the relations between nodes and edges. With these assumptions, the constraintexpressed domain knowledge can be integrated in the DNNs in multiple ways. We classify literature works in five branches, based on the injection mechanism: 1) feature space, 2) hypothesis space, 3) data augmentation, 4) regularization schemes, 5) constraints learning. Each subsection is devoted to a branch.
2.1 Feature Space
DL models’ performance strongly depends on the quality of the available training data, either labeled or not; the features in the data define the feature space whose implicit information is extracted by DNNs. The shape of the feature space is a critical issue for the performance of DL models and the ease of their training. For instance, feature engineering is a common method for improving the accuracy of purely datadriven ML models by selecting useful features and/or transforming the original ones to facilitate the learner’s task. In general, this is a difficult problem and requires much effort, both from system expert and ML practitioners [14]. In recent years, several research avenues studied the possibility to automatically explore the feature space in order to extract only the most relevant features, with most of the approaches belonging to the AutoML area [26]
[15]. The majority of current feature engineering methods aim at selecting the optimal features for a specific learning task and generally tend to reduce the feature space, by selecting only the most relevant features (feature extraction or feature compression). Furthermore, these methods are generally purely datadriven and require large amounts of data; although they aim at exploring the feature space in an efficient way, when the number of features is large this becomes a non trivial problem.
A relatively unexplored direction is the use of domain knowledge to create novel features that render explicit the information hidden in the raw data. This is a form of feature space extension with the purpose of highlighting the prior knowledge embedded in the original features but not easily extractable by a neural network. In practice, the approaches proposed in this area work on the original input features to generate an extended feature set , where is the set composed by the additional features ( is the number of added features). The new features are computed as combination (linear or nonlinear equations) of the original ones, depending on the domain constraints. For example, [1] enrich the feature space using the domainspecific information encoded by knowledge graphs. The relationships explicitly described by the knowledge graph represent constraints (soft and hard) among the original input features and can be used to create additional features, which then improve the accuracy of a supervised DNN. In a similar fashion [23]
increase the feature space with domainbased features and boost the performance of a DL model. They do not use a knowledge graph to obtain the additional features but rather an ensemble of decision trees solving a classification task on a subset of the data devoted to the training of the DL model. Each tree learns domainspecific information and partakes to the final prediction through its own score; these scores are then added as additional features to the training set. Another method to incorporate domain knowledge by enriching the feature space is discussed by
[3], which study the improvement of a DNN for the prediction of soccer match outcomes obtained through the addition of domaininspired features. The training set is a time series composed of matches among two different teams and related outcomes; their approach consists in adding a set of novel features that encapsulate i) the rating of the teams involved in the match and ii) the results obtained in the last matches by each team. These novel features are encoded as a set of linear and nonlinear equations applied to the original input .2.2 Hypothesis Space
A DNN can be characterized by its position in the so called hypothesis space
, namely the multidimensional space covering its structure and its hyperparameters. The architecture of a NN is a very relevant factor for determining its performance on different learning tasks. The hypothesis space has been explored with implicit guidance provided by domain knowledge for many years, as attested by the introduction of convolutional networks, whose structure is based on the locality assumption (e.g. pixels close to each other in an image are related). Implicit knowledge about temporal locality has also lead to a wide range of architectures targeted at handling time series and sequences, for instance recurrent NNs (RNNs), LongShort Term Memory NNs (LSTMs), Temporal Convolutional Networks
[2].In more recent years, a remarkable research effort has been devoted to exploring DNN architectures optimized for circumstances where the domain knowledge can be expressed in the form of graphs, precisely with a DL model called Graph Neural Network (GNNs) [27]; Graph Convolutional Networks (GCNN) [16] were introduced as well to exploit the same type of graphexpressible prior information. GNNs have been used in several fields [37], owning to their capability to deal with data whose structure can be described via graphs, thanks to a generalization in the spectral domain of the convolutional layers found in many deep learning networks. GCNNs most common applications involve semisupervised classification tasks, with the goal of predicting the class of unlabeled nodes in a graph – a case of graph learning. Using prior information expressible as graph has been proposed also to devise other types of NNs, namely networks whose structure resembles the LSTM’s one but where the nodes connections are determined by the prior information [20]. Similarly, [13] combine the temporal structure of RNNs with spatialbased information, namely the domain information is encoded as sequences of actions each one characterized by a time and a position in space, which are then represented through an extended RNN. As mentioned earlier, domain knowledge represented as a graph can be expressed by sets of constraints, which encode the relationships between the nodes through the edges. Currently, the vast majority of information injection methods in this area (and all those listed in this section) operate on the input features , which can be cast as concepts and related connections within the graph structure.
2.3 Data Augmentation
Besides working on the feature and the hypothesis space, another mechanism to infuse domain knowledge in DNNs regards the training data, mainly in the forms of creating exnovo entire training sets or adding new examples to existing ones following criteria defined by the domain knowledge (for instances, examples respecting certain relationships among the input features). We refer to these methodologies with the term data augmentation. Data augmentation approaches based on prior information (constraints) have been started to be explored in recent years, especially to cope with data sets of limited size and the related issue of poor generalization performance [12]. Data augmentation techniques have a strong history of success in the context of imagebased learning tasks (e.g. image classification) [33]. For image classification tasks, the training set can be augmented by applying a plethora of transformations to the images in the original training set, thus feeding the NN to be trained with a more varied set of examples. The selection of the best transformations to apply to augment the available data is a process that is typically guided by information obtained via domain experts. In particular, constraintbased domain information can be used to augment the available data, e.g. linear and nonlinear functions such as rotation, distortion, flipping, etc. For instance, [4]
propose a data augmentation technique to train a LSTM model for classifying chemical molecules. Each molecule can be described as a combination of its composing elements, encoded as a concatenation of strings. A single molecule has multiple possible encoding strings; the authors propose to expand the training set by enumerating the chemically allowed combinations for each data point/molecule. The enumeration takes place via a heuristic algorithm that enforces on the generated examples the same chemical properties of the original data points; these properties are encoded as a collection of constraints (linear combinations representing admissible chemical properties) among the input features (the concatenated strings). In this case the constraints involve both input features
and the output , as the relationships among are used to generate novel data points with the same output value (the label).[24] propose a different methodology for data set augmentation: they consider feature sideinformation, domain knowledge describing feature properties and/or feature relations. The feature sideinformation are expressed as a matrix whose rows represent the prior information associated to each feature; a similarity function is introduced to compute the pairwise similarity of different features. An augmented training data point is obtained from an original example by applying a transformation that preserves the associated label and perturbs the values of similar features. Again, the focus is on constraints among input features . Similarly, [31]
devise a data augmentation method based on similarity score among features, for the purpose of improving the results of a CNNs used for sentiment analysis. In this case the original data set is composed by labeled sentences (the training examples); each sentence can be decomposed in a set of sentiment terms, i.e. the features. The author introduce a similarity measure for the sentiments and then propose an algorithm (based on quadratic programming) to generate similar sentences from the original ones, based on the sentiment similarity score; the augmented similar examples are annotated with the same label as the corresponding original training points.
2.4 Regularization Schemes
Regularization is a widely known method to avoid overfitting in machine learning models, but it can also be exploited to inject domain knowledge. To do so, some form of prior insight, i.e constraints, is translated into a regularization term able to measure the level of consistency with the domain knowledge and guide the loss optimization process. More in general, the loss function can be defined as follows:
where represents the true loss, e.g. MSE, while represents the regularization term given a set of constraints ; these terms are weighted, respectively, with and ; the choice of these weight parameters is nontrivial and might consistently affect the final outcome ([10] address this problem exploiting lagrangian duality). In the last few years many authors worked on this class of methods proposing a variety of approaches to the problem; for instance, [25], developed a DNN, called domain adapted neural network (DANN), which exploits a mathematical formulation of the constraints , namely approximation and monotonicity constraints, whose degree of satisfaction is used as regularization term. A more general approach is introduced by [8] with Semantic Based Regularization (SBR), which provides a set of rules to translate firstorder logic formulas into fuzzy constraints, then used as penalty factors in the loss function. This is achieved by introducing a tnorm function, which can be defined in different ways, according to the desired interpretation of the domain constraints involved in the regularization, allowing for an adaptable tool. [22] present a strongly related method denoted LYRICS, a framework that introduces a declarative language to express the domain knowledge and enforces it using SBR on top of DNNs, allowing a very flexible representation of the constraints and the prior information. A stochastic approach with semantic loss is proposed by [35]
, which use a regularization term given by the probability of generating a state satisfying the domainbased constraints; during the training process, the presence of states not satisfying the desired constraints is penalized acting on the loss function.
[29] propose a SBRinspired technique for injecting domain constraints in a DNN used to extend a partial variable assignment for the Partial Latin Square problem, specifically finding feasible solutions that respect domain constraints.2.5 Learning with Constraints
DL models are often employed in specific tasks, e.g. to find solutions to optimization problems. Usually these approaches follow a twostep process where: first the model is trained with the observed data and then used to approximate an aspect of the optimization problem, e.g. the cost function. In these cases, the model’s accuracy is not the only way to measure the performance of the approach, as task related metrics might be more relevant. Lately a new paradigm emerged, namely decisionfocused learning (or endtoend learning), where DNNs are trained to directly produce good results for the end goal of the whole task. In this context, prior information for the specific optimization task is introduced to guide the training of the DNNs used to approximate the needed functions. This class of approaches shares some aspects with the regularization ones, as the penalty terms in the loss function can be applied also for endtoend learning. However, since in this case the DNNs are used within larger optimization models, the mechanism to influence their behaviour is different. In general, the DNNs for decisionfocused learning produce a proxy (intermediate) solution for the target task, given a fixed parametrization of the network, then optimized by the loss function; however, as the solution space might be not continuous, differentiability issues might arise, which are in general solved through transformations.
A first example of this method can be seen in [34]
, where it is applied to combinatorial optimization with the introduction of a continuous relaxation of the proxy solution (which is discrete), namely a
convex hull; the loss is then computed by combining the gradient with respect to the decision variables and the gradient of the model with respect to its own parameters. A slightly different contribution is given by [9], which apply endtoend learning on a stochastic optimization problem; however, in this approach the focus is on the training process, where the gradient descent updates the network parameters using two functions: one computing the number of constraint violations and one represented by a classic loss function. If the proxy solution does not satisfy the constraints, the parameters are updated using the first function, otherwise using loglikelihood loss.[17] offer a totally different method to tackle problems where prior knowledge is expressed with firstorder logic formulas; in this case the prior information is embedded directly in the DNN, where nodes, called named neurons, are labeled to mimic the logic elements in the formula; these are used to build constrained neural layers
, which produce their output according to the truth value of each named neuron in the layer.
Recently, [19] introduced an approach similar to the endtoend learning paradigm, but more general, called DL2. DL2 is a framework for explicitly embedding constraints in DNNs, specifically, it allows to translate logical formulas into loss functions, i.e. by defining recursively the corresponding mathematical equation for each term in the formula; the training is then carried out using projected gradient descent. The model is also provided with a SQLlike query language, which allows to interrogate the network on a specific input, in order to: check if the constraints are satisfied and train the model for input outside of the set of observed data; basically, the network can be challenged with queries that help improve its accuracy beyond the data available for the training – this method is called global training.
3 Related Areas
Another field where prior information has been exploited to improve performance is reinforcement learning. In this area as well the knowledge domain information can be used to improve the models and/or transform very difficult problems into more tractable ones; for example, by creating good initial conditions for the training algorithm, thus decreasing the number of required training examples and therefore providing a warm start for the reinforcement learning process [28]. Prior information has been successfully applied with remarkable benefits in various context well suited for reinforcement learning [18]. However, an important aspect has to be considered while injecting domain knowledge in this setting: uncertainty in the domain knowledge can greatly hinder the learning process if wrong decisions (caused by uncertain or incomplete information on the system state) are taken at the beginning of the learning phase [30]. Another area where domain knowledge has been shown to be beneficial is the initialization of the weights of a DNN. For instance, [36] describe a semisupervised pretraining strategy for DL models to predict the behaviour of industrial processes; the pretraining is based on domain information regarding the chemical properties and relations among the data set features. With a similar strategy [11] exploit semanticbased knowledge to address an artificial task unsolvable by standard ML techniques; the key idea is to provide “hints” to the learner about appropriate intermediate concepts.
4 Observations & Remarks
In this section we will discuss some common traits spanning over all the knowledge injection methodologies described previously and present some insights.
Feature manipulation
The features composing the data are often the target of the techniques described in Sec. 2, as they offer a practical mechanism to inject prior information in a variety of DL models. This method is especially effective when the knowledge can be expressed in the form of relationships among input features, which can then be used either to augment the number of training examples (the relations provide a guide to generate novel valid examples starting from the original ones) or to create new, more informative features that should highlight implicit information already available in the data but hard to extract. An interesting thing to be noted is the fact that acting on the feature space does not necessarily require information about labels , as the domain knowledge might involve only constraints among the
features; this suggests that this type of approach is especially well suited for unsupervised learning tasks. These approaches go against the most prevailing DL research direction in recent years, namely the development of purely datadriven models that extract all the hidden information contained in the feature space without any help. Clearly, having powerful and general models capable of good performance regardless of the domain information is a very important purpose, nevertheless we reckon that there exist many contexts that could benefit from the exploitation of domainderived prior information, and this research direction is worthy to be explored in future works.
Accuracy VS Optimization
A partially unexplored area, in our opinion, is the tradeoff between the accuracy of the DL models (e.g. high accuracy for classification tasks and low error for regression ones) and the satisfaction of the constraints imposed to the models. For instance, we already stated that in endtoend learning the goal is to optimize the neural network given a specific task, therefore we might not observe an improvement in the accuracy of the model itself (e.g. Mean Average Error), but rather an improvement over a specific task. This is also the case for regularization schemes, where the resulting models have outputs more consistent with the prior knowledge, but with no increase in accuracy in mere terms of prediction. This is an aspect that is implicit in the desire to minimize a objective function composed by multiple terms that do not point towards the same direction. Generally speaking, the majority of these approaches opt for “soft” constraints on the output of the neural network, that is the constraints are not enforced strictly (as in the case of hard constraints) but rather the optimization tries to balance the diverse terms. This problem has been only partially studied on DNNs [21, 7], although in the optimization area it is well known that minimizing an objective function with multiple terms yields poor convergence properties, mainly because the optimizer is likely to focus on one term of the objective function while ignoring the remaining ones; furthermore, enforcing multiple constraints of different natures means that the terms risk not to be commensurate. Finally, an additional complicating factor in these methods is the selection of adequate weights for the various terms (model accuracy and constraintsbased terms).
Small data sets
A nonnegligible problem for DL approaches is the requirement of large amounts of data, preferably labelled. In many scenarios, this does not happen, thus the training of deep models is hindered. Injecting domain knowledge can boost the performance of DL models when training data is scarce. For this purpose, the most obvious candidates are techniques which augment the available data, but regularization schemes and feature engineering approaches can provide benefits as well, by, respectively, “guiding” the training process of the DNN and simplifying the learning task thanks to the additional features added to the raw data. In general, techniques to integrate prior information can be extremely useful in the context of active learning
and other settings where the dearth of data cannot be bypassed. Nowadays, the majority of approaches for active learning are not guided with domain knowledge whilst there are potentially big benefits in exploiting such knowledge, for instance to drive the selection of new instances to be evaluated, an exploration that is currently guided by domainagnostic strategies based on measures such as information gain. Some recent works did preliminary work towards this direction. For instance,
[6] propose a combination of active and endtoend learning, by embedding a DNN within an optimization model for floating point variables precision tuning. As the training set is relatively small, the DNN starts with inaccurate predictions; the author use active learning to iteratively improve the DNN by retraining it on new examples, namely the solutions of the optimization model, directly depending on the domain knowledge.Evaluation metrics
A partially unexplored issue in the domain information injection area regards the best metric to evaluate the performance of the injection mechanism. Broadly speaking, most common DL techniques are measured on the basis of a single metric such as accuracy (classification tasks) or Mean Average Error or Mean Squared Error (for regression tasks). However, these metrics are not the fairest ones when one has to judge the benefits of injecting domain knowledge. For instance, a common aspect of regularization schemes and methodologies that embed constraints in the NNs is that their goal is not to simply reduce the prediction error or increase the model’s accuracy, but rather to obtain NNs whose output respects some desirable proprieties (e.g. monotonicity) or where certain relationships between input and output need to hold. In practice, this means that there are no established and straightforward methods to measure the improvements of knowledge injection methods, as the for each domain different authors chose different evaluation metrics. In many cases the authors compare their injection methods to standard DL models using specific test sets, carefully crafted for the particular task. For example, models enforcing an output with no violations of a particular constraint can be trained on data sets that contain instances violating the constraints but tested exclusively on data sets with no violations. This is acceptable in order to perform a fair comparison but with an increased risk of creating artificial experimental settings. This lack of homogeneity is not a trivial issue and it complicates the comparison of different techniques. A set of common benchmarks and key performance indicators would greatly benefit this research area.
Towards a unified framework
As an overall remark, it can be noted that the various approaches discussed previously operate in relative isolation. This is understandable and can be partially explained by the fact that domain injection techniques are, by definition, domain specific. This issue is exacerbated by the large number of possible injection mechanisms and targets. This leads to a lack of a common perspective and makes the comparison of different injection approaches harder; however, some recent attempts have been made, see for example [5]
, where multiple knowledge injection techniques are employed to boost a DNN dealing with a complex learning task. We believe that a unified methodology, or framework, for injecting domainderived constraints would be a great step for advancing the research progress in this area. Such unified framework would require a common language to express the domain knowledge; it should be a language with expressive power (many different types of relationships and concepts should be allowed to be represented) and flexible, that is capable to describe information stemming from very diverse domains. We reckon that a language based on constraints and logic predicates could be a very apt choice for this task, especially thanks to paradigms such as constraint programming and mixed integer and linear programming, which have been proven to be capable of handling a wide range of domainspecific challenges. The next step would be deciding the best information injection mechanism for the desired task; for a detailed answer followup studies need to be conducted, but some guidelines can be already provided at this stage. First, when the available data is scarce, data augmentation and feature space manipulation techniques are to be preferred, as they allow to increase the training set size by exploiting known relationships between input set features
, as well as relations between input features and output features . Secondly, if the information contained in the input data is hidden and not easy to be extracted, adhoc DNNs architectures can be extremely helpful – e.g. acting on the hypothesis space – as the domain expert knowledge can be directly injected in the neural network structure from its design to the training algorithm. Finally, regularization methods and explicit constraints learning in the neural network are very effective strategies for endtoend learning and, in general, for more complex learning tasks where the model accuracy is not the exclusive performance metrics. In this case, careful attention should be employed to choose the right tradeoff between the optimization problem and the pure minimization of the DNN’s loss function. This is a non trivial issue, as it complicates the actual development of regularization and constraints learning techniques, together with the fact that they are extremely taskspecific (e.g. a set of weights for one context would be not well suited to other ones); the practitioner implementing these approaches will be faced with steeper challenges compared, for example, to data augmentation approaches, which also tend to be more transferable.5 Conclusion
The integration of domain knowledge expressible in the form of constraints into deep neural networks is a wide research area that have seen increasing research interest in recent years. This topic has been tackled from different angles by a variety of approaches, typically in relative isolation, a fact that probably hindered research breakthrough. In this paper, we have provided a first crossdisciplinary attempt at classifying existing approaches, identifying the main classes of techniques for domain knowledge injection, and highlighting connections with related fields from the DL area. Moreover, we identified a series of common trends and issues that have been addressed and open challenges that still need to be tackled, with the hope of providing useful insights and a guidance for future research efforts.
Acknowledgments
This work has been partially supported by European H2020 FET project OPRECOMP (g.a. 732631) and ICT project AI4EU (g.a. 825619).
References
 [1] M. Atzmueller and E. Sternberg. Mixedinitiative feature engineering using knowledge graphs. In Proceedings of the Knowledge Capture Conference, pages 1–4, 2017.
 [2] S. Bai, J. Z. Kolter, and V. Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
 [3] D. Berrar, P. Lopes, and W. Dubitzky. Incorporating domain knowledge in machine learning for soccer outcome prediction. Machine Learning, 108(1):97–126, 2019.
 [4] E. J. Bjerrum. Smiles enumeration as data augmentation for neural network modeling of molecules. arXiv preprint arXiv:1703.07076, 2017.
 [5] A. Borghesi, F. Baldo, M. Lombardi, and M. Milano. Injective domain knowledge in neural networks for transprecision computing. arXiv preprint arXiv:2002.10214, 2020.
 [6] A. Borghesi, G. Tagliavini, and et al. Combining learning and optimization for transprecision computing. In Proceedings of the 17th ACM International Conference on Computing Frontiers, 2020.
 [7] F. De Tassis, M. Lombardi, and M. Milano. Teaching the old dog new tricks: Supervised learning with constraints. arXiv preprint arXiv:2002.10766, 2020.
 [8] M. Diligenti, M. Gori, and C. Sacca. Semanticbased regularization for learning and inference. Artificial Intelligence, 244:143–165, 2017.
 [9] P. Donti, B. Amos, and J. Z. Kolter. Taskbased endtoend model learning in stochastic optimization. In Advances in Neural Information Processing Systems, pages 5484–5494, 2017.
 [10] F. Fioretto, T. W. Mak, and et al. A lagrangian dual framework for deep neural networks with constraints. arXiv preprint arXiv:2001.09394, 2020.
 [11] Ç. Gülçehre and Y. Bengio. Knowledge matters: Importance of prior information for optimization. The Journal of Machine Learning Research, 17(1):226–257, 2016.
 [12] A. HernándezGarcía and P. König. Data augmentation instead of explicit regularization. arXiv preprint arXiv:1806.03852, 2018.

[13]
A. Jain, A. R. Zamir, S. Savarese, and A. Saxena.
Structuralrnn: Deep learning on spatiotemporal graphs.
In
Proceedings of the ieee conference on computer vision and pattern recognition
, pages 5308–5317, 2016. 
[14]
S. Khalid, T. Khalil, and S. Nasreen.
A survey of feature selection and feature extraction techniques in machine learning.
In 2014 Science and Information Conference, pages 372–378. IEEE, 2014.  [15] U. Khurana, H. Samulowitz, and D. Turaga. Feature engineering for predictive modeling using reinforcement learning. In ThirtySecond AAAI Conference on Artificial Intelligence, 2018.
 [16] T. N. Kipf and M. Welling. SemiSupervised Classification with Graph Convolutional Networks. In Proceedings of the 5th International Conference on Learning Representations, ICLR ’17, 2017.
 [17] T. Li and V. Srikumar. Augmenting neural networks with firstorder logic. arXiv preprint arXiv:1906.06298, 2019.
 [18] J. Luketina, N. Nardelli, and et al. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv:1906.03926, 2019.
 [19] D. D.C. T. G. C. Z. M. V. Marc Fischer, Mislav Balunovic. Dl2: Training and querying neural networks with logic. In International Conference on Machine Learning, 2019.
 [20] K. Marino, R. Salakhutdinov, and A. Gupta. The more you know: Using knowledge graphs for image classification. arXiv preprint arXiv:1612.04844, 2016.
 [21] P. MárquezNeila, M. Salzmann, and P. Fua. Imposing hard constraints on deep networks: Promises and limitations. arXiv preprint arXiv:1706.02025, 2017.
 [22] G. Marra, F. Giannini, M. Diligenti, and M. Gori. Lyrics: a general interface layer to integrate logic inference and deep learning. In ECMLPKDD 2019, 2019.
 [23] R. Miao, Z. Yang, and V. Gavrishchaka. Leveraging domainexpert knowledge, boosting and deep learning for identification of rare and complex states. In Journal of Physics: Conference Series, volume 1207, page 012016. IOP Publishing, 2019.
 [24] A. Mollaysa, A. Kalousis, E. Bruno, and M. Diephuis. Learning to augment with feature sideinformation. In Asian Conference on Machine Learning, pages 173–187, 2019.
 [25] N. Muralidhar, M. Islam, and et al. Incorporating prior domain knowledge into deep neural networks. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, 2018.
 [26] F. Nargesian, H. Samulowitz, U. Khurana, E. B. Khalil, and D. S. Turaga. Learning feature engineering for classification. In IJCAI, pages 2529–2535, 2017.
 [27] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80, 2008.
 [28] A. Silva and M. Gombolay. Prolonets: Neuralencoding human experts’ domain knowledge to warm start reinforcement learning. arXiv preprint arXiv:1902.06007, 2019.
 [29] M. Silvestri, M. Lombardi, and M. Milano. Injecting domain knowledge in neural networks: a controlled experiment on a constrained problem. arXiv preprint arXiv:2002.10742, 2020.
 [30] K. Terashima and J. Murata. A study on use of prior information for acceleration of reinforcement learning. In SICE Annual Conference 2011, pages 537–543. IEEE, 2011.
 [31] K. Vo, D. Pham, M. Nguyen, T. Mai, and T. Quan. Combination of domain knowledge and deep learning for sentiment analysis. In International Workshop on Multidisciplinary Trends in Artificial Intelligence, pages 162–173. Springer, 2017.
 [32] L. von Rueden, S. Mayer, and et al. Informed machine learning – a taxonomy and survey of integrating knowledge into learning systems. arXiv preprint arXiv:1903.12394v2, 2020.
 [33] J. Wang and L. Perez. The effectiveness of data augmentation in image classification using deep learning. Convolutional Neural Networks Vis. Recognit, page 11, 2017.
 [34] B. Wilder, B. Dilkina, and M. Tambe. Melding the datadecisions pipeline: Decisionfocused learning for combinatorial optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1658–1665, 2019.
 [35] J. Xu, Z. Zhang, and et al. A semantic loss function for deep learning with symbolic knowledge. In Proceedings of the 35th ICML, 2018.
 [36] X. Yuan, C. Ou, Y. Wang, C. Yang, and W. Gui. A novel semisupervised pretraining strategy for deep networks and its application for quality variable prediction in industrial processes. Chemical Engineering Science, page 115509, 2020.
 [37] S. Zhang, H. Tong, J. Xu, and R. Maciejewski. Graph convolutional networks: a comprehensive review. Computational Social Networks, 6(1):11, 2019.
Comments
There are no comments yet.