Causal Generative Neural Networks

11/24/2017 ∙ by Olivier Goudet, et al. ∙ 0

We introduce CGNN, a framework to learn functional causal models as generative neural networks. These networks are trained using backpropagation to minimize the maximum mean discrepancy to the observed data. Unlike previous approaches, CGNN leverages both conditional independences and distributional asymmetries to seamlessly discover bivariate and multivariate causal structures, with or without hidden variables. CGNN does not only estimate the causal structure, but a full and differentiable generative model of the data. Throughout an extensive variety of experiments, we illustrate the competitive results of CGNN w.r.t state-of-the-art alternatives in observational causal discovery on both simulated and real data, in the tasks of cause-effect inference, v-structure identification, and multivariate causal discovery.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Deep learning models have shown tremendous predictive abilities in image classification, speech recognition, language translation, game playing, and much more [Goodfellow et al.2016]. However, they often mistake correlation for causation [Stock and Cisse2017], which can have catastrophic consequences for agents that plan and decide from observation.

The gold standard to discover causal relations is to perform experiments [Pearl2003]. However, whenever experiments are expensive, unethical, or impossible to realize, there is a need for observational causal discovery, that is, the estimation of causal relations from observation alone [Spirtes et al.2000, Peters et al.2017]. In observational causal discovery, some authors exploit distributional asymmetries to discover bivariate causal relations [Hoyer et al.2009, Zhang and Hyvärinen2009, Daniusis et al.2012, Stegle et al.2010, Lopez-Paz et al.2015, Fonollosa2016], while others rely on conditional independence to discover structures on three or more variables [Spirtes et al.2000, Chickering2002]. Different approaches rely on different but equally strong assumptions, such as linearity [Shimizu et al.2006], additive noise [Zhang and Hyvärinen2009, Peters et al.2014], determinism [Daniusis et al.2012], or a large corpus of annotated causal relations [Lopez-Paz et al.2015, Fonollosa2016]. Among the most promising approaches are score-based methods [Chickering2002], assuming the existence of external score-functions that must be powerful enough to detect diverse causal relations. Finally, most methods are not differentiable, thus unsuited for deep learning pipelines.

The ambition of Causal Generative Neural Network (CGNNs) is to provide a unified approach. CGNNs learn functional causal models (Section 2) as generative neural networks, trained by backpropagation to minimize the Maximum Mean Discrepancy (MMD) [Gretton et al.2007, Li et al.2015] between the observational and the generated data (Section 3

). Leveraging the representational power of deep generative models, CGNNs account for both distributional asymmetries and conditional independencies, tackle the bivariate and multivariate cases, and deal with hidden variables (confounders). They estimate both the causal graph underlying the data and the full joint distribution, through the architecture and the weights of generative networks. Unlike previous approaches, CGNNs allow non-additive noise terms to model flexible conditional distributions. Lastly, they define differentiable joint distributions, which can be embedded within deep architectures. Extensive experiments show the state-of-the-art performance of CGNNs (Section 

4) on cause-effect inference, v-structure identification, and multivariate causal discovery with hidden variables.111Code available at Datasets available at and

2 The language of causality: FCMs

A Functional Causal Model (FCM) upon a random variable vector

is a triplet , representing a set of equations:


Each equation characterizes the direct causal relation from the set of causes to observed variable , described by some causal mechanism up to the effects of noise variable drawn after distribution , accounting for all unobserved phenomenons. For simplicity, interchangeably denotes an observed variable and a node in graph . There exists a direct causal relation from to , written , iff there exists a directed edge from to in . In the following, we restrict ourselves to considering Directed Acyclic Graph (DAG) (Fig. 1) and

is set to the uniform distribution on

, .

Figure 1: Example FCM for .

2.1 Generative models and interventions

The generative model associated to FCM proceeds by first drawing for all , then in topological order of computing .

Importantly, the FCM supports interventions, that is, freezing a variable to some constant . The resulting joint distribution noted , called interventional distribution [Pearl2009], can be computed from the FCM by discarding all causal influences on and clamping its value to . It is emphasized that intervening is different from conditioning (correlation does not imply causation). The knowledge of interventional distributions is essential for e.g., public policy makers, wanting to estimate the overall effects of a decision on a given variable.

2.2 Formal background and notations

In this section, we introduce notations and definitions and prove the representational power of FCMs.

Two random variables are conditionally independent given if . Three random variables form a v-structure iff . The random variable is a confounder (or common cause) of the pair if have causal structure . The skeleton of a DAG is obtained by replacing all the directed edges in by undirected edges.

Discovering the causal structure of a random vector is a difficult task in all generality. For this reason, the literature in causal inference relies on a set of common assumptions [Pearl2003]. The causal sufficiency assumption states that there are no unobserved confounders. The causal Markov assumption states that all the d-separations in the causal graph imply conditional independences in the observational distribution . The causal faithfulness assumption states that all the conditional independences in the observational distribution imply d-separations in the causal graph . A Markov equivalence class denotes the set of graphs with same set of d-separations.

Proposition 1.

Representing joint distributions with FCMs


denote a set of continuous random variables with joint distribution

, and further assume that the joint density function of is continuous and strictly positive on a compact subset of , and zero elsewhere. Letting be a DAG such that can be factorized along ,

there exists with a continuous function with compact support in such that equals the generative model defined from FCM .


By induction on the topological order of , taking inspiration from [Carlier et al.2016]. Let be such that and consider the cumulative distribution defined over the domain of ().

is strictly monotonous as the joint density function is strictly positive therefore its inverse, the quantile function

is defined and continuous. By construction, and setting yields the result.
Assume be defined for all variables with topological order less than . Let with topological order and the vector of its parent variables. For any noise vector let be the value vector of variables in defined from . The conditional cumulative distribution is strictly continuous and monotonous wrt , and can be inverted using the same argument as above. Defining yields the result. ∎

3 Causal generative neural networks

Let denote a set of continuous random variables with joint distribution . Under same conditions as in Proposition 1, ( being decomposable to graph , with continuous and strictly positive joint density function on a compact in and zero elsewhere), it is shown that there exists a generative neural network called CGNN (Causal Generative Neural Network), that approximates with arbitrary accuracy.

3.1 Approximating continuous FCMs with CGNN

Firstly, given , it is shown that there exists a set of networks such that the generative model defined by with defines a joint distribution arbitrarily close to .

Proposition 2.

For , let denote the set of variables with topological order less than and let be its size. For any -dimensional vector of noise values , let (resp. ) be the vector of values computed in topological order from (resp. ). For any , there exists a set of networks with architecture such that


By induction on the topological order of . Let be such that . Following the universal approximation theorem [Cybenko1989], as is a continuous function over a compact of , there exists a neural net such that . Thus Eq. 2 holds for the set of networks for ranging over variables with topological order 0.
Let us assume that Prop. 2 holds up to , and let us assume for brevity that there exists a single variable with topological order . Letting be such that (based on the universal approximation property), letting be such that for all (by absolute continuity) and letting satisfying Eq. 2 for with topological order less than for , it comes: , which ends the proof. ∎

3.2 Scoring metric

The architecture and the network weights are trained and optimized using a score-based approach [Chickering2002]. The ideal score, to be minimized, is the distance between the joint distribution associated with the ground truth FCM, and the joint distribution defined by the estimated . A tractable approximation thereof is given by the Maximum Mean Discrepancy (MMD) [Gretton et al.2007] between the -sample observational data , and an - sample sampled after . Overall, is trained by minimizing


with the number of edges in and defined as:

where kernel usually is taken as the Gaussian kernel (). The MMD statistic, with quadratic complexity in the sample size, has the good property that it is zero if and only if as goes to infinity [Gretton et al.2007]. For scalability, a linear approximation of the MMD statistics based on random features [Lopez-Paz2016], called , will also be used in the experiments. Due to the Gaussian kernel being differentiable, and are differentiable, and backpropagation can be used to learn the CGNN made of networks structured along .

It is shown that the distribution of the CGNN can estimate the true observational distribution of the (unknown) FCM up to an arbitrary precision, under the assumption of an infinite observational sample:

Proposition 3.

Let be an infinite observational sample generated from . With same notations as in Prop. 2, for every , there exists a set such that the MMD between and an infinite size sample generated from is less than .


According to Prop. 2 and with same notations, letting go to 0 as goes to infinity, consider and defined from such that for all , .

Let denote the infinite sample generated after . The score of the CGNN is .

As converges towards on the compact , using the bounded convergence theorem on a compact subset of , uniformly for , it follows from the Gaussian kernel function being bounded and continuous that , when . ∎

CGNN benefits from i) the representational power of generative networks to exploit distributional asymmetries; ii) the overall approximation of the joint distribution of the observational data to exploit conditional independences, to handle bivariate and multivariate causal modeling.

3.3 Searching causal graphs with CGNNs

The exhaustive exploration of all DAGs with variables is super-exponential in , preventing the use of brute-force methods for observational causal discovery even for moderate . Following [Tsamardinos et al.2006, Nandy et al.2015], we assume known skeleton for

, obtained via domain knowledge or a feature selection algorithm

[Yamada et al.2014] under standard assumptions such as causal Markov, faithfulness, and sufficiency. Given a skeleton on and the regularized MMD score (3), CGNN follows a greedy procedure to find and :

  • Orient each as or by selecting the associated 2-variable CGNN with best score.

  • Follow paths from a random set of nodes until all nodes are reached. Edges pointing towards a visited node reveal cycles, so must be reversed.

  • For a number of iterations, reverse the edge that leads to the maximum improvement of the score over a -variable CGNN, without creating a cycle.

    At the end of this process, we evaluate a confidence score for any edge as:


3.4 Dealing with hidden confounders

The search method above relies on the causal sufficiency assumption (no confounders). We relax this assumption as follows. Assuming confounders, each edge in the skeleton is due to one out of three possibilities: either , , or there exists an unobserved variable such that . Therefore, each equation in the FCM is extended to: , where is the set of indices of the variables adjacent to in the skeleton. Each represents the hypothetical unobserved common causes of and . For instance, hiding from the FCM in Fig. 1 would require considering a confounder . Finally, when considering hidden confounders, the above third search step considers three possible mutations of the graph: reverse, add, or remove an edge. In this case, promotes simple graphs.

4 Experiments

CGNN is empirically validated and compared to the state of the art on observational causal discovery of i) cause-effect relations (Section 4.2); ii) v-structures (Section 4.3); iii) multivariate causal structures with no confounders (Section 4.4); iv) multivariate causal structures when relaxing the no-confounder assumption (Section 4.5).

4.1 Experimental setting

(a) Samples.
(a) Losses.
Figure 2:

Samples and MMDs for CGNN models of different complexities (number of neurons) modeling the causal direction

(top row) and the anticausal direction (bottom row) of a simple example. MMDs are averaged over runs, underlined numbers indicate statistical significance at .

MMD uses a sum of Gaussian kernels with bandwidths . CGNN uses one-hidden-layer neural networks with ReLU units, trained with the Adam optimizer [Kingma and Ba2014] and initial learning rate of , with full batch size . The generated data involved from noise variables are sampled anew in each step. Each CGNN is trained for epochs and evaluated on generated samples. Reported results are averaged over 32 runs for (resp. 64 runs for ). All experiments run on an Intel Xeon 2.7GHz CPU, and an NVIDIA GTX 1080Ti GPU.

The most sensitive CGNN hyper-parameter is the number of hidden units , governing the CGNN ability to model the causal mechanisms : too small , and data patterns may be missed; too large , and overly complicated causal mechanisms might be retained. Overall, is problem-dependent, as illustrated on a toy problem where two bivariate CGNNs are learned with (Fig. 2.a) from data generated by FCM: . Fig. 2.b shows the associated MMDs averaged on 32 independent runs, and confirms the importance of cross-validating model capacity [Zhang and Hyvärinen2009].

4.2 Discovering cause-effect relations

Under the causal sufficiency assumption, the statistical dependence between two random variables and is either due to causal relation or . The CGNN cause-effect accuracy is the fraction of edges in the graph skeleton that are rightly oriented, with Area Under Precision/Recall curve (AUPR) as performance indicator.

Five cause-effect inference datasets, covering a wide range of associations, are used. CE-Cha contains 300 cause-effect pairs from the challenge of [Guyon2013]. CE-Net contains 300 artificial cause-effect pairs generated using random distributions as causes, and neural networks as causal mechanisms. CE-Gauss contains 300 artificial cause-effect pairs as generated by [Mooij et al.2016], using random mixtures of Gaussians as causes, and Gaussian process priors as causal mechanisms. CE-Multi contains 300 artificial cause-effect pairs built with random linear and polynomial causal mechanisms. In this dataset, we simulate additive or multiplicative noise, applied before or after the causal mechanism. CE-Tüb contains the 99 real-world scalar cause-effect pairs from the Tübingen dataset [Mooij et al.2016], concerning domains such as climatology, finance, and medicine. We set .

The baseline and competitor methods222 include: i) the Additive Noise Model ANM [Mooij et al.2016], with Gaussian process regression and HSIC independence test; ii) the Linear Non-Gaussian Additive Model LiNGAM [Shimizu et al.2006]

, a variant of Independent Component Analysis to identify linear causal relations; iii) The Information Geometric Causal Inference

IGCI [Daniusis et al.2012], with entropy estimator and Gaussian reference measure; iv) the Post-Non-Linear model PNL [Zhang and Hyvärinen2009], with HSIC test; v) The GPI method [Stegle et al.2010], where the Gaussian process regression with higher marginal likelihood is selected as causal direction; vi) the Conditional Distribution Similarity statistic CDS [Fonollosa2016]

, which prefers the causal direction with lowest variance of conditional distribution variances; vii) the award-winning method

Jarfo [Fonollosa2016]

, a random forest classifier trained on the ChaLearn Cause-effect pairs and hand-crafted to extract 150 features, including methods ANM, IGCI, CDS, and LiNGAM. For each baseline and competitor method, a leave-one-dataset-out scheme is used to select the best hyperparameters for each method (details omitted for brevity).

method Cha Net Gauss Multi Tüb
Best fit 56.4 77.6 36.3 55.4 58.4 (44.9)
LiNGAM 54.3 43.7 66.5 59.3 39.7 (44.3)
CDS 55.4 89.5 84.3 37.2 59.8 (65.5)
IGCI 54.4 54.7 33.2 80.7 60.7 (62.6)
ANM 66.3 85.1 88.9 35.5 53.7 (59.5)
PNL 73.1 75.5 83.0 49.0 68.1 (66.2)
Jarfo 79.5 92.7 85.3 94.6 54.5 (59.5)
GPI 67.4 88.4 89.1 65.8 66.4 (62.6)
CGNN () 73.6 89.6 82.9 96.6 79.8 (74.4)
CGNN () 76.5 87.0 88.3 94.2 76.9 (72.7)
Table 1: Cause-effect relations: Area Under the Precision Recall curve on 5 benchmarks for the cause-effect experiments (weighted accuracy in parenthesis for Tüb)

As shown in Table 1

, i) linear regression methods are dominated; ii) CDS and IGCI perform well in some cases (e.g. when the entropy of causes is lower than those of effects); iii) ANM performs well when the additive noise assumption holds; iv) PNL, a generalization of ANM, compares favorably to the above methods; v) Jarfo performs well on artificial data but badly on real examples. Lastly, generative methods GPI and

CGNN () perform well on most datasets, including the real-world cause-effect pairs CE-Tüb, in counterpart for a higher computational cost (resp. 32 min on CPU for GPI and 24 min on GPU for CGNN). Using the linear MMD approximation [Lopez-Paz2016], CGNN ( as explained in Section 3.2) reduces the cost by a factor of 5 without hindering the performance. Overall, CGNN demonstrates competitive performance on the cause-effect inference problem, where it is necessary to discover distributional asymmetries.

4.3 Discovering v-structures

Considering random variables with skeleton , four causal structures are possible: the chain , the reverse chain , the v-structure , and the reverse v-structure . Note that the chain, the reverse chain, and the reverse v-structure are Markov equivalent, and therefore indistinguishable from each other using statistics alone. This section thus examines the CGNN ability to identify v-structures.

Let us consider an FCM with causal mechanisms Identity and Gaussian noise variables (e.g., . As the joint distribution of one cause and its effect is symmetrical, the bivariate methods used in the previous section do not apply and the conditional independences among all three variables must be taken into account.

The retained experimental setting trains a CGNN for every possible causal graph with skeleton , and selects the one with minimal MMD. CGNN accurately discriminates the v-structures from the other ones , with a significantly lower MMD for the ground truth causal graph. This proof of concept shows the ability of CGNN to detect and exploit conditional independences among variables.

4.4 Discovering multivariate causal structures

Skeleton without error Skeleton with 20% of error Causal protein network
PC-Gauss 0.67 (0.11) 9.0 (3.4) 131 (70) 0.42 (0.06) 21.8 (5.5) 191.3 (73) 0.19 (0.07) 16.4 (1.3) 91.9 (12.3)
PC-HSIC 0.80 (0.08) 6.7 (3.2) 80.1 (38) 0.49 (0.06) 19.8 (5.1) 165.1 (67) 0.18 (0.01) 17.1 (1.1) 90.8 (2.6)
ANM 0.67 (0.11) 7.5 (3.0) 135.4 (63) 0.52 (0.10) 19.2 (5.5) 171.6 (66) 0.34 (0.05) 8.6 (1.3) 85.9 (10.1)
Jarfo 0.74 (0.10) 8.1 (4.7) 147.1 (94) 0.58 (0.09) 20.0 (6.8) 184.8 (88) 0.33 (0.02) 10.2 (0.8) 92.2 (5.2)
GES 0.48 (0.13) 14.1 (5.8) 186.4 (86) 0.37 (0.08) 20.9 (5.5) 209 (83) 0.26 (0.01) 12.1 (0.3) 92.3 (5.4)
LiNGAM 0.65 (0.10) 9.6 (3.8) 171 (86) 0.53 (0.10) 20.9 (6.8) 196 (83) 0.29 (0.03) 10.5 (0.8) 83.1 (4.8)
CAM 0.69 (0.13) 7.0 (4.3) 122 (76) 0.51 (0.11) 15.6 (5.7) 175 (80) 0.37 (0.10) 8.5 (2.2) 78.1 (10.3)
CGNN () 0.77 (0.09) 7.1 (2.7) 141 (59) 0.54 (0.08) 20 (10) 179 (102) 0.68 (0.07) 5.7 (1.7) 56.6 (10.0)
CGNN () 0.89* (0.09) 2.5* (2.0) 50.45* (45) 0.62 (0.12) 16.9 (4.5) 134.0* (55) 0.74* (0.09) 4.3* (1.6) 46.6* (12.4)
Table 2: Average (std. dev.) results for the orientation of 20 artificial graphs given true skeleton (left), artificial graphs given skeleton with 20% error (middle), and real protein network given true skeleton (right). denotes statistical significance at .

Consider a random vector . Our goal is to find the FCM of under the causal sufficiency assumption. At this point, we will assume known skeleton, so the problem reduces to orienting every edge. To that end, all experiments provide all algorithms the true graph skeleton, so their ability to orient edges is compared in a fair way. This allows us to separate the task of orienting the graph from that of uncovering the skeleton.

Results on artificial data

We draw samples from training artificial causal graphs and test artificial causal graphs on 20 variables. Each variable has a number of parents uniformly drawn in ; s are randomly generated polynomials involving additive/multiplicative noise.

We compare CGNN to the PC algorithm [Spirtes et al.2000], the score-based methods GES [Chickering2002], LiNGAM [Shimizu et al.2006], causal additive model (CAM) [Peters et al.2014] and with the pairwise methods ANM and Jarfo. For PC, we employ the better-performing, order-independent version of the PC algorithm proposed by [Colombo and Maathuis2014]

. PC needs the specification of a conditional independence test. We compare PC-Gaussian, which employs a Gaussian conditional independence test on Fisher z-transformations, and PC-HSIC, which uses the HSIC conditional independence test with the Gamma approximation

[Gretton et al.2005]. PC and GES are implemented in the pcalg package [Kalisch et al.2012].

All hyperparameters are set on the training graphs in order to maximize the Area Under the Precision/Recall score (AUPR). For the Gaussian conditional independence test and the HSIC conditional independence test, the significance level achieving best result on the training set are respectively and . For GES, the penalization parameter is set to on the training set. For CGNN, is set to 20 on the training set. For CAM, the cutoff value is set to .

Table 2 (left) displays the performance of all algorithms obtained by starting from the exact skeleton on the test set of artificial graphs and measured from the AUPR (Area Under the Precision/Recall curve), the Structural Hamming Distance (SHD, the number of edge modifications to transform one graph into another) and the Structural Intervention Distance (SID, the number of equivalent two-variable interventions between two graphs) [Peters and Bühlmann2013].

CGNN obtains significant better results with SHD and SID compared to the other algorithms when the task is to discover the causal from the true skeleton. Constraints based method PC with powerful HSIC conditional independence test is the second best performing method. It highlights the fact that when the skeleton is known, exploiting the structure of the graph leads to good results compared to pairwise methods using only local information. However CGNN and PC-HSIC are the most computationally expensive methods, taking an average of 4 hours on GPU and 15 hours on CPU, respectively.

The robustness of the approach is validated by randomly perturbing 20% edges in the graph skeletons provided to all algorithms (introducing about 10 false edges over 50 in each skeleton). As shown on Table 2 (middle), and as could be expected, the scores of all algorithms are lower when spurious edges are introduced. Among the least robust methods are constraint-based methods; a tentative explanation is that they heavily rely on the graph structure to orient edges. By comparison pairwise methods are more robust because each edge is oriented separately. As CGNN leverages conditional independence but also distributional asymmetry like pairwise methods, it obtains overall more robust results when there are errors in the skeleton compared to PC-HSIC.

CGNN obtains overall good results on these artificial datasets. It offers the advantage to deliver a full generative model useful for simulation (while e.g., Jarfo and PC-HSIC only give the causality graph). To explore the scalability of the approach, 5 artificial graphs with variables have been considered, achieving an AUPRC of , in 30 hours of computation on four NVIDIA 1080Ti GPUs.

Results on real-world data

CGNN is applied to the protein network problem [Sachs et al.2005], using the Anti-CD3/CD28 dataset with 853 observational data points corresponding to general perturbations without specific interventions. All algorithms were given the skeleton of the causal graph [Sachs et al.2005, Fig. 2] with same hyper-parameters as in the previous subsection. We run each algorithm on 10-fold cross-validation. Table 2 (right) reports average (std. dev.) results.

Constraint-based algorithms obtain surprisingly low scores, because they cannot identify many V-structures in this graph. We confirm this by evaluating conditional independence tests for the adjacent tuples of nodes pip3-akt-pka, pka-pmek-pkc, pka-raf-pkc and we do not find strong evidences for V-structure. Therefore methods based on distributional asymmetry between cause and effect seem better suited to this dataset. CGNN obtains good results compared to the other algorithms. Notably, Figure 3 shows that CGNN is able to recover the strong signal transduction pathway rafmekerk reported in [Sachs et al.2005] and corresponding to clear direct enzyme-substrate causal effect. CGNN gives important scores for edges with good orientation (solid line), and low scores (thinnest edges) to the wrong edges (dashed line), suggesting that false causal discoveries may be controlled by using the confidence scores defined in Eq. (4).

Figure 3: Causal protein network obtained with CGNN

4.5 Dealing with hidden confounders

As real data often includes unobserved confounding variables, the robustness of CGNN is assessed by considering the previous artificial datasets while hiding some of the 20 observed variables in the graph. Specifically three random variables that cause at least two others in the same graph are hidden. Consequently, the skeleton now includes additional edges for all pairs of variables that are consequences of the same hidden cause (confounder). The goal in this section is to orient the edges due to direct causal relations, and to remove those due to confounders.

We compare CGNN to the RFCI algorithm (Gaussian or HSIC conditional independence tests) [Colombo et al.2012], which is a modification of the PC algorithm that accounts for hidden variables. For CGNN, we set the hyperparameter fitted on the training graph dataset. Table 3 shows that CGNN is robust to confounders. Interestingly, true causal edges have high confidence, while edges due to confounding effects are removed or have low confidence.

RFCI-Gaussian 0.22 (0.08) 21.9 (7.5) 174.9 (58.2)
RFCI-HSIC 0.41 (0.09) 17.1 (6.2) 124.6 (52.3)
Jarfo 0.54 (0.21) 20.1 (14.8) 98.2 (49.6)
CGNN () 0.71* (0.13) 11.7* (5.5) 53.55* (48.1)
Table 3: AUPR, SHD and SID on causal discovery with confounders. denotes significance at .

5 Conclusion

We introduced CGNN, a new framework to learn functional causal models from observational data based on generative neural networks. CGNNs minimize the maximum mean discrepancy between their generated samples and the observed data. CGNNs combines the power of deep learning and the interpretability of causal models. Once trained, CGNNs are causal models of the world able to simulate the outcome of interventions. Future work includes i) extending the proposed approach to categorical and temporal data, ii) characterizing sufficient identifiability conditions for the approach, and iii) improving the computational efficiency of CGNN.


  • [Carlier et al.2016] Guillaume Carlier, Victor Chernozhukov, and Alfred Galichon. Vector quantile regression beyond correct specification. arXiv, 2016.
  • [Chickering2002] David Maxwell Chickering. Optimal structure identification with greedy search. JMLR, 2002.
  • [Colombo and Maathuis2014] Diego Colombo and Marloes H Maathuis. Order-independent constraint-based causal structure learning. JMLR, 2014.
  • [Colombo et al.2012] Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, 2012.
  • [Cybenko1989] George Cybenko.

    Approximation by superpositions of a sigmoidal function.

    Mathematics of Control, Signals, and Systems (MCSS), 2(4):303–314, 1989.
  • [Daniusis et al.2012] Povilas Daniusis, Dominik Janzing, Joris Mooij, Jakob Zscheischler, Bastian Steudel, Kun Zhang, and Bernhard Schölkopf. Inferring deterministic causal relations. arXiv, 2012.
  • [Fonollosa2016] José Fonollosa. Conditional distribution variability measures for causality detection. arXiv, 2016.
  • [Goodfellow et al.2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
  • [Gretton et al.2005] Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, and Bernhard Schölkopf. Kernel methods for measuring independence. JMLR, 2005.
  • [Gretton et al.2007] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, Alexander J Smola, et al. A kernel method for the two-sample-problem. NIPS, 2007.
  • [Guyon2013] Isabelle Guyon. Chalearn cause effect pairs challenge, 2013.
  • [Hoyer et al.2009] Patrik O Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. NIPS, 2009.
  • [Kalisch et al.2012] Markus Kalisch, Martin Mächler, Diego Colombo, Marloes H Maathuis, Peter Bühlmann, et al. Causal inference using graphical models with the r package pcalg. Journal of Statistical Software, 2012.
  • [Kingma and Ba2014] Durk P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR, 2014.
  • [Li et al.2015] Yujia Li, Kevin Swersky, and Richard S Zemel.

    Generative moment matching networks.

    ICML, 2015.
  • [Lopez-Paz et al.2015] David Lopez-Paz, Krikamol Muandet, Bernhard Schölkopf, and Ilya O Tolstikhin. Towards a learning theory of cause-effect inference. ICML, 2015.
  • [Lopez-Paz2016] David Lopez-Paz. From dependence to causation. PhD thesis, University of Cambridge, 2016.
  • [Mooij et al.2016] Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using observational data: methods and benchmarks. JMLR, 2016.
  • [Nandy et al.2015] Preetam Nandy, Alain Hauser, and Marloes H Maathuis. High-dimensional consistency in score-based and hybrid structure learning. arXiv, 2015.
  • [Pearl2003] Judea Pearl. Causality: models, reasoning and inference. Econometric Theory, 2003.
  • [Pearl2009] Judea Pearl. Causality. 2009.
  • [Peters and Bühlmann2013] Jonas Peters and Peter Bühlmann. Structural intervention distance (sid) for evaluating causal graphs. arXiv, 2013.
  • [Peters et al.2014] Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models.

    The Journal of Machine Learning Research

    , 15(1):2009–2053, 2014.
  • [Peters et al.2017] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference - Foundations and Learning Algorithms. MIT Press, 2017.
  • [Sachs et al.2005] Karen Sachs, Omar Perez, Dana Pe’er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721):523–529, 2005.
  • [Shimizu et al.2006] Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, and Antti Kerminen. A linear non-gaussian acyclic model for causal discovery. JMLR, 2006.
  • [Spirtes et al.2000] Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search. MIT Press, 2000.
  • [Stegle et al.2010] Oliver Stegle, Dominik Janzing, Kun Zhang, Joris M Mooij, and Bernhard Schölkopf. Probabilistic latent variable models for distinguishing between cause and effect. NIPS, 2010.
  • [Stock and Cisse2017] P. Stock and M. Cisse.

    ConvNets and ImageNet Beyond Accuracy: Explanations, Bias Detection, Adversarial Examples and Model Criticism.

    arXiv, 2017.
  • [Tsamardinos et al.2006] Ioannis Tsamardinos, Laura E Brown, and Constantin F Aliferis.

    The max-min hill-climbing bayesian network structure learning algorithm.

    Machine learning, 2006.
  • [Yamada et al.2014] Makoto Yamada, Wittawat Jitkrittum, Leonid Sigal, Eric P Xing, and Masashi Sugiyama. High-dimensional feature selection by feature-wise kernelized lasso. Neural computation, 2014.
  • [Zhang and Hyvärinen2009] Kun Zhang and Aapo Hyvärinen. On the identifiability of the post-nonlinear causal model. UAI, 2009.