From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling

This is the github supplement to the paper From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling published in the Transactions on Machine Learning Research (TMLR).

@article{komanduri2024survey,
title={From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling},
author={Aneesh Komanduri and Xintao Wu and Yongkai Wu and Feng Chen},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2024},
url={https://openreview.net/forum?id=PUpZXvNqmb},
}

Abstract: Deep generative models have shown tremendous capability in data density estimation and data generation from finite samples. While these models have shown impressive performance by learning correlations among features in the data, some fundamental shortcomings are their lack of explainability, tendency to induce spurious correlations, and poor out-of-distribution extrapolation. To remedy such challenges, recent work has proposed a shift toward causal generative models. Causal models offer several beneficial properties to deep generative models, such as distribution shift robustness, fairness, and interpretability. Structural causal models (SCMs) describe data-generating processes and model complex causal relationships and mechanisms among variables in a system. Thus, SCMs can naturally be combined with deep generative models. We provide a technical survey on causal generative modeling categorized into causal representation learning and controllable counterfactual generation methods. We focus on fundamental theory, methodology, drawbacks, datasets, and metrics. Then, we cover applications of causal generative models in fairness, privacy, out-of-distribution generalization, precision medicine, and biological sciences. Lastly, we discuss open problems and fruitful research directions for future work in the field.

Note that papers marked with a * are new references that were not included in the published version of the survey.

Preliminaries

Causality

1. Causality, Cambridge University Press, Cambridge, UK, 2 edition, 2009 [book]

2. On Pearl’s Hierarchy and the Foundations of Causal Inference, Probabilistic and Causal Inference: The Works of Judea Pearl 2022 [paper]

3. Elements of Causal Inference: Foundations and Learning Algorithms, The MIT Press 2017 [book]

Independent Component Analysis

1. Analyse générale des liaisons stochastiques: etude particulière de l’analyse factorielle linéaire, Review of the International Statistical Institute 1953 [paper]

2. Independent component analysis, a new concept? Signal Processing 1994 [paper]

3. Nonlinear Independent Component Analysis: Existence and Uniqueness Results, Neural Networks 1999 [paper]

4. Nonlinear ICA using Auxiliary Variables and Generalized Contrastive Learning, AISTATS 2019 [paper]

5. The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICA, UAI 2019 [paper]

6. Independent mechanism analysis, a new concept?, NeurIPS 2021 [paper]

7. On the identifiability of nonlinear ICA: Sparsity and beyond, NeurIPS 2022 [paper]

8. Generalizing nonlinear ICA beyond structural sparsity, NeurIPS 2023 [paper]

9. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA, CLeaR 2022 [paper]

Identifiability in Generative Models

1. Indeterminacy in Generative Models: Characterization and Strong Identifiability, AISTATS 2023 [paper]

Disentangled Representation Learning

1. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, ICML 2019 [paper]

2. Variational Autoencoders and Nonlinear ICA: A Unifying Framework, AISTATS 2020 [paper]

3. Weakly-supervised Disentanglement Without Compromises, ICML 2020 [paper]

4. On Disentangled Representations Learned From Correlated Data, ICML 2021 [paper]

5. Embrace the Gap: VAEs Perform Independent Mechanism Analysis, NeurIPS 2022 [paper]

6. Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning, ICML 2023 [paper]

Causal Representation Learning (CRL)

Observational

Static

1. CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models, CVPR 2021 [paper]

2. Weakly Supervised Disentangled Generative Causal Representation Learning, JMLR 2022 [paper]

3. SCM-VAE: Learning Identifiable Causal Representations via Structural Knowledge, IEEE Big Data 2022 [paper]

4. Learning Causally Disentangled Representations via the Principle of Independent Causal Mechanisms, IJCAI 2024 [paper]

5. Unpaired Multi-Domain Causal Representation Learning, NeurIPS 2023 [paper],

6. Causal Representation Learning Made Identifiable by Grouping of Observational Variables, ICML 2024 [paper]

7. Causal Representation Learning from Multiple Distributions: A General Setting, ICML 2024 [paper]

8. Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity, NeurIPS 2024 [paper]*

9. Identifying General Mechanism Shifts in Linear Causal Representations, NeurIPS 2024 [paper]*

10. Multi-Domain Causal Representation Learning via Weak Distributional Invariances, AISTATS 2024 [paper]

11. Multi-View Causal Representation Learning with Partial Observability, ICLR 2024 [paper]

12. A Sparsity Principle for Partially Observable Causal Representation Learning, ICML 2024 [paper]

13. Identifiability Guarantees for Causal Disentanglement from Purely Observational Data, NeurIPS 2024 [paper]*

14. From Causal to Concept-Based Representation Learning, NeurIPS 2024 [paper]

Temporal

15. Learning Temporally Causal Latent Processes from General Temporal Data, ICLR 2022 [paper]

16. Temporally Disentangled Representation Learning, NeurIPS 2022 [paper]

17. Temporally Disentangled Representation Learning under Unknown Nonstationarity, NeurIPS 2023 [paper]

18. CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process, ICML 2024 [paper]*

19. Causal Temporal Representation Learning with Nonstationary Sparse Transition, NeurIPS 2024, [paper]*

Interventional

Static

20. Interventional Causal Representation Learning, ICML 2023 [paper]

21. Linear Causal Disentanglement via Interventions, ICML 2023 [paper]

22. Learning Linear Causal Representations from Interventions under General Nonlinear Mixing, NeurIPS 2023 [paper]

23. Identifiability Guarantees for Causal Disentanglement from Soft Interventions, NeurIPS 2023 [paper]

24. Causal Component Analysis, NeurIPS 2023 [paper]

25. Nonparametric Identifiability of Causal Representations from Unknown Interventions, NeurIPS 2023 [paper]

26. Score-based Causal Representation Learning From Interventions, CRL@NeurIPS 2023 [paper]

27. General Identifiability and Achievability for Causal Representation Learning, AISTATS 2024 [paper]

28. Identifying Linearly-Mixed Causal Representations from Multi-Node Interventions, CLeaR 2024 [paper]

29. Linear Causal Representation Learning from Unknown Multi-node Interventions, NeurIPS 2024 [paper]*

30. Sample Complexity for Interventional Causal Representation Learning, NeurIPS 2024 [paper]*

Temporal

31. CITRIS: Causal Identifiability for Temporal Intervened Sequences, ICML 2022 [paper]

32. Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems, ICLR 2023 [paper]

33. BISCUIT: Causal Representation Learning from Binary Interactions, UAI 2023 [paper]

34. Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA, CLeaR 2022 [paper]

Counterfactual

35. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, NeurIPS 2021 [paper]

36. Weakly supervised causal representation learning, NeurIPS 2022 [paper]

37. Diffusion-based Causal Representation Learning, Entropy 2024 [paper]

Controllable Counterfactual Generation (CCG)

GAN-based

1. CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training, ICLR 2018 [paper]

2. Counterfactual Generative Networks, ICLR 2021 [paper]

3. Counterfactual Generation under Confounding, CML4Impact@NeurIPS 2022 [paper]

Flow/VAE-based

4. Deep Structural Causal Models for Tractable Counterfactual Inference, NeurIPS 2020 [paper]

5. High Fidelity Image Counterfactuals with Probabilistic Causal Models, ICML 2023 [paper]

6. Counterfactual Image Editing, ICML 2024 [paper]*

7. Deep Backtracking Counterfactuals for Causally Compliant Explanations, TMLR 2024 [paper]*

Diffusion-based

8. Diffusion Causal Models for Counterfactual Estimation, CLeaR 2022 [paper]

9. Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models, ECAI 2024 [paper]*

10. Modeling Causal Mechanisms with Diffusion Models for Interventional and Counterfactual Queries, TMLR 2024 [paper]

Evaluation Metrics

Causal Representation Learning/Disentanglement

Disentanglement, Completeness, and Informativeness (DCI)

• A Framework for the Quantitative Evaluation of Disentangled Representations, ICLR 2018 [paper]
• DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability, ICLR 2023 [paper]

Interventional Robustness Score (IRS)

• Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness, ICML 2019 [paper]

Mean Correlation Coefficient (MCC)

Mutual Information Gap (MIG)

• Isolating sources of disentanglement in variational autoencoders, NeurIPS 2018 [paper]

Causal Disentanglement Score (CDS)

• Exploring the Latent Space of Autoencoders with Interventional Assays, NeurIPS 2022 [paper]

Counterfactual Generation

Counterfactual Latent Divergence (CLD)

• Diffusion Causal Models for Counterfactual Estimation, CLeaR 2022 [paper]

Axiomatic Evaluation (Composition, Reversibility, Effectiveness)

• Measuring axiomatic soundness of counterfactual image models, ICLR 2023 [paper]
• Benchmarking Counterfactual Image Generation, NeurIPS 2024 [paper]

Datasets

Synthetic

Real-World

Temporal Video

Biological/Medical

Libraries

Applications

Marrying Causal Representation Learning with Dynamical Systems for Science, NeurIPS 2024 [paper]*

Learning Identifiable Factorized Causal Representations of Cellular Responses, NeurIPS 2024 [paper]*