Uncertainty in XAI: Human Perception and Modeling Approaches
Abstract
:1. Introduction
2. What Is XAI?
2.1. Why XAI?
2.2. Requirements of Explanations
2.3. Categorization of XAI Methods
2.4. Predictive Performance vs. Explainability
2.5. Human Perception and XAI
2.6. Evaluation of Explanations
3. What Is Uncertainty?
3.1. Uncertainty in ML
- Aleatoric: Irreducible, due to the non-deterministic nature of the input/output dependency and random noise in the available data. As an example, imagine a self-driving car that relies on various sensors (cameras, LiDAR) to perceive its surroundings and make navigation decisions. Sensor data can be inherently noisy due to factors such as bad weather conditions, sensor limitations, or temporary occlusions. This noise in the input data translates to uncertainty in the car’s perception of the environment and ultimately in its predictions about safe navigation paths.
- Epistemic: Reducible through additional information about the perfect predictor . It comprises two parts:
- (a)
- model uncertainty: How close is the hypothesis space with its best model choice to the perfect predictor ? It is very difficult to describe, often ignored by assuming is included in the hypothesis space:
- (b)
- approximation error: How close is the learned predictor to the best ? The error vanishes as the size of training data increases indefinitely.
3.2. Uncertainty in XAI
4. Modeling Uncertainty in XAI
4.1. Approach 1: Perturbed Input (Variation in x)
Pixel Flip**
4.2. Approach 2: Probabilistic Predictor (Variation in f)
- Deep Ensembles [88], with code repository (https://github.com/Kyushik/Predictive-Uncertainty-Estimation-using-Deep-Ensemble (accessed on 19 May 2024)) and [89];
- Bayes by Backprop [90];
- Discriminative Jackknife (via influence functions) [91];
- Laplace Approximation (https://bookdown.org/rdpeng/advstatcomp/laplace-approximation.html (accessed on 19 May 2024));
- Probabilistic Backpropagation [94], with code repository (https://ymd_h.gitlab.io/b4tf/algorithms/pbp (accessed on 19 May 2024));
- Stochastic Expectation Propagation [95];
- Calibrated Explanations [85], with code repository (https://github.com/Moffran/calibrated_explanations (accessed on 19 May 2024)) and exemplified below.
4.2.1. BNN: Monte Carlo Dropout
4.2.2. Conformal Predictor: Calibrated Explanations
4.3. Approach 3: Stochastic Explainers (Variation in e)
4.3.1. CXPlain
4.3.2. BayesLIME
4.3.3. TCAV
4.3.4. CoProNN
5. Human Perception and Uncertainty in XAI
5.1. Neural and Cognitive Aspects of Uncertainty
5.2. Uncertainty via Explanation Fragility
5.3. Effects of Communicating Uncertainty
6. Discussion
7. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
Abbreviations
ACE | Automated Concept-based Explanations |
BNN | Bayesian Neural Network |
CNN | Convolutional Neural Network |
CoProNN | Concept-based Prototypical Nearest Neighbors |
CXPlain | Causal Explanations for Model Interpretation under Uncertainty |
DNN | Deep Neural Network |
HIL | Human-in-the-Loop |
ICE | Individual Conditional Expectation |
LIME | Local Interpretable Model-agnostic Explanations |
ML | Machine Learning |
PDP | Partial Dependence Plots |
SHAP | Shapley Additive Explanations |
TCAV | Testing with Concept Activation Vectors |
XAI | Explainable Artifical Intelligence |
References
- Angelopoulos, A.N.; Bates, S. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. ar** and Occlusion Strategy for Consistent XAI Benchmarks. ar**+and+Occlusion+Strategy+for+Consistent+XAI+Benchmarks&author=Bl%C3%BCcher,+S.&author=Vielhaben,+J.&author=Strodthoff,+N.&publication_year=2024&journal=ar**v" class='google-scholar' target='_blank' rel='noopener noreferrer'>Google Scholar]
- Xu, H.; Ma, Y.; Liu, H.; Deb, D.; Liu, H.; Tang, J.; Jain, A.K. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review. ar**v 2019, ar**v:1909.08072. [Google Scholar] [CrossRef]
- Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.R.; Samek, W. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE 2015, 10, 0130140. [Google Scholar] [CrossRef]
- Simonyan, K.; Vedaldi, A.; Zisserman, A. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. ar**v 2014, ar**v:1312.6034. [Google Scholar]
- Shrikumar, A.; Greenside, P.; Kundaje, A. Learning Important Features Through Propagating Activation Differences. ar**v 2019, ar**v:1704.02685. [Google Scholar]
- Chiaburu, T.; Biessmann, F.; Hausser, F. Towards ML Methods for Biodiversity: A Novel Wild Bee Dataset and Evaluations of XAI Methods for ML-Assisted Rare Species Annotations. ar**v 2022, ar**v:2206.07497. [Google Scholar]
- Griffiths, R.R.; A Aldrick, A.; Garcia-Ortegon, M.; Lalchand, V.; Lee, A.A. Achieving robustness to aleatoric uncertainty with heteroscedastic Bayesian optimisation. Mach. Learn. Sci. Technol. 2021, 3, 015004. [Google Scholar] [CrossRef]
- Koenker, R.; Hallock, K.F. Quantile Regression. J. Econ. Perspect. 2001, 15, 143–156. [Google Scholar] [CrossRef]
- Romano, Y.; Patterson, E.; Candès, E.J. Conformalized Quantile Regression. ar**v 2019, ar**v:1905.03222. [Google Scholar]
- Wang, Z.; Ku, A.; Baldridge, J.; Griffiths, T.L.; Kim, B. Gaussian Process Probes (GPP) for Uncertainty-Aware Probing. ar**v 2023, ar**v:2305.18213. [Google Scholar]
- Lofstrom, H.; Lofstrom, T.; Johansson, U.; Sonstrod, C. Calibrated Explanations: With Uncertainty Information and Counterfactuals. ar**v 2023, ar**v:2305.02305. [Google Scholar] [CrossRef]
- Bykov, K.; Höhne, M.M.C.; Müller, K.R.; Nakajima, S.; Kloft, M. How Much Can I Trust You?—Quantifying Uncertainties in Explaining Neural Networks. ar**v 2020, ar**v:2006.09000. [Google Scholar]
- Gal, Y.; Ghahramani, Z. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, 20–22 June 2016; Balcan, M.F., Weinberger, K.Q., Eds.; Volume 48, Proceedings of Machine Learning Research. pp. 1050–1059. [Google Scholar]
- Lakshminarayanan, B.; Pritzel, A.; Blundell, C. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. ar**v 2017, ar**v:1612.01474. [Google Scholar]
- Yang, C.I.; Li, Y.P. Explainable uncertainty quantifications for deep learning-based molecular property prediction. J. Cheminform. 2023, 15, 13. [Google Scholar] [CrossRef]
- Blundell, C.; Cornebise, J.; Kavukcuoglu, K.; Wierstra, D. Weight Uncertainty in Neural Networks. ar**v 2015, ar**v:1505.05424. [Google Scholar]
- Alaa, A.M.; van der Schaar, M. Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions. ar**v 2020, ar**v:2007.13481. [Google Scholar]
- Graves, A. Practical Variational Inference for Neural Networks. In Advances in Neural Information Processing Systems; Curran Associates, Inc.: Nice, France, 2011; Volome 24. [Google Scholar]
- Bishop, C.M. Pattern Recognition and Machine Learning (Information Science and Statistics); Springer: Berlin/Heidelberg, Germany, 2006. [Google Scholar]
- Hernández-Lobato, J.M.; Adams, R.P. Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks. ar**v 2015, ar**v:1502.05336. [Google Scholar]
- Li, Y.; Hernandez-Lobato, J.M.; Turner, R.E. Stochastic Expectation Propagation. ar**v 2015, ar**v:1506.04132. [Google Scholar]
- Vovk, V.; Petej, I. Venn-Abers predictors. ar**v 2014, ar**v:1211.0025. [Google Scholar]
- Chen, C.; Li, O.; Tao, C.; Barnett, A.J.; Su, J.; Rudin, C. This Looks Like That: Deep Learning for Interpretable Image Recognition. ar**v 2019, ar**v:1806.10574. [Google Scholar]
- Schwab, P.; Karlen, W. CXPlain: Causal Explanations for Model Interpretation under Uncertainty. ar**v 2019, ar**v:1910.12336. [Google Scholar]
- Slack, D.; Hilgard, S.; Singh, S.; Lakkaraju, H. Reliable Post hoc Explanations: Modeling Uncertainty in Explainability. ar**v 2021, ar**v:2008.05030. [Google Scholar]
- Zhao, X.; Huang, W.; Huang, X.; Robu, V.; Flynn, D. BayLIME: Bayesian Local Interpretable Model-Agnostic Explanations. ar**v 2021, ar**v:2012.03058. [Google Scholar]
- Ghorbani, A.; Wexler, J.; Zou, J.; Kim, B. Towards Automatic Concept-based Explanations. ar**v 2019, ar**v:1902.03129. [Google Scholar]
- Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. ar**v 2015, ar**v:1505.04597. [Google Scholar]
- Efron, B. The Jackknife, the Bootstrap and Other Resampling Plans; Society for Industrial and Applied Mathematics: Philadelphia, PA, USA, 1982. [Google Scholar] [CrossRef]
- Deng, L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Process. Mag. 2012, 29, 141–142. [Google Scholar] [CrossRef]
- Krizhevsky, A. Learning Multiple Layers of Features from Tiny Images; University of Toronto: Toronto, ON, Canada, 2012. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. ar**v 2015, ar**v:1409.1556. [Google Scholar]
- Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. ar**v 2021, ar**v:2112.10752v2. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the CVPR09, Miami, FL, USA, 20–25 June 2009. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. ar**v 2015, ar**v:1512.03385. [Google Scholar]
- Jacovi, A.; Marasović, A.; Miller, T.; Goldberg, Y. Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual, 3–10 March 2021. [Google Scholar] [CrossRef]
- Lee, J.D.; See, K.A. Trust in Automation: Designing for Appropriate Reliance. Hum. Factors 2004, 46, 50–80. [Google Scholar] [CrossRef] [PubMed]
- Kepecs, A.; Mainen, Z.F. A computational framework for the study of confidence in humans and animals. Philos. Trans. R. Soc. B Biol. Sci. 2012, 367, 1322–1337. [Google Scholar] [CrossRef] [PubMed]
- Walker, E.Y.; Pohl, S.; Denison, R.N.; Barack, D.L.; Lee, J.; Block, N.; Ma, W.J.; Meyniel, F. Studying the neural representations of uncertainty. Nat. Neurosci. 2023, 26, 1857–1867. [Google Scholar] [CrossRef]
- Pouget, A.; Drugowitsch, J.; Kepecs, A. Confidence and certainty: Distinct probabilistic quantities for different goals. Nat. Neurosci. 2016, 19, 366–374. [Google Scholar] [CrossRef] [PubMed]
- Zhang, Y.; Song, K.; Sun, Y.; Tan, S.; Udell, M. "Why Should You Trust My Explanation”? Understanding Uncertainty in LIME Explanations. ar**v 2019, ar**v:1904.12991. [Google Scholar]
- Ghorbani, A.; Abid, A.; Zou, J. Interpretation of Neural Networks is Fragile. ar**v 2018, ar**v:1710.10547. [Google Scholar] [CrossRef]
- Yin, M.; Vaughan, J.W.; Wallach, H. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Glasgow, UK, 4–9 May 2019; pp. 1–12. [Google Scholar] [CrossRef]
Name | Local/Global | Specific/Agnostic | Modality | Input Type | Task |
---|---|---|---|---|---|
CXPlain (https://github.com/d909b/cxplain (accessed on 5 May 2024)) [98] | local | agnostic | feature attribution | image, tabular, text | classification, regression |
BayesLIME (https://github.com/dylan-slack/Modeling-Uncertainty-Local-Explainability (accessed on 5 May 2024)) [99] BayLIME (https://github.com/x-y-zhao/BayLime (accessed on 5 May 2024)) [100] | local | agnostic | feature attribution | image, tabular | classification |
TCAV (https://github.com/tensorflow/tcav (accessed on 5 May 2024)) [33] & ACE [101] | local & global | specific | concepts | image | classification |
CoProNN (https://github.com/TeodorChiaburu/beexplainable (accessed on 5 May 2024)) | local & global | specific | concepts, examples | image | classification |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chiaburu, T.; Haußer, F.; Bießmann, F. Uncertainty in XAI: Human Perception and Modeling Approaches. Mach. Learn. Knowl. Extr. 2024, 6, 1170-1192. https://doi.org/10.3390/make6020055
Chiaburu T, Haußer F, Bießmann F. Uncertainty in XAI: Human Perception and Modeling Approaches. Machine Learning and Knowledge Extraction. 2024; 6(2):1170-1192. https://doi.org/10.3390/make6020055
Chicago/Turabian StyleChiaburu, Teodor, Frank Haußer, and Felix Bießmann. 2024. "Uncertainty in XAI: Human Perception and Modeling Approaches" Machine Learning and Knowledge Extraction 6, no. 2: 1170-1192. https://doi.org/10.3390/make6020055