Explainable AI, Trustworthy AI, and the Role of Provenance
Artificial intelligence (AI) and machine learning (ML) have demonstrated superior performance in diverse fields, surpassing human capabilities in areas like healthcare, autonomous vehicles, criminal justice, banking, and finance. However, the lack of explainability in AI/ML systems presents a critical challenge. The demand for explainable AI (XAI) has driven researchers to shift their focus from complex black-box models to transparent and interpretable models. XAI aims to make AI/ML results understandable and explainable to humans, and it has gained widespread recognition in academia, industry, and government. Although various approaches and methodologies exist, each with its own strengths and limitations, provenance emerges as a promising solution.
Provenance refers to the documentation of the origin and history of data and processes within an AI/ML system. In the context of XAI, provenance highlights the importance of recording and understanding the details of workflows, facilitating human decision-making and enhancing trust in AI/ML outcomes. By capturing information such as the data sources, preprocessing steps, model training, and inference processes, provenance provides a comprehensive trail that can be analyzed and interpreted to gain insights into the decision-making process of AI/ML models. In this chapter, we delve into the fundamental concepts of XAI, trustworthy AI (TAI), and provenance, offering a comprehensive overview of the latest research and practices.
Drawing upon the domain of earth science, we emphasize the significance of provenance documentation in AI/ML systems. The earth science domain often deals with complex and interconnected processes, where understanding the lineage and transformations of data is crucial for reliable and interpretable predictions. Provenance enables scientists and analysts to trace back the origin of data, assess its quality, and understand how it has been processed and transformed during the AI/ML workflow. By examining technical approaches and methodologies, we aim to highlight the value of provenance in enhancing the trustworthiness and interpretability of AI/ML systems.
In our discussion, we propose several directions for AI/ML systems to further explore in order to leverage the potential of provenance, XAI, and TAI. We aim to provide valuable insights and inspire experts and professionals from diverse disciplines to recognize the benefits of incorporating provenance in their AI/ML workflows. This chapter serves as a comprehensive reference material, incorporating the latest advancements in XAI, TAI, and provenance, and encourages the adoption of transparent and trustworthy AI practices in various domains.
this chapter has no code