In this book we argue for the need for a new approach to data provenance and explain how recent advancements in data processing workflow automation present an opportunity to address this need. We introduce descriptive dataflow operators - a novel approach based on integrating descriptive workflow languages with a data modeling Domain-Specific Language (DSL). We review the current workflow automation technologies and propose a DSL that supports complex data transformations, enhances reproducibility, and enables precise data lineage tracking. Within the framework we introduce the concept of descriptive dataflow operators for more flexible and expressive data transformations.
Modern healthcare increasingly relies on complex data pipelines to process diverse diagnostic information, clinical records, and research data. This growing complexity, combined with emerging AI/ML applications and stricter regulatory oversight, demands sophisticated approaches to data preparation, documentation, and validation. Healthcare organizations face mounting pressure to ensure granular traceability and reproducibility of their data transformations while maintaining regulatory compliance. These challenges are particularly acute in research settings, where data provenance and quality validation become critical for scientific reproducibility and regulatory adherence.
Given the increasing complexity of healthcare data, data ingestion and transformation workflows present significant technical challenges, particularly in ensuring the reproducibility and seamless integration of diverse datasets for ML and AI model development.
We introduce the Dorieh Data Platform as an exemplar implementation of a DSL, providing a comprehensive framework for reproducible research. The platform's infrastructure supports robust data lineage documentation, validation, and error logging, making it a powerful tool for healthcare data analysis by ensuring transparent, auditable data processes and regulatory conformance.
We show how to apply this framework to analyze healthcare claims data quality, revealing insights into inconsistencies and deficiencies. Our approach demonstrates the potential for improved data management and accountability in scientific research, underscoring the necessity for precise, reproducible data transformation methodologies to produce reliable research outcomes.
Table of Contents:
Chapter 1. Introduction.- Part I. The Need, the Opportunity and the Solution.- Chapter 2. The Need for Complete Data Provenance.- Chapter 3. The Opportunity: Advancement of Workflow Automation.- Chapter 4. Solution: Descriptive Dataflow Operators.- Chapter 5. Language Design.- Part II. How to Implement It.- Chapter 6. Proof of Concept Implementation.- Chapter 7. Sample Application: Building ML-Ready.- Chapter 8. Dorieh Medicare Claims Data Pipeline.- Chapter 9. What is Next?.- Part III. The Architecture of Trust: Regulation, Provenance and Compliance-As-Code.- Chapter 10. Trust and Regulation as Delegated Understanding.- Chapter 11. The Burden Spiral: When Technology Exceeds Human Oversight.- Chapter 12. The Provenance Imperative.- Chapter 13. From Provenance to Computable Trust.- Part IV. Classification of Data Transformations.- Chapter 14. A Taxonomy of Data Transformations.- Appendix.
