Assessing Data Pipeline Quality & Sanity with Data Angiograms

Taking inspiration from a medical procedure called Coronary Angiography, this talk aims to show how one can perform an end-to-end sanity test of the Data pipeline and also perform Quality tests on the data flowing through the pipeline.

In the talk, I will demonstrate how one can perform a Data Angiogram using your favourite technologies like GitLab Pipelines, GitHub Actions, Docker, test libraries like pytest, JUnit, etc.