Data Quality Tools: How, What for, Why. T-Bank's Experience

Data quality is the foundation of successful data management. People, processes, and data quality (DQ) tools all play crucial roles in this process.

There are numerous approaches to implementing DQ tools, and there is no one-size-fits-all solution. I will share T-Bank's experience in implementing DQ tools: transitioning from a fully in-house solution to a modular approach based on open source. I will explain the reasons, motivations, and decision-making process behind this transition.

I will discuss our choice between open source libraries Soda and Great Expectations, as well as the specific components of these tools that we had to develop ourselves.

We will explore the decision-making methodology for choosing DQ implementation approaches based on different data management strategies. We will examine various implementations of DQ tools and compare their effectiveness in different scenarios.

This talk will be beneficial for both engineers and managers aiming to implement effective data quality control tools in their organizations.