How Do You Put Two Exabytes of Data in Order?

In a company with a rapidly growing amount of data, navigating through the data becomes more difficult every day. Data catalogues help in this situation, but the information in them is usually filled in by users themselves or taken from ERM links of small databases. In our internal DataCatalog we have learnt to automatically collect the Data Lineage of the YTsaurus system based on the logs of ETL operations and ad hoc calculations.

We will tell you how we are trying to become a single point of truth about all company data.