2 offline days

We will talk about the schedule, sessions, and share the information. Come to the room or join the broadcast to find out what to expect soon!

In many business tasks we rely on our DWH, Data Lake, LakeHouse, etc. In the image and likeness of how OLAP spreadsheets did it years ago. But business tasks and data processes have changed a lot since then, and for some businesses this approach is fundamentally wrong, because they have a different nature of data than they had decades ago. The speaker will talk about: how data is different in today's businesses; the approach that Google proposed in its 2015 article; the problems this approach solves; the new problems it creates, and what to do about them now.

Does it hurt to replace a well-established BI tool for years with something new, and even open source? Ilya will tell you how they chose the tool, what difficulties they encountered and how users reacted to it.

Collecting the base of almost all publicly available datasets is not that difficult. All that matters is to collect their primary sources and properly build the architecture for data set collection and analysis...

To improve the efficiency of YTsaurus, the team decided to remove blobs and store them separately from "normal" tabular data. They had to modify compaction algorithms in a special way to be able to collect "garbage" among the blocks and to provide a suitable tradoff between the disk space (space amplification) and the amount of permanently overwritten data (write amplification). They also took an approach to a number of tables, which were kept in RAM. As a result, we moved (under the guise of blobs!) some of their data to disks and reduced RAM consumption by several times, while maintaining low read times at high quantiles. In the process of implementation, the IO-stack had to be significantly improved by switching to io_uring, and the block-storage layer by adding a consistent hashing algorithm to choose the method of data replicas arrangement.

Alexey will talk about data depersonalization methods, consider the risk assessment model, usefulness and anonymization metrics. Spark and Python were used in writing the product for data depersonalization.

NiFi is a very powerful tool, and it can cover a very wide range of tasks. However, there are some tasks that make NiFi not feel very good. The speaker will talk about his view on such tasks.

A talk on how not to use NiFi, what cases NiFi can implement, how to implement them, and why not to do it.

With the exit of foreign vendors, building disaster-resistant data storages has become even more difficult. Surely, many people have faced this problem and understand the difficulties of implementing such storages based on Greenplum. Alexander will talk about possible solutions and the best ways to build them, and show the most successful approaches.

Graph challenge with three levels of complexity, gifts and an awards ceremony at the afterparty.

For OK Hadoop is a key infrastructure component: it is actively used both for product analytics implementation and for recommendation systems production. In terms of volumes it is more than 200 PB in HDFS, 50k vcores, 200 TB RAM. The speaker will talk about clustering in OC, migration of clusters to internal container cloud. In the report we expect details of the final solution, overview of the migration rake and benefits of the approach.

The architecture of a distributed opensource SQL engine is described. On exectuting queries the engine loads the data in memory . The calculation is divided into stages that can be performed on different nodes. Cross-cluster queries are supported. User defined functions are supported.

The speaker will talk about the perennial problem with data quality and detail why and how they built the data quality platform at SberHealth. He will reveal the work with great expectations, integration with the data catalog (DataHub) and tell what happens after they find "broken" data.

If you know the rules of working with ClickHouse, you can process hundreds of millions of data in a matter of seconds. After analyzing the experience of using it, the speaker will tell you about the most popular and effective ways to speed up queries. Indices — with duplicates will not help, but significantly reduce the amount of scanned data. Projections — what, how and why. Sharding — how to scale horizontally.

The speaker will talk about the Apache Flink streaming framework. You will learn about the architecture and the main components of Flink, using event stream deduplication as an example.

The speaker will talk about the move from the legacy data stack to the new version. You will learn what features have been revealed in almost half a year of using the new one, and why sometimes it's worth trying new technologies.

Astronomers are cramped on Earth: the atmosphere is in the way, Ilon Musk's satellites are in the way, and the size of the planet is too small. Now space for astronomers has become not only an object of research, but also a working platform. What new things have scientists learned with the help of space telescopes and what are the prospects?

Experts of Yandex Search and Advertising Technologies will summarize the results and award the winners and participants of the game.

Устали думать во время докладов? Тогда приглашаем подумать на творческой викторине! Совместно с Лигой Индиго проведем интеллектуальный квиз, где вы сможете испытать интеллект и эрудицию и отдохнуть в компании других участников конференции.

We will talk about international space cooperation and competition. We will discuss modern domestic instruments, as well as issues of import substitution in software and hardware for space observations. We will consider the possibilities of programmers to make useful changes in space science.

The speaker will cover the implementation and practical use of key optimizations that allow Trino and related commercial products to quickly "grind" data from your lake: using Parquet and ORC metadata to reduce the amount of read-out data (project/filter/aggregate pushdown), dynamic filtering (runtime filtering), late materialization of columns (late materialization), and as many as three local caches: metadata cache, data cache and intermediate query results cache.

How can you build a data processing pipeline using cloud services (DataProc and DataSphere), set up interaction with a Spark cluster via Jupyter laptops, and why is it convenient to do it in managed services? How can you teach the system to raise the cluster for you - exactly when you need it, and save money on it? What challenges do companies face when migrating and what solutions do they find? What are the peculiarities of cloud services? What do you need to be prepared for and what improvements might be needed?

The speaker will look at the main challenges facing the implementation of real-time tools in analytical tasks and how they have approached these challenges in their ETL tool.

The team had an ambitious task: to combine two full-fledged data warehouses Delivery Club and Yandex Food in just over eight weeks and, before backend integration, to provide reporting with basic business metrics and data on Delivery Club. Olga will tell how they implemented this project, how they collected the task scoop, evaluated them and adjusted in the process. She will also talk about the technical implementation of improvements on DWH: what solution architecture was invented, what stack was used and why. Of course, there were some pitfalls: we will discuss which ones were stepped on and how to avoid them.

Integration of OLTP and OLAP functions in a distributed database - overcoming traditional barriers towards a universal solution. Alexey will talk about their process of developing such a solution that combines OLTP and OLAP functionality to perform both transactional and analytical queries simultaneously - YDB. He will discuss the main architectural features of such a system, compare it with ClickHouse and other standard solutions, and share his experience in implementing and using this database in real projects.

A talk about the path an engineer goes through to choose their solution for Model Serving. We'll talk about cloud tools, ready-made Inference Servers, their features and selection criteria.

From the talk you will learn about data compression and encryption on disk and in memory in the context of a distributed database, the advantages and disadvantages of both approaches. The speaker will also consider other options for data transformation, such as filtering, and ways to implement them in an open source product.

The speaker will talk about the idea of using open source BI, piloting different solutions, assembling a BI tool team and evolving to meet the requirements of new users and teams migrating from proprietary software (Tableau, Power BI).

We will consider working with Single Message Transformations (SMT) in Kafka Connect in general, and in Debezium in particular. The speaker will tell what SMT is, how to use it in practice, will review the implementation process with code examples. He will cover the pitfalls, discuss customization and configuration, and provide examples of use cases in real-world scenarios.

About using dbt. How to use it, how to customize it. About writing materializations, DDL generator, and problems with temporary tables.

Vladimir will share their experience of building a data management platform around YT, tell where it is good, and where it can be supplemented with different frameworks or other analytical bases. This topic can be useful for architects and data engineers who are going to build a new DWH or revise the architecture of an existing one, and are facing the hard question of choosing technologies from the Open Source world.

We'll talk about what linearizable consistency is good for, various ways to achieve it in a distributed system and the tradeoffs it demands.

Apache Spark Streaming is versatile enough and has rich functionality. But there are tasks, where Spark Streaming is not the best solution, and it can become more of a burden than an effective solution. Evgeny will talk about the advantages and disadvantages of Spark Streaming: when it is worth using this particular tool, and when it is better to consider other options. He will also make a checklist for using Spark Streaming in projects.

The essence of the problem: since GreenPlum has unshared resources and operates at the speed of the slowest segment, situations may arise in which some resources are underutilized or utilized unevenly, which negatively affects the optimality of executed queries. In highly loaded industrial systems it is not possible to manually analyze the optimality of all requests. And some queries can have a negative impact on all processes on a GreenPlum cluster. The speaker will tell you how to solve these problems.

In the talk we will study the problem of development and testing of distributed systems, consider the TLA+ specification language and its application for program verification. In addition, we will describe the method of testing distributed systems based on the actor model, which combines the advantages of both fuzzing and TLA+.

Lessons learned in developing complex products in the face of foreign partners leaving markets, as well as development tools the company has created to make it easier for studios and freelance developers to enter the market.

We take stock, remember the bright moments and talk about our plans. Come to the room or join the broadcast, so you don't miss anything!