Reliable and scalable data pipelines at OK

Odnoklassniki has multiple recommendation systems that handle real-time requests from millions of users every day. To maintain the quality of these systems, there are hundreds of pipelines running daily that collect datasets and attributes, train models and roll them out to extensions, run models in batch mode, load attributes into the Feature Store, and much more. But what happens if some of the pipelines stop working?