S3, HDFS, POSIX... or All of the Above? Building a Data Lake the Chinese Way with CubeFS

Modern analytics platforms increasingly demand all of the following at the same time: an S3-compatible API for lakehouse architectures, an HDFS API for the Hadoop ecosystem, POSIX semantics for ML/AI workloads and legacy applications.

And they need all of this while maintaining inexpensive horizontal scalability, efficient resource utilization, and low operational costs.

Chinese hyperscale companies encountered this challenge years ago and responded by building unified storage platforms. In this talk, we'll explore how the distributed storage system CubeFS came to be, the capabilities it provides, how we adapted it for our own use cases, and the challenges we faced along the way.