Scale By the Bay Scale By the Bay

Friday, October 29, 2021

- PDT
How We Built SQL Rollups on Streaming Data
Tudor Bosman
Tudor Bosman
Rockset, Chief Architect
Karen Li
Karen Li
Rockset, Software Engineer

Rockset is a real-time analytics database for serving fast search and analytics at scale. We built SQL rollups in Rockset that can pre-aggregate data from streaming sources, like Apache Kafka and Amazon Kinesis. Using rollups can improve storage efficiency and query performance.Over the course of this project, we encountered multiple challenges in building SQL rollups on streaming data, including:
* supporting exactly-once write semantics
* executing SQL on streaming data at ingest time
* processing out-of-order arrivals correctly
In this talk, we discuss how we overcame these technical challenges to implement rollups.