Data Platform — The Challenges (Part 1)
The scalability and autonomy challenges involved in building data platforms.
#data-engineering
5 posts
Data Platform — Solution for Consumption (Part 2)
A decoupled architecture for analytical data storage, cataloging, and processing.
From Full Loads to Incremental Processing: Computing Changes Is Not Enough
How monotonicity, coordination, and the information preserved by a pipeline explain the boundaries between full loads and incremental processing.
Data Platform — Solution for Ingestion (Part 3)
An autonomous, near real-time ingestion layer built on Kafka, Avro, and object storage.
SQL Execution Model
How a SQL query becomes an AST, a logical plan, an optimized plan, and execution.