We can’t talk about a Data Platform without also thinking about an ingestion layer. It’s where most of the decentralization away from the traditional BI team model comes from, and it’s also how we take a step toward Real-Time and make it possible to build more reactive Data Products.

Before architecting the ingestion layer, I laid out the main goals I wanted to solve:

  • Fully automate ingestion, minimizing the need for Data Engineering teams to centralize any part of the process
  • Make data and its history available for analytical querying as close to Real-time as possible
  • Support building reactive, near Real-Time Data Products

Streaming Platform

We decided to start building the Ingestion Layer at the point where data would transit before being persisted to our Permanent Memory (S3). We needed to move data with the lowest possible latency to get closer to Real-Time and also make it possible for other systems and teams to consume it.

In the streaming platform, the assumptions are that data will travel with very low latency, with a guarantee that the information about its arrival order won’t be lost, and with the ability to make that data volatile to avoid unnecessary costs.

And when we talk about Big Data, the choice is almost obvious: Apache Kafka. Let’s list some of the characteristics that make it the first choice for this:

  • Distributed tool
  • Guaranteed order in data consumption
  • Fault-tolerant data storage
  • Reactive data processing
  • Useful abstractions within the data universe (KStream/KTable)
  • Compliance with the current Big Data ecosystem

To stay aligned with best practices in using the tool, we segregated the data that would compose each table in our Consumption Layer into Kafka topics, making sure that data within the same topic shared a compatible structure.

Streaming platform

Schema Governance

Knowing that the data flowing through the Streaming Platform would compose a table in the consumption layer, we needed to fit that data into a tabular format as closely as possible. And when we talk about tabular format, we mean:

  • Row-based structures
  • Typing
  • Constraints
  • Compatibility guarantees for structure evolution

The easiest option would be to think about moving data in a structure-free format, like JSON, with no constraints, almost no typing, and no native support for structure evolution. However, it’s well known that once data is already at rest, the cost of fixing it is much higher, so it makes sense to think about a structure with a stronger contract and the guarantees mentioned above.

Among the available options, I’d say the main ones are: Avro and Protobuf. The internet is full of benchmarks between these two formats, most of them pointing to no significant difference in serialization/deserialization time and data compression, leaving only the difference in how the structure is defined and managed.

In Protobuf, the structure is defined through its own file format, called .proto, which is usually compiled together with the code that will use that structure to serialize and deserialize data. Avro, on the other hand, has a simpler structure definition that can be written in .json, making it easier for humans to read and removing the need to dive into a new domain-specific language.

Beyond that, Avro’s ecosystem includes a tool called Confluent Schema Registry, which lets you store and manage data structures in a single place. These structures are also made available to applications through an API, in most cases removing the need to recompile code whenever the structure changes.

Remember that the main goals of the ingestion layer are autonomy and automation? That’s why we chose to serialize all data flowing through Apache Kafka in Avro — for the autonomy of being able to change its structure directly through a well-known format (.json).

Event schema governance

Proxy

Now that we’ve defined the place and the shape data would travel in, we still need to know how it will arrive. There are countless Apache Kafka and Confluent Schema Registry clients across many languages, capable of reaching this goal in just a few lines of code.

In practice, though, I found a lot of resistance among teams to integrating their systems with Apache Kafka and Avro, so we decided to make everyone’s life easier by putting an HTTP REST Proxy in front of it. Confluent already provides an open-source tool that does exactly this, turning an HTTP request with JSON data into Avro events and populating an Apache Kafka topic. This way, teams only need to implement a simple HTTP client to start populating Kafka topics with Avro data.

REST proxy for ingestion

Persistence

At this point, we have an easy way to collect data, a streaming platform where all data flows in Real-Time, and a guarantee that the data conforms to the structures defined through Avro and the Confluent Schema Registry. What’s left is to move that data from the Streaming Platform to the Permanent Memory of our consumption layer, so it becomes available for querying through the architecture defined in part 2 of this article.

Some tools on the market already do this job, like Confluent’s Kafka Connect, Apache Gobblin, or Pinterest’s Secor. But in my experience with them, none met all the requirements we needed, running into performance issues or, in some cases, generating files with non-customizable sizes that badly affected the Consumption Engine. That’s when we decided to build our own tool: the Dumping Machine.

The tool’s main goal is to move data from transient storage (Apache Kafka) to stationary storage (S3/HDFS). During this transition, we also need to:

  • Transform data from Avro into Parquet to get optimal performance in analytical queries
  • Translate the Avro structure into the Hive Metastore, automating the creation of new tables and updating table structures whenever the Avro structure changes
  • Add each new partition so the new data is recognized by the Computation Engines

The Dumping Machine has served this purpose for quite a while, and in every case with better performance than the other tools, making it the most obvious choice.

Data persistence

Now that the data has been translated into Parquet in our Permanent Memory (S3) and the structure and metadata have been updated in the Catalog Manager (Hive Metastore), that data is already available for querying alongside everything else.

This way, any engineering team can now send their data through a simple request to our REST Proxy, in JSON, so that data gets persisted into a table in our Permanent Memory and becomes available for analytical queries through the Consumption Engine. What’s more, that data will also flow through our Streaming Platform alongside every other team’s data, unlocking even more possibilities for building applications that consume all of this data to make real-time decisions.