Journey to Real-Time Analytics - Lakehouse Is Not the Solution

Building a real-time analytics platform seems like a solved problem. We have streaming, open formats, virtually unlimited storage, and engines capable of querying billions of records. Yet many companies are still stuck with D-1 pipelines—in which today’s data only becomes available the next day—or maintain another infrastructure to react to what is happening now.

The gap between receiving an event and using it analytically in less than a second is the subject of the series The Journey to Real-Time Analytics. Its starting point is the construction of a platform with subsecond latency. Each article will address a concrete obstacle, the available solutions, and the decisions that bring the architecture closer to—or further from—that goal.

But it is not time to present the architecture yet. Some assumptions make the problem look simpler than it really is. The first concerns the Lakehouse: data in open formats, usually on object storage, with table capabilities such as transactions, versioning, and schema evolution.

The first premise is this:

Lakehouse tools such as Apache Iceberg do not enable real-time analytics on their own.

Defining real time

The “real-time” label often hides five different clocks:

1. Ingestion: when the event enters the pipeline.
2. Commit: when the files and the new table version are published.
3. Visibility: when the data appears in a snapshot—that is, a consistent, queryable version of the table.
4. Query: how long the query takes to return.
5. End to end: how much time passes between the event and the consumer’s action.

For this series, the target is the fifth clock: subsecond end-to-end latency. Reducing commit time alone does not solve the other four.

A table may receive data every thirty seconds and take ten seconds to answer a query. Another may respond in less than a second using a snapshot that is five minutes old. Neither necessarily delivers a subsecond answer about the current state.

Apache Iceberg supports streaming writes. Spark Structured Streaming—the Apache Spark mechanism for processing continuous streams—can publish successive micro-batches to an Iceberg table. But “supports streaming” only says that the operation exists. It does not reveal end-to-end latency, how many files will be created, the publishing cost, or who will maintain the table afterward.

The mistake is in this leap:

accepts updates → accepts frequent updates → delivers real-time analytics

A feature list does not settle the question. The path of a single update, from event to result, reveals what remains hidden.

The lifecycle of an update

Consider an inventory change:

{
  "product_id": 42,
  "inventory": 7,
  "event_at": "2026-07-10T14:32:01.250Z"
}

Apache Iceberg is the main example because its specification exposes the mechanism precisely. The problem, however, does not come from the name Iceberg. It appears whenever the update path publishes versions of a table made of files. Other table formats may reorganize part of the cost, but they still need to reconcile publishing frequency with physical granularity.

For the new inventory value to appear in a query, it goes through a path much larger than “writing one row”: accumulation, publishing, reading, and result delivery.

The diagrams in this section are conceptual models. Engines, catalogs, and writers may implement the details differently.

1. Accumulation

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart LR
    classDef active fill:#FFF4DB,stroke:#B45309,color:#1F2937,stroke-width:2px;
    classDef muted fill:#F7F7F5,stroke:#D6D3D1,color:#4B5563,stroke-width:1px;
    A[1. Accumulate events] --> B[2. Publish table] --> C[3. Read new state] --> D[4. Deliver result]
    class A active;
    class B,C,D muted;

The inventory event reaches the streaming job, the application that consumes the continuous stream. Instead of publishing each event separately, the job holds the received records until the next trigger, the configured moment that closes the micro-batch and starts its processing.

If the trigger runs every ten seconds, an update may wait almost ten seconds before writing even begins. Shortening this interval reduces the wait, but also reduces the amount of data available to form each file.

2. Publishing

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart LR
    classDef active fill:#FFF4DB,stroke:#B45309,color:#1F2937,stroke-width:2px;
    classDef muted fill:#F7F7F5,stroke:#D6D3D1,color:#4B5563,stroke-width:1px;
    A[1. Accumulate events] --> B[2. Publish table] --> C[3. Read new state] --> D[4. Deliver result]
    class B active;
    class A,C,D muted;

When the trigger fires, the processing engine tasks separate records according to the table’s partitions and operations. They then write files containing new data—or the deletions required to represent an update—to S3.

The Iceberg writer organizes the produced files into manifests—indexes that record which files belong to the table—and prepares the metadata for the new snapshot. The catalog, the service that points to the current version, atomically swaps the old reference for the new one.

That swap is the commit. Until it happens, the new inventory value remains invisible. Afterward, the version exists—but it still needs to be found and queried.

3. Reading

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart LR
    classDef active fill:#FFF4DB,stroke:#B45309,color:#1F2937,stroke-width:2px;
    classDef muted fill:#F7F7F5,stroke:#D6D3D1,color:#4B5563,stroke-width:1px;
    A[1. Accumulate events] --> B[2. Publish table] --> C[3. Read new state] --> D[4. Deliver result]
    class C active;
    class A,B,D muted;

After the commit, the query engine needs to load the current metadata. Starting from the snapshot, it uses manifests, partitions, and statistics to discover which files contain data relevant to the query.

The workers, processes that execute parts of the query in parallel, open those files in object storage, read the required columns, and perform filters, joins, or aggregations. Publishing the snapshot made the update visible; it did not eliminate query planning and execution time. The consumer sees the change only after that work is complete.

4. Result

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart LR
    classDef active fill:#FFF4DB,stroke:#B45309,color:#1F2937,stroke-width:2px;
    classDef muted fill:#F7F7F5,stroke:#D6D3D1,color:#4B5563,stroke-width:1px;
    A[1. Accumulate events] --> B[2. Publish table] --> C[3. Read new state] --> D[4. Deliver result]
    class D active;
    class A,B,C muted;

The consumer receives inventory = 7 only when the query finishes. End-to-end latency includes the wait for the micro-batch, the write, the commit, snapshot discovery, and query execution. Ingestion is not visibility; visibility is not query latency.

Shortening the interval between snapshots reduces only one part of that sum and leaves less data in each publication. A seemingly simple latency adjustment changes the table’s physical shape.

From one small table to an entire platform

The cost becomes clearer at two scales: a low-volume table and a platform with 1,000 tables.

One table receiving 1 GB per hour

Assume a constant flow of 1 GB per hour. To keep the calculation simple, treat 1 GB as 1,000 MB and ignore compression, partitions, and parallelism:

Publishing interval	Data available per commit	Commits per hour
1 hour	1 GB	1
10 minutes	~167 MB	6
1 minute	~16.7 MB	60
10 seconds	~2.8 MB	360
1 second	~0.28 MB	3,600

The volume does not change. The physical granularity changes completely.

The current Iceberg documentation uses 512 MB as the default target size for data files. The actual value depends on tasks, partitions, compression, and distribution, but the contrast is enough: with a publication every ten seconds, the entire stream provides only 2.8 MB before the engine splits it.

Greater freshness means replacing large analytical files with a sequence of small publications. Now multiply that decision.

A platform with 1,000 tables

Now imagine a Data Platform with 1,000 tables in Amazon S3, each publishing every 10 seconds:

Frequency	Per table	Across 1,000 tables
Updates per minute	6	6,000
Updates per day	8,640	8,640,000
Updates in 30 days	259,200	259,200,000

That is 259.2 million commits per month, each potentially involving files, manifests, metadata, and a catalog update.

A PUT request is the call used to create or replace an object in S3. On its official pricing page, AWS uses $0.005 per 1,000 PUT requests in its S3 Tables examples. Even assuming only one billable request per commit—an overly optimistic scenario—the result would be:

259,200,000 requests ÷ 1,000 × $0.005 = $1,296/month

If each commit created four objects, the order of magnitude would rise to:

1,036,800,000 requests ÷ 1,000 × $0.005 = $5,184/month

The calculation is not an estimate of a real Iceberg deployment. The number of objects depends on the engine, writer, partitions, and volume. GETs, LISTs, catalog, compute, storage, transfer, retries, queries, and compaction are all excluded. The calculation exposes the order of magnitude hidden behind a local decision of “only ten seconds.”

There is no exact second at which Iceberg “stops working.” That is not the relevant question. Every reduction in waiting time is funded by more objects, commits, coordination, or maintenance.

Compaction

Continuous writes carry little data per commit. The Iceberg documentation itself warns that this can make the table track many small files. Any table format that materializes short micro-batches as new files must manage this fragmentation. More files mean:

• more entries in manifests;
• more objects to locate and open;
• more planning work;
• less efficient scans;
• a need to combine files afterward.

Compaction improves the final state, but does not erase the path taken. The system needs to:

1. locate and read files that have already been persisted;
2. rewrite them into larger files;
3. publish another snapshot;
4. retain and later remove files still referenced by history.

Compaction pays later for the granularity demanded earlier.

The design leaves three structural limits:

Limit	Consequence
Snapshot-based publishing	The update appears only after files, metadata, and commit
File-based granularity	Less waiting produces smaller and more numerous files
Deferred maintenance	Compaction recovers efficiency by rewriting data already persisted

Nothing prevents operating within these limits. The mistake is treating them as natural properties of a layer responsible for the latest state. Lakehouse makes more sense when applied to the problem it was built to solve.

The role of Lakehouse

Lakehouse emerged to combine the open, economical storage of the Data Lake with capabilities expected from analytical databases. In this design, a table format defines how files form a consistent table and how different engines observe its versions. Apache Iceberg contributes capabilities such as:

• safe schema and partition evolution;
• snapshots, time travel, and rollback;
• serializable isolation;
• optimistic concurrency;
• efficient planning for large tables;
• interoperability across engines.

These capabilities make Iceberg a valuable component for consistency, management, and analytical scale. They also define the role of Lakehouse: the historical layer can participate in the real-time platform without being responsible for serving the latest state.

This is the first premise of the series:

Lakehouse is not synonymous with real time. If a proposal begins and ends with update support in the table format, it still needs to explain where the hot data lives.

Main sources

• Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics
• Apache Iceberg documentation
• Apache Iceberg — Structured Streaming
• Apache Iceberg — Maintenance
• Amazon S3 pricing
• Amazon S3 performance design patterns