The first article in this series reached a conclusion that unsettles anyone hoping for a simpler answer: Lakehouse tools are not, by themselves, real-time analytics enablers. Supporting frequent updates is not the same as sustaining them at subsecond latency — the cost resurfaces as commits, snapshots, small files, and compaction.

That conclusion doesn’t remove the Lakehouse from the equation. A real-time analytics platform will still need a layer where data changes and is persisted durably. This article asks a more specific question: given that we’ll build the architecture on some Lakehouse format, which one has the mental model most aligned with a continuously changing workload?

Two kinds of tables

The traditional lakehouse works well when the flow is predictable:

data → processing → lakehouse → query

This flow assumes a specific write pattern: append. But not every table behaves this way, and the difference matters more than it seems.

Append-only tables

An append-only table (Append Table, in Paimon’s vocabulary; Log Table, in Fluss’s) only accepts new records — it never modifies or removes a row that’s already been written. An event log, a clickstream, a history of closed transactions: every new fact becomes a new row.

This is the pattern the traditional lakehouse was optimized for from the start: large columnar files, partitioning, and immutable snapshots all work well when the only operation is “append,” because appending never requires rewriting what already exists. Iceberg, Delta Lake, and Hudi handle this case well.

Primary Key tables

A Primary Key Table is defined by a primary key and accepts insert, update, and delete directly on an existing row. An e-commerce order, a user session, an account balance: any entity whose state is observed via Change Data Capture (CDC) doesn’t behave like a log that only grows. It behaves like a state that’s continuously rewritten.

That’s exactly the pattern that breaks the predictability of the traditional flow. To see why, it’s worth watching what happens to a single row over time.

Consider a customer record, observed via CDC from the customers table of a transactional system. Instead of treating these changes as an abstract block of “INSERT/UPDATE/DELETE,” it’s worth following what happens at each moment — the payload arriving with each event, and exactly where the problem lives.

The diagrams and payloads in this section are conceptual models of a timeline, not the literal representation of a specific CDC tool such as Debezium.

1. Created

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#000000",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        A["t0 · INSERT"] --> B["t1 · UPDATE"] --> C["t2 · UPDATE"] --> D["t3 · DELETE"]
    end
    timeline ~~~ EA["customer_id: 8821<br/>name: Marina Costa<br/>email: [email protected]<br/>plan: trial<br/>address: Av. Paulista, 1000 — São Paulo, Brazil"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class A active;
    class B,C,D inactive;
    class EA detail;
    class timeline invisible;

t0: the customer signs up on the platform. Up to this point, the behavior is indistinguishable from an append-only table — a new row is born.

2. Updated

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#000000",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        A["t0 · INSERT"] --> B["t1 · UPDATE"] --> C["t2 · UPDATE"] --> D["t3 · DELETE"]
    end
    timeline ~~~ EB["customer_id: 8821<br/>plan: trial → pro"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class A,C,D inactive;
    class B active;
    class EB detail;
    class timeline invisible;

t1: the customer upgrades from the trial plan to the paid plan. The t0 row doesn’t disappear from the event history — but for anyone asking “what’s this customer’s plan today,” it’s no longer valid on its own.

3. Updated again

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#000000",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        A["t0 · INSERT"] --> B["t1 · UPDATE"] --> C["t2 · UPDATE"] --> D["t3 · DELETE"]
    end
    timeline ~~~ EC["customer_id: 8821<br/>address: Av. Paulista, 1000 → Rua das Acácias, 120"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class A,B,D inactive;
    class C active;
    class EC detail;
    class timeline invisible;

t2: the customer updates their billing address. There are now three events in the history for the same key — and only the most recent one describes the customer’s current state.

4. Removed

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#000000",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        A["t0 · INSERT"] --> B["t1 · UPDATE"] --> C["t2 · UPDATE"] --> D["t3 · DELETE"]
    end
    timeline ~~~ ED["customer_id: 8821"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class A,B,C inactive;
    class D active;
    class ED detail;
    class timeline invisible;

t3: the customer requests account deletion, and the row is removed from the customers table. The current state, for that key, ceases to exist — but the four events that passed through it remain recorded in the history.

Rebuilding the snapshot

An append-only table would handle this sequence effortlessly: it would write the four events as four new rows, full stop. The problem shows up in the next question: given this history, what is the customer’s current state right now?

Answering that requires ordering the four events by key (customer_id=8821) and by time, applying each one on top of the result of the previous one until reaching a single snapshot — in this case, “customer deleted, no active row.” Doing this reconstruction on every query, reprocessing the entire history, doesn’t scale — and that’s exactly the problem that motivates the mechanisms behind Primary Key Tables, the subject of the rest of this article.

That leads to the question this article tries to answer: were today’s lakehouse formats equally designed for this kind of workload?

Supporting streaming vs. being streaming-first

Every widely discussed Lakehouse format today can participate in a streaming pipeline. That doesn’t mean they share the same mental model — and the difference becomes clear when you look at how the storage keeps the data, and what happens when someone tries to read those changes back as a stream.

Formats like Iceberg organize the table as a sequence of complete snapshots — each snapshot describes, in simplified form, the set of files that represents the entire table at that moment. To see what this means in practice, go back to the sequence for customer 8821 from the previous section and follow what happens with each event.

The diagrams in this section are conceptual models of the snapshot model, simplified for the argument — not the literal representation of how Iceberg (or any specific tool) works internally.

1. INSERT arrives

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        S1["t0 · INSERT"] --> S2["t1 · UPDATE"] --> S3["t2 · UPDATE"] --> S4["t3 · DELETE"]
    end
    timeline ~~~ SD1["customer 8821 · plan: trial"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class S1 active;
    class S2,S3,S4 inactive;
    class SD1 detail;
    class timeline invisible;

Customer 8821 enters as a new data file. That file gets registered in a manifest — an index listing which files belong to the table, along with statistics about them — and an initial snapshot (t0) is published pointing to that manifest. The table’s metadata file is updated to point to that snapshot as the latest one. If someone queries the table now, the query engine reads the metadata, discovers that the current snapshot is t0, opens the corresponding manifest, and reads the single existing data file. Up to this point, the behavior is indistinguishable from an append-only table.

In simplified form, the managed files look like this:

data/
  data-0001.parquet     # customer_id=8821 · plan=trial

metadata/
  manifest-0001.avro    # lists the files valid for snapshot t0
  snap-t0.avro          # snapshot t0 → points to manifest-0001
  v1.metadata.json      # table metadata: current snapshot is t0

2. First UPDATE arrives

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        S1["t0 · INSERT"] --> S2["t1 · UPDATE"] --> S3["t2 · UPDATE"] --> S4["t3 · DELETE"]
    end
    timeline ~~~ SD2["customer 8821 · plan: trial → pro"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class S1,S3,S4 inactive;
    class S2 active;
    class SD2 detail;
    class timeline invisible;

The plan upgrade generates a new data file with the updated version of the row, plus a marker indicating that the previous version is no longer valid. A new manifest is written referencing this updated set of files, a new snapshot (t1) is published, and the table metadata now points to t1 as the current snapshot. A query now reconstructs the state by combining the valid files from the current snapshot with the invalidation markers, arriving at the row with plan=pro — but nothing in this process records that “the plan field changed from trial to pro.” To know that, a downstream consumer would need to compare snapshot t1 with t0 and infer the difference: the snapshot itself records “how the table looks now,” not “what changed.”

data/
  data-0001.parquet     # orphaned: outside the current snapshot, but still on disk
  data-0002.parquet     # new version · plan=pro

metadata/
  snap-t1.avro           # current snapshot → only data-0002 is read

3. Second UPDATE arrives

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        S1["t0 · INSERT"] --> S2["t1 · UPDATE"] --> S3["t2 · UPDATE"] --> S4["t3 · DELETE"]
    end
    timeline ~~~ SD3["customer 8821 · address updated"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class S1,S2,S4 inactive;
    class S3 active;
    class SD3 detail;
    class timeline invisible;

The pattern repeats: the address update generates another data file, another invalidation marker on the previous version, another manifest, and a new snapshot (t2) — with the table metadata updated once more to point to it.

Here a further problem shows up: intermediate snapshots are typically discarded by retention policy or rewritten by compaction, to control storage and metadata cost. If snapshot t1 is no longer available when someone tries to audit the history, the “trial → pro” transition simply stops existing for anyone with access only to the current snapshots.

data/
  data-0001.parquet     # orphaned
  data-0002.parquet     # orphaned since t2 · address changed again
  data-0003.parquet     # new version · plan=pro, address updated

metadata/
  snap-t1.avro           # still exists, but a candidate for retention expiry
  snap-t2.avro           # current snapshot → only data-0003 is read

4. DELETE arrives

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        S1["t0 · INSERT"] --> S2["t1 · UPDATE"] --> S3["t2 · UPDATE"] --> S4["t3 · DELETE"]
    end
    timeline ~~~ SD4["customer 8821 · removed"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class S1,S2,S3 inactive;
    class S4 active;
    class SD4 detail;
    class timeline invisible;

The delete generates a final invalidation marker — with no need for a new data file, since there’s no new version of the row to write. Another manifest, another snapshot (t3), another update to the table metadata. A query now finds no valid version of customer 8821’s row in snapshot t3: it simply doesn’t appear in the result. Compared to the previous snapshot, a consumer can infer that the row disappeared — but by this point, without the intermediate snapshots, it has already lost the history of the two updates that happened along the way.

data/
  data-0001.parquet     # orphaned
  data-0002.parquet     # orphaned
  data-0003.parquet     # orphaned since t3 · row marked as removed, no new file

metadata/
  snap-t3.avro           # current snapshot — no valid file for customer_id=8821

What becomes clear

In this model, reconstructing “what changed” depends on comparing successive states — and that comparison only works if the intermediate states still exist. It’s a model built to answer “what is the table’s state now,” not “what happened to this row over time.” Reading the table as a continuous stream of changes isn’t this model’s default behavior: it’s a reconstruction someone has to do externally, one that gets more fragile the more events and time pass between one read and the next.

That leaves an open question: is there a storage structure where every change is born already recorded as an event — without depending on comparing states or retaining snapshots indefinitely? That’s what the next section explains.

How this would work with an LSM

The question that closed the previous section has an answer: a Log-Structured Merge Tree (LSM Tree) — the same family of structure used by databases like Cassandra and RocksDB to sustain frequent writes without paying the cost of rewriting everything on every change. Instead of keeping snapshots of the final state, the LSM keeps the sequence of operations that arrived, organized by key: every insert, update, or delete enters as a distinct event, without depending on comparing two states or retaining full snapshots.

That still leaves a problem: updating large columnar files on every change isn’t free — a Parquet file of hundreds of megabytes wasn’t designed to be rewritten on every UPDATE. That’s the cost the LSM Tree manages: recent writes arrive as new files at a shallow level of the tree (L0), later reorganized into deeper sorted runs by compaction. The trade-off is explicit, not a free efficiency gain — more sorted runs make reads more expensive, compacting too aggressively slows down writes — and it’s usually a configurable parameter, not a fixed constant of the structure.

Go back once more to the sequence for customer 8821 — the same one used in the snapshot model — and follow what the LSM Tree does with each event.

The diagrams and file listings in this section are conceptual models, simplified for the argument — not the literal representation of Paimon’s internal layout.

1. INSERT arrives

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        O1["t0 · INSERT"] --> O2["t1 · UPDATE"] --> O3["t2 · UPDATE"] --> O4["t3 · DELETE"]
    end
    timeline ~~~ OD1["customer 8821 · plan: trial"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class O1 active;
    class O2,O3,O4 inactive;
    class OD1 detail;
    class timeline invisible;

Customer 8821’s insert enters as a new file in L0, the shallowest level of the tree, inside the bucket responsible for that key. At the same time, the operation is already emitted to the changelog — not as a consequence of some future read, but as part of the write path itself.

bucket-03/
  L0/
    data-0001.parquet   # insert · customer_id=8821 · plan=trial

changelog/
  chg-0001              # insert · plan=trial

2. First UPDATE arrives

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        O1["t0 · INSERT"] --> O2["t1 · UPDATE"] --> O3["t2 · UPDATE"] --> O4["t3 · DELETE"]
    end
    timeline ~~~ OD2["customer 8821 · plan: trial → pro"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class O1,O3,O4 inactive;
    class O2 active;
    class OD2 detail;
    class timeline invisible;

The plan upgrade enters as one more file in L0 — not as a rewrite of the previous file. It’s just one more operation, tied to the same key. The changelog already records this change as soon as it happens: chg-0002 says, unambiguously, that the plan became pro. No need to compare anything with what came before.

bucket-03/
  L0/
    data-0001.parquet   # insert · plan=trial
    data-0002.parquet   # update · plan=pro

changelog/
  chg-0001              # insert · plan=trial
  chg-0002              # update · plan=pro

3. Second UPDATE arrives

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        O1["t0 · INSERT"] --> O2["t1 · UPDATE"] --> O3["t2 · UPDATE"] --> O4["t3 · DELETE"]
    end
    timeline ~~~ OD3["customer 8821 · address updated"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class O1,O2,O4 inactive;
    class O3 active;
    class OD3 detail;
    class timeline invisible;

The address update arrives as a third operation. With L0 accumulating files, compaction kicks in: data-0001 and data-0002 are merged into a deeper sorted run, consolidating the first two operations into a single record — plan=pro. The new L0 file (with the updated address) hasn’t been compacted yet.

bucket-03/
  L0/
    data-0003.parquet   # update · new address

L1 (sorted run)/
  run-0001.parquet      # consolidated: plan=pro (result of compacting data-0001 + data-0002)

changelog/
  chg-0001
  chg-0002
  chg-0003   # update · new address

Compaction physically reorganizes the files, but doesn’t erase anything from the changelog — chg-0001, chg-0002, and chg-0003 are still there, in the order they happened, regardless of when or how the tree decides to compact.

4. DELETE arrives

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart TD
    subgraph timeline[" "]
        direction LR
        O1["t0 · INSERT"] --> O2["t1 · UPDATE"] --> O3["t2 · UPDATE"] --> O4["t3 · DELETE"]
    end
    timeline ~~~ OD4["customer 8821 · removed"]

    classDef active fill:#FFF4DB,stroke:#B45309,color:#000000,stroke-width:2px;
    classDef inactive fill:#F7F7F5,stroke:#D6D3D1,color:#000000,stroke-width:1px;
    classDef detail fill:#EFF6FF,stroke:#2563EB,color:#000000,stroke-width:2px,stroke-dasharray:5 3;
    classDef invisible fill:none,stroke:none;

    class O1,O2,O3 inactive;
    class O4 active;
    class OD4 detail;
    class timeline invisible;

The delete enters as the fourth operation, the same way as the previous three — one more file in L0, marking key customer_id=8821 as removed. A read now combines L0 with the deeper sorted runs and finds no valid record for that key.

bucket-03/
  L0/
    data-0004.parquet   # delete · customer_id=8821

L1 (sorted run)/
  run-0001.parquet      # plan=pro, old address (doesn't reflect the delete yet)

changelog/
  chg-0001
  chg-0002
  chg-0003
  chg-0004   # delete

The central difference from the snapshot model isn’t subtle: at no point was it necessary to compare the current state with a previous one to know what changed. The four changelog entries — insert, update, update, delete — already existed, one by one, the instant each operation was written. Reading this as a stream isn’t a reconstruction; it’s just continuing to consume something the table was already producing.

The pattern that shows up in these four steps isn’t exclusive to one tool: it’s how any LSM Tree behaves, by construction. It’s worth keeping this vocabulary — sorted runs, levels, compaction — because it’s a concept this series will revisit later, in more depth, when the topic is how to serve queries on the most recent data with even lower latency.

Apache Paimon: the Lakehouse using this model

It’s on top of this structure that Apache Paimon organizes its Primary Key Tables. Each primary key defines the unique set of columns per record, and each bucket — the table’s internal partition — is an independent LSM tree, receiving the same L0-and-compaction path described in the previous section on every Flink checkpoint.

Paimon handles this trade-off by offering different table modes: the default, merge-on-read, prioritizes writes and pays the merge cost on read; copy-on-write flips that priority, rewriting more data on every change to keep reads simple.

This only makes sense as a set of three pieces, not as isolated concepts:

  • Primary Key defines what the “current row” is for each key.
  • LSM Tree explains how frequent updates are absorbed without rewriting everything on every change, and how compaction reconciles multiple versions of the same record.
  • Changelog Producer explains how those changes keep flowing to downstream consumers, closing the loop described in the previous sections.

In practice, this combination changes the table’s role in a pipeline. A common design treats the table as the final destination:

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart LR
    classDef muted fill:#F7F7F5,stroke:#D6D3D1,color:#4B5563,stroke-width:1px;
    classDef active fill:#FFF4DB,stroke:#B45309,color:#1F2937,stroke-width:2px;
    K[Kafka] --> F[Flink] --> L[Lakehouse]
    class K,F muted;
    class L active;

Paimon’s model is different:

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#F7F7F5",
    "primaryTextColor": "#1F2937",
    "primaryBorderColor": "#D6D3D1",
    "tertiaryColor": "#FFF4DB",
    "lineColor": "#6B7280",
    "clusterBkg": "#EEEDE8",
    "clusterBorder": "#D6D3D1",
    "fontFamily": "Inter, Segoe UI, Helvetica Neue, Arial, sans-serif",
    "fontSize": "15px"
  },
  "flowchart": {"curve": "basis", "nodeSpacing": 24, "rankSpacing": 30, "padding": 8}
}}%%
flowchart LR
    classDef muted fill:#F7F7F5,stroke:#D6D3D1,color:#4B5563,stroke-width:1px;
    classDef active fill:#FFF4DB,stroke:#B45309,color:#1F2937,stroke-width:2px;
    K[Kafka] --> F1[Flink] --> P1[Paimon] --> F2[Flink] --> P2[Paimon]
    class K,F1,F2 muted;
    class P1,P2 active;

Here, the Paimon table isn’t just a destination — it’s a materialized state, a source for queries, and an incremental source of changes for the next continuous transformation. This is possible because Paimon documents this goal explicitly: “streaming write can continuously produce the latest changes for streaming read” — a second Flink job can consume back exactly what changed in the table, without reprocessing everything, and feed another table or another system.

It’s this chaining — the table as an active participant, not just a final file — that explains why Paimon fits naturally into a streaming architecture.

The limit of Paimon

This article showed why Paimon is, among the formats evaluated, the one most aligned with a continuously changing workload: Primary Key, LSM Tree, and Changelog Producer form a set designed to receive and propagate mutation continuously.

But at the end of the chain, Paimon clearly documents where the data actually lives: “Under the hood, Paimon stores the columnar files on the filesystem/object-store.” Changelog, efficient updates, and incremental reads don’t change that final destination: the persisted data is still a file on durable storage.

That matters because durable storage — even well optimized, even with an LSM, even with a changelog — carries different latencies and access patterns than a consistently subsecond response requires. Continuously receiving changes and producing a changelog solves the problem of keeping a streaming lakehouse up to date. It doesn’t, by itself, solve the problem of answering a subsecond query over that same state.

Streaming ingestion is not the same thing as real-time serving.

The practical takeaway from this article: when evaluating a real-time architecture, it’s worth explicitly separating where data changes and is durably persisted from where queries are answered at the latency the use case demands. A streaming-first Lakehouse format solves the first question well. It doesn’t solve the second one on its own.

That leaves an open question for the rest of the series: if we can keep a lakehouse continuously up to date, how do we make that data available for queries at truly subsecond latency?

Main sources