The pipeline ran successfully. Every row loaded, every transformation completed, every downstream table refreshed on schedule. And still, three teams looked at the output and asked three different questions about what it actually meant.
The pipeline's job, and where that job ends
Lakeflow has made building and orchestrating data pipelines on Databricks dramatically more accessible. Declarative pipelines, automated dependency management, built-in data quality checks, and unified batch and streaming — the kind of engineering effort that used to take a dedicated platform team months to hand-build is now something a single data engineer can stand up in an afternoon.
For most organizations running on Databricks, Lakeflow has genuinely solved the orchestration problem. Pipelines run reliably, on schedule, with lineage and data quality checks built in. Structural issues get caught before they reach a dashboard — a missing column, a type mismatch, a broken dependency.
And yet ask a data engineer whether a successful pipeline run means the business can trust the output, and the honest answer is often "it depends on who's asking, and why." Not because the pipeline did anything wrong. Because a pipeline's job is to move and transform data reliably — not to decide what that data means to the people who will eventually reason over it.
Lakeflow gets trusted data into the Lakehouse reliably. What it doesn't do is connect that data into the business relationships enterprise AI needs to reason across, because it was never meant to.
Three places the gap shows up
This gap tends to surface in three recurring situations — each one familiar to anyone running a Lakehouse at enterprise scale.
1. A new category, ingested correctly, interpreted differently
The operational question: "Our billing system added a 'suspended' status for accounts under review. The pipeline picked it up perfectly, so why does finance count suspended accounts as active, while customer success counts them as churned?"
Nothing broke. The new status ingested cleanly, exactly as the source system sent it, and the pipeline did precisely what it was built to do. The ambiguity was never technical. Finance and customer success had each built their own working definition of "active" long before "suspended" existed as a status, and neither definition was written down anywhere the other team could see it. A perfectly ingested value still left two teams computing two different numbers from the same underlying data.
2. A relationship no single pipeline was built to see
The operational question: "Which customers are at risk this quarter because a delayed supplier affects components used in products tied to their open orders?"
No single Lakeflow pipeline was designed to answer that question, and it would be unreasonable to expect one to. It requires connecting a supplier delay to the components that supplier provides, those components to the products they go into, those products to open orders, and those orders to the customers behind them — a chain spanning supply chain, manufacturing, and sales data that typically live in separate pipelines built by separate teams for separate purposes.
Each pipeline does its job well. None of them was ever positioned to see the relationship that spans all of them, because that relationship was never anyone's pipeline to build.
3. A metric that means something different depending on which pipeline calculated it
The operational question: "We just learned the APAC team built their own 'active customer' pipeline last year. It excludes accounts under a trial period, and ours doesn't. How many other quiet variations like this exist across the company?"
As organizations scale their use of Lakeflow, more teams build more pipelines independently, each solving a real, local problem well. Nobody sets out to create competing definitions of "active customer", it happens gradually, pipeline by pipeline, because there was never a single, discoverable place where that definition already existed before someone started writing new transformation logic.
Why none of this is a Lakeflow problem
It's worth being precise about what's actually happening here, because it's easy to describe these patterns in a way that sounds like a critique of the platform. It isn't one.
In every example above, Lakeflow executed exactly the logic it was given, exactly on schedule, without error. The gap isn't in pipeline reliability, it's in the layer of business meaning and cross-domain relationships that sits above any individual pipeline's scope. That layer was never Lakeflow's job to build, in the same way it was never Unity Catalog's job to define what "active" means. Each platform capability is doing exactly what it was designed to do.
From governed pipelines to reusable business context
Unity Catalog provides the governed foundation — consistent access control and lineage across every table Lakeflow reads from and writes to. The next challenge for most enterprises isn't more governance. It's making business meaning reusable across the growing number of pipelines, AI experiences, and applications being built on that governed foundation.
That's where Kobai fits. Lakeflow gets trusted data into the Lakehouse reliably. Kobai connects that governed data into the entities and relationships that represent how the business actually operates — customer, order, product, component, supplier — as a connected business context model that sits over the data Lakeflow has already made trustworthy. That context can then support Genie, agents, analytics, and any other application that needs to reason consistently across domains, rather than each consumer rebuilding its own version of the same relationships.
The supplier-delay question above is a useful illustration of why this matters. Answering it well doesn't require a faster pipeline. It requires the customer, order, product, component, and supplier relationships to already exist somewhere as connected business context, so that when a disruption hits, the question can be answered directly instead of requiring someone to manually trace five different pipelines' worth of tables under time pressure.
|
Without Connected Business Context |
With Databricks + Kobai |
|
A new status ingests correctly but is classified differently by different teams. |
Business classification is defined once and reused consistently across teams. |
|
Answering a multi-domain question means manually tracing several pipelines' tables. |
Customer, order, product, and supplier relationships are already connected and queryable. |
|
New pipelines quietly reinvent definitions that already exist elsewhere. |
New pipelines and AI experiences draw from existing, discoverable business context. |
Reliable data and consistent meaning are different problems
Lakeflow has solved a genuinely difficult engineering problem: building and running reliable, well-governed data pipelines at scale, without requiring every team to hand-roll their own orchestration and quality checks. That work is real, and it's why so many data engineering teams have been able to move faster on Databricks than they could before.
Getting the data onboarded reliably was always the first step. Making the business meaning behind it consistent and reusable — across every team, every pipeline, and every AI experience built on top — is the next one.
A pipeline can run perfectly and still leave the business asking what the output means. Reliable data movement and consistent business meaning are different problems. Enterprise AI needs both.