Author: Balaswamy Kaladi: Principal Architect - Data Engineering
I have spent a lot of my career watching talented engineers spend their days on plumbing. Not on models, not on data products, not on the questions the business was actually asking, but on building and rebuilding the pipes that move data out of Workday, the ERP, and a dozen other systems. For years we treated that as simply the cost of having a data platform. Databricks Lakeflow Connect is one of the clearest signs yet that we no longer have to, and that changes where a data leader should be pointing the team.
The burden here is not anecdotal. Research from Wakefield Research and Fivetran found that data engineers spend on average 44 percent of their time building and rebuilding pipelines, and that 71 percent of leaders said their end users were making decisions on old or error-prone data. Sit with that for a moment. Close to half of a scarce, expensive team's capacity has gone into keeping the lights on, and even after all that effort, the numbers reaching the business were often stale.
We stopped questioning this because there was no obvious alternative. Every enterprise application spoke its own dialect, every source changed its schema on its own schedule, and every new report meant another pipeline to build and, later, to fix. Ingestion was genuinely the hard part, so that is where the effort went.
Lakeflow Connect resets that assumption. It provides more than 100 managed connectors into the Databricks Lakehouse, with the connectors for Workday, Salesforce, and Microsoft SQL Server now generally available and a wider set spanning enterprise applications such as ServiceNow, Microsoft Dynamics 365, and Oracle NetSuite. For the systems most of us care about, that means Workday Human Capital Management (HCM) and custom Workday reports, ERP and procurement records, and IT service data all arriving through connectors that Databricks builds and maintains, not your team.
Two details matter more than the connector count. First, each connector uses change data capture (CDC) to move only what has changed, so pipelines stay incremental and efficient rather than reprocessing everything. Second, the data lands already inside Unity Catalog governance, so lineage, access control, and quality are part of ingestion rather than a cleanup project afterward. Ingestion stops being a project you staff and starts being a capability you switch on.
Here is the shift I want every data leader to internalize: When ingestion is solved, the difficulty does not disappear, it relocates. The hard and valuable work is now what you do with unified workforce, finance, and supply chain context once it lands, and how much your teams and your systems can trust it.
The current data on this is telling. The dbt Labs 2026 State of Analytics Engineering report found that 72 percent of teams now prioritize artificial intelligence (AI) assisted coding, while only 24 percent prioritize the testing, observability, and quality controls that keep pipelines trustworthy. The report's own conclusion is one I would put on the wall: AI will not fix a messy foundation, it only makes the lack of discipline more visible. As pipelines shift toward continuous, real-time execution and agents begin acting on enterprise data, that trusted, governed context stops being a nicety and becomes an operational requirement.
This is where the human resources, finance, and supply chain stakeholders come in, because the value of solved ingestion is cross-functional. When Workday HCM data, ERP financials, and procurement and Supply Chain Management (SCM) records all land in one governed place, a head of human resources can finally see workforce trends without waiting on a custom extract, a finance leader can close and analyze faster on current numbers, and a supply chain leader can connect suppliers, orders, and inventory in one view.
None of that was impossible before, but it was slow and expensive enough that it rarely happened well.
It matters even more for the agentic direction every enterprise is heading toward. An agent is only as capable as the context it can reach. An agent that can see workforce, spend, and supply data together, all governed and current, can answer questions and take actions that a single-domain agent simply cannot. Complete, trusted enterprise context is the raw material of useful agents, and managed ingestion is how you assemble it without a year of pipeline work.
Managed connectors solve the movement of data. They do not, on their own, decide how workforce, finance, and supply chain data should be modeled together, governed, and turned into something a business user or an agent can rely on. That is the work my team focuses on. KPI Partners designs governed ingestion on the Databricks Lakehouse, using Unity Catalog so trust is built in from the first table, and turns the result into cross-functional data products through our enterprise analytics on Databricks and pre-built models for Workday, ERP, and other core systems. From there, our agentic AI practice helps teams put that governed context to work responsibly. The KPI Partners and Databricks partnership is where managed ingestion becomes trusted, AI-ready data rather than just a faster way to move rows.
If ingestion is no longer where your best engineers should be spending their days, the question becomes where that reclaimed capacity should go. I would start in four places.
Ingestion being hard was never the goal; It was a constraint we learned to live with. Lakeflow Connect removes enough of that constraint that the advantage shifts to a different question: What can you build, and what can your agents do, when workforce, finance, and supply chain data arrive governed and current by default? The teams that stop celebrating working pipelines as an achievement, and start treating trusted, unified, agent-ready data as the real deliverable, are the ones that will get the most out of this shift.
Ready to Transform Your Data Strategy? Talk to our experts and discover how KPI Partners can accelerate your data and analytics initiatives.