Author: Balaswamy Kaladi: Principal Architect - Data Engineering
Key takeaways:
- For most of the last decade, moving data out of enterprise applications was the hard, expensive part of every analytics program: Custom pipelines and a permanent maintenance tax.
- Databricks Lakeflow Connect turns ingestion into a managed capability, with 100+ connectors, including Workday and Enterprise Resource Planning (ERP) sources, change data capture (CDC), and Unity Catalog governance.
- When ingestion is solved, both the difficulty and the value move downstream, to the governed, cross-functional context your teams and your agents actually act on.
I have spent a lot of my career watching talented engineers spend their days on plumbing. Not on models, not on data products, not on the questions the business was actually asking, but on building and rebuilding the pipes that move data out of Workday, the ERP, and a dozen other systems. For years we treated that as simply the cost of having a data platform. Databricks Lakeflow Connect is one of the clearest signs yet that we no longer have to, and that changes where a data leader should be pointing the team.
The Plumbing Tax We Stopped Questioning
The burden here is not anecdotal. Research from Wakefield Research and Fivetran found that data engineers spend on average 44 percent of their time building and rebuilding pipelines, and that 71 percent of leaders said their end users were making decisions on old or error-prone data. Sit with that for a moment. Close to half of a scarce, expensive team's capacity has gone into keeping the lights on, and even after all that effort, the numbers reaching the business were often stale.
We stopped questioning this because there was no obvious alternative. Every enterprise application spoke its own dialect, every source changed its schema on its own schedule, and every new report meant another pipeline to build and, later, to fix. Ingestion was genuinely the hard part, so that is where the effort went.
What Lakeflow Connect Actually Changes
Lakeflow Connect resets that assumption. It provides more than 100 managed connectors into the Databricks Lakehouse, with the connectors for Workday, Salesforce, and Microsoft SQL Server now generally available and a wider set spanning enterprise applications such as ServiceNow, Microsoft Dynamics 365, and Oracle NetSuite. For the systems most of us care about, that means Workday Human Capital Management (HCM) and custom Workday reports, ERP and procurement records, and IT service data all arriving through connectors that Databricks builds and maintains, not your team.
Two details matter more than the connector count. First, each connector uses change data capture (CDC) to move only what has changed, so pipelines stay incremental and efficient rather than reprocessing everything. Second, the data lands already inside Unity Catalog governance, so lineage, access control, and quality are part of ingestion rather than a cleanup project afterward. Ingestion stops being a project you staff and starts being a capability you switch on.
The Hard Part Moves Downstream
Here is the shift I want every data leader to internalize: When ingestion is solved, the difficulty does not disappear, it relocates. The hard and valuable work is now what you do with unified workforce, finance, and supply chain context once it lands, and how much your teams and your systems can trust it.
The current data on this is telling. The dbt Labs 2026 State of Analytics Engineering report found that 72 percent of teams now prioritize artificial intelligence (AI) assisted coding, while only 24 percent prioritize the testing, observability, and quality controls that keep pipelines trustworthy. The report's own conclusion is one I would put on the wall: AI will not fix a messy foundation, it only makes the lack of discipline more visible. As pipelines shift toward continuous, real-time execution and agents begin acting on enterprise data, that trusted, governed context stops being a nicety and becomes an operational requirement.
Why This Matters Beyond the Data Team
This is where the human resources, finance, and supply chain stakeholders come in, because the value of solved ingestion is cross-functional. When Workday HCM data, ERP financials, and procurement and Supply Chain Management (SCM) records all land in one governed place, a head of human resources can finally see workforce trends without waiting on a custom extract, a finance leader can close and analyze faster on current numbers, and a supply chain leader can connect suppliers, orders, and inventory in one view.
None of that was impossible before, but it was slow and expensive enough that it rarely happened well.
It matters even more for the agentic direction every enterprise is heading toward. An agent is only as capable as the context it can reach. An agent that can see workforce, spend, and supply data together, all governed and current, can answer questions and take actions that a single-domain agent simply cannot. Complete, trusted enterprise context is the raw material of useful agents, and managed ingestion is how you assemble it without a year of pipeline work.
Where KPI Partners Enter
Managed connectors solve the movement of data. They do not, on their own, decide how workforce, finance, and supply chain data should be modeled together, governed, and turned into something a business user or an agent can rely on. That is the work my team focuses on. KPI Partners designs governed ingestion on the Databricks Lakehouse, using Unity Catalog so trust is built in from the first table, and turns the result into cross-functional data products through our enterprise analytics on Databricks and pre-built models for Workday, ERP, and other core systems. From there, our agentic AI practice helps teams put that governed context to work responsibly. The KPI Partners and Databricks partnership is where managed ingestion becomes trusted, AI-ready data rather than just a faster way to move rows.
What to Do with the Time You Get Back
If ingestion is no longer where your best engineers should be spending their days, the question becomes where that reclaimed capacity should go. I would start in four places.
- Move your highest-maintenance sources, starting with the ones that break most often, onto managed connectors, and retire the custom pipelines behind them.
- Put governance in at the point of ingestion, so lineage and access are never an afterthought.
- Invest the freed-up engineering time in cross-functional data products that combine workforce, finance, and supply chain context, since that is where the business value now sits.
- And prepare that context for agents deliberately, because the organizations that give their agents complete, trusted enterprise data will pull ahead of those still stitching sources together by hand.
The Real Advantage
Ingestion being hard was never the goal; It was a constraint we learned to live with. Lakeflow Connect removes enough of that constraint that the advantage shifts to a different question: What can you build, and what can your agents do, when workforce, finance, and supply chain data arrive governed and current by default? The teams that stop celebrating working pipelines as an achievement, and start treating trusted, unified, agent-ready data as the real deliverable, are the ones that will get the most out of this shift.
Ready to Transform Your Data Strategy? Talk to our experts and discover how KPI Partners can accelerate your data and analytics initiatives.
