Security as a Data Problem, Not a Cost Constraint 

Lakewatch looks like a SIEM (security information and event management) on the surface, but underneath, it applies the lakehouse approach to security. Instead of pushing logs into a closed, pay-per-ingestion system, it lands raw telemetry directly into cloud storage in open formats like Delta Lake and Iceberg. From there, the flow feels familiar to any data engineer.  

Security has always been a data problem, but traditional SIEM tools forced teams to treat it like a cost problem. Most platforms charge based on data ingestion, which leads organizations to filter, drop, or ignore large portions of their logs.

Lakewatch: Technical Highlights

An Open, AI-Native Security Layer Built on the Lakehouse. At its core, Databricks Lakewatch is an open SIEM (security information and event management) built natively on the lakehouse.

Core Capabilities  

• Lakewatch lets organizations retain and analyze full-fidelity security telemetry without sampling or data loss.
• Teams can build, manage, and deploy detection logic as code using version-controlled pipelines.
• AI-driven agents automate threat detection, correlation, and incident investigation at scale.
  

Platform Features  

• Lakewatch  stores security data in open formats like Delta Lake and Apache Iceberg within your cloud environment.
• It separates storage and compute to deliver scalable performance with optimized cost efficiency.
• It integrates ingestion, governance, and analytics using Lakeflow Connect, Unity Catalog, and the Databricks platform.
  
  

What Do Engineers Build with Lakewatch? 

1. Security Pipelines That Work Like Data Pipelines  

Ingestion, transformation, and schema normalization for security telemetry use the same frameworks already in use for analytics workloads. Streaming ingestion, schema evolution, and layered architectures are first-class patterns, not workarounds. Security pipelines become easier to maintain, test, and integrate with the rest of the data platform.

2. Detection as Code  

Teams write, version, and deploy detection logic like software. CI/CD (Continuous Integration and Continuous Deployment) workflows, peer review, and rollback are all standard. This closes one of the most frustrating gaps in traditional SIEM platforms, where detection rules often live in proprietary interfaces with no version history and no testing framework.   

3. Unified Governance Across Security and Business Data  

Unity Catalog applies consistent access control, lineage tracking, and compliance policies across all data in the lakehouse, security telemetry included. Teams stop managing governance in two separate systems and get a single, auditable view of who accessed what and when.  

4. Machine Learning on Full-Fidelity Data  

Because security data lives in the same lakehouse as everything else, ML models for anomaly detection, user behavior analytics, or predictive threat modeling can incorporate non-security features such as transaction history, application usage patterns, and access behavior. More context means better accuracy and fewer false positives. This is only possible when you retain all your data rather than sampling it down to manage ingestion costs.    

5. Security Signals in BI Dashboards  

Risk exposure and security events sit alongside operational KPIs (Key Performance Indicators) in the same dashboards leadership already uses. Instead of siloed security reporting that requires separate tools and separate training, organizations gain a unified view of business performance and security posture at the same time.  

How Does Databricks Lakewatch Work?  

Where Do Organizations Start?  

• Start by identifying visibility gaps, including dropped logs and ignored signals, and build a parallel ingestion path into the lakehouse.
• Audit current SIEM costs to understand what telemetry is being discarded and why.
• Ingest high-value data into a Bronze layer using Lakeflow Connect or Auto Loader.
• Normalize and validate data in the Silver layer using OCSF (Open Cybersecurity Schema Framework) schema.
• Establish governance and access control early using Unity Catalog.
• Introduce AI-driven detection and investigation workflows once the data foundation is stable.
• Focus on gradually shifting from ingestion constraints to operating on complete, high-fidelity data

How Can KPI Partners Accelerate Your Lakewatch Adoption?  

KPI Partners brings over 20 years of experience in analytics and lakehouse architecture. As a trusted Databricks partner, KPI Partners has designed and delivered data platform implementations across industries, specializes in lakehouse architecture, data engineering, governance, and AI-driven workflows, giving organizations a faster and lower-risk path to a working security data platform. In practice, this means organizations avoid the trial-and-error cycle that slows most implementations. Governance frameworks satisfy compliance requirements without bolted-on fixes. AI assisted workflows are built to adapt as the threat landscape changes rather than becoming brittle point-in-time rules.

Rather than treating Lakewatch as a standalone security tool, KPI Partners helps organizations extend their existing Databricks investment into security, applying the same engineering discipline and governance rigor they already rely on for their core data platform. The outcome is a security lakehouse that delivers measurable business impact faster, with lower implementation risk than building it from scratch.