The Path to a Successful Lakehouse: How We Built DPX
Author: Nageen Gandikota
The Path to a Successful Lakehouse: How We Built DPX
Part 1 of 2 | By Nageen Gandikota, Distinguished Engineer, GEICO Data Platform
Abstract
In just over a year, we built and scaled the Data Producer Experience (DPX) platform from zero to 300+ production pipelines—one of the fastest adoption curves we’ve seen at GEICO. But this isn’t a story about technology alone. It’s about changing how an organization thinks about data—from an afterthought to a first-class citizen. DPX is the unified data platform that brings data from any source into our governed lakehouse and makes it available for downstream analytics and AI, automatically. A lakehouse architecture succeeds when it abstracts complexity without sacrificing governance, and intelligent automation should encode expert knowledge into systems—not require it from every user.
A lakehouse is a data architecture that combines the open, flexible storage of a data lake with the schema, governance, and performance of a data warehouse—a single platform for both raw and curated data, accessible by analytics, ML, and BI workloads.
Why We Needed a Lakehouse
GEICO’s data landscape circa 2023 looked like many large enterprises: fragmented and complex ecosystems. A push toward democratizing data had led individual teams to bring their own technology and bespoke integrations for their specific data sources. Each approach worked locally but created systemic problems:
- Duplicated engineering efforts: Multiple teams solved the same problems independently
- Inconsistent governance: Catalog, Quality, Lineage, and Classification (the four pillars of governance) were inconsistent across different approaches and regulatory burdens
- Expertise bottlenecks: Schema management required specialized knowledge that didn’t scale
- Time-to-Value: Often measured in weeks per source due to engineering overhead
A representative onboarding looked like this: a week to align on schema with the source team, another week to build a custom ingestion script, several days to wire up Airflow scheduling and monitoring, and yet more time to integrate masking policies, register schemas in the right registries, configure data quality checks, and publish lineage—each often handled by a different specialist. By the time data was usable downstream, two to four weeks had passed and most of that effort was plumbing rather than business logic.
We realized the problem wasn’t a lack of technology—it was that our architecture demanded specialized knowledge across too many disciplines: Spark for distributed ingestion, Airflow for orchestration, security tooling for masking and tokenization, and schema modeling for the downstream layers. Every team that wanted to bring in a new source had to either employ all those skills or wait in line for someone who did. We needed a unified technology: a lakehouse that could store data in a technology-agnostic manner, with a templated approach to bring and transform data for data users—without requiring every engineer to be an expert in every tool.
The Lakehouse Vision: From Source to Insight
Our answer was the Data Producer Experience (DPX)—the unified platform that builds and operates the lakehouse on behalf of its users. DPX is built around a core principle: users describe outcomes, the platform handles implementation.
What DPX Is
DPX is a single product with three user-facing surfaces—a Web Portal, a CLI, and a REST API—backed by declarative YAML configuration. A data producer describes what they want (a source, a contract, a transformation) in any of these surfaces, and the platform takes care of everything downstream. Behind those surfaces, DPX automates the full path from a source system to curated, governed data in the lakehouse:
- Schema inference and registration: detect schema from sample data across JSON, XML, CSV, Parquet, and Avro; register and version it in MINT (our Metadata Integration platform) with backward-compatible evolution tracked over time
- Security and privacy by default: apply masking policies driven by business attributes (such as PII and PCI classifications), and orchestrate field-level tokenization and decryption based on the contract—so sensitive data is protected from the moment it lands
- Multi-engine routing: for ingestion, automatically select among Apache Spark, Apache Flink, or Airbyte based on source type; for transformation, generate dbt models or Spark patterns matched to the use case
- Pipeline orchestration: generate Airflow DAGs, deploy through our CI/CD pipelines, and configure schedules and retries from declarative configuration
- Quality and balance control: register data quality checks and balance control validation as part of every pipeline
- Lineage and discoverability: publish schema, lineage, and ownership to our enterprise data catalog and lineage system automatically so downstream consumers can find and trust the data
- Self-service access: integrate with our identity and access management to provision domain access groups and route requests through the right approvals
The lakehouse, in other words, isn’t something users build—it’s something DPX builds for them, every time they register a source.
How This Compares to the Industry
The principles behind DPX aren’t unique to GEICO—they mirror where the data industry is heading. Confluent built its streaming platform around a schema registry and data contracts that enforce schema, quality, and ownership as data is produced—a “shift-left” approach that moves responsibility to the source. Snowflake’s enterprise lakehouse centralizes governance over open Iceberg tables through its Horizon Catalog, with its Openflow service automating ingestion from virtually any source. DPX brings these same ideas—producer-owned contracts, automated governance, and any-source ingestion—into a single platform. The difference: where Confluent specializes in data in motion and Snowflake anchors on its own analytics cloud, DPX stays engine- and vendor-agnostic, orchestrating across both batch and streaming engines (Spark, Flink, Airbyte, Airflow, dbt) and abstracting that choice entirely from the user—while tuning everything to GEICO’s regulatory, security, and organizational realities.
Three-Zone Data Model
Data flows through governed zones, each with clear responsibilities:
Source Systems → S2R Pipelines → RAW ZONE
|
R2C Pipelines
|
CONFORMED ZONE
|
PRESENTATION/ANALYTICS ZONE- Raw Zone: Minimal transformation, source-faithful, full history preserved for replay and debugging
- Conformed Zone: Flattened semi-structured data, light standardization, quality validated
- Presentation/Analytics Zone: Business logic, aggregations, analytics-optimized for BI and ML. Today, engineers write the transformation logic (SQL via dbt models) and DPX handles the configuration, orchestration, and deployment. PRISM, our transformation automation initiative, is extending this with pattern-based code generation—covered in Part 2.
This zone separation is foundational to automation: each layer has clear responsibilities, which lets the platform make intelligent decisions about processing and storage.
Intelligent Multi-Engine Routing
Rather than requiring users to research and configure each engine, DPX selects the right one based on the source type and pipeline shape:
| Data Pattern | Engine | Rationale |
|---|---|---|
| Large batch files | Apache Spark | Distributed processing, memory optimization |
| Real-time change data capture (CDC) | Apache Flink | Event-time semantics, exactly-once processing |
| SaaS API sources | Airbyte | Pre-built connectors, auth management |
| SQL transformations | dbt | Declarative modeling, built-in testing |
Each abstraction layer removes hours of manual configuration from the critical path. A new data source that took 2-4 weeks to onboard end-to-end—from registration through a running production pipeline with data flowing into the Conformed zone—now reaches that state in 1-2 days.
MINT: Making Schemas Intelligent
Schema management is where data contracts become operational. Our Metadata Integration (MINT) platform goes beyond traditional schema registries to provide intelligent schema processing.
When a new data source arrives, MINT analyzes sample data to detect structure across formats (JSON, XML, CSV, Parquet, Avro), infer appropriate data types, recognize embedded structures that need flattening, and produce machine-readable schemas with table structures. Arrays in semi-structured data are automatically modeled as child tables with parent-child relationships, eliminating hours of manual schema design.
MINT also supports backward-compatible schema evolution—additive changes, deprecation, and version tracking. Auto-detection establishes the initial schema; subsequent changes from upstream flow through a contract renewal process with human checkpoints on classification (PII, PCI), masking, and encryption decisions—we cover that flow in Part 2. Not everyone is a schema expert, so MINT integrates with enterprise metadata catalogs and RDBMS sources to automatically pull existing metadata. Teams provide sample data or point to an existing source, and MINT handles the rest.
The Results
The numbers tell the story:
| Metric | Before | After | Improvement |
|---|---|---|---|
| Pipeline onboarding | 2-4 weeks | 1-2 days | 85% faster |
| Orchestration setup (PRISM Phase 0) | 8-12 hours | 20 minutes | 97% reduction |
| Engine selection | Manual research | Automatic | Zero human time |
| Schema registration | Manual documentation | Automatic inference | Days to minutes |
| Quality monitoring | Ad-hoc scripts | Built-in platform | Consistent coverage |
Beyond metrics, we achieved something harder to measure: engineers now focus on business logic, not plumbing. The platform handles the complexity they used to fight. In practical terms, faster pipelines mean GEICO’s analytics, ML, and customer-facing teams get the data they need in days instead of weeks—turning data work from a bottleneck into a competitive advantage.
A representative recent onboarding tells the story well: a producer team registered a new data source in the Portal on a Monday, completed schema enrichment that afternoon, and was running validated production pipelines—with masking applied, lineage published to the enterprise catalog, and quality checks configured—by Wednesday. The same work under the old model would have taken a month and pulled in three or four specialists.
This growth has been among the fastest adoption rates we’ve seen for a data platform initiative. DPX started in Q4 2024 and went live officially in Q1 2025. Within our first year, we supported 300+ active production pipelines across dozens of teams, with adoption accelerating as word spread.
Building the platform was only half the journey. In Part 2, we’ll share the adoption lessons and the AI-powered features we’re building on this foundation.
Continue the series: Part 2—Lessons and the AI-Powered Future →
Nageen Gandikota is a Distinguished Engineer at GEICO, where he leads the Data Producer Experience platform team. The DPX platform innovations have resulted in multiple patent applications currently under review.