The Path to a Successful Lakehouse: Lessons and the AI-Powered Future

Author: Nageen Gandikota

The Path to a Successful Lakehouse: Lessons and the AI-Powered Future

Part 2 of 2 | By Nageen Gandikota, Distinguished Engineer, GEICO Data Platform

Abstract

Data contracts change culture, not just technology. Collaborate, don’t mandate—meet teams where they are and absorb what they build. Solve the boring orchestration problems first; this builds trust and creates the foundation for AI-powered capabilities. The hardest problems aren’t technical—they’re about earning trust.

In Part 1, we explored why we needed a lakehouse architecture and how we built the Data Producer Experience (DPX) platform—from intelligent multi-engine routing to schema automation. Now we’ll share the lessons that made adoption stick, and the AI-powered future we’re building on this foundation.

Start the series: ← Part 1—How We Built DPX

Lesson 1: Data Contracts Change Culture, Not Just Technology

The problem. For years, our default pattern was “push as-is data to the warehouse and figure out the schema later.” Data engineers would receive raw dumps from upstream systems and spend days reverse-engineering what the data meant, which fields were required, and how entities related to each other. The hidden cost was that understanding schemas was always an afterthought, and the people best positioned to describe the data—the source system owners—were rarely the ones doing it.

The solution. We introduced data contracts: formal agreements between data producers and consumers about schema, quality expectations, and SLAs. The first reaction was resistance. “Just take the data we give you,” was a common refrain. Source teams viewed contracts as bureaucracy.

The breakthrough came when we reframed the conversation. Data contracts aren’t about restricting producers; they’re about establishing shared responsibility. When a source system commits to a schema, they’re making a promise. When that promise is codified and versioned, both sides can build reliable systems. To make adoption practical, we made the supporting machinery do most of the work: machine-readable schemas stored in our MINT registry (not documentation wikis), automated validation at ingestion time, clear ownership where source teams own their contracts, and versioned evolution supporting backward-compatible changes. For teams unfamiliar with schema design, we built inference utilities that generate contracts from sample data automatically.

We were also deliberate about where humans stay in the loop. Target schema evolution—adding columns, deprecating fields—can be fully automated through versioning. But accepting schema changes from upstream systems requires human judgment: Is this new column needed? Does it contain PII requiring masking? Does it need PCI scrubbing? Is it encrypted upstream, requiring decryption and re-masking for the Conformed and Presentation/Analytics zones? These questions define data contract renewal, not just registration—and they belong with people who understand the business context.

The outcome. Today, new data sources can’t be onboarded without a registered contract. It’s not optional—it’s infrastructure. We’re working toward making schema changes flow as a first-class part of upstream CI/CD, with automated workflows handling routine cases and human review where judgment is needed. RDBMS sources are our first focus area.

Lesson 2: Encode Expertise, Don’t Gatekeep It

As described in Part 1, DPX automatically routes data to the optimal processing engine—choosing between Apache Spark, Apache Flink, Airbyte, or dbt based on the data’s characteristics rather than asking the user to pick. Hiding engine selection behind automation was controversial. Senior engineers worried we were “dumbing down” the platform. But the results spoke for themselves: zero manual engine selection for standard use cases, consistent patterns regardless of who built the pipeline, and significant compute savings through automated right-sizing.

The key insight: expertise should be encoded in systems, not required from every user. Our best engineers’ knowledge now benefits everyone, not just their immediate team.

This same principle extends beyond ingestion. For transformations, our PRISM initiative applies the same philosophy: engineers write transformation logic in their preferred tools; the platform handles configuration, orchestration, and deployment. Pattern libraries offer starting points, not mandates. At every critical decision point, humans remain in control.

Lesson 3: Collaborate, Don’t Mandate

Building a platform is fundamentally different from building an application. Platforms must serve teams you haven’t met yet, solving problems you can’t anticipate.

We tried “build it and they will come”—adoption was minimal. We tried mandating platform usage—this achieved pro forma adoption but not genuine buy-in. The lesson: sustainable platform success comes from solving real problems, not organizational authority.

What actually worked: Hub-and-spoke collaboration

• Meet teams where they are. If a team had a working solution, we offered to enhance it rather than demanding migration.
• Share technology, not just documentation. We published reusable libraries, templates, and patterns. Teams could adopt pieces incrementally.
• Absorb what teams build. When teams needed capabilities we didn’t have, we incorporated their solutions back into the platform so every team benefits.

This required humility. We had to accept that our platform wouldn’t be the center of everyone’s universe. But by being genuinely helpful—not gatekeepers—we built trust that led to organic adoption.

A few patterns played out repeatedly. Some teams had built their own ingestion frameworks over years and were understandably reluctant to migrate. Rather than mandate adoption, we offered to plug DPX’s schema validation, masking, and lineage publishing into their existing flow—they kept their tooling, and gained the governance benefits without rewriting anything. Other teams came to us asking for capabilities we didn’t have yet; we built them together, and the results became part of the platform that everyone benefits from. And when teams worked around platform gaps with parallel solutions, we treated it as a forcing function rather than a violation—shipping the missing capability so the platform path became the obvious choice. Migration followed naturally once the friction was gone.

Platform success isn’t measured by compliance; it’s measured by teams choosing shared capabilities because they’re genuinely better than going alone.

Lesson 4: Solve the Boring Problems First

When we began planning PRISM, the temptation was to build everything at once: visual IDE, pattern library, AI-powered code generation. The complete vision would take 12-18 months.

We made a different choice: orchestration first.

Data engineers don’t struggle to write SQL—they’re good at that. They struggle with everything around the SQL: setting up dbt profiles, creating Airflow DAGs manually, configuring compute resources, deploying to production environments, tracking pipeline metadata.

These tasks were tedious, error-prone, and entirely automatable. So that’s what we automated first. Engineers register their transformation code, and the platform handles everything else—configuration generation, DAG creation, deployment, monitoring.

Time savings for transformation orchestration: 97% (8-12 hours reduced to 20 minutes)

With orchestration solved, we’re now accelerating the code writing itself through PRISM Phase 1: a growing library of 90+ transformation patterns, automated schema discovery, and dbt model generation from schema definitions, with IDE integration on the roadmap.

Only after validating orchestration automation did we begin planning these intelligence features. By then, we had real usage data, credibility from delivering tangible value, and a foundation to build upon rather than replace.

The lesson: solve the boring problems first. Automation of tedious tasks builds trust and creates the foundation for more ambitious capabilities.

What’s Next: The AI-Powered Future

We’re now extending DPX with AI capabilities through an internal intelligent assistant initiative that embeds AI directly into the data platform. Rather than a chatbot that answers questions, we built an operational AI that can take action. The system pairs purpose-built AI agents with platform tools that can read pipeline state, run diagnostics, and execute approved actions—so engineers can ask questions like “Why did my pipeline fail last night?” and get answers grounded in real system data, not generic guesses.

Our vision for PRISM Phase 2 takes this further: automated pipeline development with human oversight. AI agents handle the mechanical stages of pipeline development—interpreting requirements, generating code, testing, and preparing deployment—while human checkpoints govern the decisions that matter most.

This isn’t AI replacing engineers—it’s AI as a co-pilot, amplifying engineer productivity while preserving accountability. The engineer remains in the driver’s seat. The expertise stays with the humans; the tedium moves to the machines.

Reflections

The hardest problems weren’t technical—they were about trust. Engineers had to trust that automation wouldn’t diminish their work, leaders had to trust that AI oversight was genuine, and teams had to trust that the platform served them. Build that trust first, and the technology follows.

For teams embarking on similar journeys:

• Abstract complexity before adding intelligence
• Encode expertise in systems, not in gatekeepers
• Collaborate, don’t mandate—meet teams where they are
• Solve boring problems first—orchestration automation builds trust
• Preserve human oversight at critical decision points

The future isn’t AI replacing data engineers—it’s data engineers equipped with AI, building data products at unprecedented speed while maintaining enterprise governance. We’re building it at GEICO, one pipeline at a time.

Nageen Gandikota is a Distinguished Engineer at GEICO, where he leads the Data Producer Experience platform team. The DPX platform innovations have resulted in multiple patent applications currently under review. PRISM Phase 0 orchestration automation has been delivered with user validation underway; Phase 1 and Phase 2 are in development.