The Lakehouse Built for Everyone
Published by Self-published · Tech
From laptop prototypes to enterprise-scale agentic AI with Apache Iceberg and Dremio.
Who this book is for
Data engineers, analytics engineers, software developers, and technical leaders who want practical blueprints for an Apache Iceberg lakehouse, whether they are leaving a legacy cloud warehouse, adding generative AI to BI, or learning the Iceberg ecosystem.
Table of contents
- The Foundation: Data Lakehouses, Parquet, and Arrow
- The Metadata Layer: Apache Iceberg and Apache Polaris
- The Compute Ecosystem: Dremio, DuckDB, and Polars/Datafusion
- The Modern Data Platform Dilemma
- The Heart of the Lakehouse: Deep Dive into Iceberg
- Setting Up Your Dremio Open Catalog
- Ingesting Data with PyIceberg
- Analytical Engines: DuckDB and DataFusion
- When Your Laptop Isn't Enough: Knowing When to Scale
- Introducing Dremio: Scaling Down to Minimize Costs
- Federated Queries (Connecting Across Silos)
- Natural Language Analytics (AI Agent Basics)
- Using MCP Server with Local/Cloud Contexts
- Crossing the Boundary (Moving Compute Entirely to Dremio)
- Autonomous Optimization (Let the Engine Do the Work)
- Building a Strict Semantic Layer for the Organization
- Maximizing AI Quality (How the Semantic Layer Powers Agents)
- The Open Foundation (Avoiding Vendor Lock-In)
Excerpt
Data has gravity. The moment you place a terabyte of information into a proprietary database, it begins pulling your budget, your engineering time, and your architectural decisions toward that single vendor. Ten years ago, data engineers accepted this gravity as a law of physics. If you wanted fast, reliable SQL analytics, you paid a premium for an enterprise data warehouse. If you wanted cheap storage for unstructured files, you dumped them into a data lake. Organizations ran both systems simultaneously, building complex pipelines to shuffle data between the cheap swamp and the expensive fortress.
