Reference: Adapted and expanded from The 12 Data Architecture Patterns Every Data Engineer Should Master.
The Data Lake vs Warehouse debate is effectively over. Modern enterprise engineering is no longer about picking a single isolated design; it is about knowing how to compose interoperable, composable data patterns into a reliable, governed ecosystem.
1. Storage & Ingestion Foundations
1. Medallion Architecture (The Refinement Ladder)
Purpose: Progressively clean, normalize, and enrich data across quality tiers.
Tiers:
Bronze: Raw ingestion, append-only, immutable historical landing.
Silver: Cleaned, deduplicated, conformed, and joined records.
Gold: Highly aggregated, business-ready models powering BI and ML features.
Why It Works: Decouples raw ingestion from transformation, meaning you never re-pull from source systems—you simply rebuild downstream.
2. Data Lake Architecture (Store First, Schema Later)
Purpose: Cost-effective, scalable object storage for structured, semi-structured, and unstructured data.
Mechanism:Schema-on-Read. Raw data lands with zero upfront modeling; schemas are applied dynamically at query time.
Trade-off: Maximum flexibility, but without active cataloging it becomes an unmaintainable “data swamp.”
3. Data Warehouse Architecture (The BI Workhorse)
Purpose: Highly structured, curated storage optimized for fast SQL analytics and reporting.
Mechanism:Schema-on-Write. Rigorous ETL cleans and models data before persistence, delivering deterministic reporting at the cost of ingest agility.
4. Lakehouse Architecture (The Modern Standard)
Purpose: Blends the cheap, open file storage of a Data Lake (Parquet) with the ACID transactional integrity and schema enforcement of a Data Warehouse.
Core Formats: Apache Iceberg, Delta Lake, and Apache Hudi.
The “Lambda Tax”: Maintaining double the codebase—one set of logic for batch processing and another for stream processing.
6. Kappa Architecture (Everything is a Stream)
Purpose: Eliminates the dual-codebase overhead of Lambda by treating all historical and real-time data as a single continuous log.
Mechanism: Re-processing history is accomplished simply by replaying the streaming log from an earlier offset.
7. Event-Driven Architecture (Systems That React)
Purpose: Decouples distributed microservices through asynchronous pub/sub messaging queues. Services publish events and react to updates independently.
8. Modern End-to-End Streaming
Purpose: Real-time stream processing that immediately commits ACID-compliant events directly into Iceberg or Delta tables with near-zero latency.
3. Governance, Organization & Enterprise Scale
9. Data Mesh (Decentralized Domain Ownership)
Core Philosophy: Treats data as an organizational product. Domain teams (Finance, Marketing) own their own data products and pipelines, supported by a shared self-service infrastructure platform.
10. Data Fabric (The Active Metadata Plane)
Core Philosophy: A centralized, tech-driven software layer that automates discovery, maps global lineage, and enforces access policies across heterogeneous storage silos without moving the physical files.
11. Hub-and-Spoke Architecture
Purpose: A centralized, conformed data warehouse (the Hub) feeds specialized, departmental data marts (the Spokes) tailored to individual team query patterns.
12. Data Vault Architecture (Auditable Historical Modeling)
Components:
Hubs: Unique business keys (The What).
Links: Transactional relationships between keys (The How).
Satellites: Time-stamped descriptive attributes (The History).
How They Fit Together: The Composite Production Stack
Production environments do not pick a single pattern in isolation. A modern retail or financial platform combines several patterns into a unified stack:
Quick Architecture Decision Matrix
Your Business & Technical Requirement
Primary Architectural Patterns to Combine
Modern Analytics & Machine Learning Platforms
Lakehouse + Medallion Architecture + Modern Streaming
Real-Time Dashboards with Continuous Ingestion
Kappa Architecture + Event-Driven Pub/Sub
Strict Regulatory Audits & Point-in-Time History
Data Vault + Lakehouse Storage
Scaling Across Multiple Autonomous Business Units
Data Mesh (Org Model) + Data Fabric (Governance Layer)
