The medallion architecture
bigdata databricks doc-review Next Learn
- #bigdata #databricks #doc-review
- https://www.databricks.com/glossary/medallion-architecture
- ELT vs ETL
- Extract Load Transform
- bronze layer: speed of ingestion
- silver layer: like 3rd Normal Form of Boyce Codd (avoid repetitions, data types, …)
- gold layer: complex transformations, project specific, business rules
- #Next Learn lakehouse = data-lake + datawarehouse https://www.databricks.com/product/data-lakehouse
- bronze layer
- raw data from external systems
- “as is”
- metadata: load, datetime, process id, …
- goal
- quick Change Data Capture (= events on data change)
- historical archive on cold storage
- data lineage (family tree but for data), compute a trace_id
- audit
- silver layer
- cleansed “just enough” + merged + comformed = “Enterprise view” of key business entities, concepts and transactions
- goal
- self-service analytics
- serve gold layer: a source for other company’s projects
- gold layer
- db ready to consume by apps
- de-normalized, read-optimized, less joins
- #Next Learn Kimball style star schema-based data models or Inmon style Data marts
- benefits
- simple data model
- can recreate from raw data
- time travel
