02 Architecture

Architecture Diagram

The Fabric Accelerator uses a medallion architecture. The medallion layers can be configured as files, a lakehouse, or a data warehouse in OneLake, allowing for seamless switching through configuration. In this setup, the bronze layer consists of files, the silver layer is a lakehouse, and the gold layer is a data warehouse.

Dataflow and Components

1. Data Factory pipelines ingest data from both cloud and on-premises sources into the data platform, with on-premises sources requiring an OPDG.
2. Data lands in the bronze layer in OneLake as files, preferably in parquet format, without any transformation.
3. Spark notebooks then transform the raw data in the bronze layer into curated data in the silver layer as Lakehouse tables. In the silver layer, data is cleansed, flattened, and standardized while maintaining its grain, and can be transformed one-to-one or one-to-many Lakehouse table(s).
4. Data warehouse stored procedures apply business rules to data from the Lakehouse tables in the silver layer, landing it as DW tables in the gold layer. Typical activities in the gold layer include applying custom business rules, creating snapshots, merging data from multiple sources, and forming hub-spoke star schemas. A Lakehouse table from the silver layer can be transformed one-to-one, one-to-many, or many-to-one DW table(s) in the gold layer.
5. Semantic models built on the gold layer DW tables serve as the Analytic layer, also known as the diamond layer, establishing relationships between tables.
6. The orchestration of this fabric accelerator is underpinned by the ELT Framework, a metadata-driven orchestration tool that streamlines ingestion and transformation pipelines.
7. The ELT framework uses an Azure SQL (Serverless) database for metadata, mirrored into the Fabric workspace. Semantic models built from ELT metadata provide real-time reporting via direct lake Semantic Models.
8. Power BI serves as the analytics layer, supported by PBI Copilot for self-service capabilities.