Enterprise Analytics Modernisation with Microsoft Fabric & OneLake
Consolidating legacy relational databases into a unified Microsoft Fabric Medallion Lakehouse with Delta Parquet storage and Direct Lake Power BI semantic models.
Challenge, Approach & Delivery Outcomes
The enterprise struggled with multiple isolated operational databases, an overnight batch ETL pipeline that frequently delayed operational reporting, and executive Power BI dashboards that suffered from slow import refreshes.
AgenorIT architected a modern Medallion Lakehouse on Microsoft Fabric OneLake, utilizing PySpark notebooks for Bronze-to-Silver delta processing, automated SQL transformations for Gold aggregates, and Direct Lake semantic models for Power BI.
Direct Lake query execution with sub-second response times across large datasets, legacy overnight batch windows replaced with frequent automated delta syncs, and significant cloud infrastructure savings through unified capacity.
Measured Technical & Operational Impact
Instant dashboard interactions querying OneLake Delta Parquet directly without memory imports.
Replaced an overnight batch processing window with automated incremental synchronization.
Consolidated disparate compute engines into shared Microsoft Fabric capacity.
Unified multi-system freight, fleet telematics, and financial data in OneLake.
Client Context & Environment
A leading freight and supply chain operator headquartered in Melbourne managed a distributed fleet across Australia. Operational data was fragmented across multiple on-premises SQL Server instances, cloud ERP databases, and IoT telematics feeds.
Leadership required unified, real-time visibility into route profitability, fleet utilization, and customer SLA compliance, but existing legacy reporting infrastructure could not keep pace with transaction growth.
The Engineering Challenge
The legacy data architecture suffered from systemic scalability and reliability bottlenecks:
- Prolonged Batch Latency: Overnight data warehouse processing delayed morning operational reports.
- Power BI Refresh Timeouts: Large data volumes forced Power BI semantic models to hit dataset thresholds and timeout during scheduled imports.
- Data Duplication & Licensing Costs: Multiple business units maintained independent copies of identical datasets across SQL instances, multiplying cloud compute spend.
- Data Governance Gaps: Lack of centralised column-level security and lineage tracking created compliance audit risks.
Chronological Delivery & Concrete Artefacts
AgenorIT designed and deployed a Greenfield Microsoft Fabric architecture over a structured 10-week engagement:
OneLake Medallion Lakehouse Architecture
Established a unified Fabric Lakehouse structured into Bronze (raw ingestion), Silver (cleansed & deduplicated), and Gold (curated dimensional star schemas) layers using Delta Parquet open storage format.
- •Medallion architecture specification
- •Delta Lake schema and V-Order indexing strategy
Synapse Data Engineering & Delta Pipelines
Engineered automated Data Factory pipelines and PySpark notebooks for incremental delta ingestion, replacing fragile monolithic stored procedures with idempotent transformations.
- •Fabric Data Factory pipeline definitions
- •PySpark transformation notebooks in Git integration
Direct Lake Semantic Modeling for Power BI
Built enterprise Power BI semantic models operating in Direct Lake mode, allowing executive reports to query Gold Lakehouse tables directly from OneLake storage with zero data replication lag.
- •Direct Lake semantic model schema (TMDL)
- •Executive fleet and financial KPI dashboards
Security, Fabric Capacity & FinOps Governance
Configured Microsoft Purview sensitivity labels, Row-Level Security (RLS) policies, and Fabric Capacity (F-SKU) auto-pause rules to optimize monthly consumption spend.
- •Fabric security and access matrix
- •Capacity metrics monitoring workbook
Components Deployed in Production
Platform
Storage
Semantic Modeling
Transformations
Ingestion
Governance
Key Engineering Outcomes
The Microsoft Fabric implementation provided the organisation with a single source of truth, enabling instantaneous executive reporting and real-time operational decision-making.
- Consolidated records across disparate operational databases into a single governed OneLake repository.
- Accelerated executive report queries to responsive sub-second response times using Direct Lake.
- Accelerated data refresh cycles from overnight batch intervals to continuous incremental syncs.
- Substantially lowered cloud data platform operational costs through capacity consolidation.
Engineered Data Platform Topology
Detailed enterprise architecture diagram illustrating data ingestion, medallion lakehouse processing, and analytical serving layers.
Enterprise lakehouse reference architecture on Microsoft Fabric: Multi-source ingestion across Bronze/Silver/Gold tiers with sub-second Power BI Direct Lake querying and unified governance boundaries.
Discuss a Similar Architecture for Your Organisation
Speak directly with Gurinder Singh about architecting your cloud migration, Entra identity system, or high-performance digital product.