Enterprise Analytics Modernisation with Microsoft Fabric & OneLake
Consolidating legacy relational databases into a unified Microsoft Fabric Medallion Lakehouse with Delta Parquet storage and Direct Lake Power BI semantic models.
Challenge, Approach & Verified Outcome
The enterprise struggled with 6 isolated SQL databases, an 8-hour overnight batch ETL pipeline that frequently failed, and executive Power BI dashboards that timed out during morning operational reviews.
AgenorIT architected a modern Medallion Lakehouse on Microsoft Fabric OneLake, utilizing PySpark notebooks for Bronze-to-Silver delta processing, automated SQL transformations for Gold aggregates, and Direct Lake semantic models for Power BI.
Direct Lake query execution under 800ms across 120M+ rows with zero import refresh lag, 8-hour batch windows replaced with 12-minute real-time delta syncs, and 45% reduction in cloud data infrastructure costs.
Measured Technical & Operational Impact
Instant dashboard interactions querying OneLake Delta Parquet directly without memory imports.
Reduced from an 8-hour overnight batch processing window with automated error recovery.
Eliminated redundant dedicated SQL pools and per-user semantic caching licenses.
Unified multi-system freight, fleet telematics, and financial data in OneLake.
Client Context & Environment
A leading freight and supply chain operator headquartered in Melbourne managed a distributed fleet across Australia. Operational data was fragmented across multiple on-premises SQL Server instances, cloud ERP databases, and IoT telematics feeds.
Leadership required unified, real-time visibility into route profitability, fleet utilization, and customer SLA compliance, but existing legacy reporting infrastructure could not keep pace with transaction growth.
The Engineering Challenge
The legacy data architecture suffered from systemic scalability and reliability bottlenecks:
- Prolonged Batch Latency: Overnight data warehouse processing took up to 8 hours, meaning morning operational reports were already stale before dispatchers opened them.
- Power BI Refresh Timeouts: Large data volumes forced Power BI semantic models to hit 10GB dataset limits and timeout during scheduled scheduled imports.
- Data Duplication & Licensing Costs: Multiple business units maintained independent copies of identical datasets across SQL instances, multiplying cloud compute spend.
- Data Governance Gaps: Lack of centralised column-level security and lineage tracking created compliance audit risks.
Chronological Delivery & Concrete Artefacts
AgenorIT designed and deployed a Greenfield Microsoft Fabric architecture over a structured 10-week engagement:
OneLake Medallion Lakehouse Architecture
Established a unified Fabric Lakehouse structured into Bronze (raw ingestion), Silver (cleansed & deduplicated), and Gold (curated dimensional star schemas) layers using Delta Parquet open storage format.
- •Medallion architecture specification
- •Delta Lake schema and V-Order indexing strategy
Synapse Data Engineering & Delta Pipelines
Engineered automated Data Factory pipelines and PySpark notebooks for incremental delta ingestion, replacing fragile monolithic stored procedures with idempotent transformations.
- •Fabric Data Factory pipeline definitions
- •PySpark transformation notebooks in Git integration
Direct Lake Semantic Modeling for Power BI
Built enterprise Power BI semantic models operating in Direct Lake mode, allowing executive reports to query Gold Lakehouse tables directly from OneLake storage with zero data replication lag.
- •Direct Lake semantic model schema (TMDL)
- •Executive fleet and financial KPI dashboards
Security, Fabric Capacity & FinOps Governance
Configured Microsoft Purview sensitivity labels, Row-Level Security (RLS) policies, and Fabric Capacity (F-SKU) auto-pause rules to optimize monthly consumption spend.
- •Fabric security and access matrix
- •Capacity metrics monitoring workbook
Components Deployed in Production
Platform
Storage
Semantic Modeling
Transformations
Ingestion
Governance
Verified Outcomes
The Microsoft Fabric implementation provided the organisation with a single source of truth, enabling instantaneous executive reporting and real-time operational decision-making.
- Consolidated 120M+ records across 6 disparate databases into a single governed OneLake repository.
- Reduced report query latency from 45+ seconds to under 800 milliseconds using Direct Lake.
- Accelerated data refresh cycles from 8-hour batch intervals to 12-minute incremental syncs.
- Cut monthly cloud data platform operational costs by 45% through Fabric capacity consolidation.
Live App Store Listing & Interactive Experience
Explore the live product on the Apple App Store or access the web deployment running in production.
Production-verified enterprise lakehouse reference architecture on Microsoft Fabric: 120M+ daily records ingested across Bronze/Silver/Gold tiers with sub-second Power BI Direct Lake querying and Essential Eight / ISO 27001 governance boundary.
Discuss a Similar Architecture for Your Organisation
Speak directly with our principal engineers about architecting your cloud migration, Entra identity system, or high-performance digital product.