Unifying Global Sports Data with a Metadata-Driven Databricks Lakehouse

A leading international sports governing body needed a modern, scalable solution to unify and analyse sports-related data from over 40 federations worldwide. With diverse data sources and formats, the organisation sought to centralise data management, streamline analytics, and upskill their internal teams, all while ensuring data quality and governance. Read on to discover how we unified global sports data with a Metadata-Driven Databricks Lakehouse technology.

Key Challenges

Fragmented Data Sources and Formats The organisation received data in multiple formats (JSON, XML, CSV, Excel) from federations around the world, making ingestion and standardisation a complex, manual process.

• Lack of a Centralised Analytics Platform Without a unified data platform, analytics and reporting were slow, inconsistent, and difficult to scale.

  
  

• Need for Automated, Configurable Data Processing Manual data validation and mapping led to inefficiencies and potential data quality issues.

• Skills Gap in Modern Data Engineering The Data Engineering team required hands-on mentoring to adopt Lakehouse architecture and Databricks best practices.

How Did Primus Help?

To address these challenges, Primus designed and implemented a metadata-driven Databricks Lakehouse platform, delivering:

• Automated, Configurable Data Ingestion: Built a robust ingestion framework to process and validate data from any source or format, mapping it into a central reporting data model.

  
  

• Modern Data Modelling: Leveraged the medallion architecture (bronze, silver, gold layers) for scalable, high-quality data processing.

  
  

• End-to-End Automation: Deployed infrastructure using ARM Templates, automated Databricks setup with Azure DevOps, and established CI/CD pipelines for code deployment.

  
  

• Centralised Analytics: Enabled the organisation to perform advanced analytics on a single, high-quality data platform, unlocking new insights and efficiencies.

  
  

• Technical Mentoring: Provided in-depth training and mentoring to the Data Engineering team, focusing on Databricks, Lakehouse architecture, Python, and PySpark.

Results

• Unified Data Platform: The organisation now analyses global sports data from 40+ federations in a single, centralised Lakehouse, driving faster and more reliable insights.

  
  

• Automated Data Quality: Data is validated, mapped, and processed automatically, reducing manual effort and improving accuracy.

  
  

• Empowered Internal Teams: The Data Engineering team is now skilled in Databricks and Lakehouse best practices, ensuring long-term self-sufficiency.

  
  

• Scalable, Future-Proof Architecture: The metadata-driven approach and automation ensures the platform can easily adapt to new data sources and future requirements.

Technologies Used:

Databricks, Delta Live Tables, Unity Catalog, Delta Lake, Python, PySpark, Azure Data Lake, Azure Functions, Azure Key Vault, Azure DevOps Pipelines, ARM Templates, PowerShell, Git.

Want to learn more about the impact the right talent can have on your business? Let's Connect