Nando's Modern Data Platform with Data Mesh and Data Contracts
Discover how Nando's transformed its data platform using data mesh and data contracts, the challenges they faced, and the solutions that led to their success.
About Nando's
Nando's is a South African fast-casual restaurant chain known for its Portuguese flame-grilled peri-peri chicken. Founded in Johannesburg in 1987, it now has over 1,200 locations in 30 countries. Curious to learn more about Nando's? Discover their story here!
In this post, we share Nando's journey in transforming their data platform using data mesh and data contracts. We highlight the tools they used, the challenges they encountered, and the solutions that proved effective. Additionally, we offer key insights that contributed to their success in building a robust data platform.
Nando’s Data Landscape
Data is crucial to Nando’s success. Providing accurate, timely insights enables the business to address its primary challenges effectively. Nando’s utilises data from over 50 sources in various formats, including:
Sales Data: From tills, in-house platforms, and Deliveroo.
Sustainability Data: Information on oil recycling, chicken donations, and waste collections.
Kitchen Data: Data from smart ovens and kitchen display systems.
Customer Loyalty Data: Details from customer loyalty programmes.
Nando’s Old Data Platform (Pre-2021)
In the previous data platform, different teams maintained separate copies of data, leading to fragmented efforts across various cloud environments. This decentralised approach resulted in:
Data Inconsistencies: Multiple versions of the same data.
Increased Storage Costs: Duplication of data led to unnecessary expenses.
Limited Collaboration: Teams worked in silos, making it difficult to achieve a unified view of the organisation's data.
Challenges Faced
Tracking Shared Data: Uncertainty about data shared with external companies led to confusion and compliance issues.
Understanding Data Changes: Teams struggled to identify who would be impacted by changes to specific data tables, risking disruption.
Data Delivery Deadlines: Data needed to be cleaned and transformed by 8 AM daily to support staff planning.
Reliance on External Data: Real-time updates were delayed due to slow data from external sources.
Operational Efficiency: Data engineers aimed to meet deadlines without starting work excessively early, and sought to minimise system alerts while maintaining smooth data processes.
Transition to a Modern Data Platform (Post-2021)
To tackle these challenges, Ben Watson, the Data Engineering Tech Lead, and his team developed a new data platform through a two-stage approach.
Stage 1: Centralised Data Team
By establishing a unified data team, Nando’s aimed to create a single source of truth. Teams were instructed to hand over their data for centralised cleaning, documentation, transformation, analysis, and exposure. However, limitations soon became apparent:
Scalability Issues: One team couldn't manage all the data domains effectively.
Domain-Specific Needs: Teams struggled with unique requirements, such as custom test pipelines.
Scheduling Complexities: Meeting diverse scheduling needs within the central platform was challenging.
Limited Flexibility: Domain teams lacked full control over their data processes.
Stage 2: Embracing Data Mesh and Data Contract
Recognising the benefits of domain ownership, they shifted to a model where domain teams owned their data. This approach allowed:
Enhanced Control: Teams could build data-driven applications tailored to their needs.
Better Understanding: Data remained with those who best understood it.
Unified Strategy with Flexibility: Maintained a unified data strategy while embracing the data mesh concept.
Nando’s architecture is built on Google Cloud Platform (GCP), enabling each domain team to manage its own GCP project. Teams store raw data in BigQuery, which is transformed into reporting tables using Dataform, referred to as "Data Ponds." Collectively, these Data Ponds form Nando’s Data Lake.
Nandos Data Mesh/Data Contract Architecture
Now let’s have a closer look at this architecture :
Extract and Load
Nando's efficiently ingests data into BigQuery using built-in GCP tools. For complex data sources, they develop custom Cloud Functions or utilise Fivetran. Ingesting raw data in its original format allows for easier identification and cleaning of issues later in BigQuery.
Transformation
To ensure high-quality transformed data, Nando's uses Dataform, a Google Cloud tool for SQL-based transformations in BigQuery.
Dataform helps data analysts create, test, and manage SQL workflows. Transformations are stored in a Git repository, allowing version control and collaboration among teams.
Dataform supports over 1,000 transformations across multiple projects at Nados. it uses JavaScript injections to identify the dependencies between each transformation and builds a DAG
Metadata at the top of each file in Dataform help with organising, documenting, testing, and scheduling these queries:
Consume
Suggestion: Starting from Nando’s shares data in various ways:
Nando’s shares data through various methods:
Transformed Data in BigQuery: Shared across Dataform instances via an in-house Data Contract Service.
CSV Reports: Generated using Looker’s scheduling feature.
Live Data Streaming: Certain datasets are streamed to services using Hightouch (a reverse ETL tool).
Ad-Hoc Reporting: Analysts use Vertex AI notebooks for exploratory data science.
Looker’s Semantic Layer: Provides users with easy data analysis capabilities.
Domain Champions: Promote data use across Nando’s, crucial for the platform's success.
A core data and analysis team is responsible for:
Providing tooling to enable domain teams to run their own Data Ponds.
Defining data governance.
Managing Looker.
Processing core datasets such as financial data, opening hours, and restaurant master data.
For more details on the architecture, please refer to this article.
Strong Data Culture at Nando's
Nando's promotes a strong data culture to avoid isolated data efforts. They created a #data-community Slack channel for over 300 members to share support and ideas. Domain teams have their own data ponds, and data champions help others improve their data skills. Looker dashboards are organised by role, ensuring users get the data they need without feeling overwhelmed.
Summary
In this post, we explored Nando's transition from an outdated data platform to a modern one, utilising data mesh and data contracts. We discussed the tools implemented, the challenges faced, and the effective solutions adopted. Collaboration within the data team was highlighted as essential to prevent isolation, along with the importance of buy-in to ensure effective data usage. Ultimately, building a successful data platform requires not only technical expertise but also a strong cultural focus.
Stay tuned for our upcoming posts where we'll delve deeper into this topic.
Further Reading
If you're interested in learning more about data contracts, we recommend the following resources:
Join The Conversation!
We hope you found this post helpful. If you have any questions, comments, or experiences you'd like to share, we'd love to hear from you!
