Lecture 4: Data Mesh

Question 1

Reasons for Data Mesh

Answer 1

• Fail to bootstrap
• Fail to scale resources
• Fail to scale consumers
• Fail to materialize data-driven value

Question 2

Operational Data Plane

Answer 2

• Running the business

- Serving the users

Question 3

Analytical Data Plane

Answer 3

• Optimising the business

- Improving the user experience

Question 4

Timeline

Answer 4

Data Warehouse -> Data Lake -> Multi-model data architecture on cloud

Question 5

Easiest way to scale a monolithic solution

Answer 5

• Technical decomposition
• Ingest, process, serve
• This led to teams that are responsible for one task (change doesn’t happen locally)

Question 6

Data Engineers in Monolithic Architecture

Answer 6

• Data engineers are siloed and expected in the monolithic piece in the middle built by highly specialized people
• This leads to disconnected executions of, start with one end and never finish

Question 7

Monolithic architecture

Answer 7

• Centralised Architecture
• Hyper-specialized Silo Delivery
• Disconnected execution

Question 8

Data Mesh principles

Answer 8

• Domain-driven Data Ownership Architecture
• Data as a product
• Self-service Infrastructure as a Platform
• Federated Computational Governance

Question 9

Domain-driven Data Ownership Architecture

Answer 9

• Bring ownership to the domains

Flowing data -> Serving data at source
One canonical model -> Multiple models
Source of truth -> Most relevant copy
Pipeline as first concern -> Domains internal implementation
Technology-driven composition -> Domain-oriented distribution

Question 10

Data as a product

Answer 10

• Ownership comes with accountability and responsibility of sharing that data as a product
• Discoverability & Understanding

Data as an asset -> Data as a product
Byproduct -> Product
Data as an input of compute -> Data compute as one unit

Question 11

Self-service Infrastructure as a Platform

Answer 11

• Addressing the cost of ownership -> Enable autonomy
• Self-serve platform empowers the data domain teams to create and share valuable and useful data with less overhead

Big platforms -> Protocols
Specialists -> Generalists
Imperative -> Declarative
Mechanisms -> Experiences

Question 12

Federated computational governance

Answer 12

• Enable interoperability
• Enable ecosystems to play

Centralised team of experts -> federated team
Responsibility for data quality -> Responsibility for defining aspects of security
Responsibility for canonical data modelling -> Responsible for modelling polysemes
Measure success based on data volume -> Measure success based on value-generated

Question 13

What is data mesh for shift?

Answer 13

Data Mesh is a paradigm shift in architecture to bring the operational and analytical plane together with a feedback loop

Question 14

Challenging task seen from the Governance Perspective

Answer 14

Balance between global (central authority) and local governance (autonomous components)

Question 15

Enforcing GDPR compliance of data processing across Data Mesh

Answer 15

Data product should implement a set of mechanisms (e.g., checking access token) to add GDPR-aware processing of data and provide a set of policy enforcement points for configuring those mechanisms. These capabilities should be part of the management or control interface (port) of the data product

Question 16

Managed Spark service should provide .. to the domain team

Answer 16

1. Programmatic API
2. Multi-tenancy
3. Support Generalists

Question 17

Standardization efforts

Answer 17

• Data product interface (OpenAPI)
• Data Query Language (SQL)
• Data Modeling (ER)

Question 18

Cross-cutting polices that the federated governance system should enforce globally

Answer 18

1. Data privacy and protection
2. Data Quality
3. Data access control and audit

Question 19

Data Mesh Approach

Answer 19

• Cultural change
• Alignment across operational and analytic data domains
• A distributed architecture for on-prem and multi-cloud data