Data management Flashcards
Data management patterns
Database per Service Shared database Saganew API Compositionnew CQRS Event sourcing Application events
Data management patterns: context
Let’s imagine you are developing an online store application using the Microservice architecture pattern. Most services need to persist data in some kind of database. For example, the Order Service stores information about orders and the Customer Service stores information about customers.
Data management patterns: problem
What’s the database architecture in a microservices application?
Data management patterns: forces
Services must be loosely coupled so that they can be developed, deployed and scaled independently
Some business transactions must enforce invariants that span multiple services. For example, the Place Order use case must verify that a new Order will not exceed the customer’s credit limit. Other business transactions, must update data owned by multiple services.
Some business transactions need to query data that is owned by multiple services. For example, the View Available Credit use must query the Customer to find the creditLimit and Orders to calculate the total amount of the open orders.
Some queries must join data that is owned by multiple services. For example, finding customers in a particular region and their recent orders requires a join between customers and orders.
Databases must sometimes be replicated and sharded in order to scale. See the Scale Cube.
Different services have different data storage requirements. For some services, a relational database is the best choice. Other services might need a NoSQL database such as MongoDB, which is good at storing complex, unstructured data, or Neo4J, which is designed to efficiently store and query graph data.
Data management patterns: solution
Keep each microservice’s persistent data private to that service and accessible only via its API. A service’s transactions only involve its database.
The service’s database is effectively part of the implementation of that service. It cannot be accessed directly by other services.
There are a few different ways to keep a service’s persistent data private. You do not need to provision a database server for each service. For example, if you are using a relational database then the options are:
- Private-tables-per-service – each service owns a set of tables that must only be accessed by that service
- Schema-per-service – each service has a database schema that’s private to that service
- Database-server-per-service – each service has it’s own database server.
Private-tables-per-service and schema-per-service have the lowest overhead. Using a schema per service is appealing since it makes ownership clearer. Some high throughput services might need their own database server.
It is a good idea to create barriers that enforce this modularity. You could, for example, assign a different database user id to each service and use a database access control mechanism such as grants. Without some kind of barrier to enforce encapsulation, developers will always be tempted to bypass a service’s API and access it’s data directly.
Data management patterns: result benefits
Helps ensure that the services are loosely coupled. Changes to one service’s database does not impact any other services.
Each service can use the type of database that is best suited to its needs. For example, a service that does text searches could use ElasticSearch. A service that manipulates a social graph could use Neo4j.
Data management patterns: result drawbacks
Implementing business transactions that span multiple services is not straightforward. Distributed transactions are best avoided because of the CAP theorem. Moreover, many modern (NoSQL) databases don’t support them. The best solution is to use the Saga pattern. Services publish events when they update data. Other services subscribe to events and update their data in response.
Implementing queries that join data that is now in multiple databases is challenging.
Complexity of managing multiple SQL and NoSQL databases
Data management patterns: issues
API Composition - the application performs the join rather than the database. For example, a service (or the API gateway) could retrieve a customer and their orders by first retrieving the customer from the customer service and then querying the order service to return the customer’s most recent orders.
Command Query Responsibility Segregation (CQRS) - maintain one or more materialized views that contain data from multiple services. The views are kept by services that subscribe to events that each services publishes when it updates its data. For example, the online store could implement a query that finds customers in a particular region and their recent orders by maintaining a view that joins customers and orders. The view is updated by a service that subscribes to customer and order events.
Data management patterns: related patterns
- Microservice architecture pattern creates the need for this pattern
- Saga pattern is a useful way to implement eventually consistent transactions
- The API Composition and Command Query Responsibility Segregation (CQRS) pattern are useful ways to implement queries
- The Shared Database anti-pattern describes the problems that result from microservices sharing a database
Shared database: context
Let’s imagine you are developing an online store application using the Microservice architecture pattern. Most services need to persist data in some kind of database. For example, the Order Service stores information about orders and the Customer Service stores information about customers.
Shared database: problem
What’s the database architecture in a microservices application?
Shared database: forces
Services must be loosely coupled so that they can be developed, deployed and scaled independently
Some business transactions must enforce invariants that span multiple services. For example, the Place Order use case must verify that a new Order will not exceed the customer’s credit limit. Other business transactions, must update data owned by multiple services.
Some business transactions need to query data that is owned by multiple services. For example, the View Available Credit use must query the Customer to find the creditLimit and Orders to calculate the total amount of the open orders.
Some queries must join data that is owned by multiple services. For example, finding customers in a particular region and their recent orders requires a join between customers and orders.
Databases must sometimes be replicated and sharded in order to scale. See the Scale Cube.
Different services have different data storage requirements. For some services, a relational database is the best choice. Other services might need a NoSQL database such as MongoDB, which is good at storing complex, unstructured data, or Neo4J, which is designed to efficiently store and query graph data.
Shared database: solution
Use a (single) database that is shared by multiple services. Each service freely accesses data owned by other services using local ACID transactions.
Shared database: result benefits
A developer uses familiar and straightforward ACID transactions to enforce data consistency
A single database is simpler to operate
Shared database: result drawbacks
Development time coupling - a developer working on, for example, the OrderService will need to coordinate schema changes with the developers of other services that access the same tables. This coupling and additional coordination will slow down development.
Runtime coupling - because all services access the same database they can potentially interfere with one another. For example, if long running CustomerService transaction holds a lock on the ORDER table then the OrderService will be blocked.
Single database might not satisfy the data storage and access requirements of all services.
Shared database: related
Database per Service is an alternative approach
Saga: context
You have applied the Database per Service pattern. Each service has its own database. Some business transactions, however, span multiple service so you need a mechanism to ensure data consistency across services. For example, lets imagine that you are building an e-commerce store where customers have a credit limit. The application must ensure that a new order will not exceed the customer’s credit limit. Since Orders and Customers are in different databases the application cannot simply use a local ACID transaction.
