Deep Dive - Change Data Capture

Question 1

What is Change Data Capture (CDC)?

Answer 1

CDC involves reading a database’s transaction log and streaming out the transactions as they happen to Kafka/Kinesis/DB. It is used to share database changes to downstream apps or datastores. It is an improvement on making a batch query to the same database.

For MySQL, the transaction log is called binlog (Binary Log). It is an append-only log that records all data-manipulating operations (INSERT, UPDATE, DELETE).

A CDC process will monitor the binlog and propogate the changes to downstream systems (Kafka, other DB).
Debezium is open-source software that does such a task.

CDC is good for tracking DELETEs, which is often difficult to do when trying to query the original DB.

Question 2

What are some use cases for CDC?

Answer 2

1. Data replication for analytics. Maybe an Insights team wants to query database about sales, view cart, other events. Want to separate the primary DB from the analytics DB. Can use CDC for this.
2. The ChatGPT question that you were asked. Only talk to the prompt service if database table for Prompts has a new entry.

Question 3

How fast are CDC solutions?

Answer 3

a few milliseconds

Question 4

What are some limitations of log based CDC?

Answer 4

some operations (ALTER, TRUNCATE) are not captured in transaction log.

If destination DB is down, transaction log should be kept intact (huh?)