Data Science at Scale Flashcards
How have companies changed in the big data era and what has enabled this?
Become more social, customer-orientated and dynamic. They have done this by collecting data, learning from the data, and improving and adapting in response. This is because of cheaper storage and processing, faster networks, and free open-source tools.
What technological shift enabled widespread data analytics?
Cloud-based infrastructure (e.g., AWS, GCP) and Infrastructure as a Service solutions from internet giants like Google, Amazon, and Microsoft.
How has big data transformed marketing?
Customer profiling, targeted ads, and personalised communication and recommendations.
What are the 3 V’s of big data?
Volume, Velocity and Variety
What are the four fundamental functionalities that Data-intensive applications are built from?
Database - Store data so it can be retrieved later.
Caching - Store the results of expensive operations to be used again soon.
Indexing - Allow users to efficiently search the data.
Batch Processing - Periodically run specific routines on large amounts of accumulated data.
What are the three important things that a Data-intensive application needs to be?
Reliable, Scalable, Maintainable
What does it mean for a system to be reliable?
It performs the function the user expected. It can tolerate the user making mistakes. Its performance is good enough for the requires use case, under the expected load and volume. It prevents any unauthorised access.
What are faults and failures?
A fault is when a component (hardware/software) of the system works in an unexpected way, and a failure is when the entire system stops providing the service.
What are Hardware Faults and what measures can be taken to stop them?
Usually when a HDD, memory module or PSU stops working. In large data centres this is common. We can use hardware measures such as RAID for HDD’s, redundant PSU’s, and hot-swappable CPU’s.
What are Software Faults?
When the software stops working. These are harder to anticipate, and can be present on many nodes of a system causing widespread failures.
What is Scalability?
A system’s ability to cope with increased load
What is Load?
A measure of the amount of use of a system, for example: requests per second, number of players, read/write ratio.
What is performance?
How well the system is responding to the load, for example: response time, or time taken to process a dataset. The average and distribution are both important.
What is Vertical Scaling?
Upping the specs of the current system, this does not scale linearly, and has limited fault tolerance.
What is Horizontal Scaling?
Increasing the amount of devices in the system, which scales better and has better fault tolerance.
What is Maintainability?
The overall cost to maintain a system operational and updated.
What is Operability?
How easy it is for the operation team to keep it running. Includes good monitoring, automation, and predictable behaviour.
What is Simplicity?
How easy it is for new people working on the system to understand it, without reducing functionality, just accidental complexity.
What is Evolvability?
How easy it is to make changes and update the system, which is closely linked with simplicity.
What is a data model in the context of data storage?
A structure that maps real-world entities (e.g., objects in code) to how they are stored (e.g., tables, JSON).
What is an ORM (Object-Relational Mapping)?
A system that maps classes in code to a relational database schema.
In the relational model, how are one-to-many relationships handled?
By using separate tables with foreign keys pointing to the parent table.
What structured field types can be used within relational DBs for complex fields?
XML or JSON fields within a table.
What is a document model in NoSQL?
A system where data is stored in documents (e.g., JSON, XML) representing semi-structured data.
Why are document models considered more flexible than relational models?
They don’t require a rigid schema; each document can be different.
What’s a key weakness of document models?
Difficulty handling many-to-many relationships efficiently.
What are graph models used for?
Representing complex many-to-many relationships using nodes and edges.
In graph models, what are nodes and edges?
Nodes are entities or objects; edges are the relationships between them.
What is a NoSQL database?
A database that doesn’t use traditional relational models; prioritizes scalability and availability.
What is the CAP Theorem?
In a distributed system, you can only choose two: Consistency, Availability, and Partition Tolerance.
What does consistency mean in CAP?
Every read returns the most recent write or an error.
What does availability mean in CAP?
Every request gets a response, even if it’s not the most up-to-date.
What does partition tolerance mean in CAP?
The system continues functioning despite network partitions.
What are the four types of NoSQL databases?
Document, Key-value, Wide-column, and Graph databases.
What is a key-value store?
A NoSQL database that stores data as key-value pairs, like a dictionary.
What is a wide-column store?
A database model where data is stored in rows and columns, but columns can vary between rows.
How is data organized in wide-column stores?
By column families instead of rows.
What is a property graph in a graph database?
A graph where nodes and edges can have associated properties.
What is the main advantage of using a graph store over relational or document models?
Better handling of complex and interconnected data.
What does schema-on-read mean in document databases?
The structure of data is defined when it’s read by the application, not enforced when written.
Why is schema-on-read useful?
t’s good for heterogeneous data or when you can’t control the data structure, like tweets or logs.
How does schema flexibility in document DBs compare to relational DBs?
Document DBs allow format changes without changing the schema; relational DBs often need downtime and schema updates.
What is “locality” in document databases?
It means storing a whole document as a continuous string (like JSON), keeping related data together.
What’s a drawback of locality in document databases?
Even if you need only part of the document, the DB loads the whole thing, which can be inefficient.
What are the two main jobs of a database?
Store data and retrieve data.
What is a transactional workload?
A write-heavy workload (e.g., bank transactions).
What is an analytics workload?
A read-heavy workload (e.g., dashboards, reports).
What is a storage engine?
The part of the database that handles how data is written to and read from disk.
What is a log-structured database?
A database that appends all writes to a log file.
Why are appends fast in a log file?
Because they avoid random disk access and just add to the end of the file.
What’s the downside of using a plain log for reads?
You have to scan the whole file (O(n) time complexity).
What is a hash index?
A map of keys to byte offsets in a log file.
What is the benefit of a hash index?
Fast lookups for keys.
What is a limitation of a hash index?
It doesn’t support range queries and must fit in memory.
Why are log files split into segments?
To prevent a single log file from growing too large.
What is compaction?
A process that merges segments and removes duplicates or deleted records.
What does SSTable stand for?
Sorted String Table.
What is the key property of SSTables?
They store keys in sorted order and only once per segment.
Why are SSTables efficient for reads?
Because they support binary search and smaller indexes.
Do SSTables still need an index?
Yes, but only for some keys (sparse index).
How is data written to an SSTable?
Data is first written to a memtable (e.g., an AVL tree) in memory. When full, the memtable is flushed to disk as a new SSTable file.
How does reading from an SSTable work?
First, check the memtable; if not found, search the newest SSTable, then older ones.
