Lecture 3-5

Question 1

What is the difference between MapReduce and Spark?

Answer 1

MapReduce writes all data that Mapppers send to file first. Spark attempts to avoid doing so, keeping as much data in RAM possible.

Question 2

Resilient Distributed Dataset. - How does Spark address faul tolerance in distributed computation?

Answer 2

RDDs are either persistent datasets, or declarative recipes of how to create a dataset from other RDDs.

Question 3

How can Spark be interpreted as generalization of MapReduce?

Answer 3

It offers 20+ more communication patterns. (for example joins between datasets)

Question 4

What is lambda in Spark?

Answer 4

Implicit function definitions. A lambda is typically a parameter to high-level operators of Spark.

Question 5

What is a DataFrame?

Answer 5

A DataFrame is a conceptual layer on top of RDDs. It is a table with column names and types. Thanks to this information, Spark can execute queries on DataFrames more efficiently.

Question 6

What is MLLIB

Answer 6

Set of ready-to-run machine learning algorithms that work on RDDs.

Question 7

OLAP (On Line Analytical Processing)

Answer 7

Users run a few complex read-only queries that search through huge tables.

Question 8

Why are column stores are better than row-wise storage?

Answer 8

When data is compressed per column, value distribution is more regular than if values from different columns are mixed.

Question 9

What is zone-map?

Answer 9

Storage trick. Keep simple statistics (Min, Max) for large tuple ranges. Avoid reading a zone in the table if WHERE condition ask for values that outside [Min, Max] range of the zone.

Question 10

What are the goals of Distribution and Partitioning?

Answer 10

Distribution: Which machine gets which rows. Ensure that data gets split evenly. Partitioning: Often done on a time-related column. Speed up queries by skipping partitions. Second goal is data lifecycle management. -> Keep the last X days of data in X partitions, each day drop the oldest one.

Question 11

ORC and Parquet

Answer 11

Columnar data storage. Data is placed column after column. Applications that read only few columns can skip over the unused columns.

Question 12

What is the difference between database systems in the cloud and classic on-premise architectures?

Answer 12

Queries are not run on machines where the data is. Each database system will have to fetch data from a cloud storage service such as S3.

Question 13

How can cloud database systems achieve locality?

Answer 13

If they are running on cloud instances with a disk. Such disks are for caching only, empty at start.

Question 14

What is a serverless database system?

Answer 14

The scaling and start/stop of the database is fully handled by the system.

Question 15

What is the problem with updating data on cloud? How to update data in cloud?

Answer 15

Each persistent update must write a new S3 file. Data may arrive all the time in small quantities/ This leads to very many small files. Solution: Batch data in the update pipeline. Only go to cloud when you have 100MB.

Question 16

What is vectorization or just-in-time compilation for overhauling of SQL query engine?

Answer 16

Take a bunch of rows and process them together. Columnar format is a must. Column is treated as an array and you iterate over that array.

Question 17

Why is MapReduce a step backwards for database access?

Answer 17

• a sub-optimal implementation, in that it uses brute force instead of indexing;
• not novel at all — it represents a specific implementation of well known techniques developed nearly 25 years ago;
• missing most of the features that are routinely included in current DBMS;
• incompatible with all of the tools DBMS users have come to depend on.”