Spark

Question 1

For iterative map/reduce implementations, what negatively affects performance (2)

Answer 1

1. Data needs to be loaded from disk on every iteration
2. Results are saved to HDFS, with multiple replications

Question 2

What is Apache Spark?

Answer 2

an open-source, distributed framework designed for big data processing and analytics that takes advantage of in-memory processing

Question 3

How is data handled with in-memory processing? (2)

Answer 3

1. Data is loaded in memory before computation
2. Kept in memory during successive steps

Question 4

In what circumstance is in-memory processing suitable?

Answer 4

1. when new data arrives at a fast pace (streams)
2. for real-time analytics and exploratory tasks
3. when iterative access is required
4. when multiple jobs need the same data

Question 5

What is an in-memory processing approach?

Answer 5

the technique of storing and manipulating data that is stored in the computer’s main memory (RAM), as opposed to traditional disk-based storage

Question 6

Where do Spark components run?

Answer 6

In Java virtual machines

Question 7

What is the Spark cluster manager?

Answer 7

Spark’s standalone cluster manager, YARN which is mainly suited for Hadoop or Mesos which is more generic) keeps track of resources available

Question 8

What do Spark applications consist of? (2)

Answer 8

1. Driver process
2. (A set of) Executor processes

Question 9

How are spark applications run? (5)

Answer 9

1. User submits spark application to the cluster manager (via the application)
2. The application’s driver process is run on one of the nodes in the cluster
3. The driver program breaks down the application into tasks which are then distributed to executor processes running on worker nodes in the cluster
4. Executors execute the tasks assigned to them and the results from each task are sent back to the driver program for aggregation
5. Once all tasks are completed, the Spark application terminates, and the final results are returned to the user or saved to an external storage system

Question 10

What is an RDD?

Answer 10

a partitioned collection of records

Question 11

How do you control spark applications?

Answer 11

through a driver process called the SparkSession

Question 12

How are RDDs created? (2)

Answer 12

1. by reading data form an external storage system (for instance HDFS, Hbase, Amazon S3, Cassandra, …)
2. from an existing collection in the driver program.

Question 13

What are the two types of spark operation?

Answer 13

1. Transformation
2. Action

Question 14

What is a transformation?

Answer 14

A lazy operation to build an RDD from another RDD

Question 15

What is an action? (3)

Answer 15

An operation to take an RDD and return a result to driver//HDFS

Question 16

Describe how operations are evaluated lazily in Spark (3)

Answer 16

1. Transformations are only executed when they are needed
2. Only the invocation of an action will trigger the execution chain
3. Enables building the actual execution plan to optimise the data flow

Question 17

When is it necessary to use low-level API? (3)

Answer 17

1. Control over physical data across a cluster is needed
2. Some very specific functionality is needed
3. There is legacy code using RDDs

Question 18

Describe the “micro-batch” approach?

Answer 18

(accumulates small batches of input data and then processes them in
parallel

Question 19

What are Spark low-level APIs?

Answer 19

APIs to write applications that operate on RDDs directly

Question 20

What are the three execution modes?

Answer 20

1. Local
2. Client
3. Cluster

Question 21

Which is the most common mode for execution spark programs?

Answer 21

Cluster mode

Question 22

What is the difference between client and cluster mode

Answer 22

In cluster mode, the cluster manager launches the driver process and executor processes on worker nodes inside the cluster meaning the cluster manager is responsible for maintaining all Spark Application–related processes
In client mode, the driver process is run on the same machine that submits the application, meaning that the that the client machine is responsible for maintaining the Spark driver process, and the cluster manager maintains the executor processses

Question 23

what is a narrow operation?

Answer 23

operations, which are applied to each record independently e.g. map, flatMap

Question 24

what is a wide operation?

Answer 24

operations which involve records from multiple partitions (and are costly)