Week 6: Apache Spark

Question 1

Apache Spark

Answer 1

It’s a unified engine for distributed data processing. Spark extends the MapReduce programming model with abstraction that allows efficient data reuse.

Question 2

Resilient Distributed Datasets (RDD’s)

Answer 2

It allows for efficient data reuse. RDD’s are read-only, fault-tolerant collections of records that can be operated in parallel. They’re data structures that serve as the core unit of data in Spark. It can be manipulated, its partitioning controlled, and can be made persistent in memory. All RDD’s may or may not be materialised.

Question 3

RDD: Lineage

Answer 3

Each RDD knows its lineage, which is how it was derived from other datasets, and can compute its partitions from data in stable storage. Only RDD’s that can be reconstructed after failure can be referenced by the user’s programme.

Representation can track lineage and provide transformations that can be composed arbitrarily. A representation includes a set of partitions, a set of dependencies on the parent RDD’s, a function for computing the dataset based on its parents, and the metadata about the partitioning scheme and data placement.

Question 4

RDD: Persistence

Answer 4

Users can specify which RDD’s they’ll reuse and choose storage strategies for them.

Question 5

RDD: Partitioning

Answer 5

Users can specify how records of RDD’s are partitioned across machines based on the key in each record.

Question 6

Filter Example

rdd = sc.parallelize([1,2,3,4,5])
rdd.filter(lambda x: x % 2 == 0).collect()

Answer 6

Output
[2,4]

Question 7

Map Example

rdd = sc.parallelize([2,3,4])
rdd.map(lambda x: range(1,x)).collect()

Answer 7

Output
[[1],[1,2],[1,2,3]]

Question 8

FlatMap Example

rdd = sc.parallelize([2,3,4])
rdd.flatMap(lambda x: range(1,x)).collect()

Answer 8

Output
[1,1,2,1,2,3,]

Question 9

RDD: Storage Strategies

Answer 9

By default, RDD’s are stored in RAM. If there’s not enough RAM, the RDD’s are spilled to the disk. Users can also store RDD’s only on the disk, replicate RDD’s across machines and use flags to persist, or set the persistence priority on each RDD to specify which in-memory data should spill to the disk first.

Question 10

RDD: Lazy Evaluation

Answer 10

Spark executes only after seeing the first action, and it lazy evaluates transformations by recording metadata for them. Lazy evaluation provides a global view, so Spark can optimise the required calculations by grouping operations together and recovers from failures and slow workers.

Question 11

RDD: Narrow Dependencies

Answer 11

Each partition of the parent RDD is used by at most 1 partition of the child RDD. It allows for pipeline execution on the node. It has efficient failure recovery, as only lost parent partitions need to be recomputed and recomputation can be parallel.

Question 12

RDD: Wide Dependencies

Answer 12

Multiple child partitions may depend on a single partition of the parent RDD. It requires data from all parent partitions, in order to be available and shuffled across nodes. Failure recovery involves many RDD’s and complete re-execution may be needed.

Question 13

RDD: Advantages

Answer 13

1. Distributed Shared Memory (DSM), which is a global address space that applications can read and write to in arbitrary locations.
2. RDD’S are created by coarse-grained transformations (applied to the entire dataset), but reads can be fine-grained (read from a specific location).
3. RDD’s are restricted to applications performing bulk writes, but provide efficient fault tolerance.
4. RDD’s are read-only, so Spark can run backup copies of slow tasks without accessing the same memory. Spark can distribute data over different nodes to run computations in parallel.
5. Runtime can schedule tasks based on data locality to improve performance.
6. RDD’s degrade gracefully when there isn’t enough memory to store them.

Question 14

RDD: Applications

Answer 14

They’re great for batch applications that perform the same operations on the entire dataset.

RDD’s aren’t suitable for applications that make asynchronous fine-grained updates to the shared state.

Question 15

Spark Programming Interface

Answer 15

The driver programme defines the RDD’s, invokes actions on them, and tracks the RDD’s lineage. The processes read data blocks from the distributed file system and store RDD partitions in RAM. The processes comprise of workers which receive tasks from the driver programme.

Question 16

Spark’s Cluster Mode Components

Answer 16

1. Cluster Manager: allocates resources across applications.
2. Driver Programme: must listen for and accept incoming connections from its executors throughout its lifetime. Should run close to workers, meaning it should be in the same local area network.
3. Executor: this is a process that performs computations and stores data.
4. Task: unit of work that’ll be sent to the executor.
5. SparkContext: connects to the cluster manager, acquires executors, sends application code and tasks to executors.

Question 17

pySpark

Answer 17

This is Spark’s Python interface.

Question 18

pySpark Commands: Start Shell

Answer 18

Start shell
./bin/pyspark –master local

Question 19

pySpark Commands: Shart Shell With k Worker Threads

Answer 19

./bin/pyspark –master local[k]
(ideally, k = number of cores)

Question 20

pySpark Commands: Create RDD

Answer 20

tf = sc.textFile”file/////usr/share/dict/words”)
(tf is the point to the file. No loading is performed)
(sc is the SparkContext variable, which is created automatically)

Question 21

pySpark Commands: Parallelised Collections

Answer 21

d=[1,2,3,4,5]
parallel_col = sc.parallelize(d)
(sc is the SparkContext variable, which is created automatically)

Question 22

pySpark Commands: Count

Answer 22

parallel_col.count()
(parallel_col is a previously defined parallelised column)

Question 23

pySpark Commands: Filter

Answer 23

lines_nonempty = tf.filter(lambda x: len(x) > 0)
(tf is the point to the file. No loading is performed)

Question 24

pySpark Commands: Map

Answer 24

nums = sc.parallelize([1,2,3,4])
squared = nums.map(lambda x: x * x).collect()
for num in squared:
print(num, “,”)
Output: 1,4,9,16

Question 25

pySpark Commands: FlatMap

Answer 25

x = sc.parallelize([“a b”, “c d”])
y = x.flatMap(lambda x: x.split(‘ ‘)
Output: [‘a’, ‘b’, ‘c’, ‘d’]

Question 26

pySpark Commands: Union

Answer 26

rddA.union(rddB)

Question 27

pySpark Commands: Intersection

Answer 27

rddA.intersection(rddB)

Question 28

pySpark Commands: Subtract

Answer 28

rddA.subtract(rddB)

Question 29

pySpark Commands: Cartesian

Answer 29

rddA.cartesian(rddB)

Question 30

pySpark Commands: Join

Answer 30

rddA.join(rddB,[number of reduce tasks])

Question 31

pySpark Commands: Reduce

Answer 31

rdd = sc.parallelize([1,2,3,4,5])
sum = rdd.reduce(lambda x, y: x + y)
sum
Output:15

Question 32

pySpark Commands: Fold

Answer 32

Works the same way as reduce, but first argument is type of output we want to return.

rdd = sc.parallelize([1,2,3,4,5])
sum = rdd.fold(0.0, lambda x, y: x + y)
sum
Output:15

Question 33

Accumulators

Answer 33

Accumulators only works for operations that are both associative and commutative (i.e. addition)

Example:
accum = sc.accumulator(0)
accum
Output: Accumulator<id=0, value=0>
sc.parallelize([1,2,3,4,5,6]).foreach(lambda x: accum.add(x))
accum.value
Output: 15

Question 34

Storage Strategy: MEMORY_ONLY

Answer 34

useDisk: False
useMemory: True
deserialised: False
replication: 1

Question 35

Storage Strategy: MEMORY_ONLY_2

Answer 35

useDisk: False
useMemory: True
deserialised: False
replication: 2

Question 36

Storage Strategy: MEMORY_ONLY_SER

Answer 36

useDisk: False
useMemory: True
deserialised: False
replication: 1

Question 37

Storage Strategy: DISK_ONLY

Answer 37

useDisk: True
useMemory: False
deserialised: False
replication: 1

Question 38

Storage Strategy: MEMORY_AND_DISK

Answer 38

useDisk: True
useMemory: True
deserialised: True
replication: 1

Question 39

Storage Strategy: MEMORY_AND_DISK_SER

Answer 39

useDisk: True
useMemory: True
deserialised: False
replication: 1

Question 40

Transformation

Answer 40

In Apache Spark, transformations modify an existing dataset to create a new RDD. It returns a dataset. They are executed in a lazy manner, which means that they are chained together to build a strategic execution plan.

Question 41

Actions

Answer 41

In Apache Sparks, actions do the actual calculations and return counts, lists of elements, or nothing at all. Actions force the execution of the entire lineage of transformations leading up to the action.

Question 42

Spark’s Cluster Mode Workflow

Answer 42

SparkContext connects to the cluster manager.

SparkContext communicates with the Executors.