Spark

Question 1

What are the differences between these 3 types of programming?

object-oriented
declarative
functional

Answer 1

object-oriented: Python, objects are mutable

declarative: SQL

functional: Spark, objects are immutable

Question 2

What is MapReduce?

Answer 2

The two parts of a simple distributed-computing framework.

“Map” functions/algorithms: should be easily parallelizable

“Reduce” functions/algorithms: should be commutative and associative.

Question 3

What are the two types of Spark operations?

Answer 3

Transformations and Actions

Question 4

How is Spark similar to git?

Answer 4

They both keep track of the “roadmap” of how the dataset gets from one state to the next. This is what makes Spark’s RDDs (Resilient Distributed Datasets) resilient: if one worker node breaks, the “instructions” for doing its job are still intact.

Question 5

Why are RDDs resilient?

Answer 5

Because Spark keeps track of the “roadmap” of how the dataset gets from one state to the next. So if one worker node breaks, the “instructions” for doing its job are still intact.

Question 6

What are key characteristics of Spark RDDs or Dataframes?

Answer 6

• Immutable
• Resilient

Question 7

Example imports/code to create a Spark dataframe from an RDD

Answer 7

Just these 2 lines needed to create an RDD
from pyspark import SparkContext
sc = SparkContext(“local[*]”, “temp”)

This needed for Dataframes
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

df = my_rdd.toDF()

Question 8

How does caching Spark dataframes work?

Answer 8

Add .cache() at the end of a chain of transformations.

The next action called after that will run all the transformations, including the caching.

The 2nd time, 3rd time, etc this new df is referenced with an action, it will be using its cached version & won’t have to recompute all the transformations again.

Question 9

Do both RDDs and Dataframes optimize the order of transformations?

Answer 9

Yes.