Spark and Python for Big Data

Question 1

When to use spark ?

Answer 1

When it no longer makes sense to fit all your data on RAM or into a single machine

Question 2

Local system vs Distributed system characteristics

Answer 2

Distributed :
leverages power of many machines
easier to scale adding new machines unlike local
fault tolerance: if one machine fails the whole system can still go on

Question 3

What is Hadoop ?

Answer 3

Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation.

Question 4

What is HDFS ?

Answer 4

Hadoop Distributed File System : allows user to work with very large datasets

Question 5

What is MapReduce ?

Answer 5

A programming paradigm that allows calculations on different machines of the cluster in two steps : map which is a local calculation, and reduce which combines the results over the results to yield a single result.

Question 6

What are the trackers involved in mapreduce ?

Answer 6

Job tracker (master node) sends code to run on the task trackers. the task trackers allocate CPU and memory for the tasks AND monitor the tasks on worker nodes

Question 7

What is Spark ?

Answer 7

Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

EASE OF USE and SPEED

Question 8

How does Spark relate to MapReduce ?

Answer 8

Spark is a flexible alternative to MapReduce

Question 9

Spark vs MapReduce ? (NOT hadoop vs Spark)

Answer 9

• MapReduce requires files to be stored in HDFS, Spark does NOT, wide variety of data formats
• Spark can perform operations up to xI00 times faster than MapReduce (Spark keeps most of the data in RAM after each transformation, and can spill over to disk if the memory is filled)

Question 10

What is RDD ?

Answer 10

Resilient Distributed Dataset (idea at the core of Spark)

Question 11

What are the 4 main features of RDDs ? (DFPS)

Answer 11

• Distributed collection of data
• Fault tolerance
• Parallel operation
• Ability to use many data sources (not limited to HDFS)

Question 12

What are 3 main characteristics of RDDS ? (ILC)

Answer 12

immutable, lazily evaluated, and cacheable

Question 13

What are the two types of Spark operations ?

Answer 13

• transformations : recipe to follow
• actions : Perform the recipe and return something

Because spark is lazy, with a large dataset you have to be, you don’t want to go on calculating all the transformations until you are sure you want to PERFORM them (action)

Question 14

RDD vs Dataframe ?

Answer 14

Since spark 2.0, we are moving toward dataframe syntactically but RDD at the core

Question 15

What language is Spark written in ?

Answer 15

Scala, so Scala API gets latests features, Scala is written in Java, sometimes takes one extra release cycle to get to python, another to get to R

Question 16

Can Spark be run on a single machine ?

Answer 16

Realistically spark is run on a cluster, AWS or google Cloud. These clusters will almost always be Linux based (real world)