Introduction

Question 1

What language is Spark written in?

Answer 1

SCALA which is based on JAVA

Question 2

Cluster Manager

Answer 2

Acquire’s resources, worker nodes, executors and task required to perform the work.

Question 3

Partition

Answer 3

Large task are split into chunks to be sent to a different node for processing.

Question 4

Executors

Answer 4

Executors are contained within each node and perform task to work in parallel with eachother. Each executor uses a seperate Java Virtual Machine

Question 5

worker node

Answer 5

Any node that can run application code in the cluster.

Question 6

Task

Answer 6

A unit of work that will be sent to one executor

Question 7

Job

Answer 7

A Parallel computation consisting of multiple task that gets spawned in response to a Spark action.

Question 8

Stage

Question 9

Cluster Managers

Answer 9

Program that controls the how the cluster processes data.

Question 10

Spark Standalone

Answer 10

A basic built-in cluster manager.

Question 11

Apache Mesos

Answer 11

A general cluster manager that can also run Hadoop MapReduce and service applications.

Question 12

Hadoop Yarn

Answer 12

The resource manager used in Hadoop 2

Question 13

Kubernetes

Answer 13

An open-source service for automating deployment, scaling, and management of containerized applications.

Question 14

What are the 4 spark core services?

Answer 14

Spark SQL, Spark Streaming, MLIB Machine Learning, GraphX

Question 15

What type of Dataframe does Spark use?

Answer 15

Spark SQL uses a distributed DataFrame

Question 16

What is spark streaming?

Answer 16

Spark Streaming offers realtime data processing that can take input’s from multiple sources, integrated with machine learning, then output to different data storage systems.

Question 17

What is Spark MLIB

Answer 17

Spark Machine Learning provides a set of tools for ML that are optimized to use with paralleized execution which enables processing of big data.

Question 18

What is GraphX?

Answer 18

GraphX is a library of tools used to traverse netwroks, display paths and visualize connections. (think relationship data between entities like flights, social networks )

Question 19

What is DataBricks?

Answer 19

A commercial, for-profit, company, founded by two of the creators of Apache Spark. The application provides a complete data engineering and science collaborative environment to develop spark applications