05 Spark DF Intervals

Question 1

Difference between Hadoop & Spark in-terms of intermediate result storage

Answer 1

Hadoop will stores intermediate result on disk.
Spark does not store intermediate result on disk but stores in memory.

Question 2

Full Form of RDD

Answer 2

Resilient Distributed Dataset

Question 3

Use of RDD

Answer 3

used for transformation

Question 4

What are the steps before execution of a query

Answer 4

query –> spark sql engine –> catalyst optimizer –> RDD –> Execute

Question 5

Relation between spark dataframe, table & RDD

Answer 5

Spark table and dataframe are logical view of RDD as all the tables are later converted to RDD.

Question 6

Properties of RDD

Answer 6

1. RDD will partition main file into multiple files. So if it has to store a file it will divide the main file into multiple small files and store it.

Question 7

What are transformations

Answer 7

they are operations applied on dataframes to produce a new dataframe.

Question 8

Types of transformations

Answer 8

Narrow and wide transformations

Question 9

Define Narrow Transformation

Answer 9

while executing the file is converted to rdd which will break the file into multiple small files. Now when we need to apply transformations like where the operation is performed on each partition and finally the output is generated.
there is not shuffle in this.

Question 10

Define wide transformation

Answer 10

while executing the file is converted to rdd which will break the file into multiple small files. Now when we need to apply transformations like group by the operation is performed on each partition and finally the output should be combined from each partition which causes shuffle.

Question 11

How does spark divide its hierarchy(job)

Answer 11

job –> stages –> tasks

Question 12

how are jobs decided

Answer 12

1. based on dependencies on a table jobs are decided.
2. each action is divided into 1 job.

Question 13

how are jobs executed

Answer 13

1. spark creates a logical query plan for each job
2. jobs are broken at each wide transformation.
3. each wide transformation will create 2 stages.

Question 14

features of stages

Answer 14

1. the last operation of a stage is either a wide transformation or an action.
   stages can not run parallelly.

Question 15

features of a task

Answer 15

1. if data has x partitions; transformation will be applied on each partition hence x tasks will be created.
2. output of each task is stored in an exchange buffer.
3. number of tasks depends on number of partitions.
4. tasks run parallelly

Question 16

what is shuffle operation

Answer 16

movement of data from write exchange buffer of 1 stage to the read exchange buffer of the other stage. (sharing of data from 1 stage to another stage).

Question 17

Number of stages in query planning

Answer 17

There are 4 stages

Question 18

What are 4stages of query planning

Answer 18

1. Analysis - in this stage the SQL engine will go to the catalog and check if the variables are present or not.
2. Logical optimisation- in this stage an optimised logical plan is created by spark based on multiple points
3. Physical planning - in this spark will calculate cost running in different ways and choose the best way to run the query.
4. Code generation- using the selected physical plan spark will create rdd and execute the query.