General

Question 1

how do you cache a dataframe

Answer 1

.persist
.cache.count
doesn’t have to be count but has to be an action that will touch every single record

Question 2

How can you select where your cache is stored

Answer 2

.persist(storage level)

Question 3

the default storage level for persist and cache is

Answer 3

MEMORY_AND_DISK

Question 4

how do you un cache data

Answer 4

.unpresist.count

Question 5

How can you determine the storage level of your data frame

Answer 5

.storageLevel

Question 6

How do you register a function as a dataframe function

Answer 6

val function_udf = udf(stringConcat(_:ParamType…):ReturnType)

Question 7

how do you register a function as SQL Function

Answer 7

spark.udf.register(“new_function_name”, function signature)

Question 8

how can you create a table from a dataframe

Answer 8

.write.saveAsTable(“table_name”)

Question 9

how can you set the number of partitions for a shuffle

Answer 9

spark.conf.set(“spark.sql.shuffle.partitions”,50)

Question 10

how do you get the number of partitions available in a given dataframe

Answer 10

.rdd.getNumPartitions

Question 11

how do you repartition a dataframe

Answer 11

.repartition(2)

Question 12

how can you change the number of partitions on a single node

Answer 12

.coalesce(2)

Question 13

which causes a shuffle repartition or coalesce

Answer 13

repartition

Question 14

How do you enable adaptive query execution

Answer 14

spark.conf.set(“spark.sql.adaptive.enabled”, true)

Question 15

What are the elements of an Apache Spark Execution Hierarchy

Answer 15

Job, Stages, and Tasks

Question 16

Adaptive Query Execution re-optimizes the query plan in the middle of the query execution based on accurate runtime statistics T/F

Answer 16

True

Question 17

With AQE, Logical optimization and physical planning is removed

Answer 17

False

Question 18

what does spark.sql.autoBroadcastJoinThreshold do

Answer 18

Configures the maximum size in bytes for a table that will broadcast to all worker nodes when performing a join

Question 19

How do you turn off dynamic partitions coalescing

Answer 19

spark.conf.set(“spark.sql.adaptive.coalescePartitions.enabled”,false)

Question 20

What allows you to control how complex types are printed on schemas

Answer 20

.printSchema(1)

Question 21

How do you set infer schema

Answer 21

.option(“inferSchema”, true)

Question 22

How do you make a dataframe into a table or a view

Answer 22

createOrReplaceTempView()