Information

Question 1

• drop(): columns or name

Question 2

• Split is a function: storesDF.withColumn(“storeValueCategory”, split(col(“storeCategory”), “_”)[0])

Question 3

• CAST is a COL method: col(“storeId”).cast(StringType())

Question 4

• Explode is a function: storesDF.withColumn(“productCategories”, explode(col(“productCategories”)))

Question 5

• regexp_replace is a function: storesDF.withColumn(“storeDescription”, regexp_replace(col(“storeDescription”), “^Description: “, “”))

Question 6

• na.fill: storesDF.na.fill(value = 30000, subset = “sqft”) value, list or string or tupple of column names. Subset can be a list or string.

Question 7

• dropDuplicates or drop_duplicates which is an alias: DataFrame.drop_duplicates(subset = None) or storesDF.dropDuplicates(subset = [“id”]). subset should be a list or tuple.

Question 8

• approxCountDistinct or approx_count_distinct is used with the .agg: storesDF.agg(approx_count_distinct(col(“division”), 0.15).alias(“divisionDistinct”)) ,
  the default is 0.05 if we don’t specify.

Question 9

• mean is a function that can be used with .agg: storesDF.agg(mean(col(“sqft”)).alias(“sqftMean”)) take a col or col_name (shorthand for df.groupBy().agg()).

Question 10

• describe is a function that takes column_names string or list. df.describe([‘age’,’size’]).show(),df.describe(‘age’,’size’).show()

Question 11

• orderBy is a function that takes str, list, or Column with an ascending parameter bool defaults to True. df.orderBy([“age”, “name”], ascending=[False, False])

Question 12

• sample is a function that takes as parameter: storesDF.sample(withReplacement = False, fraction = 0.10, seed = 123)

Question 13

• printSchema(level: Optional[int] = None) has level parameter specifies how many levels to print for nested schemas.

Question 14

• for udf sql: spark.udf.register(“function_name”, udf_function, returnType:Optional) then in sql we call the function by the function_name

Question 15

• for udf python: assessPerformanceUDF = udf(assessPerformance,IntegerType()), storesDF.withColumn(“result”, assessPerformanceUDF(col(“customerSatisfaction”)))

Question 16

• to create a dataframe from a list, spark.createDataFrame(years, IntegerType())

Question 17

• .cache() is by default MEMORY_AND_DISK, it takes no parameters

Question 18

• .persist() is by default MEMORY_AND_DISK but takes storageLevel as parameter. df.persist(storageLevel = StorageLevel.DISK_ONLY)

Question 19

• spark.sql.adaptive.coalescePartitions.enabled is used to configure whether DataFrame partitions that do not meet a minimum size threshold are automatically coalesced into larger partitions during a shuffle.

Question 20

• spark.sql.shuffle.partitions is used to adjust the number of partitions used in wide transformations like join(). spark.conf.set(“spark.sql.shuffle.partitions”, “32”)

Question 21

• from unix time to timestamp: storesDF.withColumn(“openDateString”, from_unixtime(col(“openDate”), “EEE, MMM d, yyyy h:mm a”))

Question 22

• from unix to dayofyear (storesDF.withColumn(“openTimestamp”, col(“openDate”).cast(“Timestamp”)).withColumn(“dayOfYear”, dayofyear(col(“openTimestamp”))))

Question 23

• for joins: joinedDF = StoresDF.join(other = employeesDF, on= “storeId”, how = “inner”), the how could be inner, cross, outer, full, fullouter, full_outer, left, leftouter, left_outer, right, rightouter, right_outer, semi, leftsemi, left_semi, anti, leftanti and left_anti.

Question 24

• for the join, when using, df = a.join(b, on=[“column1”, “column2”]) PySpark sees this as a list of column names and automatically matches column1 in DataFrame a with column1 in DataFrame b, and column2 in a with column2 in b. With df = a.join(b, on=[co(“column1”), col(“column2”)]) you are passing column expressions instead of simple column names. PySpark sees each col(“column1”) as an individual expression without any direct context to the DataFrames a or b. It doesn’t know whether col(“column1”) refers to a or b — which causes ambiguity error.

Question 25

• for outer joins, the JOIN can perform them and there’s no outer function

Question 26

• for cross joins use: df.crossJoin(df2.select(“height”))

Question 27

• Brodcast joins: storesDF.join(broadcast(employeesDF), “storeId”)

Question 28

• for unions, storesDF.union(acquiredStoresDF) does a position-wise union between the dataframes and storesDF.unionByName(acquiredStoresDF, allowMissingColumns = False) does the union resolving columns by name rather than position.

Question 29

• to write in parquet partitioned, storesDF.write.mode(“overwrite”).partitionBy(“division”).parquet(filePath)

Question 30

• to write in json, storesDF.write.json(filePath), spark.read.schema(schema).format(“json”).load(filePath)

Question 31

• to read parquet, spark.read.load(path = filePath, schema = None, format = “parquet”)
  *to read CSV, spark.read.schema(schema).csv(filePath)

Question 32

• MEMORY_AND_DISK_2 store as much data as possible in memory on two cluster nodes while storing any data that does not fit in memory on disk to be read in when needed

Question 33

• spark.sql.autoBroadcastJoinThreshold is used to configure the maximum size of an automatically broadcasted DataFrame when performing a join

Question 34

• spark.sql.adaptive.skewedJoin.enabled Spark properties is used to configure whether skewed partitions are automatically detected and subdivided into smaller partitions when joining two DataFrames together

Question 35

• storesDF.drop() to return a new DataFrame from DataFrame storesDF without columns that are specified by name

Question 36

• using a filter, storesDF.filter(col(“sqft”) <= 25000), storesDF.filter((col(“sqft”) <= 25000) | (col(“customerSatisfaction”) >= 30))

Question 37

• storesDF.withColumn(“modality”, lit(“PHYSICAL”)), column modality is the constant string “PHYSICAL”

Question 38

• To rename columns, storesDF.withColumnRenamed(“division”, “state”).withColumnRenamed(“managerName”, “managerFullName”)

Question 39

• na.fill(value = “hello”, subset = [“col_name1”,”col_name2”]) or it’s alias .fillna(), subset can be str, tuple or list

Question 40

• DataFrame.dropDuplicates(), DataFrame.distinct() and DataFrame.drop_duplicates() used to return a DataFrame with no duplicate rows

Question 41

*storesDF.agg(approx_count_distinct(col(“division”)).alias(“divisionDistinct”)) returns the a dataframe where column divisionDistinct is the approximate number of distinct values in column division

Question 42

• both df.head(3) and df.take(3) returns the first 3 rows as a list. df.head() with no nb returns it as Row.

Question 43

• filter() and where() will both return a new DataFrame only containing rows that meet a specified logical condition

Question 44

• difference between coalesce and repartition, DataFrame.repartition(n) will split a DataFrame into n number of new partitions with data distributed evenly. DataFrame.coalesce(n) will more quickly combine the existing partitions of a DataFrame but might result in an uneven distribution of data across the new partitions.

Question 45

• DataFrame.select() is classified as a transformation.

Question 46

• absolute value, storesDF.withColumn(“customerSatisfactionAbs”, abs(col(“customerSatisfaction”)))

Question 47

• The Spark driver is responsible for scheduling the execution of data by various worker nodes in cluster mode.

Question 48

• A shuffle induces a stage boundary

Question 49

• A partition is a collection of rows of data that fit on a single machine in a cluster.

Question 50

• A Stage identifies multiple narrow operations that are executed in sequence. In short, a stage groups together narrow transformations that can be executed in sequence without data movement, and each stage boundary is created when a wide transformation that requires a shuffle occurs.

Question 51

• Spark execution/deployment modes include: Client mode, Cluster mode, Local mode.

Question 52

• A failed driver node will cause a Spark job to fail.

Question 53

• The MEMORY_ONLY storage level will store as much data as possible in memory and will recompute any data that does not fit in memory as it’s called. The MEMORY_AND_DISK storage level will store as much data as possible in memory and will store any data that does on fit in memory on disk and read it as it’s called.

Question 54

• to return the first two characters of a column, use storesDF.withColumn(“division”, col(“division”).substr(l, 2)). The position is not zero based, but 1 based index. We can also use the function substring df.select(substring(df.s, 1, 2).alias(‘s’))

Question 55

• to drop rows that have a nan value in any column, df.na.drop(how = “any”), to drop if all columns have nan, df.na.drop(how = “all”)

Question 56

• extracts the value for column sqft from the first row of DataFrame storesDF, storesDF.first().sqft

Question 57

• to create a tempview and query it: storesDF.createOrReplaceTempView(“stores”), spark.sql(“SELECT storeId, managerName FROM stores”)

Question 58

• Job, Stage, Task are the units of work performed by Spark from largest to smallest.

Question 59

• groupBy takes list, str or Column.

Question 60

• UDF using pyspark instead of sql, assessPerformanceUDF = udf(assessPerformance, IntegerType()) storesDF.withColumn(“result”, assessPerformanceUDF(col(“customerSatisfaction”)))

Question 61

• Executors are processing engine instances for performing data computations which run on a worker node.

Question 62

• Garbage collection is important because Spark jobs will fail or run slowly if memory is not available for new objects to be created.

Question 63

• To find the number of characters in each row, storesDF.withColumn(“managerNameLength”, length(col(“managerName”)))

Question 64

• if you have a Spark application running with 1 driver and 1 worker node, and the worker node fails. Spark will ensure completion because worker nodes are fault-tolerant.

Question 65

Worker nodes are machines that host the executors responsible for the execution of tasks.

Question 66

A task is a combination of a block of data and a set of transformers that will run on
a single executor

Question 67

A Stage is a group of tasks that can be executed in parallel to compute the same set of operations on potentially multiple machines

Question 68

A shuffle is the process by which data is compared across partitions.

Question 69

Transformations are business logic operations that do not induce execution while actions
are execution triggers focused on returning results

Question 70

The df.select() is always classified as a narrow transformation

Question 71

Spark’s execution/deployment mode determines where the driver and executors are
physically located when a Spark application is run

Question 72

An out-of-memory error occurs when either the driver or an executor does not have enough
memory to collect or process the data allocated to it.

Question 73

A broadcast variable is entirely cached on each worker node so it doesn’t need to be
shipped or shuffled between nodes with each stage.

Question 74

Spark DataFrames built on top of RDDs datastructures (Resilient Distributed Datasets).

Question 75

lower is a function that turns characters to lower case, storesDF.withColumn(“storeCategory”,
lower(col(“storeCategory”)))

Question 76

Sum of the values in column sqft in DataFrame
storesDF grouped by distinct value in column division,
storesDF.groupBy(“division”).agg(sum(col(“sqft”)))