Spark Structured API

Question 1

What is the problem with Resilient Distributed Datasets (RDDs)?

Answer 1

Resilient Distributed Datasets are low-level objects that are hard to understand and maintain when coding Spark jobs. Efficiency is also an issue as RDD’s are executed directly without optimization.

Question 2

What is the Spark SQL Module?

Answer 2

Adds structure to data with high-level API’s: Dataframes, Datasets and SQL.

Provides benefits like better readability, type checking and faster execution.

Works with Catalyst Optimizer for performance improvements.

Question 3

What are Resilient Distributed Datasets (RDD)?

Answer 3

Low-level API with manual optimization, code is harder to read and maintain.

Question 4

What are DataFrames?

Answer 4

Tables with schema (columns and rows), distributed structure inspired by Python and R, commonly used due to its simplicity and performance.

Question 5

What are DataSets?

Answer 5

Adds type safety by working with domain-specific types, preferred when strict control over data and compile-time error checking are required.

Question 6

What is Catalyst Optimizer?

Answer 6

Analyzes and optimizes structured queries and converts DataFrames, DataSets and SQL into optimized RDD’s.

Question 7

What is type safety?

Answer 7

DataFrames: no compile-time type checking; runtime errors are possible.
DataSets: Type-safe; compile-time errors prevent invalid operations.

Question 8

Why are Structured API’s (DataFrames/DataSets/SQL) more optimized than RDD’s?

Answer 8

Catalyst Optimizer.