Spark Flashcards
1
Q
What is
AQE
A
Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan, which is enabled by default since Apache Spark 3.2.0.
2
Q
What is
Shuffle
A
A shuffle is operation when data needs to move between executors.
3
Q
How to
Decrese size of spark partitions
A
df.coalesc(<number of partitions or list of columns>)
4
Q
How to
Decrese size of spark partitions
A
df.coalesc(<number of partitions or list of columns>)
5
Q
A
