DataFrameWriter Flashcards
1
Q
How can you save the dateframe to the tmp folder
A
.write
.option(“path”,”tmp”)
.save
2
Q
each partition creates how many files
A
1
3
Q
How many partitions are written out
A
200 unless you repartition
4
Q
How do you repartition
A
.repartition(number of partitions)
5
Q
How do you overwrite date
A
.write.mode(SaveMode.Overwrite)
6
Q
What is the default file format
A
parquet
7
Q
how can you write files as json
A
.write.format(“json”)
8
Q
What is the default compression
A
snappy
9
Q
how can you specify a compression type
A
.write.option(“compression”, “snappy”)
10
Q
How can you specify the partition by column on writing the data
A
.write.partitionBy(“column_name”)
11
Q
how can you read in a parquet file
A
spark.read.parquet(“file path”)
12
Q
partition by takes what as an argument
A
string
