04 Working With Spark Data Spurces & Sinks

Question 1

What are spark data sources.

Answer 1

Sources from where spark reads data.

Question 2

Different types of data sources

Answer 2

1. External data sources.
2. Internal data sources.

Question 3

Name some external data sources.

Answer 3

Oracle, SQL, Casandra, snowflake, redshift, khafka

Question 4

Eg of internal storage system.

Answer 4

HDFS, Azure, Amazon S3

Question 5

What are data sinks

Answer 5

Place where output is stored.it can be both internal and external.

Question 6

Name file format where schema is well-defined.

Answer 6

AVRO & Parquet.

Question 7

Can we mention schema in json file

Answer 7

no

Question 8

in what files can we mention schema information in the file

Answer 8

parquet & AVRO

Question 9

suggested file format to write in

Answer 9

Parquet

Question 10

how can we manage data layout

Answer 10

by using partitioning we can partition data on 1 or more than 1 column using which we can organize data really well.

Question 11

if there are partitioning how will data frame reader read it

Answer 11

data frame reader will read all the subdirectories

Question 12

what will happen when we apply filter on partitioned data

Answer 12

in this case it will not go to all the files but it will go to the required data.