6. Programming (Python, Scala, Java)

Question 1

How do you read and process large CSV or JSON files using Python?

Answer 1

You can use libraries like Pandas for CSV files and the built-in json module for JSON files.

Question 2

What are Python libraries for data processing, like Pandas and PySpark?

Answer 2

Pandas is used for data manipulation and analysis, while PySpark is used for large-scale data processing.

Question 3

Explain how you would handle schema evolution in JSON data.

Answer 3

You can manage schema evolution by using flexible data structures and versioning your schemas.

Question 4

How would you implement a sliding window algorithm in Python for streaming data?

Answer 4

You can use collections.deque to maintain a fixed-size window of data.

Question 5

How do you manage memory optimization in PySpark?

Answer 5

You can optimize memory usage by tuning Spark configurations and using efficient data formats.

Question 6

What are Python decorators, and when would you use them in ETL scripts?

Answer 6

Decorators are functions that modify the behavior of other functions. They can be used for logging or performance monitoring in ETL scripts.

Question 7

How would you parallelize data processing in Python?

Answer 7

You can use the multiprocessing module or libraries like Dask to parallelize data processing.

Question 8

How does Scala handle immutability and parallel processing in Spark?

Answer 8

Scala uses immutable collections and functional programming principles to enable safe parallel processing in Spark.

Question 9

Explain the purpose of Python generators in handling large datasets.

Answer 9

Generators allow you to iterate over large datasets without loading them entirely into memory.

Question 10

How do you handle exceptions in Python for robust ETL workflows?

Answer 10

You can use try-except blocks to catch and handle exceptions, ensuring the ETL process continues smoothly.