Azure Data Lake

Question 1

What is the Parquet file type, and where are they used?

Answer 1

0. Structured
1. A Parquet file is a columnar storage file format that is designed for efficient data processing, particularly in large-scale data systems. It was created as part of the Apache Hadoop ecosystem and is commonly used in big data platforms such as Apache Spark, Apache Hive, and Apache Impala. Parquet is optimized for both storage efficiency and performance, especially for analytical and read-heavy workloads.
2. Columnar Format: Unlike row-based formats (like CSV or Avro), Parquet stores data column by column. This means that when querying data, only the necessary columns need to be read, significantly reducing I/O operations and improving query performance, especially for large datasets.
3. Efficient Compression: Parquet supports highly efficient compression and encoding schemes (such as Snappy, Gzip, LZO, etc.) at the column level. This reduces storage requirements and speeds up query processing by only decompressing the necessary data.
4. Splittable: Parquet files can be split into chunks, making them ideal for distributed data processing frameworks like Hadoop and Spark, where multiple nodes can read parts of a file in parallel.
5. Schema: Like Avro, Parquet files are schema-based, meaning the structure of the data is stored with the file itself, allowing different applications to interpret the data consistently.
6. Supports Complex Data Types: Parquet can handle complex data types such as nested structures, arrays, and maps, which makes it versatile for different types of data and use cases.
7. Compatibility: Parquet is compatible with various big data tools and languages, including Apache Hive, Apache Drill, Apache Impala, Apache Spark, and even cloud platforms like AWS (Amazon S3) and Azure Data Lake.

Question 2

What is the Avro file type, and where are they used?

Answer 2

0. Structured
1. An Avro file is a binary file format used for serializing data, typically in big data processing and storage systems.
2. Avro is part of the Apache Hadoop ecosystem and is often used with Apache Kafka, Apache Spark, and Apache Hive.
3. Schema-Based: Avro uses a JSON-based schema to define the structure of the data. This schema is stored with the data, ensuring that the data can be read and written consistently even across different systems.
4. Compact & Efficient: Avro files are compressed binary formats, making them more efficient in terms of storage space and data transfer compared to text formats like JSON or CSV.
5. Splittable: Avro files are designed to be splittable, meaning large Avro files can be divided into smaller chunks, making them ideal for distributed systems like Hadoop, where parallel processing is common.

6 Interoperability: Since Avro stores schema information with the data, it enables cross-language interoperability. Applications written in different programming languages (such as Java, Python, etc.) can easily read and write Avro data.

7. Schema Evolution: Avro supports schema evolution, meaning you can modify the schema of your data (e.g., adding new fields or removing existing ones) while maintaining backward and forward compatibility. This feature is useful when managing large datasets over time.

Question 3

What can I store in Azure Data Lake Storage Gen2?

Answer 3

Structured: Avro, Parque.
Structured: CSV, JSON, XML.
Unstructured: Images, Video.

Question 4

What is Azure Data Lake Storage Gen2 running on top of?

Answer 4

Azure Blob Storage Service.

Question 5

How do I create a Gen2 Azure Data Lake?

Answer 5

By creating an Azure Storage VBBlob Account and checking the Enabling higherarchy?

Question 6

What are examples of structured file types for Azure Data Lake?

Answer 6

Avro
Parque

Question 7

What are examples of semi-structured file types for Azure Data Lake?

Answer 7

XML
JSON
YAML

Question 8

What are examples of unstructured file types for Azure Data Lake?

Answer 8

images
movies

Question 9

What performance levels can I have with Azure Data Lake?

Answer 9

Standard (Standard Blob)
Premium (Premium Blob)

Question 10

For Azure Data Lake, do all the Azure Storage functions still work as Azure Data Lake? Is it built on top of Azure Storage?

Answer 10

Most functions but not all?

Question 11

For Azure Data Lake, do I still get lifecycle and tiered storage?

Answer 11

Yes

Question 12

In Azure Data Lake, you get a hierarchical namespace; what do you also get in permissions?

Answer 12

Do you get POSIX permissions?

Question 13

Do you access the Azure Storage Account API or Azure Data Lake data?

Answer 13

You can, but you do not have the hierarchical namespace or POSIX permissions; you should use the Azure Storage Account API DFS, and if required, the driver called

Question 14

What is the Azure Data Lake DFS?

Answer 14

It’s the distributed files system that Azure Data Lake uses and is accessed using the API.

Question 15

What is the Azure Data Lake ABFS Driver?

Answer 15

It’s the ABFS driver that Azure Data Lake allows systems like Hadoop and Sparc to interact with Azure Data Lake.