Ch. 1 - Get started with data engineering on Azure

Question 1

What is Data Integration?

Answer 1

Establishing links between data sources to enable access to data across multiple systems.

Question 2

What is data transformation?

Answer 2

Transforming operational data into suitable structure and format for analysis. Often part of extract, transform, and load process.

Question 3

What is data consolidation?

Answer 3

Combining data that has been extracted from multiple data sources into a consistent structure - supports analytics and reporting.

Question 4

What is operational data?

Answer 4

Operational data is typically transactional data that is generated and stored by apps, often in a non-relational or relational database.

Question 5

What is analytical data?

Answer 5

Analytical data has been optimized for analysis and reporting, often in a data warehouse.

Question 6

What is streaming data?

Answer 6

Refers to perpetual sources of data that generate data values in-real time, relating to specific events (IoT).

Question 7

What is a data lake?

Answer 7

Storage repository that holds large amounts of data in native, raw formats. Optimized for scaling to MASSIVE volumes, comes from multiple heterogeneous sources, may be structured, semi-structured, or unstructured.

Store everything in its original, untransformed state.

Question 8

What is a data warehouse?

Answer 8

Centralized repo of integrated data from one or more disparate sources. Stores current and historical data in relational tables that are organized into a schema that optimizes performance for analytical queries.

Data engineers are responsible for designing and implementing relational data warehouses, and managing regular data loads into tables.

Question 9

What is Apache Spark?

Answer 9

Parallel processing framework that takes advantage of in-memory processing and a distributed file storage.

Data engineers need to be proficient with Spark, using notebooks and other code artifacts to process data in a data lake and prepare it for modeling and analysis.

Question 10

How is Azure Data Lake Gen2 Hadoop compatible?

Answer 10

1. Can treat data as if its HDFS, stored in one location, and access it via compute tech. (Azure Databricks, Azure HDInsight, and Azure Synapse Analytics) without moving the data. Also have access to parquet (columnar) format.

Question 11

Explain ADLG2 Security Features

Answer 11

2. Supports Access Control Lists (ACLs) and Portable Operating System Interface (POSIX) permissions that don’t inherit the permissions of the parent dir. Security is configurable via Hive, Spark, or Azure Storage Explorer.

Question 12

Explain ADLG2 File Storage

Answer 12

3. Stores data into a hierarchy of directories, like a file system for ease of navigation.

Question 13

Explain ADLG2 Data Redundancy

Answer 13

4. Data redundancy - Data Lake Storage takes advantage of Azure Blob replication models with LRS / GRS options.

Question 14

How are Blob files stored?

Answer 14

Blobs allow for large amounts of unstructured (“object”) data in a flat namespace within a blob container. Names include “/” characters to organize blobs into virtual “folders”, but are actually stored as a single-level hierarchy in a flat namespace. Accessed via HTTP/HTTPS.

Question 15

How does Azure Data Lake Storage Gen2 compare to Azure Blobs?

Answer 15

Azure Data Lake Storage Gen2 builds on blob storage and optimizes I/O of high-volume data by using a hierarchical namespace that organizes blob data into directories, and stores metadata about each directory and the files within it.

This structure allows operations, such as directory renames and deletes, to be performed in a single atomic operation.

Question 16

What are the four stages of processing big data?

Answer 16

Ingest, Store, Prep and Train, Model and Serve

Data lakes have a fundamental role in a wide range of big data architectures. These architectures can involve the creation of:

An enterprise data warehouse.
Advanced analytics against big data.
A real-time analytical solution.

Question 17

What is Data Ingestion in processing big data?

Answer 17

Ingest - acquire source data. I.e. batch movement of data in Azure Synapse Analytics or Azure Data Factory.

Real-time ingestion could be Apache Kafka for HDInsight or Stream Analytics.

Question 18

What is Data Store in processing big data?

Answer 18

The store phase identifies where the ingested data should be place (Azure Data Lake Storage Gen2 for big data).

Question 19

What is Prep and train phase in processing big data?

Answer 19

ID’s technologies that are used to perform data prep and model training. Azure Synapse Analytics, Azure Databricks, Azure HDInsight, Azure Machine Learning

Question 20

What is Model and Serve in processing big data?

Answer 20

Present data to the users. Microsoft Power BI, Azure Synapse Analytics.

Question 21

What is Azure Synapse Link?

Answer 21

Azure Synapse Link is a data integration feature that synchronizes operational data from services like Azure Cosmos DB, Azure SQL Database, SQL Server, and Microsoft Dataverse in near real-time for analytics in Azure Synapse.

Question 22

What does Microsoft Purview do?

Answer 22

Microsoft Purview is a unified data governance solution that catalogs data assets—including those in Azure Synapse—so data engineers can easily discover data, understand lineage, and track it across pipelines.

Question 23

What are common use cases for Azure Synapse Analytics?

Answer 23

Large-scale Data Warehousing
Advanced Analytics
Data Exploration
Real Time analytics
Data integration

Question 24

What is a SQL Serverless Pool?

Answer 24

On-demand SQL query processing, primarily used for work with data in a data lake.

It is not good for transactional data requiring millisecond response times.

Question 25

What is a SQL Dedicated Pool?

Answer 25

Enterprise-scale relational database instances used to host data warehouses in which data is stored in relational tables.

Question 26

What does the OPENROWSET syntax do?

Answer 26

Enables querying of various serverless SQL pool formats (CSVs, JSON, and Parquet files)

SELECT TOP 100 *
FROM OPENROWSET(
BULK ‘https://mydatalake.blob.core.windows.net/data/files/*.csv’,
FORMAT = ‘csv’,
PARSER_VERSION = ‘2.0’,
FIRSTROW = 2) AS rows

The PARSER_VERSION is used to determine how the query interprets the text encoding used in the files. Version 1.0 is the default and supports a wide range of file encodings, while version 2.0 supports fewer encodings but offers better performance. The FIRSTROW parameter is used to skip rows in the text file, to eliminate any unstructured preamble text or to ignore a row containing column headings.

https://learn.microsoft.com/en-us/training/modules/query-data-lake-using-azure-synapse-serverless-sql-pools/3-query-files

Question 27

What is Azure Blob storage?

Answer 27

Used to store unstructured data such as videos, audio, metadata, log files, text, and binary.

It can be accessed via Representation State Transfer (REST) - one way of implementing web service endpoints.

Question 28

What is a Hierarchical namespace?

Answer 28

Means that all directories within a storage account will contain metadata describing their structure and content which improves operations such as moving an entire directory structure to a different parent directory, as only the metadata will need to be updated.

Question 29

Difference between Blob Storage and ADLS Gen2?

Answer 29

This makes operations such as renaming and deleting directories atomic and quick. For example, if you have 100 files under a directory in Blob Storage, renaming that directory would require 100 metadata operations.

Question 30

What are Azure Files?

Answer 30

Azure Files offers fully managed file shares in the cloud that are accessible via the industry standard Server Message Block (SMB) protocol, Network File System (NFS) protocol, and Azure Files REST API. Azure file shares can be mounted concurrently by cloud or on-premises deployments. SMB Azure file shares are accessible from Windows, Linux, and macOS clients.

File shares are provisioned via Azure Storage Account V2 or premium File Share Tier.

Question 31

What are Azure Queues?

Answer 31

Used to store a large number of messages that can be accessed asynchronously between the source and the destination.

Asynchronously means a sender can add a message to the queue and the receiver can retrieve it later when ready

Question 32

What is Azure Queue Storage?

Answer 32

Storage queues can be used ​for simple asynchronous message processing. They can stor​e up to 500 TB of data (per storage account) and each message can be up to 64 KB in size

You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue may contain millions of messages, up to the total capacity limit of a storage account. Queues are commonly used to create a backlog of work to process asynchronously.

Question 33

How is Azure Service Bus used?

Answer 33

A fully managed enterprise messaging service used to reliably transfer business data (sales, purchases, inventory movements, journals) among decoupled systems through Azure Queues.

Question 34

What are Azure Tables?

Answer 34

Azure tables refer to key-value stores for structured, non-relational data,.

Azure Table Storage = cost-effective basic option

Azure Cosmos DB (the premium service) offering advanced features like global distribution, flexible consistency models, and serverless compute

Question 35

What are Azure Managed Disks?

Answer 35

Virtual hard disks that are mounted to an Azure VM, available standard HDD, standard SSD, premium SSD, and ultra disks.

Question 36

What are Azure VNets?

Answer 36

Ties all resources such as VMs, storage accounts, and databases together securely in a private network.

Provides 4 main services:

Security: provides secure connectivity within azure, using basic VNet, VNet Peering, and Service endpoints.

Networking: Provides networking beyond the Azure Cloud and into the internet and hybrid clouds using express routes, private endpoints, and point-to-site and site-to-site VPNs.

Filtering: Provides networking filtering/firewalls that can implemented either via network or app security groups.

Routing: provides network routing abilities that allow configuration network routes using route tables and Border Gateway Protocol.

Question 37

What is Azure Synapse?

Answer 37

Azure Synapse is Microsoft’s unified analytics platform that integrates data ingestion, data warehousing, and big data analytics in one environment, enabling teams to ingest, transform, and analyze data at scale