Data Analytics in Azure

Question 1

Batch vs Stream Processing

Answer 1

Batch processing, in which multiple data records are collected and stored before being processed together in a single operation.

Stream processing, in which a source of data is constantly monitored and processed in real time as new data events occur.

Question 2

Advantages of Batch Processing

Answer 2

Large volumes of data can be processed at a convenient time.

It can be scheduled to run at a time when computers or systems might otherwise be idle, such as overnight, or during off-peak hours.

Question 3

Disadvantages of Batch Processing

Answer 3

The time delay between ingesting the data and getting the results.

Dependencies between data observations as all of a batch job’s input data must be ready before a batch can be processed. Problems with data, errors, and program crashes that occur during batch jobs bring the whole process to a halt.

Question 4

General four step architecture for stream processing

Answer 4

1. Event generates data
2. Data is captured at a streaming source
3. Data is processed
4. Results are written to an output (a.k.a sink)

Question 5

Stream processing sources in Azure

Answer 5

Azure Event Hubs
Azure IoT Hub
Azure Data Lake Store Gen 2
Apache Kafka

Question 6

Stream processing sinks in Azure

Answer 6

Azure Event Hubs
Azure Data Lake Store Gen 2
Azure blob storage
Azure SQL database
Azure Synapse Analytics
Azure Databricks
Microsoft Power BI

Question 7

Azure Stream Analytics

Answer 7

Azure Stream Analytics is a service for complex event processing and analysis of streaming data.

Stream Analytics is used to:

• Ingest data from an input
• Process the data by using a query to select, project and aggregate data values
• Write the results to an output

Question 8

Apache Spark on Microsoft Azure

Answer 8

Apache Spark is a distributed processing framework for large scale data analytics. You can use Spark on Microsoft Azure in the following services:

Azure Synapse Analytics
Azure Databricks
Azure HDInsight

Question 9

Delta Lake

Answer 9

Delta Lake is an open-source storage layer that adds support for transactional consistency, schema enforcement, and other common data warehousing features to data lake storage.

Question 10

Azure Data Explorer

Answer 10

is a standalone Azure service for efficiently analyzing data. You can use the service as the output for analyzing large volumes of diverse data from data sources such as websites, applications, IoT devices, and more.

Question 11

Kusto Query Language (KQL)

Answer 11

language that is specifically optimized for fast read performance – particularly with telemetry data that includes a timestamp attribute.