Design and Develop Data Processing

Question 1

What is Spark structured streaming?

Answer 1

Structured Streaming is the Apache Spark API that lets you express computation on streaming data in the same way you express a batch computation on static data. The Spark SQL engine performs the computation incrementally and continuously updates the result as streaming data arrives.

Question 2

What kind of Azure Stream Analytics window is fixed size, repeating, non-overlapping and in which events cannot belong to more than one window?

Answer 2

Tumbling Window

Question 3

Define Azure Stream Analytics Window: Sliding

Answer 3

Produces an output only when an event occurs.

Question 4

Describe a non-relational database.

Answer 4

They are good for varying data types and simple, fast queries. Typically non-relational databases utilize Cassandra or something similar.

Question 5

Describe the types of ADF triggers.

Answer 5

Schedule Trigger: Invokes a pipeline on a wall-clock schedule.

Tumbling Window Trigger: A trigger that operates on a periodic interval, while also retaining state.

Event-based Trigger: A trigger that responds to an event.

Question 6

List the 5 types of Azure Stream Analytics Windows.

Answer 6

Tumbling,
Hopping,
Sliding,
Session, 
Snapshot

Question 7

Describe Stream Analytics Tumbling Window.

Answer 7

Repeat, do not overlap, and an event cannot belong to more than one tumbling window. “Tell me the count of Tweets per time zone every 10 seconds.”

Question 8

Describe Stream Analytics Hopping Window.

Answer 8

Hop forward in time by a fixed period. “Tumbling windows that can overlap and can be emitted more often than the window size.” “Every 5 seconds give me the count of tweets over the last 10 seconds.”

Question 9

Describe Stream Analytics Sliding Window.

Answer 9

Only output events for points in time when the content of the window actually changes, ie, when an event enters or exits the window. Events can belong to more than one sliding window.

“Alert me whenever a topic is mentioned more than 3 times in under 10 seconds.”

Question 10

Describe Stream Analytics Session Window.

Answer 10

Group events that arrive at similar times, filtering out periods of time where there is no data.

“Tell me the count of tweets that occur within 5 minutes of each other.”

Question 11

Describe Stream Analytics Snapshot Window.

Answer 11

Group events that have the same timestamp.

Question 12

What are recommended optimization techniques for Azure Databricks?

Answer 12

Group jobs into pools by weight.

Optimize Watermark Policies to improve performance by better handling late data.

Question 13

How can PolyBase reduce the cost of transformation on a big data architecture?

Answer 13

By allowing for ELT which could greatly reduce the transformation needed on the data.

Question 14

What coding languages can be used for cells in Databricks?

Answer 14

Scala, R, Python & SparkSQL. NOT C#

Question 15

What are 3 valid inputs for Stream Analytics?

Answer 15

IoT Hub
Blob Storage
Event Hubs

Question 16

How are you charged using Data Factory?

Answer 16

Pipeline Orchestration

Inactive Pipelines

Question 17

What are the two options for storing Reference Data in Azure Stream Analytics?

Answer 17

SQL Database

Blob Storage

Question 18

What is a Databricks cluster?

Answer 18

A group of compute resources.

Question 19

How long is pipeline-run monitoring data kept in ADF?

Answer 19

45 days.

Question 20

What data ingestion tools can be used for maximum parallelization in ADLS?

Answer 20

DistCp: -m (mapper)
Azure Data Factory: parallelCopies
Sqoop: fs.azure.block.size, -m (mapper)

Question 21

Describe what a Stream Analytics window is used for.

Answer 21

Windows are subsets of events that fall within some period of time. The window is a simple way to work with the time component of query logic in the system.