Instructor's Method - 6/16/2021

Question 1

Azure Data Factory

Answer 1

Ingest - copy data 90+ different sources

Transform - map data flows. Code-free workflows

Orchestration - end-to-end workflows =>

This is a Standalone-Tool

Question 2

Diff bet Azure Data Factory and Synapse Pipeline

Answer 2

Barely - they have same codebase

Question 3

Mapping Data Flows

Answer 3

Visually design ETL pipelines

Once data flow is created, run in

Question 4

Spark

Answer 4

Open-source data processing engine built around speed, ease of use, and sophisticated analytics

Compute engine designed for distributed data processing at scale

In-memory engine that is up to 100 times faster than Hadoop

Largest open-source data project

Multi-language support - Scala, Java, SQL, R & Python

Question 5

Spark SQL

Answer 5

Batch processing

Question 6

Spark Streaming

Answer 6

Stream processing

Question 7

Spark Pool

Answer 7

Node size

Number of nodes

Apache Spark version

Different library versions that will be installed

Question 8

Spark Pool

Answer 8

Node size

Number of nodes

Apache Spark version

Different library versions that will be installed

Auto-pause (auto-termination time)

Question 9

How to set header

Answer 9

.option(“header”,”true”)

Question 10

Which is Native Spark function to analyze data

Answer 10

describe()

Question 11

What is display() function used for

Answer 11

in-built function of spark to show data in data frame

Question 12

Difference between show() and display()

Answer 12

show() is native function of spark

display() is from Synapse and shows data in html

Question 13

Show data in text format

Answer 13

spark.read.parquet(“filename”)

Question 14

Show data in tabular format

Answer 14

using display(Dframe, True)