Azure Databricks

Question 1

What is Azure Spark?

Answer 1

It is an open-source, streaming distributed computing system for big data and analytics. It can process big data across multiple nodes using Python, Scala, and R.

Question 2

What is the difference between Hadoop and Apache Spark?

Answer 2

Hadoop is a batch system; data is batched processed, SPark is stream processes, and the node keeps it in memory.

Question 3

Is Apache Spark in memory or on disk?

Answer 3

In-memory

Question 4

Can you stream data into Apache Spark, analyze it, and then store and then reanalyze the data using Spark?

Answer 4

Yes, you can store the analyzed streamed data in different storage systems, such as HIVE, HDFS, Cloud DStorage, etc. You can then query it using Spark and explore it further.

Question 5

Where do Spark results stored?

Answer 5

HDFS, distributed FS, Cloud Storage, SQL, No SQL

Question 6

What are the layers of the Apache Spark architecture?

Answer 6

1. Libs (Spark SQL, ML Lib, Spark R)
2. Spark Core
3. Resource Management
4. Datastore

Question 7

In Apache Sparc, what are the three workspaces?

Answer 7

1. Spark SQL
2. ML Lib
3. Spark R

Question 8

What is Azure Databricks?

Answer 8

It is an Apache Spark-based big data and machine learning platform.

Question 9

We are primarily an Azure orgnization but need a big data solution to run on AWS; what are my options?

Answer 9

Azure Databricks can be run on AWS.

Question 10

For Azure Databricks, what are the three environments supported?

Answer 10

1. SQL (Run SQL queries on a data lake, create visualizations and dashboards)
2. Data science and engineering (Collaborative workspace for working on big data pipeline and analytics)
3. Machine Learning (End to end ML, train, serving)

Question 11

For Azure Databricks, what are the pricing models?

Answer 11

1. Standard (All common Databricks capabilities, fully hosted)
2. Premium (Aditional management, RBAC, monitoring and logs, AD Passthrough)

Question 12

Is Azure Databricks fully hosted?

Answer 12

Yes

Question 13

Is Azure Databricks fully managed?

Answer 13

Yes

Question 14

I wnat to use Azure Databricks, and what is my use to log in using their SD creds? What options do I have?

Answer 14

Use the premium pricing tier; this has an AD passthrough.

Question 15

I require passthrough authentication for Databricks to use Azure Entra ID, can i use Standard tier?

Answer 15

No, you need premium tier to support passthrough.

Question 16

What is Azure Databricks?

Answer 16

Azure Databricks is a spark streaming platform for building, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale.

Question 17

Where is data stored using Azure Databricks?

Answer 17

1. It is store in Azure Data Lake using HDFS
2. Temporary storage is local

Question 18

What external storage can Azure Databricks usise apart from HDFS with Azure Data Lake?

Answer 18

External databases like Azure SQL Database, Azure Synapse.

Question 19

How does Azure Databricks analyze the streamed data?

Answer 19

1. It uses code like;
2. Python (using pySpark)
3. Scala
4. Java
5. R

Question 20

What are the use cases for Azure Databricks?

Answer 20

1. Data processing scheduling and management, in particular ETL
2. Generating dashboards and visualizations
3. Managing security, governance, high availability, and disaster recovery
4. Data discovery, annotation, and exploration
5. Machine learning (ML) modeling, tracking, and model serving
6. Generative AI solutions

Question 21

Can zure Databricks be used for Gen AI?

Answer 21

Yes

Question 22

Can zure Databricks be used for ML?

Answer 22

Yes

Question 23

Can Azure Databricks handle near-real-time data ingestion into Databricks Lakehouse?

Answer 23

Yes, Azure Databricks can ingest data into Azure Databricks Lakehouse.

Question 24

What is Azure Databricks Lakehouse?

Answer 24

Combines the best of a data lake and datawherehouse.