01 Understanding Big Data And PySpark

Question 1

Components of YARN

Answer 1

Resource manager, Node manager, Application manager

Question 2

Process of YARN

Answer 2

Resource manager will take requests from the team client

Node manager will send node status reports

Application manager will allocate cpu and RAM for the request to process

Question 3

Semi structured data

Answer 3

Key and value pair

RDBMS

Eg .JSON, .XML

Question 4

Lakehouse

Answer 4

Spark without Hadoop

Question 5

Process of HDFS

Answer 5

Main data will be broken down into blocks

Each block stored in 1data node

Name node will have all details about block and block location

Question 6

features of distributed approach

Answer 6

horizontal

Easily scalable

Cheap

Fault tolerance

Question 7

structured data

Answer 7

Rows and columns

COBOL & RDBMS

Question 8

Map/Reduce

Answer 8

Dividing problems into simple parts and solving accordingly

Question 9

Approach to solve big data problem

Answer 9

Monolithic & Distributed

Question 10

Big data problems

Answer 10

Volume, velocity, variety

Question 11

Unstructured data

Answer 11

No structure

E.g - images and audio

Question 12

Monolithic

Answer 12

Vertical

Difficult to scale

No fault tolerance

Expensive

Question 13

Data lake

Answer 13

Spark with Hadoop

Question 14

Components HDFS

Answer 14

Name node

Data node

Question 15

Hadoop Architecture

Answer 15

YARN, HDFS, Map/Reduce

Question 16

Types of Data

Answer 16

structured, semi-structured, unstructured

Question 17

Structure of DataLake

Answer 17

data lake is where all data is dumped; there is a middle layer named data warehouse where data from datalake after ETL is stored.

Question 18

Data warehouse

Answer 18

Initially all the data would stored in lakehouse which led to data disturbances. So now day is tired in data warehouse after ETL from where data is give to powerbi.

Question 19

Which type of distribution does apache spark have

Answer 19

Distributed distribution.

Question 20

2 layers of spark ecosystems

Answer 20

Set of core layer
Set of DSL (domain specific language)

Question 21

Layers of core layer

Answer 21

Distributed set of engine
Set of core API’s

Question 22

What does distributed set of engine do

Answer 22

Runs dat processing workload.
Responsible for map reduce, fault tolerance.

Question 23

What are set of core API’s

Answer 23

We can write logic in scala , R, java, python

Question 24

What are set of DSL

Answer 24

They have 4 groups
Spar SQL & Df - allows to use sql to get data from db
Streaming - allow to process continuous and unbound data
Mllib- have ML Libraries
Graphx - has graph processing libraries.

Question 25

Advantages of spark ecosystem

Answer 25

1. Abstraction - hides the fact that we are working on cluster of computers 2. Unifies platform- combines capabilities of all DSL & API 3. easy to use

Question 26

Types of Structured Data

Answer 26

1. Time Series 2. Cross Sectional 3. Panel Data

Question 27

what is Time Series data?

Answer 27

data collected for a single observational unit over a period of time.

Question 28

what is Cross Sectional data

Answer 28

data collected for a multiple observational unit at a point of time.

Question 29

what is Panel Data

Answer 29

data collected for a multiple observational unit over a period of time.