Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Big Data > Big Data- Lec 2 (Hadoop) > Flashcards

Big Data- Lec 2 (Hadoop) Flashcards

1
Q

what are the 2 components that Hadoop is built on?

A

HDFS and Mapreduce

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

t/f

Hadoop is a centralized file system. All data is stored in one location making it easier to gather information.

A

false - hadoop is a distributed file system. purpose is to promote sharing of dispersed files.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q
  1. t/f

A distributed file system is something where users are able to see everything at their end. ( Back end and front end are the same).

  1. Which of the following does a DFS not provide?

a. distributed storage
b. fault tolerance
c. low throughput data access

A
  1. false- users dont see the distribution at their level
  2. c - dfs provides HIGH throughput
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

t/f - hadoop infrastructure

  1. Hadoop is powerful but not scalable. it can only handle a few compute nodes and terabytes of information
  2. Hadoop is very cost-effective since it uses low cost commodity hardware
  3. hadoop is efficient since it distributes the data and processes it individually but never at the same time.
A
  1. false - it is scalable. it can handle thousands of compute nodes and PB of info
  2. t
  3. F - yes data is distributed, but its efficient bc the data is processed in parallel
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

t/f

  1. hadoop is an open source system
  2. hadoop provides analysis of only structured data
  3. compute nodes in hadoop can offer local computation and storage

4 . Apache spark is a software library that allows us to do distributing processing of small datasets across computer ‘clusters’

A
  1. T
  2. F - both struc and unstruc
  3. T
  4. F
    - apache HADOOP is the library
    -data sets are huge (big data )
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

which is true about hadoop?

  1. The Hadoop framework includes master and slave nodes for hdfs and MapReduce
  2. master node for hdfs is job-tracker and for map reduce its name node
  3. slave node for hdfs is task-tracker and for MapReduce its data node.
  4. hadoop is made up of only one component in its platform which is hive.
A
  1. T
  2. F- Master node:
    - hdfs = name node
    - map reduce= job tracker
  3. F- slave node:
    - hdfs = name node
    - map reduce= job tracker

4.false, it has many components (hive, hbase, spark etc)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

which of the following is not correct about hadoop stack?

  1. pig is a component of hadoop used for system management and maintenance. It helps to grab or kill applications
  2. Hive is a data warehouse-like tool that is built on top of mapreduce.
  3. Hadoops main framework revolves around HDFS and Mapreduce
  4. Zookeeper allows you to automate distributed data flow using a special language and do parallel computation.
A
  1. F - system management is done by zoo keeper
  2. F - its built on top of hdfs
  3. T
  4. F - Pig allows you to do this using language called Piglatin
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

T/F

  1. Hdfs is responsible for distributed processing across computer clusters
  2. To transfer data, you can use scoop
  3. To process streaming data ( that is very fast), you use hbase
  4. Hive is used to summarize data and to SQL-like queries
  5. To deal with large tables in a distributed database, use apache spark as it can handle large datasets very quickly.
A
  1. F - Mapreduce
  2. T
  3. F- use spark
  4. T
  5. F - for large tables- use hbase
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Which of the following is not true about what Hadoop is used for:

a. helps us deliver mashup services (GPS data, clickstream data)

b. helps give context for data (friends networks, social graphs)

c. creates a central data warehouse for our data for easy access

d. keeps apps running through edit logs, query logs

e. aggregates data-exhaust (messages,posts, blogs etc).

A

c- not one of the uses

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Describe hadoops server roles

A

hadoop has

  1. distributed data processing = map reduce

master= job tracker
slave= task tracker

  1. distributed data storage = hdfs

master= name node
slave= data node

masters - manage and track all interactions
slaves - actually do the job

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

which of the following is not an assumption / goal of HDFS?

  1. assumes there will always be a second copy and thus minimizes replications to avoid redundancy
  2. hard failure - always possible
  3. we are always gonna be dealing with large data sets and hdfs goal is to handle this
  4. hdfs wants to provide streaming data access which is best for batch processing
  5. follows a simple coherency model -write once- read many ( best for transaction data)
  6. portability across different hardware and software platforms.
A
  1. F - this is not an assumption or a goal of hdfs
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

________ refers to the overall perf of the system

a. optimization
b. through-put access

A

b

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

A distributed file system that provides high throughput access

a. hadoop
b. hdfs

A

b - hdfs

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

what is not true about hdfs?

a. it breaks data into small blocks ( 128 MB = default)

b. blocks are stored on the name node

c. blocks are replicated to other nodes to accomplish fault tolerance

d. Data node keeps track of where all the blocks are kept through metadata and indexing each block

A

b- false bc blocks are stored on data node

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

describe rack awareness script

A

rack awareness script is responsible for keeping track of which racks have which nodes and are they working

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

_______ responsible for holding status of the data nodes inside a rack. points something out if there are faulty datanodes and helps racks talk to each other.

A

switch

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Collection / stack of diff nodes is called a:

a. node shelf
b rack
c. holding block

A
18
Q

Name the HDFS daemons

A
  • name node
  • data node
  • secondary name node
19
Q

which is false about the name node:

a. this is what the clients talk to when they want to locate a file or add/copy/move/delete something

b. A successful request means name node returns list of relevant hadoop clusters where the data live

c. it keeps track of the metadata which is all files and blocks in the system

d. it stores all the high-importance files as those require extra storage

A

b. F - list is of the data nodes where the data lives

d. F - name node doesn’t do any storing

20
Q

T/F

  1. job tracker sends heartbeats to namenode to let it know job is going well
  2. every 5th heartbeat is a block report
  3. name node builds meta data from block reports
  4. if name node is down, hdfs can still run but only for short while.
  5. heartbeat say if DN is alive and block report says what blocks the DN has
A
  1. F - dn sends heartbeat to nn
  2. F- 10th heartbeat
  3. T
  4. F - no nn = no hdfs
  5. T
21
Q

what is not true about fsimage?

a. fsimage is a snapshot of the system

b. fsimage snapshot contains: DN, NN, block reports, rack awareness scripts

c. refreshes every 24 hours

d. snapshot is taken after NN has started

A

d- F

Fsimage is a snapshot taken right when NN starts not after

22
Q

T/F

  1. fsimage happens when NN starts and edit logs are for after NN has started
  2. Edit logs keep track of changes made to file
A
  1. T

2.T

23
Q

t/f

  1. DN processes a job in the Hadoop file system
  2. a functional file system can have 0 to many DN’s.
  3. data across DN’s is replicated parallely, meaning at the same time. this is called a data pipeline.
A
  1. F- DN stores data - duhh
  2. F - to be functional, you need many DN’s
  3. F- its replicated sequentially , in a pipeline manner
24
Q

Which is not true about the data node?

a. once location is provided by the NN, clients can talk to DN directly

b. DN connects to NN at startup but can receive requests directly from the client.

c. DN undergoes data replication to improve fault tolerance.

A
  1. T
  2. F- only NN can give requests to DN
  3. T
25
Q

Data replication is necessary for all the following except:

a. fault tolerance
b. hardware/software failure
c. improve speed of a result once inquiry is made
d. Network partition

A

c

26
Q

T/F

  1. replication factor is always preset by the platform provider
  2. data is replicated through a pipeline
  3. ideal replications is 3 (default) where 2 copies stay with one rack and one is put on a different rack. This helps to protect the data.
A
  1. F - can be decided by user
  2. t
  3. t
27
Q

explain process of replication pipeline

A
  1. client wants to write file.txt and makes request to name node
  2. File is broken into blocks
  3. Name node looks at rack script
  4. Tells client write to x,y,z dn
  5. Copied to dn x in rack 1
  6. This is copied to dn y in rack 2
  7. This is copied to dn z also in rack 2
28
Q

whats not true about rack awarness?

a. helps you to not loose data if a whole rack fails
b. assumption that in rack is higher bandwidth with lower latency
c. fails to keep bulky flows in rack

A

c - does keep bulky flows in rack

29
Q

whats not true about rack awarness?

a. helps you to not loose data if a whole rack fails
b. assumption that in rack is higher bandwidth with lower latency
c. fails to keep bulky flows in rack

A

c - does keep bulky flows in rack

30
Q

which is not true about DN failure?

a. dn failure happens when there is no heartbeat going to nn

b. when this happens NN finds the blocks it contains through the client, makes copies using meta data and unregisters the dead/failed node

c. can occur due to data corruption

d. to maintain data integrity, you should apply checksum checking

A

b. f- find the blocks it contains through rack script

31
Q

Re-replication. Which is not a step?

  1. NN consults Metadata and finds affected data
  2. NN consults Rack A script
  3. NN tells job tracker to replicate the needed DN
A

3- F - DN can be told to replicate directly

32
Q

which is false about the Secondary Name node?

a. it can replace the NN when it fails

b. main job is to perform periodic checkpoints

c. it can download NN fsimages and edit logs, join them and re-upload them to NN

A

a= false

33
Q

Which is false about the Name node

a. NN failure shouldn’t affect hdfs as long the system has enough space

b. If NameNode can be restored, secondary can reestablish the most current metadata snapshot

c. in the instance that a machine is lost, try your best to retrieve the old NN

d. to make cluster believe its the original NN - use Domain Name System

A
  1. F - NO NN = NO HDFS
  2. T
  3. F - just make a new one
  4. T
34
Q

T/F

  1. the 2nd part of Hadoop is MapReduce: this is responsible for data transformation
  2. data is split into chunks and processed one at a time
  3. the output of a map task in MR = input of reduce task
  4. MR is not responsible for
    a. scheduling tasks
    b. monitoring tasks
    c. redoing failed tasks
    d. sending reports for failed tasks to hdfs
A
  1. F - responsible for Data processing
  2. F - processed in parallel
  3. T
  4. d - DOESNT DO THAT
35
Q

Which is false about MR architecture

a. input is where file is broken into chunks (depends on user or default) called new-splits

b. in shuffle/sort, data records that are similar are sent
to the reducer while the others are discarded

c. in map task, records are outputted in the form of key-value pairs based on the task given (COUNT, SUM, AVG)

d. reducers job is to aggregate - means 3 similar records are counted as a 3 instead of separate 1’s

e. this process of MR is acyclic

A

a. F - chunks are called INPUT splits

b - F

in shuffle/sort, similar data is contained together
ALL data gets sent to Reducer

36
Q

what are the 4 phases of the MR process?

A

INPUT

  1. splitting
  2. mapping
  3. shuffling
  4. reducing

OUTPUT

37
Q

MR request is sent to which master node?

A

job tracker

38
Q

Name the 2 MR daemons

A

job tracker and task tracker

39
Q

Which is not true about the job tracker?

a. accepts MR jobs from client applications

b. talks to DN to find the location of the data it needs to work on

c. locates an available task tracker through slotting system and gives it the job

d.if task tracker is unavailable- JT will do the job itself

A

b-F -JT talks to NN about the data location

d. F- JT will not do the job itself

40
Q

Which is not true about the task tracker?

a. configured with a set of slots = how many tasks it can take

b. accepts map and reduces operations from JT. shuffle is not in its scope.

c. TT updates JT about a job’s status

d. TT doesn’t need to send heartbeats to JT since its already sending job status updates.

A

b- f

TT accepts all 3

d. f

TT does send heartbeats - says 1. if its available , 2. how many available slots it has

41
Q

t/f

  1. All master nodes and slave nodes contain both MR and HDFS components
  2. each master node has 2 components: resource manager (MR) and HDFS
  3. DN stores metadata
A
  1. t

2t
3. F - NN = metadata, DN = actual file data

Big Data (2 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions