Hadoop Tools

Question 1

Mapreduce 4 processes

Answer 1

Mapping
Shuffling
Sorting
Reducing

Question 2

What is sqoop?

Answer 2

Sql to hadOOP
Import / export data to and from sql into hadoop

Question 3

What is Flume?

Answer 3

Flume is used to collect and store unstructured or semistructured data into Hadoop.

This data could be videos, pictures, emails, or websites.

Keeps a balance between data read and write speeds

Question 4

Describe an Agent (Flume)

Answer 4

3 components:
Source - accepts incoming data and forwards to channel
Channel - Receives data and buffers them until the sink consumes them, avoiding data loss
Sink - Receives data from the channel and stores into HDFS

Question 5

What is HBase?

Answer 5

HBase is a column oriented, graph based database built for the Hadoop file system. It is designed to provide quick random access to huge amounts of data. Users can store data in HDFS directly or through HBase.

Question 6

Describe how HBase stores data differently from row storage

Answer 6

It uses columns instead of rows

Question 7

What are the 4 components of HBase?

Answer 7

Master Server, Region Server, Regions, & Zookeeper

Question 8

What is Hive?

Answer 8

It is a data warehouse framework for querying and analyzing data stored in HDFS.
An open-source tool for data analysis

Also created to address the stressful programming associated with MapReduce
Unlike PIG, Hive is for processing structured* data only
Can process Terabytes of data in seconds vs RDBMS

Question 9

In HBase, what is the Master Server?

Answer 9

It assigns regions to region servers with the help of Apache ZooKeeper. It handles load balancing across those servers. It is responsible for schema changes.

Question 10

In HBase, what is Zookeeper?

Answer 10

Receives keep-alive heartbeats from HRegion and HMaster
servers. Keeps track of meta data of server

Question 11

In HBase, describe the Region Server and Regions.

Answer 11

These region servers have regions that communicate with the client and handle data related operations, like reading and writing requests for all regions under it. The region server decides the size of the region following the region size thresholds

Question 12

If MapReduce works, why use other tools?

Answer 12

MapReduce requires knowledge of programming java and or python. Complex codes need to be written.

1 line of PIG Latin script approx. 20 lines of MapReduce Java code

Faster deployment, built in functions to achieve complex tasks.

Question 13

What is Apache PIG?

Answer 13

An open-source high-level data analyses tool built upon MapReduce
It represents large datasets as data flows

Like a pig that eats anything, PIG accepts any type of data - structured, unstructured, and semi-structured data.

Question 14

What are the components of Apache PIG? How does it handle scripts?

Answer 14

It is made up of two components – PIG Latin and PIG Runtime envnmt.
PIG Latin - scripting language (similar to SQL)

PIG Latin scripts are internally converted to Map and Reduce
tasks by PIG MapReduce Engine

Question 15

Describe the HBase architecture

Answer 15

Question 16

How do you connect, and login to sqoop?

Answer 16

– connect: Connection string to the RDBMS
–connect jdbc:mysql://<ip>/<database>
--username: database username
--password: database password
--username ADMIN --password P@\$\$w0rd</database></ip>

Question 17

How do you set the import table, and target directory in sqoop?

Answer 17

● –table: the RDBMS table to use
○ –table STUDENT_RECORDS
● –target-dir: where the data should be imported into in HDFS
○ –target-dir /DESTINATION_DIR

Question 18

How do you connect, login, select the table, and export a table from sqoop to sql?

Answer 18

● – connect: Connection string to the RDBMS
○ –connect jdbc:mysql://<ip>/<database>
● --username: database username
● --password: database password
○ --username ADMIN –password P@\$\$w0rd
● --table: the RDBMS table to export to ○ --table STUDENT_RECORDS
● --export-dir: where the data should be exported from in HDFS ○ --export-dir /SOURCE_DIR</database></ip>

Question 19

Sqoop - How to list databases? Tables?

Answer 19

● list-databases: list all databases in the RDBMS to which sqoop is connected
○ Sqoop list-databases –connect jdbc:mysql://localhost/ – username ADMIN password P@$$w0rd
● list-tables: list all tables in RDBMS database to which sqoop is connected
○ Sqoop list-tables –connect jdbc:mysql://localhost/DATABASE_NAME –username ADMIN password P@$$w0rd

Question 20

What is an agent’s source in flume?

Answer 20

Incoming stream of data

Question 21

What is an agent’s channel in flume?

Answer 21

The channel is a transient storage that buffers between source and sink.

Question 22

What is an agent’s sink in flume?

Answer 22

The sink is what receives data from the channel, and stores in HDFS (Think funnel)

Question 23

In flume, there are 3 types of data flow. Recite them, and explain what each means.

Answer 23

● Multi-hop Flow: event goes through two or more flume agents before reaching the its destination.
● Fan-out Flow: an event flows from one source to multiple channels.
● Fan-in Flow: an event is transferred from many sources to one channel.

Question 24

In flume, how do you specify each component in code?

Answer 24

agent.sources = specifies the source
agent.channels = specifies the source
agent.sinks = specifies the target

Question 25

Given the flume agents source, what command specifies the data file path?

Answer 25

agent.sources.flumesource.type= agent.sources.flumesource.spoolDir = path to data file agent.sources.flumesource.fileHeader = false

Question 26

In flume, how do you specify a channel’s type, and capacity?

Answer 26

agent.channels.memoryChannel.type = agent.channels.memoryChannel.capacity = agent.channels.memoryChannel.transactionCapacity =

Question 27

In flume, how do you link the hdfs path, as well as the file type?

Answer 27

agent.sinks.flumeHDFS.type = hdfs
agent.sinks.flumeHDFS.hdfs.path = output directory agent.sinks.flumeHDFS.hdfs.fileType = DataStream agent.sinks.flumeHDFS.hdfs.rollCount = 0

Question 28

Recite all the sqoop commands you know

Answer 28