Basics

Question 1

Hadoop uses _____ as its basic data unit

Answer 1

Key value pairs

Question 2

Difference between SQL and Mapreduce

Answer 2

SQL = Declaritive

Mapreduce Specify the steps to solve the problem

Question 3

Hadoop is best used as a write _____ read _____ type of data store

Answer 3

Once

Many

Question 4

Under the MapReduce model, the data processing primitives are called

Answer 4

mappers and reducers

Question 5

A _____ is a pair of consecutive words

Answer 5

Bigram

Question 6

MapReduce uses ____ and _________ as its main data primitives

Answer 6

Lists

Key Value Pairs

Question 7

In the MapReduce framework you write applications by specifying the ______ and the _______

Answer 7

mapper

reducer

Question 8

In simple terms the mapper is meant to __________ the input into something that the reducer can ___________ over

Answer 8

filter and transform

aggregate over

Question 9

The input format for processing multiple files is usually

Answer 9

list()

Question 10

The input format for processing one large file, such as a log file is

Answer 10

list()

Question 11

Authentication can be based on 1. ______________ or if security is turned on 2._______________

Answer 11

1. user.name query parameter (as part of the HTTP query string)
2. then it relies on Kerberos.

Question 12

Security features of Hadoop consist of

Answer 12

authentication,
service level authorisation,
authentication for Web consoles and data confidentiality.

Question 13

The simplest way to do authentication is

Answer 13

using kinit command of Kerberos.

Question 14

What is Yarn

Answer 14

Yet Another Resource Negotiator - A cluster management technology.
A large-scale, distributed operating system for big data applications.

Question 15

Your MapReduce programs then process data, but they usually don’t read any HDFS files directly. Instead they …

Answer 15

Instead they rely on the MapReduce framework to read and parse the HDFS files into individual records (key/ value pairs), which are the unit of data MapReduce programs do work on.

Question 16

After distributing the data to different nodes, the only time nodes communicate with each other is at the

Answer 16

“shuffle” step

Question 17

The ________ servers as the base class for mappers and reducers

Answer 17

MapReduceBase class

Question 18

To serve as a mapper, a class should

Answer 18

Inherit the MapReduceBase class and implement the Mapper interface

Question 19

The single method on the mapper interface is

Answer 19

void map(K1 key,
         V1 value,
         OutputCollector output,
         Reporter reporter
        ) throws IOException
// processes an individual key value pair

Question 20

What does the output collector do?

Answer 20

Receives the output of the mapping process

Question 21

What does the reporter do

Answer 21

provides the option to record extra information about the mapper as the task progresses.

Question 22

What are some useful mapper implementations predefined by Hadoop

Answer 22

IdentityMapper
InverseMapper
RegexMapper
TokenCountMapper

Question 23

IdentityMapper

Answer 23

Implements Mapper and maps inputs directly to outputs

Question 24

InverseMapper

Answer 24

Implements Mapper and reverses the key/value pair

Question 25

RegexMapper

Answer 25

Implements Mapper and generates a (match, 1) pair for every regular expression match

Question 26

TokenCountMapper

Answer 26

Implements Mapper and generates a (token, 1) pair when the input value is tokenized

Question 27

The single method on the reducer interface is

Answer 27

void reduce(K2 key,
                    Iterator values,
                    OutputCollector output,
                    Reporter reporter
                   ) throws IOException

Question 28

With multiple reducers, we need some way to determine the appropriate one to send a (key/value) pair outputted by a mapper. The default behavior is to

Answer 28

hash the key to determine the reducer. Hadoop enforces this strategy by use of the HashPartitioner class.

Question 29

One of the fundamental principles of MapReduce’s processing power is the splitting of the input data into chunks. You can process these chunks in parallel using multiple machines. In Hadoop terminology these chunks are called

Answer 29

input splits

Question 30

The size of each (input) split should be small enough for

Answer 30

a more granular parallelization

If all the input data is in one split, then there is no parallelization.

Question 31

The size of each split should be small enough for a more granular parallelization. (If all the input data is in one split, then there is no parallelization.) On the other hand, each split shouldn’t be so small that

Answer 31

the overhead of starting and stopping the processing of a split becomes a large fraction of execution time.

Question 32

The principle of dividing input data (which often can be one single massive file) into splits for parallel processing explains some of the design decisions behind Hadoop’s generic FileSystem as well as HDFS in particular. For example, Hadoop’s FileSystem provides the class FSDataInputStream for file reading rather than using Java’s java. io.DataInputStream. FSDataInputStream extends DataInputStream with …

Answer 32

random read access, a feature that MapReduce requires because a machine may be assigned to process a split that sits right in the middle of an input file. Without random access, it would be extremely inefficient to have to read the file from the beginning until you reach the location of the split.

Question 33

What is the difference between HDFS blocks and input splits?

Answer 33

Input splits are the logical division of records whereas HDFS blocks are a physical division of the input data.

Question 34

If Hadoop works on key value pairs. How does it consider the lines of an input file by default?

Answer 34

Each line of the input file is a record. The key value pair consists of the byte offset (key) and the line (value) respectively.

Question 35

The way an input file is split up and read by Hadoop is defined by

Answer 35

one of the implementations of the InputFormat interface.

Question 36

__________ is the default input format implementation

Answer 36

TextInputFormat

Question 37

TextInputFormat is the default Input- Format implementation It’s useful for

Answer 37

input data that has no definite key value, when you want to get the content one line at a time.

Question 38

KeyValueTextInputFormat

Answer 38

Each line in the text files is a record. The first separator character divides each line. Everything before the separator is the key, and everything after is the value. The separator is set by the key.value.separator.in.input. line property, and the default is the tab (\t) character.

Question 39

SequenceFileInputFormat

Answer 39

An InputFormat for reading in sequence files. Key and Value are user defined

Question 40

What is a sequence file?

Answer 40

a Hadoop- specific compressed binary file format

Question 41

What are sequence files optimised for?

Answer 41

passing data between the output of one MapReduce job to the input of some other MapReduce job.

Question 42

NLineInputFormat

Answer 42

Same as TextInputFormat, but each split is guaranteed to have exactly N lines. The mapred.line.input.format. linespermap property, which defaults to one, sets N.

Question 43

The input data to your MapReduce job does not necessarily have to be some external data. In fact it’s often the case that

Answer 43

the input to one MapReduce job is the output of some other MapReduce job.

Question 44

In creating your own InputFormat class you should subclass the ___________ class, which takes care of ____________

Answer 44

FileInputFormat
File Splitting (dividing files into splits)

Question 45

All of the main InputFormat classes packaged with Hadoop subclass ___________, which implements the ___________ method but leaves _____________ abstract for the subclass to fill out.

Answer 45

FileInputFormat
getSplits()
getRecordReader()

Question 46

FileInputFormat’s getSplits() implementation tries to divide the input data into roughly the number of splits specified in __________ In practice, a split usually ends up being the size of a block, which defaults to ____ in HDFS.

Answer 46

numsplits

64 MB

Question 47

In using FileInputFormat you focus on _______________

Answer 47

customising RecordReader