MapReduce And ACID vs BASE

Question 1

MapReduce

Answer 1

The way that Google splits the job of finding data into tasks for separate machines

Like a distributed GROUP BY

Question 2

Phases

Answer 2

Map phase: Find and aggregate key values on each node. done on all nodes in parallel
Shuffle phase: group all like categories in each node
Reduce phase: totals the number of things in each category

Question 3

ACID properties of big data

Answer 3

Atomicity
Consistency
Isolation
Durability

Question 4

Atomicity

Answer 4

All or nothing. In a transaction all the operations must succeed or fail as a group
-> ensures data integrity

Achieved through COMMIT and ROLLBACK transactions

Question 5

Consistency

Answer 5

Data must be consistent before and after a transaction

Achieved through Forward recovery, backward recovery

Question 6

Isolation

Answer 6

Transactions never interfere with each other

Achieved through Locking

Question 7

Durability

Answer 7

Transactions are permanent even if they fail

Achieved through: Forward recovery, backward recovery

Question 8

Base properties (a great alternative only if query results can handle some inconsistencies)

Answer 8

Basic Availability
Soft state
Eventual Consistency

Question 9

Basic availability

Answer 9

A Big data alternative that makes it able to tolerate partial failure (failure of a node)

Question 10

Soft state

Answer 10

State of the system is in flux and may change over time
(Correctness of big data is not that important)

Question 11

Eventual consistency

Answer 11

May not be consistent in the short run but will eventually become consistent as more data is added

Question 12

Hadoop ecosystem

Answer 12

Tools that make Hadoop easy to use for people without Java programming skills.

Question 13

Elements of the Hadoop ecosystem

Answer 13

Hive, Sqoop, Pig, Flume, Hbase, Impala

Question 14

Hive

Answer 14

DW system that works with HDFS and it’s not relational.
HiveQL is a declarative (what) SQL like query language. Processes queries into MapReduce jobs.
Works best on large sets of data; doesn’t return small sets of data quickly.

Question 15

Pig

Answer 15

Pig Latin scripting language. Procedural (how).
Compiles pig Latin into MapReduce jobs.
Good for data transformation

Question 16

Flume

Answer 16

(Web click streams one)
Harvests large sets of data from server log files. Can be configured to import data on a regular schedule and can move data into HDFS.

Question 17

Sqoop

Answer 17

SQL to Hadoop
Converts data back and forth between a relational dbms and Hadoop

Question 18

HBase

Answer 18

A NoSQl database that works directly with HDFS
Does not rely on Map-Reduce
Suitable for fast processing of small data sets
Very good at quickly processing sparse data sets
Used for Facebook messaging system

Question 19

Impala

Answer 19

Supports SQL queries that pull data directly from HDFS
Works well for processing large datasets into small results set