Week 11 (IO patterns)

Question 1

why would we not want to load or store data the way hadoop does out of the box

Answer 1

inject data from original source w/o storing in hdfs

feeding MR output to next process

Question 2

2 general ways to modify the way data is loaded on disk

Answer 2

input format: configure how contiguous chunks of input are generated from blocks in hdfs

record reader: configure how records appear in the map phase

Question 3

2 general ways to modify the way data is stored on disk

Answer 3

Question 4

what are the roles fo input format in hadoop

Answer 4

make sure data is there

split input blocks and files into logical chunks to be assigned to a map task

create record reader to be used to create key,val pairs from raw input split

Question 5

what type of view does inputsplit represent of the split

Answer 5

byte-oriented

Question 6

what is partition pruning

Answer 6

configure if files are loaded into MR based on name of file

Question 7

what is the goal of a reccomendation sys

Answer 7

predict the rating or preference that a user would give to an item

Question 8

what is collabaritive filtering

Answer 8

the process of identifying similar users and reccomending what similar users like

Question 9

in collab filtering, when are users similar

Answer 9

if their vectors are close according to some distance measure (jaccard or cosine distance)

Question 10

big n of collab filtering and then what it eventually ends up being

m = num of customers

n = num of product/catalog items

Answer 10

O(MN)

ends up being O(M+N)

Question 11

what does item to item collab filtering do

Answer 11

matches each of the users purchased items to similar items

combines those into reccomendation list