11 - Scaling Up: Big Data

Question 1

Parallel processing

Answer 1

• parallelism: work on separate pieces at the same time
• – challenges = coordination, mutability, blocking
• distributed computing: same but != CPU, machine
• – challenges = sending instruction, fault tolerance, data storage and retrieving

Question 2

Programming: imperative, declarative

Answer 2

• imperative = direct orders, manual scheduling and data ctl, optimize perf possible (C, C++, Java, Matlab)
• declarative = state goals, data automatically managed and stored, automatic scheduling but not necessarily efficient (SQL, R, Python can be)

Question 3

Queue computing

Answer 3

• master (or name) node(s) = main address
• worker node = where computation is performed
• scheduler = decides which job, which resources

Question 4

Databases

Answer 4

• SQL = structured query language

- NOSQL (beyond)

Question 5

Big data

Answer 5

• PageRank: sort website by qlty (Google)
• MapReduce
• – map = apply fct° to every e/ of list
• – reduce = aggregate e/ and summarize

Question 6

Further big data

Answer 6

• distributed computing: Hadoop, Spark, Dask, DAGs

- cloud computing: Spark