Infrastructure & Architecture

Question 1

scale-up (vertical scaling)

Answer 1

upgrading existing machine (ram, processor, storage)

Question 2

scale-out (horizontal scaling)

Answer 2

adding more machines to network, unlimited scaling

Question 3

Symmetric Multi-processing (SMP)

Answer 3

traditional notebooks/desktops, multiple processors share same memory and storage (bus bottleneck)

Question 4

Massively Parallel Processing (MPP)

Answer 4

Each processor has its own dedicated memory and storage (vendor lock-in)

Question 5

Cluster architecture

Answer 5

many computers connected to work as a single system. (Like MPP, but this implies multiple processors per node).

Question 6

how are nodes connected in cluster

Answer 6

usually gigabit ethernet, 8-64 per rack, racks connected by another level of network or switch

Question 7

Only pro of MPP over cluster

Answer 7

faster message passing between nodes. Ideal for single, vertical solutions like data warehousing \

Question 8

commodity hardware

Answer 8

standardized, market priced hardware cheaper than proprietary solutions

Question 9

distributed computing

Answer 9

tasks split into smaller units processed simultaneously across machines

Question 10

challenges of distributed computing

Answer 10

splitting or assigning tasks/parallelization, resource allocation, fault-tolerance, aggregating results

Question 11

solutions to challenges of distributed computing

Answer 11

use big data framework to hide complexity of distributed computing from developers

Question 12

Grid computing

Answer 12

collection of computer resources from multiple locations, each node perform different task, commonly used for variety of purposes

Question 13

what is HPC

Answer 13

high-performance computing, use of GPU increasing

Question 14

NIST reference architecture definition

Answer 14

National Institute of Standards and Technology (NIST) recommendation for big data system architecture

Question 15

NIST Reference Architecture components

Answer 15

1. Big Data Framework Provider (process, storage, network)
2. Data Provider
3. Application provider (ex: bi analytics, ML)
4. Data Consumer
5. System Orchestrator (integrate components, meet goals, allocate resources)

Question 16

Big Data Application Provider definition

Answer 16

Component for which programs must be written to process the data (from collection to visualization and user access)

Question 17

4 jobs of data provider component

Answer 17

besides just collecting data:
scrubbing sensitive information
create metadata describing data source, access rights, usage policies
enforce access/authorizations

Question 18

4 extra components of big data architecture that professor describes, not explicitly in NIST

Answer 18

Analytical data store
Analysis and reporting

Real-time message ingestion (and buffer)
stream processing

Question 19

batch vs stream

Answer 19

analytical algorithms:
launched over large amounts of stored data
vs
continuously running over potentially infinite data as soon as its collected

Question 20

Lamda Architecture

Answer 20

Two pipelines, cold path for batch, hot path for real-time/stream

Question 21

Kappa Architecture

Answer 21

One hot path for real-time/stream