Data Pipelines on Cloud (Computing)

Question 1

IaaS vs PaaS

Answer 1

Outsource virtual machines to cloud (AWS EC2)
Outsource data ecosystem to cloud (AWS EMR)

Question 2

Amazon Machine Image (AMI)

Answer 2

Help launch an instance by defining software, settings, etc (you did it, m4.large)

Question 3

Amazon EMR

Answer 3

Automates configuring of clusters of EC2 instances that run the big data framework of your choice (Hadoop, Spark, Hbase, Hive, Presto etc)

Question 4

Example of data workload with AWS EMR

Answer 4

Upload to AWS S3
EMR launches EC2 instances specified
EMR pulls data from S3 and begins execution
EMR transfers output data to S3

Question 5

Amazon EMR cluster

Answer 5

Collection of EC2 instances

Question 6

3 types of nodes in EMR cluster

Answer 6

Master - Manage cluster, schedule tasks, monitor health
Core Nodes - Run tasks, store data in HDFS
Task Nodes (optional) - only help process taks, not storing data

Question 7

On-demand vs Spot instance

Answer 7

Spot are unused EC2 instances available for less cost, but dropped when data-center needs them

Question 8

Which nodes are best to use spot instances for?

Answer 8

Task nodes, because maybe you lost compute capacity but no data is lost

Question 9

AWS EBS

Answer 9

Elastic Block Storage

Question 10

HDFS vs EMRFS

Answer 10

Both options for DFS within EMR, but EMRFS is specific implementation of hadoop file system atop Amazon S3, and you don’t need to use EBS in this case

Question 11

A cluster cannot but stopped, only:

Answer 11

terminated which is why you should store output data you want to keep, anything stored on cluster will get deleted!

Question 12

Step

Answer 12

User-defined unit of processing (algorithm manipulating data)

Question 13

Cluster lifecycle (5 steps)

Answer 13

starting (launch EC2s)
bootstrapping (install Hadoop, Spark, etc)
running (Step runs)
waiting (time after step runs)
terminating (after manual shutdown)