Data Analytics

Question 1

How many hours is data available in the moving time window that Kinesis Stream uses?

Answer 1

24 hours (can be increased to 7 days for additional cost)

Question 2

How many MB does a single Shard in Kinesis allow for ingestion and consumption?

Answer 2

1 MB for Ingestion

2 MB for Consumption

Question 3

How many Shards does a Kinesis Stream have when newly created?

Answer 3

1

Question 4

What’s the size of a single Kinesis Data Record?

Answer 4

1 MB

Question 5

How quickly is data delivered using Kinesis Firehose?

Answer 5

Near-Real-Time, anything between 1-60 seconds (depends on the amount being ingested, i.e. how quickly the 1 MB buffer it uses is filled up).

Question 6

What are the 11 valid destinations for Kinesis Firehose?

Answer 6

• Amazon OpenSearch/Elasticsearch Service
• Amazon Redshift
• Amazon S3
• Http endpoints
• Datadog
• Dynatrace
• LogicMonitor
• MongoDB
• New Relic
• Splunk
• Sumo Logic

Question 7

How quickly is data delivered through Kinesis Streams?

Answer 7

In Real-Time (~ 200 ms)

Not to be confused with Kinesis Firehose, that delivers Near-Real-Time only!

Question 8

What’s the right product to use when (potentially complex) real-time SQL processing is required?

Answer 8

Kinesis Data Analytics

Question 9

What are the six 3rd party big data products does Amazon EMR provides as a managed service?

Answer 9

• Spark
• Hadoop (incl. Pig)
• HBase
• Hive
• Hudi
• Presto

Question 10

Is Amazon EMR a Multi-AZ or Single-AZ product?

Answer 10

Single-AZ

Question 11

What compute products can be used with Amazon EMR (i.e. which compute products are used to run EMR)?

Answer 11

EC2 & EKS

Question 12

What’s the master node used for with Amazon EMR?

Answer 12

• manages the cluster and its health
• distributes workloads
• acts as the NAME node within MapReduce
• allows SSH access to the cluster
• if it’s the only node in the cluster: runs MapReduce workload

Question 13

What are core nodes used for with Amazon EMR?

Answer 13

• provide the HDFS (Hadoop File System)
• run task trackers
• can run MapReduce workload

Note: losing a core node means losing HDFS and track of tasks => should not be run on Spot instances!
Note #2: Multi-node clusters have at least one core node.

Question 14

What are task nodes used for with Amazon EMR?

Answer 14

• run MapReduce workload

Note: ideal to be run on Spot instances

Question 15

What’s EMRFS?

Answer 15

S3-based file system for EMR. Can be used to store results of EMR workloads to ensure resilience with EMR.

Question 16

What’s the right product to use when you want to directly query S3 data via Redshift?

Answer 16

Redshift Spectrum

Question 17

Is Amazon Redshift a Multi-AZ or Single-AZ product?

Answer 17

Single-AZ

Question 18

What’s the role of the Leader Node in Amazon Redshift?

Answer 18

Receive query input and distribute it to Compute nodes for execution

Question 19

If you want to customize the network options for Amazon Redshift, what do you need to enable?

Answer 19

Enhanced VPC Routing

Question 20

At which intervals are automatic snapshots taken with Amazon Redshift?

Answer 20

Every ~8 hours or ~5 GB

Question 21

What are valid data sources for Amazon Redshift (name 7)?

Answer 21

Amazon S3
Amazon RDS
Amazon DynamoDB
Amazon EMR
AWS Glue
AWS Data Pipeline 
SSH-enabled host on Amazon EC2 or on-premises

Question 22

What are the available retention periods available for automatic snapshots taken with Amazon Redshift?

Answer 22

Anything between 1 day (default) up to 35 days.

Question 23

What are valid data sources for AWS Batch?

Answer 23

• AWS Step Functions
• AWS Lambda
• Amazon EventBridge
• Amazon S3

Question 24

What’s the right product to use for long-running (> 15 minutes) compute tasks?

Answer 24

NOT AWS Lambda!

Use AWS Batch, EC2, ECS instead for example

Question 25

Is AWS Batch serverless?

Answer 25

No

Question 26

How many records per second does a single Shard in Kinesis allow for ingestion and consumption?

Answer 26

1000 records / second

Question 27

What is default limit for the number of Shards in Kinesis?

Answer 27

500, but can be increased unlimited

Question 28

What does a Kinesis Shard consist of?

Answer 28

Partition Key, Sequence Number, Data

Question 29

What are the two main file systems used with Amazon EMR and what are their key differences?

Answer 29

HDFS and EMRFS

HDFS is fast, but ephemeral.
EMRFS is slower, but persistent as backed by S3.

Question 30

What’s the Amazon Kinesis Client Library (KCL) and when would you use it?

Answer 30

Library for reading and processing data from an Amazon Kinesis data stream. Removes some of the heavy-lifting when working with stream data, therefore more efficient than using the Kinesis API directly.

Question 31

What are the differences between AWS Glue and AWS Data Pipeline in regards to the compute infrastructure (and control of that infrastructure)?

Answer 31

• AWS Glue is serverless (but uses Apache Spark behind the scenes). No direct control on the compute resources.
• AWS Data Pipeline spins up EMR clusters and EC2 instances, which can be accessed directly.

Question 32

What are the differences between AWS Glue and AWS Data Pipeline in regards to the engines they use?

Answer 32

• AWS Glue uses a serverless Apache Spark engine and generates Scala or Python code
• AWS Data Pipeline uses Amazon EMR and through that is flexible on the engine (Spark, Hive, Hudi, Pig, etc.)

Question 33

What AWS service makes use of Apache Flink capabilities?

Answer 33

Kinesis Data Analytics