AWS Data Pipeline

Question 1

Is Datapipeline a serverless product?

Answer 1

Yes. data pipe requires no infrastructire to be deployed.

Question 2

For Datapipeline, What are data nodes?

Answer 2

Data nodes are the names and locations and formats of your data, like S3, DdesynamoDB, Redshift. You can think of them as the reader module that reads in the data to the pipeline.

Question 3

For Data pipeline, what inputs can I use?

Answer 3

Redshift
RDS
DynamoDB
S3

Question 4

For Datapipeline, What are the schedules?

Answer 4

It is part of a pipeline definition and defines when the definition is to be executed. This can be bassed on pipeline activation or based time.

Question 5

For Datapipeline, What are the actions?

Answer 5

This would be is the data available

Question 6

When you run your Data pipeline is it realtime?

Answer 6

No, things are scheduled.

Question 7

For Datapipeline, What are the resources

Answer 7

These are EC2 instances and EMR clusters.

Question 8

For Datapipeline, What are the resources

Answer 8

These are EC2 instances and EMR clusters.

Question 9

Datapipeline, What are actions?

Answer 9

these are ways you can receive updated on the For Datapipeline, status.

Question 10

For Datapipeline, What are the actions?

Answer 10

these are ways you can receive updated on the For Datapipeline, status.

Question 11

For Datapipeline, can i process data in real time?

Answer 11

No, Datapipeline is about processing data but not in a real-time fashion.

Question 12

What cna I do with Datapipelne?

Answer 12

• Export data from DynamoDB to s3
• Import data from s3 to DynamoDB
• RDS MySQL to s3
• RDS incramental copy to s3
• RDS s3 to RDS Table
• RDS Table to Redshift
• RDS incremetal to Redshift
• Load data from Redshift to RDS

Question 13

I what to stream live data from RDS to Redshift, how cna I use Datapipeline?

Answer 13

You can not do it with Datapipeline, Datapipeline is only used on a schedule and not for streaming live.

Question 14

I need to batch shift and transform data from RDS to S3, how cna I do this?

Answer 14

You can use Datapipeline

Question 15

What is pre-condition?

Answer 15

You check for the existence of your data,

• S3 files or directories exist
• DynamoDB table exists
• RDS queries
• Redshift queries

Question 16

I wnat to get an SNS message when a Data pipeline activity fails, how cna I achieve this?

Answer 16

You cna set SNS to send messages from DP activities fail.

Question 17

What is a Datapipeline activity?

Answer 17

It is a something thet works on your data,

• EMR Job
• Pig
• Hive
• Hadoop
• Shell command
• SQL

Question 18

What is the schedule?

Answer 18

It is when the Datapipeline should run, it is part of the Datapipeline definition, and you cna define parameters like,
- Start time
- Interval
- End
Example: Start today at noon and run for 1hr.

Question 19

How do I set up a cluster for datapipeline to run?

Answer 19

You do not set up a cluster, Datapipeline is a serverless product.

Question 20

What is AWS datapipeline?

Answer 20

automate and schedule regular data movement and data processing activities in AWS

Question 21

Where can the data sources be located?

Answer 21

• on-prem

- AWS

Question 22

Can I set up a schedule for the datapipeline to run?

Answer 22

Yes as part of the datapipeline definition you can set up a schedule to run, this could be like every hour or a data time, but there are lots of options.

Question 23

When would I use datapipeline vs AWS Glue?

Answer 23

TBD