Data Ingestion and Transformation

Question 1

What are the five steps to data discovery in a project?

Answer 1

1. Define Business Value
2. Identify the data consumers
3. Identify your data sources
4. Define your storage, catalog, and access needs
5. Define your processing needs.

Question 2

What is the best definition of data discovery?

Answer 2

The process of finding and understanding relevant data sources within an organization and the relationships between them

Question 3

What are the stages of a workflow in modern data architecture?

Answer 3

1. Ingest
2. Storage
3. Catalog
4. Process
5. Deliver
6. Security and Governance (covers all other five parts of workflow)

Question 4

A data engineer at a large e-commerce company has built various data processing pipelines on AWS that need to run on daily, weekly, and monthly schedules. They want to implement an orchestration layer to automate the scheduling and operation of these pipelines. Which tools would BEST fit this requirement?

Answer 4

AWS Step Functions with AWS Lambda and AWS Glue to schedule and automate data-driven workflows

Question 5

What are the key benefits to using a CI/CD pipeline?

Answer 5

1. Faster time-to-insight by automating data pipeline builds, tests, and deployments
2. Improved data quality and reliability by catching issues early in the development process
3. Reduced operational overhead by automatically managing infrastructure and deployments

Question 6

Define Infrastructure as Code

Answer 6

IaC refers to managing and provisioning infrastructure through machine-readable definition files instead of physical hardware configuration or interactive configuration tools.

Question 7

What sections exist in a CloudFormation template?

Answer 7

1. Format, Version, and Description
2. Parameters
3. Resources
4. Outputs

Question 8

What are the main benefits of using AWS Serverless Application Model?

Answer 8

1. Simplified development and deployment process
2. Local testing capabilities
3. Seamless AWS services and resources integration

Question 9

What are the three key components of AWS Site-to-Site VPN?

Answer 9

1. VPN gateway on the AWS side
2. Customer gateway on the on-premises side
3. An encrypted VPN connection

Question 10

What are spot instances?

Answer 10

Spot Instances are spare Amazon EC2 computing capacity offered at significantly discounted prices compared to On-Demand Instance pricing. The prices fluctuate based on supply and demand, but you can set a maximum price you’re willing to pay. These instances can be used for batch processing jobs or non-critical workloads that can tolerate interruptions such as data preprocessing, model training, or batch analytics jobs.

Question 11

What is the primary purpose of AWS PrivateLink?

Answer 11

To provide private connectivity between AWS services and applications without using the public internet

Question 12

What are the stages of the data engineering lifecycle?

Answer 12

1. Generation
2. Storage
3. Ingestion
4. Transformation
5. Serving

Question 13

What are the five Vs of data?

Answer 13

Variety
Volume
Velocity
Veracity / Validity
Value

Question 14

What AWS services support stateful data transfer?

Answer 14

Amazon ElastiCache
Amazon RDS

Question 15

What AWS services support stateless data transfer?

Answer 15

Lambda
API Gateway
Amazon S3

Question 16

What topics should you look for when troubleshooting and doing performance optimization?

Answer 16

1. Bottlenecks
2. High processing times, memory usage, or higher I/O operations
3. Algorithms, partition strategy, or parallel processing
4. Resource allocation
5. Caching

Question 17

What steps should you go through when troubleshooting problems in a pipeline?

Answer 17

1. Check logs
2. Verify data at different stages of process
3. Implement incremental processing
4. Add retries
5. Test

Question 18

What is needed to set up a JDBC connection (Java DB Connectivity)?

Answer 18

1. Prepare the JDBC driver
2. Configure security groups
3. Use a connection URL, Username, and PW

Question 19

What is needed to set up an ODBC connection (Open DB connectivity)?

Answer 19

1. Select an ODBC driver
2. Install an ODBC driver
3. Set up an ODBC data source name (DSN)
4. Specify a DSN

Question 20

What services exist to help cache data in your APIs?

Answer 20

1. ElastiCache
2. API Gateway
3. Amazon CloudWatch - monitor and track usage of API

Question 21

How can you scale an API?

Answer 21

1. Use auto-scaling in Lambda functions
2. Configure API Gateway to handle high concurrency and availability requirements.

Question 22

What are the two types of data architecture?

Answer 22

Operational architecture - the functional requirements
Technical architecture - how the data is ingested, stored, transformed, and served.

Question 23

What types of isses can cause a pipeline failure?

Answer 23

1. Data quality problem - ex wrong file format
2. Code errors
3. Endpoint errors - ex a service is temporarily offline
4. Dependency errors

Question 24

Which services can you use for data pipeline orchestration?

Answer 24

1. AWS Data Pipeline
2. AWS Glue
3. Step Functions
4. MWAA

Question 25

What are best practices for data pipelines?

Answer 25

1. Distributed processing (Spark) 2. Auto scaling 3. Data partitioning 4. fault-tolerant storage (S3 & EFS) 5. Backups (S3 or AWS Backup) 6. Monitoring (CloudWatch) 7. Validation and quality checks 8. Automated Testing 9. CI / CD

Question 26

A finance company has developed a machine learning (ML) model to enhance its investment strategy. The model uses various sources of data about stock, bond, and commodities markets. The model has been approved for production. A data engineer must ensure that the data being used to run ML decisions is accurate, complete, and trustworthy. The data engineer must automate the data preparation for the model's production deployment. Which solution will meet these requirements?

Answer 26

SageMaker ML Lineage Tracking creates and stores information about the steps of an ML workflow. SageMaker ML Lineage Tracking gives you the ability to establish model governance and audit standards. SageMaker ML Lineage Tracking helps to ensure that the data being used to run ML decisions is accurate, complete, and trustworthy.

Question 27

An ecommerce company runs several applications on AWS. The company wants to design a centralized streaming log ingestion solution. The solution needs to be able to convert the log files to Apache Parquet format. Then, the solution must store the log files in Amazon S3. The number of log files being created varies throughout the day. A data engineer must configure a solution that ensures the log files are delivered in near real time. Which solution will meet these requirements with the LEAST operational overhead?

Answer 27

Configure the applications to send the log files to Kinesis Data Firehose. Configure Firehose to invoke a Lamdba function that converts the log files to Parquet format. Configure Firehose to deliver the Parquet files to an output S3 bucket. You can use Kinesis Data Firehose to deliver log files to Amazon S3 with the least operational overhead. You can use a data-transformation Lambda function with Kinesis Data Firehose. This solution can convert log files to the correct format before the log files are delivered to Amazon S3.

Question 28

A company has deployed a data pipeline that uses AWS Glue to process records. The records include a JSON-formatted event and can sometimes include base64-encoded images. The AWS Glue job is configured with 10 data processing units (DPUs). However, the AWS Glue job regularly scales to several hundred DPUs and can take a long time to run. A data engineer must monitor the data pipeline to determine the appropriate DPU capacity. Which solution will meet these requirements?

Answer 28

Inspect the job monitoring section of the AWS Glue console. Review the results of the previous job runs. Visualize the profiled metrics to determine the appropriate number of DPUs. You can use the job run monitoring section of the AWS Glue console to determine the appropriate DPU capacity for this scenario. The job monitoring section of the AWS Glue console uses the results of previous job runs to determine the appropriate DPU capacity.

Question 29

An Amazon Kinesis application is trying to read data from a Kinesis data stream. However, the read data call is rejected. The following error message is displayed: ProvisionedThroughputExceededException. Which combination of steps will resolve the error? (Select TWO.)

Answer 29

Increase the number of shards within the stream to provide enough capacity for the read data calls. The ProvisionedThroughputExceededException error is caused by the capacity quotas of the data stream exceeding its provisioned amount. A sustained rise of the stream's output data rate can cause this issue. To resolve the issue, you can increase the number of shards within your stream to provide enough capacity for the read data calls to consistently succeed. 2. Make the application retry to read data from the stream. The ProvisionedThroughputExceededException is caused by the capacity quotas of the data stream exceeding its provisioned amount. A sustained rise of the stream's output data rate can cause this issue. A solution that retries the Kinesis application will eventually lead to completions of the requests.