Section 12 - AWS Kinesis

Question 1

What is AWS Kinesis?

Answer 1

Kinesis is originally a Greek word, meaning the movement or motion. Amazon Kinesis deals with data that is in motion, or streaming data.

Streaming Data?

Data generated continuously by the thousands of data sources, which typically send in the data records similtaneously and in small size(kilobytes)

• Financial Transactions
• Stock prices
• Game data (as the gamer plays)
• Social media feeds
• Location tracking data (Uber)
• IoT sensors
• Clickstream
• Log files

Question 2

What are the 4 core service of AWS Kinesis?

Answer 2

• Kinesis Video Streams
  
  • ​Amazon Kinesis Video Streams makes it easy to securely stream video from connected devices to AWS for analytics, machine learning (ML), and other processing.
• Kinesis Data Streams
  
  • Amazon Kinesis Data Streams is a scalable and durable real-time data streaming service that can continuously capture gigabytes of data per second from hundreds of thousands of sources.
• Kinesis Data Firehose
  
  • Capture, transform, load data streams into AWS data stores to enable near-real-time analytics with BI tools
• Kinesis Data Analytics
  
  • Analyze, query and transform streamed data in real-time using standard SQL. Store the results in an AWS data store.

Question 3

Kinesis Data Streams?

Answer 3

• Producers
  
  • Devices which produce data for streaming
  • e.g. IoT device
• Kinesis Streams
  
  • Data is stored in Shard
  • Data is stored for 24 hrs, with a max of 7 days retention
• ​Consumers
  
  • ​Consume stored data and apply business logic
  • e.g EC2 instance, Lambda functions …

Question 4

AWS Kinesis Shards?

Answer 4

Kinesis streams are made up of shards, each shard is a sequence of one or more data records and provides a fixed unit of capacity.

• Five reads per second
• The max total read rate is 2MB per second
• 1,000 write per second
• The max total write rate is 1 MB per second

NB: The data capacity of the stream is determined by the number of shards. if the data rate increases, you can increase capacity on your stream by increasing the number of shards.

Question 5

AWS Kinesis Video Streams?

Answer 5

Securely stream video from connected devices to AWS

• Videos can be used for analytics and machine learning.

Question 6

AWS Kinesis Data Firehose?

Answer 6

• Producers
  
  • Devices produce data
  • e.g IoT
• Kinesis Firehose
• No shards
• No data retention

Question 7

AWS Kinesis Data Analytics?

Answer 7

• Producers
  
  • Devices produce data
  • e.g. IoT
• Data is pushed to Firehose
• You can run SQL query against incoming data and store the results.
• Real-time analytics

Question 8

AWS Kinesis Exam Tips?

Answer 8

Question 9

AWS Kinesis Shards and Consumers?

Answer 9

• The kinesis client library running on your consumers create a record processor for each shard that is being consumed by your instance
• If you increase the number of shards, the KCL will add more record processors on your consumers
• CPU utilisation is what should drive the quantity of consumer instances you have, NOT the number of shards in your Kinesis stream.
• Use an Auto Scalling group, and base the scaling decisions on CPU load on your consumers.