Amazon Kinesis Data Firehose | Adding Data to Delivery Streams Flashcards
Can I keep a copy of all the raw data in my S3 bucket?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Yes, Firehose can back up all un-transformed records to your S3 bucket concurrently while delivering transformed records to destination. Source record backup can be enabled when you create or update your delivery stream.
How do I add data to my Amazon Kinesis Data Firehose delivery stream?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
You can add data to an Amazon Kinesis Data Firehose delivery stream through Amazon Kinesis Agent or Firehose’s PutRecord and PutRecordBatch operations. Kinesis Data Firehose is also integrated with other AWS data sources such as Kinesis Data Streams, AWS IoT, Amazon CloudWatch Logs, and Amazon CloudWatch Events.
What is Amazon Kinesis Agent?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Agent is a pre-built Java application that offers an easy way to collect and send data to your delivery stream. You can install the agent on Linux-based server environments such as web servers, log servers, and database servers. The agent monitors certain files and continuously sends data to your delivery stream. For more information, see Writing with Agents.
What platforms does Amazon Kinesis Agent support?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Agent currently supports Amazon Linux and Red Hat Enterprise Linux.
Where do I get Amazon Kinesis Agent?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
You can download and install Amazon Kinesis Agent using the following command and link:
On Amazon Linux: sudo yum install –y aws-kinesis-agent
On Red Hat Enterprise Linux: sudo yum install –y https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm
From GitHub: awlabs/amazon-kinesis-agent
How do I use Amazon Kinesis Agent?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
After installing Amazon Kinesis Agent on your servers, you can configure it to monitor certain files on the disk and then continuously send new data to your delivery stream. For more information, see Writing with Agents.
What is the difference between PutRecord and PutRecordBatch operations?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
PutRecord operation allows a single data record within an API call and PutRecordBatch operation allows multiple data records within an API call. For more information, see PutRecord and PutRecordBatch.
What programming languages or platforms can I use to access Amazon Kinesis Data Firehose API?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Data Firehose API is available in Amazon Web Services SDKs. For a list of programming languages or platforms for Amazon Web Services SDKs, see Tools for Amazon Web Services.
How do I add data to my Firehose delivery stream from my Kinesis stream?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
When you create or update your delivery stream through AWS console or Firehose APIs, you can configure a Kinesis stream as the source of your delivery stream. Once configured, Firehose will automatically read data from your Kinesis stream and load the data to specified destinations.
How often does Kinesis Data Firehose read data from my Kinesis stream?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Kinesis Data Firehose calls Kinesis Data Streams GetRecords() once every second for each Kinesis shard.
From where does Kinesis Data Firehose read data when my Kinesis stream is configured as the source of my delivery stream?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Kinesis Data Firehose starts reading data from the LATEST position of your Kinesis data stream when it’s configured as the source of a delivery stream. For more information about Kinesis data stream position, see GetShardIterator in the Kinesis Data Streams Service API Reference.
Can I configure my Kinesis data stream to be the source of multiple Firehose delivery streams?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Yes, you can. However, note that the GetRecords() call from Kinesis Data Firehose is counted against the overall throttling limit of your Kinesis shard so that you need to plan your delivery stream along with your other Kinesis applications to make sure you won’t get throttled. For more information, see Kinesis Data Streams Limits in the Kinesis Data Streams developer guide.
Can I still add data to delivery stream through Kinesis Agent or Firehose’s PutRecord and PutRecordBatch operations when my Kinesis data stream is configured as source?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
No, you cannot. When a Kinesis data stream is configured as the source of a Firehose delivery stream, Firehose’s PutRecord and PutRecordBatch operations will be disabled. You should add data to your Kinesis data stream through the Kinesis Data Streams PutRecord and PutRecords operations instead.
How do I add data to my delivery stream from AWS IoT?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
You add data to your delivery stream from AWS IoT by creating an AWS IoT action that sends events to your delivery stream. For more information. See Writing to Amazon Kinesis Data Firehose Using AWS IoT in the Kinesis Data Firehose developer guide.
How do I add data to my delivery stream from CloudWatch Logs?
Adding Data to Delivery Streams
Amazon Kinesis Data Firehose | Analytics
You add data to your Firehose delivery stream from CloudWatch Logs by creating a CloudWatch Logs subscription filter that sends events to your delivery stream. For more information, see Using CloudWatch Logs Subscription Filters in Amazon CloudWatch user guide.
