Amazon Kinesis Data Firehose | Data Delivery by Amazon Kinesis Data Firehose Flashcards
How do I add data to my Amazon Kinesis Data Firehose delivery stream from CloudWatch Events?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
You add data to your Firehose delivery stream from CloudWatch Events by creating a CloudWatch Events rule with your delivery stream as target. For more information, see Writing to Amazon Kinesis Data Firehose Using CloudWatch Events in the Kinesis Data Firehose developer guide.
How often does Amazon Kinesis Data Firehose deliver data to my Amazon S3 bucket?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
The frequency of data delivery to Amazon S3 is determined by the S3 buffer size and buffer interval value you configured for your delivery stream. Amazon Kinesis Data Firehose buffers incoming data before delivering it to Amazon S3. You can configure the values for S3 buffer size (1 MB to 128 MB) or buffer interval (60 to 900 seconds), and the condition satisfied first triggers data delivery to Amazon S3. Note that in circumstances where data delivery to the destination is falling behind data ingestion into the delivery stream, Amazon Kinesis Data Firehose raises the buffer size automatically to catch up and make sure that all data is delivered to the destination.
How often does Amazon Kinesis Data Firehose deliver data to my Amazon Redshift cluster?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
For Amazon Redshift destination, Amazon Kinesis Data Firehose delivers data to your Amazon S3 bucket first and then issues Redshift COPY command to load data from your S3 bucket to your Redshift cluster. The frequency of data COPY operations from Amazon S3 to Amazon Redshift is determined by how fast your Redshift cluster can finish the COPY command. If there is still data to copy, Firehose issues a new COPY command as soon as the previous COPY command is successfully finished by your Redshift cluster.
How often does Amazon Kinesis Data Firehose deliver data to my Amazon Elasticsearch domain?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
The frequency of data delivery to Amazon Elasticsearch Service is determined by the Elasticsearch buffer size and buffer interval values that you configured for your delivery stream. Firehose buffers incoming data before delivering it to Amazon Elasticsearch Service. You can configure the values for Elasticsearch buffer size (1 MB to 100 MB) or buffer interval (60 to 900 seconds), and the condition satisfied first triggers data delivery to Amazon Elasticsearch Service. Note that in circumstances where data delivery to the destination is falling behind data ingestion into the delivery stream, Amazon Kinesis Data Firehose raises the buffer size automatically to catch up and make sure that all data is delivered to the destination.
How is data organized in my Amazon S3 bucket?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Data Firehose adds a UTC time prefix in the format YYYY/MM/DD/HH before putting objects to Amazon S3. The prefix translates into an Amazon S3 folder structure, where each label separated by a forward slash (/) becomes a sub-folder. You can modify this folder structure by adding your own top-level folder with a forward slash (for example, myApp/YYYY/MM/DD/HH) or prepending text to the YYYY top-level folder name (for example, myApp YYYY/MM/DD/HH). This is accomplished by specifying an S3 Prefix when creating your delivery stream.
What is the naming pattern of the Amazon S3 objects delivered by Amazon Kinesis Data Firehose?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
The Amazon S3 object name follows the pattern DeliveryStreamName-DeliveryStreamVersion-YYYY-MM-DD-HH-MM-SS-RandomString, where DeliveryStreamVersion begins with 1 and increases by 1 for every configuration change of the delivery stream. You can change delivery stream configurations (for example, the name of the S3 bucket, buffering hints, compression, and encryption) with the Firehose Console or the UpdateDestination operation.
What is the manifests folder in my Amazon S3 bucket?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
For Amazon Redshift destination, Amazon Kinesis Data Firehose generates manifest files to load Amazon S3 objects to Redshift cluster in batch. The manifests folder stores the manifest files generated by Firehose.
How do backed up Elasticsearch documents look like in my Amazon S3 bucket?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
If “all documents” mode is used, Amazon Kinesis Data Firehose concatenates multiple incoming records based on buffering configuration of your delivery stream, and then delivers them to your S3 bucket as an S3 object. Regardless of which backup mode is configured, the failed documents are delivered to your S3 bucket using a certain JSON format that provides additional information such as error code and time of delivery attempt. For more information, see Amazon S3 Backup for the Amazon ES Destination in the Amazon Kinesis Data Firehose developer guide.
Can a single delivery stream deliver data to multiple Amazon S3 buckets?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
A single delivery stream can only deliver data to one Amazon S3 bucket currently. If you want to have data delivered to multiple S3 buckets, you can create multiple delivery streams.
Can a single delivery stream deliver data to multiple Amazon Redshift clusters or tables?
Data Delivery by Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose | Analytics
A single delivery stream can only deliver data to one Amazon Redshift cluster and one table currently. If you want to have data delivered to multiple Redshift clusters or tables, you can create multiple delivery streams.
