Amazon Kinesis Data Firehose | Creating Delivery Streams Flashcards
What is a destination?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
A destination is the data store where your data will be delivered. Amazon Kinesis Data Firehose currently supports Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk as destinations.
How do I create a delivery stream?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
You can create an Amazon Kinesis Data Firehose delivery stream through the Firehose Console or the CreateDeliveryStream operation. For more information, see Creating a Delivery Stream.
What compression format can I use?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Data Firehose allows you to compress your data before delivering it to Amazon S3. The service currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Amazon Redshift.
How does compression work when I use the CloudWatch Logs subscription feature?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
You can use CloudWatch Logs subscription feature to stream data from CloudWatch Logs to Kinesis Data Firehose. All log events from CloudWatch Logs are already compressed in gzip format, so you should keep Firehose’s compression configuration as uncompressed to avoid double-compression. For more information about CloudWatch Logs subscription feature, see Subscription Filters with Amazon Kinesis Data Firehose in the Amazon CloudWatch Logs user guide.
What kind of encryption can I use?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Data Firehose allows you to encrypt your data after it’s delivered to your Amazon S3 bucket. While creating your delivery stream, you can choose to encrypt your data with an AWS Key Management Service (KMS) key that you own. For more information about KMS, see AWS Key Management Service.
What is data transformation with Lambda?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Firehose can invoke an AWS Lambda function to transform incoming data before delivering it to destinations. You can configure a new Lambda function using one of the Lambda blueprints we provide or choose an existing Lambda function.
What is source record backup?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
If you use data transformation with Lambda, you can enable source record backup, and Amazon Kinesis Data Firehose will deliver the un-transformed incoming data to a separate S3 bucket. You can specify an extra prefix to be added in front of the “YYYY/MM/DD/HH” UTC time prefix generated by Firehose.
What is error logging?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
If you enable data transformation with Lambda, Firehose can log any Lambda invocation and data delivery errors to Amazon CloudWatch Logs so that you can view the specific error logs if Lambda invocation or data delivery fails. For more information, see Monitoring with Amazon CloudWatch Logs.
What is buffer size and buffer interval?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Data Firehose buffers incoming streaming data to a certain size or for a certain period of time before delivering it to destinations. You can configure buffer size and buffer interval while creating your delivery stream. Buffer size is in MBs and ranges from 1MB to 128MB for Amazon S3 destination and 1MB to 100MB for Amazon Elasticsearch Service destination. Buffer interval is in seconds and ranges from 60 seconds to 900 seconds. Please note that in circumstances where data delivery to destination is falling behind data writing to delivery stream, Firehose raises buffer size dynamically to catch up and make sure that all data is delivered to the destination.
How is buffer size applied if I choose to compress my data?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Buffer size is applied before compression. As a result, if you choose to compress your data, the size of the objects within your Amazon S3 bucket can be smaller than the buffer size you specify.
What is the IAM role that I need to specify while creating a delivery stream?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Data Firehose assumes the IAM role you specify to access resources such as your Amazon S3 bucket and Amazon Elasticsearch domain. For more information, see Controlling Access with Amazon Kinesis Data Firehose in the Amazon Kinesis Data Firehose developer guide.
What privilege is required for the Amazon Redshift user that I need to specify while creating a delivery stream?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
The Amazon Redshift user needs to have Redshift INSERT privilege for copying data from your Amazon S3 bucket to your Redshift cluster.
What do I need to do if my Amazon Redshift cluster is within a VPC?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
If your Amazon Redshift cluster is within a VPC, you need to grant Amazon Kinesis Data Firehose access to your Redshift cluster by unblocking Firehose IP addresses from your VPC. Firehose currently uses one CIDR block for each available AWS Region: 52.70.63.192/27 for US East (N. Virginia), 52.89.255.224/27 for US West (Oregon), and 52.19.239.192/27 for EU (Ireland). For information about how to unblock IPs to your VPC, see Grant Firehose Access to an Amazon Redshift Destination in the Amazon Kinesis Data Firehose developer guide.
Why do I need to provide an Amazon S3 bucket while choosing Amazon Redshift as destination?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
For Amazon Redshift destination, Amazon Kinesis Data Firehose delivers data to your Amazon S3 bucket first and then issues Redshift COPY command to load data from your S3 bucket to your Redshift cluster.
What is index rotation for Amazon Elasticsearch Service destination?
Creating Delivery Streams
Amazon Kinesis Data Firehose | Analytics
Amazon Kinesis Data Firehose can rotate your Amazon Elasticsearch Service index based on a time duration. You can configure this time duration while creating your delivery stream. For more information, see Index Rotation for the Amazon ES Destination in the Amazon Kinesis Data Firehose developer guide.
