Amazon Kinesis Data Firehose | Preparing and Transforming Data in Amazon Kinesis Data Firehose

Question 1

Can I change the configurations of my delivery stream after it’s created?

Preparing and Transforming Data in Amazon Kinesis Data Firehose

Amazon Kinesis Data Firehose | Analytics

Answer 1

You can change the configuration of your delivery stream at any time after it’s created. You can do so by using the Firehose Console or the UpdateDestination operation. Your delivery stream remains in ACTIVE state while your configurations are updated and you can continue to send data to your delivery stream. The updated configurations normally take effect within a few minutes.

Question 2

How do I prepare and transform raw data in Amazon Kinesis Data Firehose?

Preparing and Transforming Data in Amazon Kinesis Data Firehose

Amazon Kinesis Data Firehose | Analytics

Answer 2

Amazon Kinesis Data Firehose allows you to use an AWS Lambda function to prepare and transform incoming raw data in your delivery stream before loading it to destinations. You can configure an AWS Lambda function for data transformation when you create a new delivery stream or when you edit an existing delivery stream.

Question 3

How do I return prepared and transformed data from my AWS Lambda function back to Amazon Kinesis Data Firehose?

Preparing and Transforming Data in Amazon Kinesis Data Firehose

Amazon Kinesis Data Firehose | Analytics

Answer 3

All transformed records from Lambda must be returned to Firehose with the following three parameters; otherwise, Firehose will reject the records and treat them as data transformation failure.

recordId: Firehose passes a recordId along with each record to Lambda during the invocation. Each transformed record should be returned with the exact same recordId. Any mismatch between the original recordId and returned recordId will be treated as data transformation failure.

result: The status of transformation result of each record. The following values are allowed for this parameter: “Ok” if the record is transformed successfully as expected. “Dropped” if your processing logic intentionally drops the record as expected. “ProcessingFailed” if the record is not able to be transformed as expected. Firehose treats returned records with “Ok” and “Dropped” statuses as successfully processed records, and the ones with “ProcessingFailed” status as unsuccessfully processed records when it generates SucceedProcessing.Records and SucceedProcessing.Bytes metrics.
data: The transformed data payload after based64 encoding.

Question 4

What Lambda blueprints are available for data preparation and transformation?

Preparing and Transforming Data in Amazon Kinesis Data Firehose

Amazon Kinesis Data Firehose | Analytics

Answer 4

Firehose provides the following Lambda blueprints that you can use to create your Lambda function for data transformation:

General Firehose Processing: This blueprint contains the data transformation and status model described above. Use this blueprint for any custom transformation logic.

Apache Log to JSON: This blueprint parses and converts Apache log lines into JSON objects, with predefined JSON field names.

Apache Log to CSV: This blueprint parses and converts Apache log lines into CSV format.

Syslog to JSON: This blueprint parses and converts Syslog lines into JSON objects, with predefined JSON field names.

Syslog to CSV: This blueprint parses and converts Syslog lines into CSV format.