13_ProcessRecords Flashcards
What does the ForEach scope do?
Splits a payload collection and processes the individual elements and returns the original payload.
What happens in a ForEach scope if an error occurs?
Stops processing and invokes an error if one element throws an exception.
What does the ForEach scope return?
The original paylaod (regardless of any modifications made inside the scope)
What can a collection be (for a ForEach scope)?
can be any supported content type including applicationjson, aplication/java and application/xml
What does the batch job provide?
The ability to split large messages and process them asynchronously in a batch jon.
Where is the batch job scope available?
Enterprise edition only.
What is created especially for processing data sets?
Batch Job scope.
What splits large or streamed messages into individual records, performs actions upon each record, handles record level failures that ocure so batch job is not aborted, reports on the results and potentially pushes the prcessed output to other systems or queues.
Batch Job scope.
Example use cases for Batch Job scope.
• Integrating data sets to parallel process records
– Small or large data sets, streaming or not
• Engineering “near real-time” data integration
– Synchronizing data sets between business applications
– Like syncing contacts between NetSuite and Salesforce
• Extracting, transforming, and loading (ETL) info into a target system
– Like uploading data from a flat file (CSV) to Hadoop
• Handling large quantities of incoming data from an API into a legacy
system
A batch job contains one or more ?
Batch steps
What are the 3 different phases in a batch job?
Load and dispatch (behind the scene work), Process (asynchronoulsy processes the records and contains one or more batch steps), On Complete (optional - reports summary of records processed, rovised insight into which records failed so you can address issues).
Do all records have to finish processing in one step before any of them are send to the next step?
No
Batch records are queued and scheduled in blocks of? and a threading profile of ?? threads per job?
100 records, 16 threads
Can you customise the configuration for the block and threads?
Yes, as it may improve the batch’s performance in certain use cases
Variables created BEFORE a batch job are available where?
in ALL batch steps
Variables created inside a batch step are available where?
Only in the batch step that they were created.
What are the 3 options if a record fails in a batch job?
Stop processing (the entire batch, the in-flight steps are finished but other steps are skipped and the on complete phase invoked), Continue processing (you should handle failed records using the batch step filters), Continue processing until a max number of failed records is reached (at that point, the on complete phase is invoked).
What happens when the record fails and it stops processing the entire batch?
• In-flight steps are finished, but all other steps are skipped and the on complete phase is
invoked
What happens when the record fails and it continues processing?
You need to specify how subsequent batch steps should handle failed records
– To do this, use batch step filters that are covered in the next section
What happens when the record fails and it continues processing until a max number of failed records is reached?
At that point, the on complete phase is invoked
What are the 2 attributes that are used to filter a record for a batch job?
An accept expression and an accept policy
What is an accept expression?
Used in a batch job to filter the records, it’s an inline expression that will decide if that part of the collection will be processed. e.g. #[not var.exists]
What is an accept policy?
Used in a batch job to filter out records, either set to ALL, NO_FAILURES (default) or ONLY_FAILURES
Batch Aggregator scope do?
It accumulates records, it can upload them in bulk e.g. instead of inserting 100 individual records it will do it all in 1 insert.
What can you use to accumulate records?
Batch Aggregator
Does the ForEach scope process the elements sequentially?
Yes
Examples of complex batch jobs.
Created especially for processing data sets
– Splits messages into individual records and performs actions upon each record
– Can have multiple batch steps and these can have filters
– Record-level data is persisted across steps using variables
– Can handle record level failures so the job is not aborted
– The batch job returns an object with the results of the job for insight into which
records were processed or failed
