Introduction to Standalone Connector

Question 1

What is standalone connector?

Answer 1

It handles the communication between the source system and SAP Signavio Process Intelligence

Question 2

When can standalone connector be used

Answer 2

if it’s not covered by one of the standard connectors in Process Intelligence (or any other third party systems)

Question 3

What does standalone connector do?

Answer 3

extracts data from the source system, transforms it to event log format, and is then uploaded to Process Intelligence to be analyzed

Question 4

How do the ETL scripts run?

Answer 4

externally (outside of Process Intelligence) but uses the API to push the data to a process within the system.

Question 5

What 4 components does the standalone connector use?

Answer 5

– A collection of extraction and transformation SQL scripts
– A configuration file in YAML format
–A SQLite database to ensure the correct data is loaded each time in case of regular loads
– A java application for triggering the actual extraction, transformation and loading

Question 6

Where does the connector pull and store data?

Answer 6

Uses a SAP Technical user to pull data from the sources system and store in an S3 bucket

Question 7

How does the connectors download the files?

Answer 7

From the transformed S3 buckets, it uses Athena to generate an eventlog file and download this file

Question 8

What does the connector do to the event log files?

Answer 8

uploads the event log file to the Process Intelligence API

Question 9

How to get an automated ETL to work?

Answer 9

setup the virtual machine (to set up the environment)

Question 10

[setting up virtual machine] 1. what are specifications for the host

Answer 10

10gb ram
single core CPU
10gb free disk space
network connectivity to both signavio and source system

[size may vary depending on use case]

Question 11

[setting up virtual machine] 2. what do you install?

Answer 11

JDK 8 or higher
-oracle
-oracle JDK

Question 12

[setting up virtual machine] 3 - If the connector is running a windows environment what do you need to install as well

Answer 12

Microsoft visual C++ 2010 Resdistrubtable Package

Question 13

[setting up virtual machine] 4. what do you download last

Answer 13

download and unpack the signavio connector provided by SAP Signavio

Question 14

Why may you have a dedicated staging environment?

Answer 14

In most cases it may be faster and better-suited for process mining. It also enables you to use multiple source systems.

Question 15

when staging and environment setup, what else will you need in the case of AWS?

Answer 15

an account is required with both S3 for data storage and Athena for running the transformation scripts

Question 16

What happens once the environment setup is finished

Answer 16

the connector needs to be configured to fit the specific use case. This is done in the config.yaml file provided by SAP

Question 17

What does the config.yaml file contain?

Answer 17

defines the actions required by the connector and the connection configurations, table extractions, and event collector configurations

Question 18

[connector configuration - config.yaml] 1. you will require apiToken and url - where do you get these

Answer 18

1. PI > under process settings > API > New Token
2. same settings as API token under Upload Endpoint

Question 19

[connector configuration - config.yaml] 2. in the staging area what inputs do you need? (can be provided by your admin)

Answer 19

1. access key & secret key : access information provided by the account creator
2. athena Workgroup : name of athena workgroup if multiple exist
3. region : region your AWS instance runs in
4. bucket: name of the s3 bucket data will be stored in
5. athenaDatabase : name of your database in Athena

Question 20

[connector configuration - config.yaml] 3. what are typical input information required to establish a connection?

Answer 20

1. source - name
2. user - user in the system with neccessary rights
3. password - for the user
4. logfile - name of the extraction log file
5. verbosity - amount of detail stored in the log file

Question 21

Whats next after establishing connection?

Answer 21

extractor configuration aka. define the extraction and necessary data by looking at the parameters for delta load under tableSyncConfigurations

Question 22

[extractor configuration] 1. what are general parameters for each table that should be extracted

Answer 22

1. name - name for the table in staging area
2. data source - specify it incase of muliple source systems
3. sqlFilePath - name of sql file for extraction
4. keyColumn - the column of the table used to merge multiple rows in staging
5. mostRecentRowColumn - used to identify the newest row with most recent information

Question 23

[extractor configuration] what are additional parameters for?

Answer 23

for scheduled data loads in case new data needs to be extracted on a regular basis

Question 24

[extractor configuration] 2. what are some additional parameters?

Answer 24

1. name - table column that distingusishes different delta load eg. creation or change date
2. initial - initial date or ID to start extraction
3. date case - format eg. YYYY-MM-dd and type eg. date
4. ID case - idformat i.e format of the column and type e.g id

Question 25

[extractor configuration] what do extraction scripts have to ensure no data is missed or loaded twice during multiple data loads?

Answer 25

a condition, which usually makes use of a left and right boundary (normally dates) to extract data between these values

Question 26

[extractor configuration] 3. what is right and left boundary

Answer 26

left - initial one for the first load and the values from the previous load - they’re saved in the previously mentioned log files on each load

right - usually the current date or the newest ID

Question 27

after we have our source system and extraction information what is the next step?

Answer 27

transformation of our source data into the event log format

Question 28

To transform our data into event log format what do we need?

Answer 28

3 columns (case ID, event name, timestamp) under eventCollectorConfigurations

Question 29

[Transformation Configuration] 1. the event log format needs 3 columns, which includes what?

Answer 29

– name : distinguishing different events during transformation
– collectFrom : where source tables should be queried
– sqlFilePath - name of the transformation script for this event
–mapping : to map query results to necessary event log columns - usually contain caseID, timestamp and activityname
– attributes : for date attributes the format needs to be specified - name (of event), type (i.e date) and date format

Question 30

How can the connector be started?

Answer 30

can be started as a Java application.

First, go to the source directory of the connector then execute ‘java -jar signavio-connector.jar <command></command>’ to begin

Question 31

Based on the tableSyncConfiguration what is the ‘extract => ‘ command?

Answer 31

takes raw data from source system defined in extraction scripts and uploads it to the stage area where it’s saved as raw talbes

Question 32

Based on the tableSyncConfiguration what is the ‘createschema =>’ command?

Answer 32

generates the schema for the raw tables

Question 33

Based on the tableSyncConfiguration what is the ‘transform => ‘ command?

Answer 33

optimizes the raw table schema and merges row updates in case of changes, updates on data the extracted will be recognised based on keyColumn and mostRecentRowColumn parameters

Question 34

Based on the eventCollectorConfiguration what is the ‘eventlog =>’ command?

Answer 34

creates event log out of the staging system based on transformation scripts and uploads it to PI