Data Files and Serialization

Question 1

What is data integration?

Answer 1

Data integration is combing data, which exists in different resources, and presenting them to the user.

File transfer - when two systems needs to communication they need to produce shared files which they both can make use of.

Shared database - is often used because it is language & platform independent.

Handling different data formats depending on the use case.

Common data transfer mechanism is essential - communication between system should be independent of language & platform.

Question 2

What is File Based data integration?

Answer 2

When two or more applications make use of shared files.

The integrator will be responsible for setting the format and other common protocols, for when the use of these files are executed.

Question 3

Mention a few pros for File Based Data Integration.

Answer 3

Files are popular because they compatible with all popular languages.

Integrators does only need to know input/output, not how the consuming apps works.

Highly flexible.

No extra tools needed.

Easy to understand

Question 4

Mention a few cons for File Based Data Integration.

Answer 4

Scalability - Adaptability - Security - Data synchronisation.

Also a format must be agreed upon sharing files.

A more desirable approach is Shared database, Messaging or RPC

Question 5

Which file types/formats are popular in Data Files?

Answer 5

CSV - XML - JSON

TEXT (not used for Integration, mostly used logging errors/processes, testing or to read tokens)

Question 6

When would you use text files as your data file?

Answer 6

When logging errors/processes, testing or to read tokens

Question 7

When could you use CSV files as Data file?

Answer 7

To move large amount of data (fast because no overhead like JSON OR XML

To view a data table (very human readable)

To better organise data

Question 8

What pros are there for using CSV as Data file?

Answer 8

Human readable and easy to modify.
Faster to handle (no overhead).
It is easy to generate CSV files.
Very popular, many use CSV and therefore it is well know.

Question 9

What cons are there for using CSV as Data file?

Answer 9

It can only handle basic data, a comma would break the structure since it is used as separator.

No distinction between text & numeric values.

Problems when importing CSV to SQL (No distinction between NULL and quotes)

Question 10

When is JSON used as a Data file?

Answer 10

When creating micro-services & RESTful services

When storing data temporarily. Much easier than storing in a database.

Often used as configuration files & manifest.

Question 11

What pros are there for using JSON as Data file?

Answer 11

Easy to read - both for machines and humans

Easy to parse - Most languages support parsing JSON.

Better performance than XML(faster, easier to read)

Very popular..

Question 12

What cons are there for using JSON as Data file?

Answer 12

Can’t use comments

It is encapsulated. The structure needs to be divided into seperate objects if you want to add new data.

Question 13

When could you use XML files as Data file?

Answer 13

Heavily used in SOA.

In SOAP. when you need complicated objects

Configuration files and manifests for apps (Maven,
Gradle etc.)

Standard for Office file formats

Question 14

What pros are there for using XML as Data file?

Answer 14

Great at describing data using attributes

Lightweight when parsing as it has a strict syntax

Question 15

What cons are there for using XML as Data file?

Answer 15

Data redundancy, which leads to large size and storage costs.

Can not hold arrays.

Question 16

Which filetypes are used as configuration files?

Answer 16

JSON, XML

Question 17

Which filetypes can be used to transfer data from on db to another?

Answer 17

CSV, XML, JSON

Question 18

which filetype is often used for logs & reading encrypted files?

Answer 18

TEXT

Question 19

What is Serialisation?

Answer 19

Serialisation is the process of converting an object into a stream of bytes to be stored in memory, a database, or a file.

An example is storing an object to XML or JSON.

Serialisation is also called marshalling in some contexts.

Question 20

What is the reversed process of Serialisation?

Answer 20

Deserialisation (Unmarshalling).

Question 21

Mention some use cases of Serialisation.

Answer 21

Sending objects to a remote application via API.

Maintaining security or user-specific information across
applications - bytes.

A method of storing data.

Question 22

Mention 4 serialisers we have used in class.

Answer 22

JSON serialiser

Msgpack

Pickle

Protobuf

Question 23

What is characteristic about JSON serialiser?

Answer 23

Schema-less - we only need storage and not the representation when they are stored.

Self describing and easy to read by both human & machine.

Supported by most languages.

Standard when transferring data over the web.

Question 24

What is characteristic about Msgpack?

Answer 24

Schema-less - we only need storage and not the representation when they are stored.

It lets you exchange data among multiple languages like JSON.

A lot like JSON structure wise, but msgpack is also faster & smaller than JSON.

Question 25

What is characteristic about Pickle?

Answer 25

Schema-less - we only need storage and not the representation when they are stored.

Only used in Python

Can serialise most python objects

Relatively fast.

Less secure as it can trigger (if deserialised) malicious code.

Question 26

What is characteristic about Protobuf?

Answer 26

Used when serialising structured data, because it has a defined Schema.

It provides type checking.

Good integration with many languages - Not python though.

It involves an IDL (interface description language).

Not Human readable.

Question 27

What is the main purpose of serialisation?

Answer 27

To save the state of an object in order to be able to recreate it when needed.