Big Data Lecture 07 Data Models

Question 1

How is JSON/XML stored in memory?

Answer 1

Using a tree:<br></br><ul><li>JSON: Nodes are either values, objects or array, edges are keys in objects with annotations on them (graph is directed downwards).</li><li>XML: Nodes are tags, whose values is the text inside, the attributes and their values are attached to the respective node (graphs is undirected).</li></ul>

Question 2

What is XML information set?

Answer 2

The way parse tree is stored in memory:<br></br><ul><li>Document Information Item (child is the main element, stores version metadata),</li><li>Element Information Item (stores local name, children, attributes, parent),</li><li>Text Information Item (stores character string, and owner element),</li><li>Attribute Information Item (stores local name, normalized value (without quotes) owner element).</li></ul>

Question 3

What is validation and how does it relate to well-formedness?

Answer 3

Well-formedness is against a language, validation is done later, it is our constraint on the structure and values within the language.

Question 4

What is the difference between validation and annotation?

Answer 4

Validation is just True/False, while annotation is screening the data and adding metadata/changing value structure to prepare the data. Annotation also includes conversion of the data into correct data types in the storage.

Question 5

Why do we do data validation?

Answer 5

When data is validated against a schema, it is not heterogeneous anymore, it is homogeneous instantly (w.r.t. to the schema) and we can use that to pre-load/query the data faster.

Question 6

What are cardinality markers?

Answer 6

<ul><li>Required, must be exactly once,</li><li>repeated, *, zero or more,<br></br></li><li>optional, ?, zero or one,</li><li>no name, +, one or more.</li></ul>

Question 7

What is JSound?

Answer 7

Validation schema for JSON:<br></br><ul><li>type the wanted type using quotes: “string”, “integer”,</li><li>everything is optional by default, put a “!” up fron to make it required,</li><li>use “item” if you want to store anything you want,</li><li>you can nest [] and {} just like in any item.</li></ul>

Question 8

When is data set heterogeneous or homogeneous?

Answer 8

If it follows a schema, it is homogeneous, with respect to the schema.

Question 9

HBase runs on top of HDFS, how so it is still fast?

Answer 9

<ol><li>It is storing stuff using MemStore and cache, which are fast,</li><li>it shortcircuits DataNodes in HDFS.</li></ol>

Question 10

What is the difference between atomic and structured types?

Answer 10

<ul><li>Atomic cannot be reduced, e.g. int,</li><li>structured are nested, e.g. arrays and objects.</li></ul>

Question 11

What atomic types are there? (7)

Answer 11

<ul><li>Strings,</li><li>numbers,</li><li>booleans,</li><li>dates and times,</li><li>time intervals,</li><li>binary,</li><li>null.</li></ul>

Question 12

What is the difference between lexical and value space?

Answer 12

Value space is the actual value, lexical value is the encoding in characters.<br></br><br></br>There can be a big number of lexical values connected to one actual value.

Question 13

What is the difference and relation of subtype and supertype?

Answer 13

Subtype’s value space is a subset of supertype’s value space.

Question 14

What structured types are there? (2)

Answer 14

Maps (e.g. JSON Object),<br></br>and Lists (e.g. JSON Array).

Question 15

When is adding of default values done in the schema?

Answer 15

During the annotation phase.

Question 16

On which data representation is schema validation done?

Answer 16

It is done when the document is well-formed already on the stored representation in the memory.

Question 17

What are the rules for JSON Schema?

Answer 17

{<br></br>    “type” : “here write the type”,<br></br>    “required”: “here specify required properties”<br></br>    “properties”: {<br></br>                         “fake_name”: { “type” : “give type here”}<br></br>                         }<br></br>    “additionalProperties”: “here define if there can be more stuff, true by default”<br></br>}<br></br>Making type true makes it validate with anything, it will be always okay. Type false means that this attribute must not exist.

Question 18

What is impedance mismatch?

Answer 18

If lexical values in the file do not correspond well do the values represented in the programming language.

Question 19

How does XML validation work?

Answer 19

We have schemas stored in the domain xmlns:xs=”https://www.w3.org/2001/XMLSchema” and we use those to validate our schema.<br></br><br></br>We use complexType to make stuff with attributes.<br></br><br></br>We use sequence for repeated elements, we specify repetitions for each element in the sequence.

Question 20

What is schema of schemas?

Answer 20

Usually, JSON schema is also a JSON document. Same for XML schema, or JSound. So there is, naturally, also schema for validating the schema. This is called <i>schema of schema!</i>

Question 21

What are DataFrames?

Answer 21

Collections of valid JSON objects.

Question 22

What is a dataset from an abstract perspective?

Answer 22

A list of maps!

Question 23

What is a table w.r.t. a dataframe?

Answer 23

It is a special type of a dataframe, one that has no nesting.

Question 24

What is Parquet?

Answer 24

JSON compression schema, it does not store attribute names once again, to save space.

Question 25

What is P1DT7H2M3S?

Answer 25

Lexical value of a date.

Question 26

Can XML without a validation schema be valid?

Answer 26

No, there is nothing to validate it against.