Big Data Lecture 12 Querying Trees

Question 1

Why is not true that we should always build as many indices as possible?

Answer 1

Indexing on a certain attribute takes space, and computational resources. So even though it makes possibly the query on the attribute faster, it does not mean we should do it.

Question 2

What is the goal of JSONiq?

Answer 2

We can query unstructured JSON data in data independent way.

Question 3

What does RumbleDB actually do?

Answer 3

Rumble is an engine that connects the logical expressions of JSONiq to physical implementation underneath (HDFS, WCS, you name it!).

Question 4

What kind of language is JSONiq?

Answer 4

<ul><li>Declarative (we say that we want, and it just happens),</li><li>functional (you can nest it like a boss),</li><li>set-based (everything is a sequence).</li></ul>

Question 5

What is a data LakeHouse?

Answer 5

Datalake warehouse, something that can be queried from top-level, but at the lower level it is just a data lake.

Question 6

How is data represented in JSONiq?

Answer 6

Everything is just a sequence of items, even nothing () and a single item (“lol”).

Question 7

What happens when you enter valid JSON into JSONiq?

Answer 7

It evaluates to itself.

Question 8

Can JSONiq item sequences be nested?

Answer 8

No, it naturally removes the nesting, e.g. ((1), 2) == (1, 2).

Question 9

Are all types in JSONiq comparable like in MongoDB?

Answer 9

No.

Question 10

Can we compare two arrays in JSONiq?

Answer 10

Yes, the predicate will be evaluated as an existential quantifier.

Question 11

What are FLWOR expressions?

Answer 11

<ul><li>For,</li><li>Let,<br></br></li><li>Where,</li><li>Order by,</li><li>Return.</li></ul>

Typical pattern in JSONiq, under the hood they are nicely optimized.

Question 12

What are tuple streams in JSONiq?

Answer 12

Formalism to explain combinations of operations, how they chain together, and how they are applied to each other and in order.

Question 13

How are types formally expressed in JSONiq?

Answer 13

Using the given type, plus quantification (+/*/?/nothing if required).

Question 14

How is query of JSONiq parsed in the background?

Answer 14

<ol><li>Abstract Syntax Tree is created,</li><li>using VisitorPattern converted to ExpressionTree,</li><li>the ExpressionTree is optimized for execution,</li><li>conversion to Iterator Tree (volcano iterators).</li></ol>

Question 15

What are volcano iterators?

Answer 15

Objects that we can open(), ask hasNext(), next() and close().

Question 16

Is iterator tree composed only of volcano iterators?

Answer 16

No, there can also be tuple stream in between.

Question 17

What are 3 possible ways of execution?

Answer 17

<ol><li>Materialized execution (everything is materialized on the go before the next iteration of the algorithms is applied),</li><li>streamed execution (only necessary components are materialized for the next step),</li><li>parallel execution (separate execution on different machines, and let those materialize each part).</li></ol>

Question 18

What are the (dis)advantages of each type of execution?

Answer 18

<ul><li>Materialized: memory overhead,</li><li>streamed: function call overhead (time),</li><li>parallel: incompleteness of data, YARN executor initialization overhead.&nbsp;</li></ul>

Question 19

As what data types does Spark handle sequences?

Answer 19

RDDs (heterogeneous) or DataFrames, if possible (for homogeneous, faster).

Question 20

What is a UDF?

Answer 20

User-defined-function, anything we pack and send into Spark to handle it internally (if not native, it can be slow).

Question 21

Do we have to push processes down manually?

Answer 21

No, they are pushed down automatically as much as possible.