Hive

Question 1

What is Hive?

Answer 1

Hive is application that runs over the Hadoop framework and provides a SQL like interface for processing/querying data.

Question 2

Where is the default location of Hive’s data in HDFS?

Answer 2

/user/hive/warehouse

Question 3

What is an External table?

Answer 3

An external table is just a table that is stored outside of the hive warehouse. Hive does not manage its storage.

Question 4

What is a managed table?

Answer 4

Managed tables are Hive owned tables where the entire lifecycle of the tables’ data are managed and controlled by Hive.

Question 5

What is a Hive partition?

Example?

Answer 5

Hive Partition is a way to organize large tables into smaller logical tables based on values of columns. Lets say we had a US census table which contains zipcode, city, state, and other columns. Creating a partition on state splits the table into around 50 partitions, when searching for a zipcode with in a state (state=’CA’ and zipCode =’92704′) results in faster as it need to scan only in a state=CA partition directory.

Question 6

What’s the benefit of partitioning?

Answer 6

Its helps to organize the data in logical fashion and when we query the partitioned table using partition column, it allows hive to skip all but relevant sub-directories and files.

Question 7

What does a partitioned table look like in HDFS?

Answer 7

each partition is stored in a different directory

Question 8

What is a Hive bucket?

Answer 8

A hive Bucket is dividing the data into more manageable form resulting in multiple files or directories. it is used for efficient querying.

Question 9

What does it mean to have data skew and why does this matter when bucketing?

Answer 9

data skew means that you have an uneven distribution of data. This matters when bucketing because you need to split things up evenly or otherwise it will take time to compute your queries

Question 10

What does a bucketed table look like in HDFS?

Answer 10

each bucket is stored in a file

Question 11

What is the Hive metastore?

Answer 11

Hive metastore (HMS) is a service that stores metadata of both types of tables (managed and external)

Question 12

What is beeline?

Answer 12

Beeline is a thin client that also uses the Hive JDBC driver but instead executes queries through HiveServer2, which allows multiple concurrent client connections and supports authentication

Question 13

What does Cluster Computing refer to?

Answer 13

Cluster computing depicts a system that consists of two or more computers or systems, often known as nodes. These nodes work together for executing applications and performing other tasks.

Question 14

What is a Working Set?

Answer 14

A working set is the set of data you working

Question 15

What does RDD stand for?

Answer 15

Resilient Distributed Datasets. It is a immutable distributed collection of objects.

Question 16

What does it mean when we say an RDD is a collection of objects partitioned across a set of machines?

Answer 16

RDD partitions referred to as “splits”. Each partition in the RDD serves as the chunk of data for one task. The Resilience of RDDs in enabled by their lineage. Each RDD contains the instructions for recomputing itself from prior data. This lets Spark recompute individual partitions if they are lost. If our data processing reads from disk, does 3 transformations (3 maps) and then writes to disk, our RDD will contain that information. If some of our partitions are lost after the second transformation, Spark has the information in that lineage to recompute them from prior steps stored in memory (if they exist), or from disk.

Question 17

Why do we say that MapReduce has an acyclic data flow?

Answer 17

It completes each step of the process without ever going back to to a previous step.

Question 18

Explain the deficiency in using Hive for interactive analysis on datasets. How does Spark alleviate this problem?

Answer 18

Hive has to do MapReduce and read and write to disk. Spark is up to x100 faster because it can do the same thing right from memory.

Question 19

What is the lineage of an RDD?

Answer 19

A lineage will keep track of what all transformations has to be applied on that RDD, including the location from where it has to read the data.

Question 20

What are the 4 ways provided to construct an RDD?

Answer 20

1) using parallelize method
from a Scala collection

2) use a transformation
(to output the transformed RDD)

3) reading a file
(loading a file creates a RDD)

4)persist of an RDD
returns back the new RDD with new persist level

Question 21

What does it mean to transform an RDD?

Answer 21

Transformations produce an RDD from another RDD and so don’t cause execution and changes the data within the RDD provided a function.

Question 22

What does it mean to cache an RDD?

Answer 22

It means to store the RDD in the cache of the Worker Node to be able to be reused again without recomputing it.

Question 23

What does it mean to perform a parallel operation on an RDD?

Answer 23

RDD operations are executed in parallel on each partition. Tasks are executed on the Worker Nodes where the data is stored.

Question 24

Why does Spark need special tools for shared variables, instead of just declaring, for instance, var counter=0?

Answer 24

because of closures, global variables can only be seen by the driver and not the task closure, so each task takes a copy of the global variables and updates it independently from other worker nodes.

Question 25

What is a broadcast variable?

Answer 25

Broadcast variables in Apache Spark is a mechanism for sharing variables across executors that are meant to be read-only.

Question 26

What is an accumulator?

Answer 26

Accumulators are shared variables which are only “added” through an associative and commutative operation and are used to perform counters (Similar to Map-reduce counters) or sum operations

Question 27

What are some transformations available on an RDD?

Answer 27

Maps, filter, Reducebykey

Question 28

What are some actions available on an RDD?

Answer 28

count, collect, take, top

Question 29

What is a shuffle in Spark?

Answer 29

A shuffle occurs when data is rearranged between partitions. This is required when a transformation requires information from other partitions, such as summing all the values in a column.

Question 30

How can we see the lineage of an RDD?

Answer 30

toDebugString