Big Data for Dummies

Question 1

What are the three Vs of big data?

Answer 1

Extremely large volumes of data; extremely high velocity of data, extremely wide variety of data

Question 2

Why is big data important?

Answer 2

Enables organizations to gather, store, manage and maniuplate vast amounts of data at the right speed, the at the right time, to gain the right insights.

Question 3

Data warehouses vs. data marts

Answer 3

Data warehouses can be too complex and large and didn’t offer the speed and agility that the business required. The answer was a further refinement of hte data being managed through data marts. Data marts were focused on specific business issues and more streamlined, supporting the business need for speedy queries.

Question 4

Data warehouses are typically fed in…

Answer 4

Batch intervals, like daily or weekly. Limits in real-time business and consumer environments

Question 5

What is a BLOB

Answer 5

Binary large objects – stores an unstructured data element. ODMS (object database management system) stores the BLOB as an addressable set of pieces so that we could see what was in there.

Question 6

What is advantage of object database

Answer 6

Includes a programming language and a structure for the data elements so that it is easier to manipulate various data objects without programming and complex joins.

Question 7

What are some of the technologies a the heart of big data? (4)

Answer 7

1. Virtualization 2. Parallel processing 3. Distributed file systems 4. In-memory databases

Question 8

Different approaches to handling data exist based on whether it is data in motion or data at rest. What is data in motion vs data at rest

Answer 8

Data in motion would be used if a company is able to analyze the quality of its products during the manufacturing process to avoid costly errors. Data at rest would be used by a business analyst to better understand customers’ current buying patterns.

Question 9

Is big data a single technology? What does it help companies gain?

Answer 9

Big data is a combo of old and new technologies that helps companies gain actionable insight.

Question 10

What are the 5 components of the cycle of big data management

Answer 10

1. Capture 2. Organize 3. Integrate 4. Analyze 5. Act

Question 11

Why is validation an important issue in big data management

Answer 11

If your organization is combining data sources, it is critical that you have the ability to validate that these sources make sense when combined. Also, certain data sources may contain sensitive information, so you must implement sufficient levels of security and governance.

Question 12

Where would you start in big data management?

Answer 12

Start with the problem you’re trying to solve. That will dictate the kind of data that you need and what the architecture might look like.

Question 13

How do you determine what performance requirements will be when setting up a big data management system?

Answer 13

Your needs will depend on the nature of hte analysis you are supporting. You will need the right amount of computational power and speed. Some analysis will be real time but you will be storing some amount of data as well. -How much data will my organization need to manage today and in the future? -How often will my organization need to manage data in real time or near real time? -How much risk can my organization afford? Is my industry subject to strict security, compliance and governance requirements?

Question 14

Why do you need redundancy in your data management system?

Answer 14

So you are protected from unanticipated latency and downtime

Question 15

What is in a big data tech stack?

Answer 15

Question 16

What makes big data big?

Answer 16

It relies on picking up lots of data from lots of sources.

Question 17

Why are APIs important in the big data stack?

Answer 17

To get massive amounts of data in, you need integration. Open application programming interfaces (APIs) will be core to any big data architecture. Interfaces exist at every level and between every layer of hte stack. Without integration services, big data can’t happen.

Question 18

Why does big data need different infrastructure than traditional data

Answer 18

To support an unanticipated or unpredictable volume of data. So it’s based on a distributed computing model. This means that data may be physically stored in many different locations and can be linked together through networks, the use of a distrihbuted file system, and various big data analytic tools and applications.

Question 19

What is a distributed computing model

Answer 19

This means that data may be physically stored in many different locations and can be linked together through networks, the use of a distrihbuted file system, and various big data analytic tools and applications.

Question 20

Why is redundant physical infrastructure important

Answer 20

Because we’re dealing with so much data from so many different sources. Redundancy comes in many forms. If your company has created a private cloud, you will want to have redundancy built within th eprivate environment so that it can scale out to support changing workloads. In some cases, this redundancy may come in the form of a Software as a Service (SaaS) offering that allows companies to do sophisticated data analysis as a service.

Question 21

Why use SaaS for redundant physical infrastructure?

Answer 21

Lower costs, quicker startup and seamless evolution of the underlying tech

Question 22

Security infrastrcuture is important why

Answer 22

If you have to comply with regulations or keep customer info secure you will need to take into account who is allowed to see the data and under what circumstances they are allowed to do so.

Question 23

What is an operational data source

Answer 23

In big data you have to incorporate all the data sources that will give you a complete picture of your business and see how the data impacts the way you operate your business. In the past this was highly structured data managed in a relational database. But operational data now has to encompass a broader set of data sources, including unstructured sources such as customer and social media data

Question 24

What characteristics does a good operational data source have?

Answer 24

1. Represent systems of record that keep track of the critical data required for real-time, day to day operation of the business
2. Continually updated based on transactions wherever they take place
3. Blend structured and unstructured data
4. System that scales to support many users on a consistent basis.

Question 25

What do mapreduce type computing technologies accomplish

Answer 25

They provide the ability to process massive amounts of data efficiently, cost-effectively, and in a timely fashion.

Question 26

What does MapReduce do?

Answer 26

Designed by Google to efficiently execute a set of functions against a large amount of data in batch mode. The map component distributes the programming problem or tasks across a large number of computers

Question 27

What does Big Table do?

Answer 27

Developed by Google to be a distributed storage system intended to manage highly scalable structured data. Stores huge volumes of data across commodity servers

Question 28

What does Hadoop do?

Answer 28

Derived from MapReduce and Big Table. Hadoop allows applications based on MapReduce to run on large clusters of commodity hardware. Designed to parallelize data processing across computing nodes to speed computation and hide latency. Two major components:

1. Massively scalable distributed file system that can support huge amounts of data
2. Massively scalable MapReduce engine that computes results in batch.

Question 29

What is change between dealing with lots of data previously and dealing with it now in big data

Answer 29

Big data changes what you can do with that information. Can anticipate and solve business problems and react to opportunities.

Question 30

When would you need to deal with data in real time vs not

Answer 30

1. Monitoring traffic data – real time.
2. Analyze big set of data to mine for patterns – not real time

Deciding when you need which informs the technology purchases that you make

Question 31

What is structured data

Answer 31

Data that has a defined length and format: numbers, dates, and groups of words and numbers called strings. You can query it with SQl

Question 32

What is a string

Answer 32

Structured data but words and numbers

Question 33

What is machine generated data vs human generated data

Answer 33

Web log data, sales data….vsinput data, clickstream data, moves you make in a game

Question 34

What is data persistence

Answer 34

How a database retains versions of itself when modified. The great grandaddy of persistent data stores is the relational database management system (RDMS).

Question 35

What is a schema

Answer 35

In a relational database, a schema is a structural representation of what is in the database. The tables, the fields in the tables, and the relationships between the two.

Question 36

How do you update, read and delete data in a relational model?

Answer 36

SQL (structured query language). Tables can be queried using a common key (customer ID)