11.1 Big Data Flashcards
What are the 3 defining features of big data?
Volume, velocity, variety
What does volume mean in big data?
There is too much data for it all to fit on a conventional hard drive or even a server. Data has to be stored over multiple servers, each of which is composed of many hard drives.
What does velocity mean in big data?
Data on the servers is created and modified rapidly. The servers must respond to frequently changing data within a matter of milliseconds.
What does variety mean in big data?
The data held on the servers consists of many different types of data from binary files to multimedia files.
What problems come from the way big data is structured?
Big data is unstructured making it difficult to analyse the data. Conventional databases are not suited to storing big data because they require the data to conform to a row and column structure. Conventional databases do not scale well across multiple servers.
How is useful information extracted from big data?
to extract useful information machine learning is used to discern patterns in the data.
What are some examples of big data?
Continuously monitored banking interactions and data from surveillance systems.
Why can’t big data use conventional programming paradigms?
When data is stored over multiple servers, as is the case with big data, the processing associated with using the data must also be split across multiple machines. This would be incredible difficult with conventional programming paradigms as the machines would all have to be synchronised to ensure that no data is overwritten or otherwise damaged.
What is functional programming?
A solution to the problem of processing data over multiple machines. The programs are stateless (meaning that they have no side effects) and make use of immutable data structures. The programming paradigm also supports higher-order functions.
What are the benefits of functional programming?
easier to write correct, efficient, distributed code than with procedural programming techniques.
Why is the fact-based model used to represent data?
Because big data doesnt conform to the row and column format typically used to represent data.
How is data represented in the fact-based model?
each individual piece of information is stored as a fact. Facts are immutable and can’t be overwritten.
What does immutable mean?
they never change once created
What are timestamps in big data?
They are stored with each fact to indicate the date and time at which piece of information was recorded.
Why are timestamps in big data used?
Seeing as facts are never deleted or overwritten, multiple different values could be held for the same attribute. So they are used to allow a computer to discern which value is most recent.
What are the benefits of the fact-based model?
As the facts are immutable using the model for storing big data reduces the risk of accidentally losing data due to human error. Also the model gets rid of an index for the data and instead appends new data to the dataset as it’s created.
How does representing big data using graph schema work?
This model uses graphs consisting of nodes and edges to graphically represent the structure of a dataset. Nodes in a graph represent entities and can contain the properties of the entity. Edges are used to represent relationships between entities and are labelled with a brief description of the relationship.
Why are timestamps not used in graph schema models?
It is assumed that each node contains the most recent information.
Explain the 2 ways in which a graph schema model would look using 3 entities?
The graph schema is represented as 3 circles. The properties of each entity are listed inside of the circles. Arrows linking the circles represent the relationships between the nodes.
An alternative is to represent the entity’s properties in rectangles and joined to the entities with a dashed line. The dashed lines do not represent relationships, just that the property belongs to the entity.
