Azure Data Fundamentals

Question 1

Storage Account

Answer 1

Azure Storage is a core Azure service that enables you to store data in:
Blob containers - scalable, cost-effective storage for binary files.
File shares - network file shares such as you typically find in corporate networks.
Tables - key-value storage for applications that need to read and write data values quickly.

Question 2

Blob Storage

Question 3

Tables

Question 4

Files

Question 5

Entity

Answer 5

Data structures in which this data is organized often represents Each entity typically has one or more attributes, or characteristics

Question 6

Structured Data

Answer 6

Fixed Schema, tabular, often stored in db’s,

Question 7

Semi Structured Data

Answer 7

data is information that has some structure, but which allows for some variation between entity instances; One common format for semi-structured data is JavaScript Object Notation (JSON)

Question 8

Unstructured

Answer 8

Not all data is structured or even semi-structured. For example, documents, images, audio and video data, and binary files might not have a specific structure.

Question 9

Data Stores

Answer 9

there are 2 - files stores and databases

Question 10

BLOB

Answer 10

Binary Large Objects such as video, audio, images, and application specific documents, stored as raw binary files

Question 11

Optimized File Formats

Answer 11

Avro, ORC, Parquet
While human-readable formats for structured and semi-structured data can be useful, they’re typically not optimized for storage space or processing. Over time, some specialized file formats that enable compression, indexing, and efficient storage and processing have been developed.

Question 12

Avro

Answer 12

Avro is a row-based format. It was created by Apache. Each record contains a header that describes the structure of the data in the record. This header is stored as JSON. The data is stored as binary information. An application uses the information in the header to parse the binary data and extract the fields it contains. Avro is a good format for compressing data and minimizing storage and network bandwidth requirements.

Question 13

ORC (Optimized Row Columnar)

Answer 13

organizes data into columns rather than rows. It was developed by HortonWorks for optimizing read and write operations in Apache Hive (Hive is a data warehouse system that supports fast data summarization and querying over large datasets). An ORC file contains stripes of data. Each stripe holds the data for a column or set of columns. A stripe contains an index into the rows in the stripe, the data for each row, and a footer that holds statistical information (count, sum, max, min, and so on) for each column.

Question 14

Parquet

Answer 14

another columnar data format. It was created by Cloudera and Twitter. A Parquet file contains row groups. Data for each column is stored together in the same row group. Each row group contains one or more chunks of data. A Parquet file includes metadata that describes the set of rows found in each chunk. An application can use this metadata to quickly locate the correct chunk for a given set of rows, and retrieve the data in the specified columns for these rows. Parquet specializes in storing and processing nested data types efficiently. It supports very efficient compression and encoding schemes.

Question 15

Data Documents

Answer 15

Collective form in which data exist; common types:
Datasets, Databases, Datastores, Data Warehouses, Notebooks

Question 16

Dataset

Answer 16

logical grouping of data

Question 17

Database

Answer 17

structured or semi-structured data that can be quickly access and searches;

Question 18

Datastores

Answer 18

unstructured or semi-strucutred data (example Azure Data Lake)

Question 19

Non-Relational Databases

Answer 19

often referred to as NoSQL databases. There are 4 common types: Key-value; document (similar to key-value where the value is the JSON document); column family; graph

Question 20

Data Lake Store (Gen 2)

Answer 20

hierarchical data storage for analytical data lakes, work with structure, semi, and unstructured; To create an Azure Data Lake Store Gen2 files system, you must enable the Hierarchical Namespace option of an Azure Storage account.

Question 21

Azure File Storage

Answer 21

Many on-premises systems comprising a network of in-house computers make use of file shares. A file share enables you to store a file on one computer, and grant access to that file to users and applications running on other computers. This strategy can work well for computers in the same local area network, but doesn’t scale well as the number of users increases, or if users are located at different sites.
Azure Files enables you to share up to 100 TB of data in a single storage account.
Offers Standard and Premium tiers

Question 22

Azure Table Storage

Answer 22

NoSQL storage solution, key-value, The key in an Azure Table Storage table comprises two elements; the partition key that identifies the partition containing the row, and a row key that is unique to each row in the same partition.
Azure Table Storage allows you to store key-value data as the cheapest per GB rate. Cosmos DB also have a key-value option, but is not as cost effective.

Question 23

Storage Account

Answer 23

needs to be set up first before other storage services are added

Question 24

Azure Cosmos DB

Answer 24

Azure Cosmos DB is a highly scalable cloud database service for NoSQL data; example of NoSQL database; Cosmos DB uses indexes and partitioning to provide fast read and write performance and can scale to massive volumes of data. You can enable multi-region writes, adding the Azure regions of your choice to your Cosmos DB account so that globally distributed users can each work with data in their local replica. Includes API’s for Mongo DB, Table API (key-value tables), Cassandra API (column family storage), Gremlin API (graph structures)

Question 25

Synapse Analytics

Answer 25

unified, end-to-end solution for large scale data analytics, combines benefts of SQL Server, data lake and open source Apache Spark. All services within this can be managed through a single Azure Synapse Studio.

Question 26

Data Explorer

Answer 26

standalone service for efficiently analyzing data

Question 27

Kusto Query Language

Answer 27

to query data explorer tables

Question 28

Manual Sharding

Answer 28

With a relational data store, when data transaction volumes get too high and database performance suffers, which of the following is a common yet difficult method of distributing those data transactions over multiple servers

Question 29

Paginated Report

Answer 29

designed to be printed or shared; formated to fit well on page; display all the data in table; pixel perfect;

Question 30

Dynamic Data Masking

Answer 30

Dynamic data masking (DDM) limits sensitive data exposure by masking it to non-privileged users. It can be used to greatly simplify the design and coding of security in your application. Dynamic data masking helps prevent unauthorized access to sensitive data by enabling customers to specify how much sensitive data to reveal with minimal impact on the application layer. DDM can be configured on designated database fields to hide sensitive data in the result sets of queries. With DDM the data in the database is not changed.

Question 31

Transaction Optimized Storage

Answer 31

provides better a cost profile for frequently accessed files. It costs more to store the file, but much less to access it. And it supports Standard Tier of storage

Question 32

Azure Data Factory

Answer 32

has the ability to move data from one source to another destination with processing along the way. Azure Data Migration is more about simple movement of data from one source to another.

Question 33

Azure Cache for Redis

Answer 33

Azure Cache for Redis provides an in-memory data store based on the Redis software. Redis improves the performance and scalability of an application that uses backend data stores heavily. It's able to process large volumes of application requests by keeping frequently accessed data in the server memory, which can be written to and read from quickly. Redis brings a critical low-latency and high-throughput data storage solution to modern applications.

Question 34

Column-store data

Answer 34

data organized in to columns, faster at aggregating values for analytics; NoSQL store of SQL-Like databases; great for vast amount of data; great when you only need a few columns;

Question 35

Balanced Tree

Answer 35

a common data structure for storing index

Question 36

Data Consistency

Answer 36

when data being kept in two different place and whether the data exactly match; Strongly consistent - every time you query you get consistent data; Eventually Consistent - when you request data you may get back inconsistent data within 2 seconds

Question 37

Datamart

Answer 37

subset of data warehouse; has single business focus;

Question 38

Data Lake

Answer 38

centralized storage repository that holds a vast amount of raw data (big data) in either semi or unstructured format; Hording for data scientist;