ChatGPT Missed ?s

Question 1

IOPS (intro)

Answer 1

Input/Output Operations per second, measures performance of data access in storage systems, crucial metric for high throughput in big data systems

Question 2

Inverse of the 80-20 Pareto Rule (intro)

Answer 2

Before 80% of data would be used and 20% not, now it is the reverse

Question 3

HDFS federation (storage)

Answer 3

Multiple independent NameNodes managing namespace (helps scalability)

Question 4

Secondary NameNode (storage)

Answer 4

Different than standby node, this one takes stores and compacts edit logs (that grow too large) provide checkpoints to NameNode (snapshots)

Question 5

When is replication better than erasure coding?(storage)

Answer 5

when fast access of lost data is more important than storage optimization

Question 6

Repetition level vs Definition level Parquet (storage)

Answer 6

repetition - how deep in nested structure
definition - defined or null

Question 7

3 Parquet compression techniques (storage)

Answer 7

1. dictionary encoding - low cardinality
2. run-length encoding - long runs of same value
3. bit-packing - reduce numb of bits requires

Question 8

Shuffle vs Sorting Confusion (mapreduce)

Answer 8

shuffle is moving data in key groups, sorting happens on the reducers so that keys are processed in ordered manner

Question 9

Why is commutativity imporant in map reduce operations? (Mapreduce)

Answer 9

data gets reordered during the shuffle phase

Question 10

why is non-associativity important in map reduce operations? (mapreduce)

Answer 10

intermediate results must be combined in any order without changing fnial outcome

for sum it doesnt matter, but if we wanted to find variance, order and context matters

Question 11

Hashing (mapreduce)

Answer 11

Assigns each key numerical value to ensure all instances with same key go to same reducer, while still attempting to balance workload.

Question 12

What are the (ε, δ)-guarantees in approximation algorithms? (Streaming)

Answer 12

in the context of approximating stream - epsilon is the error margin we are willing to accept, and delta is the probability that the algorithm fails to give a good approximation.

Question 13

reservoir sampling (streaming)

Answer 13

when a new value arrives into streaming system, we can probabilistically decide whether to add it to collection (replace a slot) or discard it

Question 14

Count-min sketch algorithm (streaming)

Answer 14

used to estimate frequency, uses polylog function to store more data in smaller dataset

Question 15

Hyperloglog (streaming)

Answer 15

estimates number of unique elements by hashing (also acheives polylog)

Question 16

rollback recovery (streaming)

Answer 16

logs state at regular intervals to revert to previous state if fails

Question 17

5 streaming algorithm (streaming)

Answer 17

1. one pass
2. small space for state
3. fast update state
4. fast computation
5. approximation with conf. guarantee

Question 18

why do NoSQL databases avoid joins and rigid schemas (nosql)

Answer 18

complex queries slow down performance.

schemas make it harder to partition and scale

Question 19

schemaless indexing (nosql)

Answer 19

efficient querying even in databases that don’t enforce rigid structures

Question 20

BASE philosophy (nosql)

Answer 20

Basically available, soft-state, eventually consistent.

(avail > consistency)

Question 21

polyglot persistence (nosql)

Answer 21

using different databases for different taks within same application

Question 22

PACELC (nosql)

Answer 22

theres another tradeoff in addition to partition tolerance and availability, which is latency and consistency

Question 23

partition tolerance (nosql)

Answer 23

if there are network errors while partition happens, can still
operate

Question 24

mutli-model nosql databases

Answer 24

support multiple types of data models

Question 25

datalakehouse (data platforms)

Answer 25

metadata layer which is enhanced by delta lake which keeps a log of json files that keep track of versions over time, using checkpoints divided by timer periods so you don’t have to scan entire log to access a specific version

Question 26

data steward (data platforms)

Answer 26

enforces data governance policies

Question 27

3 key aspects to data management in platform (data platforms)

Answer 27

data integration
data quality
metadata

Question 29

active metadata (data platforms)

Answer 29

use open APIs to hook into every piece of data platform and get real time information about all the data so automating governance tasks

Question 30

data versioning significanc in pipelines (data platforms)

Answer 30

tracks changes to datasets, rolls back to earlier version of data if a trasnformation error occurs).

Question 31

two best practices for ensuring continous improvement in DataOps (data platforms)

Answer 31

feedback loops (always monitor data pipeline performance)
automated testing (catch issues early)

Question 32

5 charactersitics of cloud computing according to NIST (cloud computing)

Answer 32

1. on demand self service
2. broad network access
3. resource pooling
4. elasticity
5. measure service

Question 33

Define cloud migration (migration)

Answer 33

Moving data and business operations, applications etc from on premises to remote cloud provider server

Question 34

6 common migration strategies (migration)

Answer 34

1. Rehost “lift and shift”
2. Replatform “make some adjustments”
3. Repurchase “on prem CRF –> salesforce”
4. Refactor “rewrite applications to be cloud-native… AWS LAMBDA”
5. Retire “decommission outdates applications”
6. Retain “hybrid - keep some on”

Question 35

When is the rehost strategy (“lift and shift”) usesful (migration)

Answer 35

No time to redesign

Question 36

key steps in migration process (migration)

Answer 36

Assessment (review current system)
Planning (design cloud architecture)
Migration
Testing
Optimization

Question 37

How do cloud providers ensure data privacy and compliance during and after migration (migration)

Answer 37

AWS KMS (key management service) ensure data remains secure, AWS shield protects DDoS attack

encryption (SSL/TLS)

Question 38

What two things does a hybrid cloud solution blend? (migration)

Answer 38

Control (on premises) and scalability (c cloud)

Question 39

2 tools that monitor cloud costs? (MIGRATION)

Answer 39

AWS Cost Explorer
Azure Cost Management

Question 40

Example of pub/sub model real world (cloud streaming)

Answer 40

In a stock trading platform puiblishers (so the stock exchanges) send updates on stock prices and consumers (traders) receive only the messages related to stocks they are interested in

Question 41

3 common processing techniques applied to event streams (cloud streaming)

Answer 41

filtering
enriching (adding context)
aggregation

Question 42

What is AWS Kinesis (cloud streaming)

Answer 42

a cloud service that enables real-time data ingestion and processing. Divides streams into shards to be processed in parallel

Question 43

“priority queue pattern” (cloud streaming)

Answer 43

processes high-priority events faster

Question 44

pipes and filters pattern (cloud streaming)

Answer 44

breaks complex task into smaller pieces (filters), pipes connect them

Modular (scalable and reusable)

Question 45

real-time system security concern (cloud streaming)

Answer 45

unauthorized access (need policies)
data breaches (encyrption)

Question 46

real world example of company using streaming system (cloud streaming)

Answer 46

Netflix to monitor users activity and streaming quality, to quickly detect and resolve issues

Question 47

2 important industries that benefit from real-time analytics (cloud streaming)

Answer 47

Finance - stock price for time-sensitive decisions
Healthcare - detect emergencies and trigger intervention