Data Engineer Whiteboard Flashcards
What are the high-level whirteboard areas
Ask questions
Dataflow
Data Integrity/Quality
Monitoring and alerting
Testing - unit/load
Scalability
Infosec compliance
What questions should you ask?
Restate problem
Scope
Edge cases
Any tests?
Ask questions about the data
Summerize Data Engineering from data chaos to data insights
Data - raw data chaos
Analytics - data analytics unlocks patterns, tends, and correlations
Insights - Data insights. Actionable indights emerge, driving informed decision making
Wisdom - organizational wisdom . The origanization gains wisdom, leveraging insight for growth
what are the data engineering flowchart sections?
Source, ingestion, transformation, serving, analytics, ML, reverse ETL, storage and undercurents
What are the undercurrents?
Security, data management, data ops, data architecture, orchestration, software engineering
What are the data engineering best practices?
Proactive data monitoring, schema drift management, continuous documentation, data security measures, version control and backups
Summary of fundamentals of data engineering
Data engineering The discipline focuses on preparing “big data” for analytical or operational uses.
Use cases
Data engineering lifecycle
Data pipelines
Batch vs. stream processing
Data engineering best practices
Data engineering vs. artificial intelligence
What is data engineering?
The discipline focuses on preparing “big data” for analytical or operational uses.
What is the Data engineering lifecycle
The stages, from data ingestion to analytics, encompass integration, transformation, warehousing, and maintenance.
What are data pipelines?
A visual flow of the entire data engineering process, highlighting how data moves through each stage.
What is batch vs stream processing?
Distinguishing between processing data in large sets (batches) versus real-time (stream) processing.
What are the Data engineering best practices?
Established methods and strategies in data engineering to ensure data integrity, efficiency, and security.
What is Data engineering vs. artificial intelligence?
Differentiating the process of preparing data for AI applications from using AI to enhance data engineering tasks.
