GCP Data Engineer Quick Terms Flashcards
3 V’s of Big Data
1.) Volume – Scale of data being handled by systems (can it be handled by a single server?)2.) Velocity – speed in which its being processed3.) Variety – The diversity of data sources, formats, and quality
What is a Data Warehouse?
1.) Data Warehousea. Structured and/or processedb. Ready to usec. Rigid structures – hard to change, may not be the most up to date either
What is a data lake?
2.) Data Lakea. Raw and/or unstructuredb. Ready to analyze – more up to date but requires more advanced tools to queryc. Flexible – no structure is enforced
4 Stages of a Data Pipeline
1.) Ingestion2.) Storage3.) Processinga. ETL – Data is taken from a source, manipulated to fit the destinationb. ELT – data is loaded into a data lake and transformations can take place laterc. Common transformations: Formatting / Labeling / Filtering / Validating4.) Visualization
Cloud Storage
o Unstructured object storageo Regional, dual-region, or multi-regiono Standard, nearline, or cold lineo Storage event triggers (pub/sub) (Usually, first steps in a cloud data pipeline)
Cloud Bigtable
o Petabyte-scale NoSQL databaseo High-throughput and scalabilityo Wide column key/value data Time-series, transactional, IoT data
Cloud BigQuery
o Petabyte-scale analytics DWo Fast SQL queries across large datasetso Foundations for BI and AIo Useful public datasets
Cloud Spanner
o Global SQL-based relational databaseo Horizontal scalability and HAo Strong consistencyo Not cheap to run, usually used in financial transactions
Cloud SQL
o Managed MySQL, PostgreSQL, and SQL Server instanceso Built-in backups, replicas, and failovero Does not scale horizontally, but does scale vertically
Cloud Firestore
o Fully managed NoSQL document databaseo Large collections of small JSON documentso Realtime database with mobile SDKso Strong consistency
Cloud Memorystore
o Managed Redis instanceso In-memory DB, cache, or message brokero Bult-in HAo Vertically scalable with increasing the amount of RAM
Cloud Storage (GCS) at a high level
o Fully managed object storageo For unstructured data: Images, videos, etc.o Access via API or programmatic SDKo Multiple storage classeso Instant access in all classes, also has lifecycle managemento Secure and durable (HA and maximum durability)
GCS Concepts, what is GCS, where can buckets be?
o A bucket is a logical container for an objecto Buckets exist within projects (named within a global namespace)o Buckets can be:o Regional $ o Dual-regional $$ (HA)o Multi-regional $$$ (Uses all datacenters in a region, lowest latency as well)
4 GCS Storage Classes
o Standardo Nearlineo Coldline o Archive
Describe standard GCS Storage Class
$0.02 per GB99.99% regional availability>99.99% availability in multi and dual-regions
Describe Nearline GCS Storage Class
30 day minimum storage$0.01 per GB up / down99.9% regional availability99.95% availability in multi and dual regions
Describe Coldline GCS Storage Class
90 days minimum storage$.004 per GB stored$02 up/down99.9% regional availability99.95% availability in multi and dual region
Describe Archive GCS Storage Class
365 minimum storage$.0012 per GB stored$0.05 per GB up/down99.9% regional availability99.95% availability in multi and dual regions
Objects in cloud storage (encryption, changes)
o Encrypted in flight and at resto Objects are immutable to change you must overwrite (atomic operation)o Objects can be versioned
name the 5 “advanced” features of GCS
o Parallel uploads of a single objecto Integrity checking – pre-calculate an md5 hash, compared to the one Google calculateso Transcoding for compressiono Requester can pay, if desiredo Pub/Sub notifications New files are commits that trigger a data pipeline
What is Cloud Transfer Service?
o Transfers from a source to a sink (bucket), supported sources: S3, HTTP, GCP Storageo Transfers can be filtered based on names/dates Schedule it for one run or periodically (can delete in source or destination after transfer is confirmed)
What is BigQuery Data Transfer Service?
o Automates data transfer to BigQueryo Data is loaded on a regular basiso Backfill can recover from gaps or outageso Supported sources: Cloud Storage, Merchant Center, Google Play, S3, Teradata, Redshift
What is a transfer appliance?
physical rack storage device, 100 TB and 480 TB versions
What are the top 3 features of Cloud SQL?
o Managed SQL instances (creation, replication, backups, patches, updates)o Multiple DB engines (MySQL, PostgreSQL, SQL Server)o Scalability – vertically to 64 cores and 416 GB of RAM, HA options are available
Describe regional configuration of Cloud SQL
o Regional Replication: 3 read-write replicas Every mutation requires a write quorumThis is different from traditional HA in that it’s a read AND a write replica in each zone.
Regional Cloud SQL best practices
o Design a performant schemao Spread reads/writes around the database, avoid write hot spotso Co-locate compute workloads in the same regiono Provision nodes to keep average CPU utilization under 65%
Multi-regional Cloud SQL benefits
5 9s SLA – 99.9999% Reduce latency with distributed data External consistency• Concurrency control for transactions, this guarantees transactions are executed sequentially even across the globe
Multi-regional Cloud SQL best practices:
Design a performant schema to avoid hotspots Co-locate write-heavy compute workloads in the same region as the leader Spread critical workloads cross two regions Provision nodes to keep average CPU under 45%
Cloud SQL data model
• Data model:o Relational database tableso Strongly typed (you must conform to a strict schema)o Parent-child relationships, declared with primary keys and create an interleaved table
Cloud SQL transactions
• Transactions:o Locking read-writeo Read-Onlyo Partitioned DMLo Regular transactions using ANSI SQL best practices
Top 2 features of Cloud MemoryStore
o Fully managed Redis instanceo 2 tiers: Basic tier – make sure your app can withstand full data flush Standard tier – adds cross zone replication and automatic failover
Benefits of managed Redis (MemoryStore)
o No need to provision VMso Scale instances with minimal impacto Private IPs and IAMo Automatic replication and failover
Cloud MemoryStore use cases
o Session cache – store logins or shopping cartso Message queue – loosely couple micro serviceso Pub/sub – message queue, but also look at pub/sub
Storage options: Low latency vs. Warehouse
Low latency (use Cloud Bigtable)– petabyte scale– single-key rows– Time series or IoT data Warehouse (use BigQuery)- Petabyte scale- Analytics warehouse- SQL queries
Storage options: Horizontal vs. Vertical scaling
Horizontal scaling (Cloud Spanner)- ANSI SQL- Global replication- High Availability and consistencyVertical Scaling (use Cloud SQL)- MySQL or PostgreSQL- Managed service- High availability
Storage options: NoSQL vs Key/Value
NoSQL- Fully managed document database- Strong consistency- Mobile SDKs and offline dataKey/Value- Managed Redis instances- Does what Redis does
What is MapReduce
distributed implementation of the map and reduce programming model, common interface to program these operations while abstracting away all of the systems management
4 Core modules of Hadoop & HDFS
Hadoop Common – base files Hadoop Distributed File System (HDFS) – distributed fault tolerant file system Hadoop YARN – resource management / job scheduling Hadoop MapReduce – their own implementation
Apache Pig
language for analyzing large datasets, essentially an abstraction for MapReduceo High level framework for running MapReduce jobs on Hadoop clusters
Apache Spark
general purpose cluster-computing frameworko Pretty much replaced Hadoop, uses Hadoop, much faster Hadoop stores data in blocks on disk before, during, and after computation Spark stores data in memory, enabling parallel operations on that data
Hadoop vs. Spark
Hadoop- Slow disk storage- High latency- Used for: slow, reliable batch processingSpark- Fast memory storage- Low latency- Stream processing- 100x faster in-memory- 10x faster on disk
Apache Kafka
distributed streaming platform, designed for high-throughput and low-latency pub/sub stream of recordso Handles >800 billion messages per day at LinkedIn
Kafka vs. Pub/Sub
Kafka- Guaranteed message ordering- Tuneable message retention- Polling (Pull) subscriptions only- UnmanagedPub/Sub- No message ordering guarantee- 7 day maximum message retention- Pull or Push subscriptions- Managed
Top 2 Benefits of Pub/Sub
o Global messaging and event ingestiono Serverless and fully managed, processes up to 500 million messages per second
Top 4 features of Pub/Sub
o Multiple pub/sub patterns, one to many, many to one, and many to manyo At least once delivery is guaranteedo Can process messages in real-time or batch with exponential backoffo Integrates with Cloud Dataflow
Pub/Sub use cases
o Distributing workloadso Asynchronous workflows – order processing, order, packaging, shippingo Distributing Event Notificationso Distributed Loggingo Device Data Streaming
2 types of delivery method for pub/sub subscriptions:
o Pull (default) – ad-hoc request, messages must be acknowledged, or they will remain at the top of the queue and you won’t get the next messageo Push – will send new messages to an endpoint, must be HTTS with a valid cert
Pub/Sub integration facts
o Fully supported by Cloud Dataflowo Client libraries for popular languages (python)o Cloud Functions can be triggered by eventso Cloud Run to be the receiver of a push subo IoT Core
Pub/Sub delivery model
o You may receive a message more than once in a single subscriptiono Message Retention Duration (default 7 days) – undelivered messages are deleted
Pub/Sub lifecycle – when does a sub expire?
if No pulls / no pushes – subs expire after 31 days
Standard pub/sub model limitations
o Acknowledged messages are no longer available to subscriberso Every message must be processed by a subscription
Pub/Sub: Seek
o Seek – you can rewind the clock and retrieve old messages up to the retention window, you can also use this to seek to a point in the future Useful in case of an outage most commonly
Pub/Sub: Snapshot
o Snapshot – save the current state of the queue, this enables replay Useful if you have new code and you’re not sure what it’ll do, you can snapshot back to a certain point and move forward, then seek back to replay
Pub/Sub: Ordering messages
o Ordering messages – Use timestamps when final order matters, order still isn’t guaranteed but you do have a record of that time. If you absolutely must guarantee order, consider an alternative system
Pub/Sub: Access Control
o Use service accts for authorization, granting per-topic or per-subscription permissions Grant limited access to publish or consume messages
Define: Cloud Dataflow
• Cloud Dataflow – fully managed, serverless ETL tool, using Apache Beam.o Supports: SQL, Java, and Python Real-time and batch processing
Define: Pipeline Lifecycle
• You can run pipelines on your local machine, this is the preferred way to fix bugs• Pipeline design considerations:o Location of datao Input data structure and formato Transformation objectiveso Output data structure and location
Dataflow: ParDo
• ParDo – defines the distributed operation/transformation to be performed on the PCollection of data. These can be user defined functions or pre-defined.
Dataflow: PCollections (Characteristics)
o Data types – may be of any data type but must all be of the same type. The SDK includes built-in encoding o Access – individual access to elements is not supported, transforms are performed on allo Immutable – cannot be changed once createdo Boundedness – no limit to the number of elements that a PCollection can containo Timestamp – associated with every element of the collection, assigned by the creation
Dataflow: Core Beam transforms (6)
o ParDo – generic parallel processing transformso GroupByKey – processing collections of KVP’so CoGroupByKey – used when combining multiple key collections, performs relational joino Combine – requires you to provide a function to provide the logic, multiple pre-built functions are available, sum, min, max…o Flatten – Multiple collections become oneo Partition – how the elements of the PCollection are split up
Dataflow Security Mechanisms
o Only users with permission can submit pipelineso Any temp data during execution is encryptedo Any communication between workers happens on a private networko Access to telemetry or metrics is controlled by project permissions
Describe GCP Service Account usage
Cloud Dataflow service uses the Dataflow Service Account Account is automatically created on flow creation Manipulates job resources on your behalf Assumes the “Cloud Dataflow service agent role” Read/write access to project resources (recommended not to change this)Worker instances will use the Controller service account Used for metadata operations (ex: determine size of file on storage) Also, able to use user-managed controller service acct, enabling fine grained access-control
Dataflow: Regional Endpoints
• Regional Endpoints – specifying a regional endpoint means all the worker instances will persist in that region, this is best for:o Security and complianceo Data localityo Resiliency
What use case does Dataflow address
o Used for migrating MapReduce jobs to Cloud Dataflow
Define: Cloud Dataflow SQL
o Develop and run Cloud Dataflow jobs from the BigQuery web UI