App Integration Services Flashcards
Azure Queue Storage
Azure Queue Storage is a service offered as part of Azure Storage, designed to provide a cloud-based messaging system. It enables applications to communicate with each other asynchronously by using message queues. This is particularly useful for decoupling components of a distributed application, improving scalability, availability, and resilience.
Key Features
-Basic poll/pull messasing via HTTP with at least once delivery guarantee
-Simple message queuing service using REST-based protocol only
-Supports 64 KB messages, with unliomited queues/clients, up to 500TB queue size
-Pricing based on the number of operations, data stored, data transferred
Resource Hierarchy
1- Publisher/Receiver: Apps send/receive messages /XQM, JSON, TXT, Binary)
2- Storage Account: Built on top of Azure Storage. Can contain multiple queues/services
3- Queue: Endpoint for4 messages to be published to and received (poll/pull)
Azure Service Bus
Azure Service Bus is a fully managed enterprise messaging service provided by Microsoft Azure. It is designed for complex messaging scenarios where reliable and asynchronous communication is required between applications or services, making it suitable for both cloud and hybrid environments. Azure Service Bus enables integration across various systems, even when they are developed using different technologies or platforms.
Key Features
-Advanced push messaging with FIFO, at most once, at least once guarantees
-Supports sessions, dead lettering, duplicate detection, and more
-Maximum message size of 256KB/100MB, queue size of 1GB/80GB, 10.000 queues
-Different pricing tiers available. Pay for operations, queue size, active queues
Pricing
-Basic (Shared Capacity): Max 256KB message, Supports queues only, variable pricing.
-Standard (Shared Capacity): Max 256KB message, Queues and Topics, variable pricing.-
-Premium (Dedicated Capacity): Max 1MB message, Queues, Topics, zone redundancy, Predictable pricing and performance.
Resource Hierarchy
1- Publisher/Receiver: Apps send/receive messages (XML, JSON, TXT, Binary) with AMQP/HTTPS/WebSock.
2- Namespace: Container for service bus. Includes pricing, security, redundancy properties.
3: Queue: FIFO message delivery queue. Supports sessions, partitioning, de-dupe, more.
Azure Queue Storage VS Azure Service Bus
Message Delivery
-Azure Service Bus: Guaranteed with at-least-once and FIFO.
-Azure Queue Storage: At-least-once, but no strict FIFO.
Messaging Pattern
-Azure Service Bus: Supports pub-sub with topics.
-Azure Queue Storage: Queue-based (one-to-one) messaging.
Message Size Limit
-Azure Service Bus: Up to 1 MB (standard) or 100 MB (premium).
-Azure Queue Storage: Up to 64 KB.
Advanced Features
-Azure Service Bus: Sessions, Transactions, Dead-lettering.
-Azure Queue Storage: Basic queue functionality.
Use Case Complexity
-Azure Service Bus: Complex enterprise scenarios.
-Azure Queue Storage: Simple message queuing.
Azure Event Grid
Azure Event Grid is a fully managed event routing service provided by Microsoft Azure. It is designed to facilitate the handling of events in a distributed architecture by allowing seamless integration between event publishers and event handlers (subscribers). Event Grid provides reliable, scalable, and low-latency event delivery for real-time applications.
Key Features
-Enables event-based communication between services, promoting decoupling and scalability.
-Custom event sources via HTTP POST requests.
-Guarantees at-least-once delivery of events with built-in retries and dead-lettering for undeliverable events.
-Supports millions of events per second with low latency.
Pricing
-Basic (Integration): Supports HTTP/S, User publisher/subscripber push model, Pricing based on number of operations performed
-Standard (IoT/Streaming): Adds MQTT support, Uses publisher/subscriber pull model, Pricing based on throughput and operations
Event Grid Topics
Event Publisher:
-Azure or custom services that send events.
-Examples: Azure services like Blob Storage, Resource Groups, or custom applications emitting events.
-Publishers are the sources of events that feed into Event Grid for further processing.
Event Handler:
-Azure or custom services that receive and act on events.
-Examples: Azure Functions, Logic Apps, Event Hubs, or any webhook endpoint.
-Event handlers are responsible for processing the events and triggering workflows or actions.
Topics (System/Custom):
-A collection of related events published by a service or custom application.
-System Topics: Automatically created by Azure for events from Azure services (e.g., Blob Storage).
-Custom Topics: Defined by users to group events from their applications or services.
-Topics help organize and categorize events for better management.
Event Subscription:
-A connection between an event handler and a topic.
-Specifies how the event handler will receive events from the topic, including filtering and delivery configurations.
-Subscriptions ensure that events are delivered reliably and can include advanced filtering to ensure only relevant events are sent to a handler.
Azure Event Hubs
Azure Event Hubs is a fully managed, big data streaming platform and event ingestion service provided by Microsoft Azure. It is designed to handle large volumes of real-time data streams, making it a central component for data ingestion and processing pipelines.
Key Features
-Ingest millions of events per secord from any source in near-realtime
-Supports AMQP, HTTPS, Apache Kafka, and more, to integrate with various platforms
-Enables real-time / micro-batch processing, with optimal capture to Azure Storage
-Different pricing tiers available for different features/limits
Pricing - Event Hub Overview
-Basic (Multi-tenant): Pay per million events.
-Standard (Multi-tenant): Pay per million events, Adds private link, firewall, geo-disaster recovery, and pay-for-Capture.
-Premium (Multi-tenant*): Includes ingress events. Adds partition scale-out, audit logs, and included Capture pricing.
-Dedicated (Single-tenant): Includes ingress events. Includes all features. Exclusive single-tenant offering.
Architecture - Key Components
Event Hub Namespace:
-A management and pricing unit for Event Hubs.
-Acts as a container for multiple Event Hubs and is used for grouping related hubs under the same billing and configuration.
Event Hub:
-The core component that represents an event data stream.
-Supports multiple protocols, such as AMQP, Kafka, and HTTPS, for sending and receiving events.
-Serves as the entry point for data ingestion from various sources.
Partition:
-A logical division of an Event Hub for parallel processing.
-Events are distributed across partitions to enable scalability.
-Each partition is processed independently, ensuring high throughput and fault tolerance.
Consumer Group:
-A “view” of the Event Hub that allows multiple consumers to independently process events.
-Each consumer group maintains its own offset (progress), enabling scenarios like analytics, monitoring, and logging to run concurrently without affecting each other.
Azure Event Hubs vs. Azure Event Grid
Purpose:
Azure Event Hubs: Big data streaming and event ingestion.
Azure Event Grid: Real-time event routing and pub-sub messaging.
Delivery Model:
Azure Event Hubs: Pull-based (consumer fetches events).
Azure Event Grid: Push-based (events sent to subscribers).
Retention:
Azure Event Hubs: Configurable (up to 7 days).
Azure Event Grid: No retention; events delivered immediately.
Event Size Limit:
Azure Event Hubs: Up to 1 MB per event.
Azure Event Grid: Up to 1 MB per event.
Use Cases:
Azure Event Hubs: IoT telemetry, big data pipelines, real-time analytics.
Azure Event Grid: Real-time notifications, serverless automation.
Integration:
Azure Event Hubs: Works with Stream Analytics, Synapse, Kafka.
Azure Event Grid: Deep integration with Azure services like Functions, Logic Apps, Blob Storage.
Parallel Processing:
Azure Event Hubs: Supports partitioning for parallel consumer processing.
Azure Event Grid: No partitioning; events routed individually.
Filtering:
Azure Event Hubs: No filtering; all events consumed by clients.
Azure Event Grid: Advanced filtering based on event metadata.
Key Takeaways:
Azure Event Hubs: Best for high-throughput data ingestion, big data pipelines, and IoT scenarios.
Azure Event Grid: Ideal for lightweight, real-time event routing and automating serverless workflows.
Azure API Management
Azure API Management (APIM) is a fully managed service provided by Microsoft Azure that allows organizations to create, publish, secure, monitor, and manage APIs. It acts as a gateway to protect backend services and enables efficient communication between clients and APIs in a secure and scalable manner.
Key Features:
-The ability to provide easy access to API documentation for your customers of your API
-The API Gateway Service that includes a range of other features.
Pricing
Basic v2 (SLA %99,95): Designed for development and testing. Optional Developer Portal.
Standard v2 (SLA %99,95): Production-ready with advanced features. Includes Developer Portal. Supports outbound network to isolated backends.
Architecture - Key Components
APIM Resource:
-The parent resource in Azure API Management.
-Determines critical aspects such as URI, pricing tier, identity, scalability, and networking configurations.
-It acts as the foundation for managing all API-related activities within a specific instance.
Developer Portal:
-A fully customizable and self-service website for API consumers.
-Provides API documentation, allows developers to explore and test APIs, and offers a way to onboard new API users easily.
-Can be branded and customized to match organizational requirements.
API Gateway:
-The centralized entry point for all API traffic.
-Responsible for ensuring security, managing routing, applying policies (e.g., rate limiting, caching), and abstracting backend services from direct access.
-Protects backend resources by managing authentication and controlling access.
Management Plane:
-A toolset for administrators to manage and configure the APIM resource.
-Used for defining policies, monitoring API usage, managing security, and configuring settings like user roles and notifications.
-Accessible via Azure Portal, CLI, or APIs.
API Gateway - Features
Rate Limits & Quotas:
-Limit requests per second to the API and enforce quotas.
Security Validation:
-Verify credentials at the gateway (keys, JWT tokens, certificates, etc.).
Caching:
-Store responses to reduce latency and backend load.
Transform:
-Modify both requests and responses with policy statements.
Self-Hosting:
-Deploy the API Gateway in any environment using a Docker container.
Monitoring:
-Collect and analyze logs, metrics, and API traces.
Azure Logic Apps
Azure Logic Apps is a cloud-based platform provided by Microsoft Azure that enables users to automate workflows and integrate applications, data, services, and systems without writing complex code. It is a low-code/no-code solution, ideal for business process automation, system integration, and data orchestration.
Key Features
-GUI Workflow Editor: Easily edit simple to complex workflows with triggers, connectors, and actions.
-Variety of Connectors: Supports a large number of connectors to access internal and external services.
Pricing
Consumption (Multi-tenant)
-Charged per execution of triggers and actions.
-Does not support VNet integration.
-Develop one workflow.
Standard (Single-tenant)
-Charged per hour based on the plan selected.
-Supports VNet integration and private endpoints.
-Develop many workflows.
Architecture - Key Components
Logic App Resource:
Parent resource. Determines pricing, app settings, connection strings, and more.
Workflow:
Use the GUI designer (or edit the JSON code) to create a workflow.
Triggers and Actions:
The first step to commence your workflow, such as a schedule.
Connectors:
Use built-in, managed, or custom connectors to access services.
