Connecting Your Services Together Flashcards
What is a message?
- Message contains raw data
- A message contains the data itself, not a reference
- The sending component expects the destination component to process the mesage in a certain way. The integrity of the overall system may depend on both the send and reciever doing a specifc job
What are events?
- An event is a lightweight notification that indicates that something happened.
- Event can be sent to multiple receivers (or none at all)
- Events a often intended to “fan out” or have a large number of subscribers for each publisher
- The publisher of the event has no expectation about the action of receiving component takes
- Some events are discrete units and unrealted to other events
- Some events are part of a related and ordered series
What is one of the main consideration in choosing between messages or events?
Does the sending ocmponent expect the communication to be processed in a particular way by the destination component?
What are the two Azure services that offer messaging services?
- Azure Storage Queues
- Azure Service Bus
What is an Azure Queue Storage?
is a service to store large number of messages that can securely accessed from anywhere in the world using a simple REST-based interfaces
They can contain millions of messages, limited only by the capacity of the storage account that owns it.
What is an Azure Service Bus
is a message broker system intended for enterprise applications.
Apps that use this service may have multiple communication protocols, data contracts, higher security requirements and can include both cloud and on premise services.
Service Bus is built on top of the dedicated messaging infrastructure designed for these scenarios.
What are Azure Service Bus Topics?
They’re like queues but can have multiple subscribers.
i.e. A topic may be “Listen” for a music app. There are two subscribers, ListenHistory and AddToFavorites. Each subscriber receives its own copy of the message to process.
What are the benefits of queues
Increase reliability
* message can stay in queue until the conmponent is available to process
Message delivery guarentees
* At-least-once delivery => delivered at least one. Keep in mind if you have more than one instance of your application, each instance may get the message
* At-Most-Once-Delivery => each message IS NOT guaranteed for delivery. Very small chance it may never. But there is NO chance it will be delivered twice (Referred to automatic duplicate detection)
* First-in-First-Out => If you need messages to be processed in order, select this option
Transactional Support
* Group messages in a transaction (i.e. Shopping Cart, Charge Account, Shipping)
What to choose between Service Bus Topics vs Service Bus Queues
Topics
* Multiple receievers
Queues
* Need an At-Most-Once delivery
* FIFO
* Transactional Support
* Receive message w/o polling
** Need to handle messages larger thatn 64 kb but less than 100 mb. (standard tier message is limited to 256 kb)
Queue size will not exceed 1 TB*
* Want to publish and consume batches of messages or multiple destinations
Another part of the Microsoft documentation says 80 GB for queue max size
Use Queue Storage if
- Need an audi trail of all messages passing through queue
- Expect queue to exceed 1 TB
- Want to track progress for processing a message inside a queue
Explain a good scenario for Event Grid
Someone upload a new song to Azure. A message is then sent to all subscribers subscribing to the Topic (i.e. Genre)
What is Event Grid?
is a fully managed event routing service (running on top of the Azure Service Fabric).
It provides a dynamically scalable, low-cost messaging system that allow publishers to notify subscribers about a status change
What are the 5 components of Event Grid?
- Events
- Event Sources
- Topics => the endpont where publishers send events
- Event Subscriptions => The endpoint or built-in mechanism to route events. Subscriptions are also used by handlers to filter incoming events intelligently
- Event Handlers
What is the max event/message size for Event Grid
64 kb
What are some of the schema fields for an Event Grid message?
- topic
- subject
- id
- eventType
- eventTime
- data {}
- dataVersion
- metadataVersion
What are System Topics?
They are built-in from Azure. You cannot see them in your subscription but you can subscribe to them.
What are custom topics?
Custom topics are application and third-party topics. When you create or are assigned access to a custom topic, you see that custom topic in your subscription.
What is an event subscription?
Event Subscriptions define which events on a topic an event handler wants to receive. A subscription can also filter events by their type or subject, so you can ensure an event handler only receives relevant even
What are some of the event sources for Event Grid?
- Azure Subscriptions and Resource Groups
- Container Registry
- Event Hubs
- Service Bus
- Storage Accounts
- Media Services
- Azure IoT Hub
What are some of the Event Handlers for Event Grid?
- Azure Functions
- Azure Logic Apps
- Webhooks
- Event Hubs
- Service Bus
- Storage Queues
- Microsoft Power Automate
Why you should use Event Grid
- Simplicity
- Advanced Filtering => Subscriptions have close control over the vents they receive from a topic
- Fan-out => you can subscribe to an unlimited number of endpoints to the same events and topics
- Reliability => retrieves event delivery for up to 24 hours for each subscription
- Pay-per-event => Pay for only the event that you transmit
What are Event Hubs?
Similar to Event Grid, however it is optimized for extremely high thoughput, a large number of publishers, security, and resilency
Same publisher/subscriber pattern
What are partitions in Event Hubs?
When an Event Hub receives communications, it divides them into partitions.
Subscribers can always use buffers to “catch up”. By default events stay in the buffer for 24 hours before expiring. The buffers are called partitions because the data is deviced amounst them. Each partition has a sperate set of subscribers.
Note, an event message can be set to a specific partition
What is Capture in Event Hubs?
Event Hub can send all your events immediately to Data Lake or Azure Blob Storage for inexpensive, permanent persistence
Authentication in Event Hubs
All publishers are authenticated and issued a token. This means Event Hubs can accept events from external devices and mobile apps, without worrying that fraudulent data from prankers could ruin our analysis.
Choose Event Hubs if
- You need to support authenticating a large number of publisers
- You need to save a stream of events to Data Lake or Azure Blob Storage
- You need aggregation or analytics on your event system
- You need reliabile messaging or resiliency
Can handle millions of events ver second with low latency. Great for real time analytics
Choose Service Bus over Storage Queue if
- You need an at-most-once delivery guarantee.
- You need a FIFO guarantee (if no other settings preempt the default FIFO order)
- You need to group messages into transactions.
- You want to receive messages without polling the queue.
- You need to provide role-based access to the queues.
- You need to handle messages larger than 64 KB but smaller than 256 KB for the standard tier or 100 MB for the premium tier.
- Your queue size won’t grow larger than 80 GB.
- You’d like to be able to publish and consume batches of messages.
Choose Service bus too if you want to send with topics
Choose Storage Queue over Service Bus if
- You need a simple queue with no particular additional requirements.
- You need an audit trail of all messages that pass through the queue.
- You expect the queue to exceed 80 GB in size.
- You want to track progress for processing a message inside the queue.
Example of sending a message to a Service Bus Client
// Create a ServiceBusClient object using the connection string to the namespace.
await using var client = new ServiceBusClient(connectionString);
// Create a ServiceBusSender object by invoking the CreateSender method on the ServiceBusClient object, and specifying the queue name.
ServiceBusSender sender = client.CreateSender(queueName);
// Create a new message to send to the queue.
string messageContent = “Order new crankshaft for eBike.”;
var message = new ServiceBusMessage(messageContent);
// Send the message to the queue.
await sender.SendMessageAsync(message);
To receive a message from Service Bus Client
// Create a ServiceBusProcessor for the queue.
await using ServiceBusProcessor processor = client.CreateProcessor(queueName, options);
// Specify handler methods for messages and errors.
processor.ProcessMessageAsync += MessageHandler;
processor.ProcessErrorAsync += ErrorHandler;
await args.CompleteMessageAsync(args.Message);
Azure CLI to check messages in a queue
az servicebus queue show \
–resource-group [sandbox resource group name] \
–name salesmessages \
–query messageCount \
–namespace-name <namespace-name></namespace-name>
What filters are available for Subscription Queue Topics?
- Boolean (True means send all messages for the subscription and false means no messages are to be delivered for the subscription, basically disabling messages)
- SQL Filters (Uses the SQL where clause and if true, delivers the messages)
- Correlation (Holds a set of conditions that are matched again the properties of each message, if there is a match, the messages are delivered)
Example of sending a topic
// Create the client
await using var client = new ServiceBusClient(connectionString);
// Create sender and specifiy the topic
ServiceBusSender sender = client.CreateSender(topicName);
string message = “Cancel! I have changed my mind!”;
var message = new ServiceBusMessage(message);
// Send the message to the topic.
await sender.SendMessageAsync(message);
To receive a message from a subscription
processor = client.CreateProcessor(topicName, subscriptionName, options);
// Specify the handler method for messages.
processor.ProcessMessageAsync += MessageHandler;
// Specify the handler method for errors.
processor.ProcessErrorAsync += ErrorHandler;
// Complete the message. The message is deleted from the subscription.
await args.CompleteMessageAsync(args.Message);
What is the maximum storage size for an Azure Storage Queue
500 TB
Need to verify this. I believe for Azure Service Bus is 80 GB and if you need more, they recommended Azure Queue Storage.
How many messages per second can an Azure Storage Queue handle?
2000
What is the maximum size for an mesage in Azure Service Bus
100 MB
What is the maximum size for a message in Azure Storage Queue
64 KB
What formats does Azure Storage Queue accept
JSON, XML, a simple string
Are Azure Storage Queues available in what storage accounts?
V1 and V2
Blob Storage accounts only offer Blobs
Some best practices for Azure Storage Queues
- Should choose the same location as your source or destination components
- Choose the approperiate replication model
- Consider Access tiers (which only apply to blobs) and standard or premium accounts. Premium uses SSD
- Require secure transfer if sensitive infomration may pass through a queue
Azure CLI for creating a storage account
az storage account create
–name
-g
–kind (i.e. StorageV2
–sku (i.e. Standard_LRS)
Authorization types in Azure Queues
- Azure Active Directory
- Shared Key (or account key)
- SAS
ACI to display a connection string for a storage account
az storage account show-connection-string
-g
-n
–output tsv
Example of sending a Queue Message
QueueClient queueClient = new QueueClient(connectionString, queueName);
Response<SendReceipt> response = await queueClient.SendMessageAsync("This is a message");</SendReceipt>
string messageJson = JsonSerializer.Serialize(objectData);
Response<SendReceipt> response = await queueClient.SendMessageAsync(messageJson);</SendReceipt>
Example on how to receive a message
Response<QueueMessage> response = await queueClient.ReceiveMessageAsync();</QueueMessage>
QueueMessage message = response.Value;
Console.WriteLine($”Message id : {message.MessageId}”);
Console.WriteLine($”Inserted on : {message.InsertedOn}”);
Console.WriteLine($”Message (raw) : {message.Body}”);
Event Hubs
What is Azure Event Hubs
Azure Event Hubs is a cloud-based, event-processing service that can receive and process millions of events per second. Event Hubs acts as a front door for an event pipeline, to receive incoming data and stores this data until processing resources are available.
Event Hubs
What is Azure Event Hubs
Azure Event Hubs is a cloud-based, event-processing service that can receive and process millions of events per second. Event Hubs acts as a front door for an event pipeline, to receive incoming data and stores this data until processing resources are available.
What is the maximum size for a Event Hubs messages
1 MB
What would be a scenario that is best for using AMQP (Advanced Message Queuing Protocol) with Azure Event Hubs
For publishers that send data frequently, AMPQ is better performance
(however higher initial session overhead because of a persistence bidirectinoaly socket and TLS or SSL/TLS has to be set up first)
What would be a scenario that is best for using HTTP with Azure Event Hubs
More intermittent publishing, HTTPS is better
(requires more overhead with each request but there is no initial session overhead)
Situation where to use Apache Kafka in Event Hubs
With endpoints compatible with Apache Kafka
What are consumer groups?
An event hub consumer group represents a specific view of an event hub data stream
What are the pricing tiers for Event Hubs?
- Basic
- Standard
- Premium
- Dedicated
Max Retention Period in Days: 1, 7, 90, 90
Storage Retention: 84 GB, 84 GB, 1 TB per PU, 10 TB, per CU
Can purchase more storage retention for Premium and Dedicated
Azure CLI Commands to create an Event Hub
az eventhubs namespace create
–name
// Create the event hub
az eventhubs eventhub create
–name
–namespace
Testing resilency in Event Hubs
Messages are still delivered to subscribers even after the Event Hub goes down. You can test this by temporarly disabling the Event Hub in the portal, then re-enabled. The receiving app should receive messages again
You can then check the following metric
* Throttled Requests => The number of throttled request because the throughput exceeded unit usage
* Active Connections
* Incoming/Outgoing
What is an Event Grid?
Event Grid aggregates all your events and provides routing from any source to any destination. Event Grid is a service that manages event routing and delivery from many sources and subscribers. This process eliminates the need for polling and results in minimized cost and latency.
What is the capabilities of Event Grids?
- It’s simple
- It can filter events
- It supports multiple subscribers
- It’s reliabile
- Its throughput is high
- It has built-in events
- Is supports custom events
What are some of the Event Handlers for Event Grid?
- Azure Functions
- Azure Logic Apps
- Azure Automation
- Azure Event Hubs
- Azure Service Bus
By default, how many partitnios will a new Event Hub have?
4
What is the maximum size for a single publication (individual or batch) allowed by Azure Event Hubs?
256 KB
Diagram for Azure Event Hubs
High level overview of Event Hubs, Event Grids, and Azure Service Bus
Using the Event Hub and Event Grid together
