AWS Messaging and Middleware Overview

AWS Middleware and Messaging Overview

> SQS, SNS, MQ, Kinesis, Pinpoint, IoT broker

 

1. AWS Messaging Services

AWS Messaging services enable similar and different software systems in the cloud to communicate and exchange information asynchronously. These Software systems can be incompatible and in different languages, and could be on different platforms.

 

2. Amazon SQS (Simple Queue Service)

Amazon SQS is a distributed queuing webservice for exchanging messages between software systems asynchronously. SQS is used to develop large scale distributed systems with decoupled components without worrying about creating and managing the queue.

 

SQS comes in two modes: Standard and FIFO (First-In-First-Out). General Use Case: One to one messaging, sending emails, form and file processing.

FIFO queue maintains the order of the messages that are produced and consumed providing a once-only delivery mechanism. Use Case: guaranteed order, once only delivery.

Standard queue attempts to preserve the order but is flexible, and it provides at least once delivery. The message in Standard queue can be delivered more than once and standard Queue offers high throughput. Use Case: Faster messaging, can be duplicates, messages sent out of order.

 

As the volume of your requests grows, Amazon SQS is scaled horizontally. Messages in SQS are encrypted with Server-side encryption and are secure during their lifetime in Queue.

There is a visibility timeout on the message of 30 sec by default and Max 12 hrs, to prevent other consumers from processing the same message again.

SQS Limitations:

• Message size is limited to 256 KB
• By default the messages in SQS will be deleted after 4 days, but this can be extended up to 14 days
• In Standard SQS, the message could be delivered more than once

SQS Best Practices:

• Depending on how long it takes to process a message, extend the message’s visibility timeout to the max time it takes to process and delete the message.
• If you don’t know how long it takes to process a message from SQS, create heartbeat for consumer process and extend the visibility timeout accordingly.
• Long polling helps reduce the Amazon SQS cost. While implementing long polling, use one thread per queue, instead of one thread for all the queues. Using one thread for all the queues might cause your application to become unable to process messages available in other queues.
• Use Unique message Ids to prevent the same message being processed by the consumer.

 

3. Amazon SNS (Simple Notification Service)

SNS is a fully managed Publish/subscribe asynchronous push notification system where a producer publishes a message and multiple subscribers receive messages. Pub/Sub messaging model is used to decouple applications and to increase the application’s performance, scalability and reliability.

SNS is best suited for use cases such as event notification, publishing message to multiple subscribers, sending dedicated message to each of the SQS Queue.

Messages pushed to SNS will be persisted across multiple availability zones to prevent loss. SNS sends a message with a subscription URL to all subscribers and the subscribers need to confirm the subscription to receive the message.

SNS can send messages to multiple services including SQS, Lambda, HTTP(s), Email, SMS etc.. Amazon SNS can be used in combination with SQS, so the message published to SNS will be sent to each SQS and consumer of each SQS will receive the message.

SNS Subscribers

Amazon SNS provides encrypted topics to protect your messages from unauthorized and anonymous access. When a message is published to the encrypted topic, the messages are immediately encrypted using AES-GCM algorithm and a Customer Master Key issued with Key Management Service.

The messages are decrypted as they are delivered to subscribing endpoints

Using SNS, a message can be send to a variety of platforms such as iOS, Android, Java, NodeJS, Python, .NET and other SDKs from AWS.

SNS Message:

Message pushed to SNS topic contain the following.

• MessageId – Unique Id of the message
• Timestamp – Message creation timestamp
• TopicArn – Topic to which the message is published
• Type – Type of delivery
• UnsubscribeUrl – Link to unsubscribe from this topic
• Message – Message Payload
• Subject – Subject field
• Signature – Base64 encoded signature of the message
• Signature version – Version of SNS signature

SNS Limitations

• The message from SNS may be delivered more than once
• In case of subscriber failure, message may be delivered for a configured number of times.

 

4. Amazon MQ

Amazon MQ is a managed messaging service for Apache ActiveMQ. AWS manages the provisioning, setup, patching, and maintenance of ActiveMQ. Existing applications can be easily migrated and connect to Amazon MQ using standard APIs and protocols such as JMS, NMS, AMQP, STOMP, MQTT and WebSocket.

 

Amazon MQ allows for high availability and durability of messages. Latency in Amazon MQ is minimal (single digit millisecond) between message producer and consumer. Messages placed in Amazon MQ are always stored redundantly across multiple Availability Zones. Amazon MQ supports distributed transactions. Broker name should be unique in an AWS account and there is a limit of 200GB storage capacity per broker.

 

A Lambda function can be triggered when a message is placed on Amazon MQ. In a hybrid cloud environment, an ActiveMQ in On-premises data centre can be integrated with Amazon MQ on the cloud to exchange messages. Refer to this documentation for connecting your Java application with Amazon MQ and also for creating message producer and consumer.

 

Amazon MQ Integration

Best Practices of Amazon MQ (from AWS documentation)

• Amazon MQ encrypts messages at rest and in transit using its securely managed encryption keys. It is highly recommended to use client encryption as well.
• To optimize Amazon MQ for slow consumers, set the concurrentStoreAndDispatchQueues attribute to false.
• Use right broker instance type for better throughput

 

In the case of multi-producer and multi-consumer applications, it can be highly inefficient to create a large number of connections on demand for multiple producers. Use connection pooling instead of using PooledConnectionFactory class. Don’t use PooledConnectionFactory on message consumers.

 

Amazon MQ Limitations

• Number of brokers per broker instance type, per AWS account is limited to 20
• Number of security groups limited to per broker is 5
• Storage capacity per broker is limited to 200GB

 

5. Amazon Kinesis

Amazon Kinesis is a fully managed web service for processing big data in real time. Kinesis is helpful in collecting, processing and analyzing real-time streaming data to get insights into the information. Using Kinesis the data can be analyzed as it arrives and respond instantly. Using Kinesis the Video, audio, application logs, website click streams, and IoT telemetry can be analyzed.

 

Kinesis is highly scalable and can handle any amount of streaming data and can process data from hundreds and thousands of sources with little latency. Kinesis is capable of processing terabytes of data per hour from various data streams. Kinesis keeps the order of the messages.

 

Kinesis Integration

Use cases include collecting and analyzing sensor data such as temperature, humidity sensors, or any IoT based data (cars, smart meters etc). It is also useful in ingesting all market data reliably and build real-time analytics.

Kinesis Limitations

• Data blob size is limited to 1 MB
• By default message is deleted from the stream after 24 hrs, but can be extended up to 7 days

 

6. Amazon Pinpoint

Amazon Pinpoint is a fully managed marketing and analytics service to drive insights from marketing and sales campaigns and determine how to effectively communicate with the customer base. Using Amazon Pinpoint, a marketer can define a campaign schedule to send targeted messages to app end users.

 

Amazon Pinpoint can be integrated with Mobile and Web applications to get insights into how the customers are interacting with your apps. It also helps in sending targeted email, SMS, push notifications, and voice messages and also track how the customers respond to your messages- i.e Number of messages delivered, opened or clicked. This is useful for sending promotional or educational content to engage and retain your users.

Some of the use cases where Amazon Pinpoint is useful are:

• Aggregation of data from all Campaigns
• Demographic charts that provide info on the devices on which your app is installed
• Analytics about user engagement and campaign metrics
• Analytics about application usage time and customer retention
• Revenue charts that provide details on customer purchases and revenue generated by it
• User authentication analytics

 

7. Amazon IoT Broker

AWS IoT message broker is a managed high throughput publisher/subscriber broker service that enables the sending and receiving of messages to and from AWS IoT. Client sends a message to AWS IoT to a topic. This message broker sends a message to all the registered clients for that topic. Using AWS IoT, the messages between all IoT devices can be transmitted securely with low latency.

 

Based on the message volume, the message broker can be scaled automatically. AWS IoT supports one-to-one messaging or one-to-million broadcast notification systems. More AWS IoT features are listed in AWS Documentation.

 

==END