Enterprise Java

AWS Messaging Services: Choosing the Right Service

1. AWS Messaging Services

AWS Messaging services enable similar and different software systems in the cloud to communicate and exchange information asynchronously. These Software systems can be incompatible and in different languages, and also could be on different platforms. AWS messaging services are highly available, highly scalable and highly reliable. AWS Supports multiple messaging services for different types of use cases. In this Article, we’ll discuss various AWS messaging services and picking the right messaging service for depending on the application requirements.

Messaging services allow different software systems, often incompatible and developed in different programming languages on different platforms to communicate and exchange information.

2. Amazon SQS (Simple Queue Service)

Amazon SQS is a distributed queuing webservice for exchanging messages between software systems asynchronously. SQS is used to develop large scale distributed systems with decoupled components without worrying about creating and managing the queue. SQS comes in two flavors, Standard and FIFO (First-In-First-Out). FIFO queue maintains the order of the messages that are produced and consumed and it provides exactly-once delivery mechanism. Standard queue attempts to preserve the order but is flexible, and it provides at least once delivery. The message in Standard queue can be delivered more than once and standard Queue offers high throughput.

As the volume of your requests grows, Amazon SQS is scaled horizontally.
Messages in SQS are encrypted with Server-side encryption and are secure during
their lifetime in Queue.

If your application needs faster messages processing, but don’t care about the message order or duplicate delivery of the message, Standard queue is the best option. For guaranteed order and exactly once delivery, use FIFO queue. When a consumer receives the message and processes it, the message remains in the queue until they are removed explicitly or message retention time is expired. Amazon SQS sets visibility timeout on message (30 sec by default and Max 12 hrs) to prevent other consumers from processing the same message again.

SQS is best useful in cases of one to one messaging, sending emails, form and file processing.

AWS Messaging Services - SQS Integration
SQS Integration

2.1 SQS Limitations

  • Message size is limited to 256 KB
  • By default the messages in SQS will be deleted after 4 days, but this can be extended up to 14 days
  • In Standard SQS, the message could be delivered mre than once

2.2 SQS Best Practices

  • Depending on how long it takes to process a message, extend the message’s visibility timeout to the max time it takes to process and delete the message.
  • If you don’t know how long it takes to process a message from SQS, create heartbeat for consumer process and extend the visibility timeout accordingly.
  • Long polling helps reduce the Amazon SQS cost. While implementing long polling, use one thread per queue, instead of one thread for all the queues. Using one thread for all the queues might cause your application to become unable to process messages available in other queues.
  • Use Unique message Ids to prevent the same message being processed by the consumer.

3. Amazon SNS (Simple Notification Service)

Amazon SNS is a fully managed Publish/subscribe asynchronous push notification system where message producer publishes a message and multiple subscribers receive messages. Pub/Sub messaging model is used to decouple applications and to increase the application’s performance, scalability and reliability. Messages pushed to SNS will be persisted across multiple availability zones to prevent loss of messages. SNS sends a message with subscription URL to all subscribers and the subscribers need to confirm the subscription to receive the message.

SNS allows to send messages to multiple services including SQS, Lambda, HTTP(s), Email, SMS etc.. Amazon SNS can be used in combination with SQS, so the message published to SNS will be sent to each SQS and consumer of each SQS will receive the message.

AWS Messaging Services - SNS Subscribers
SNS Subscribers

Amazon SNS provides encrypted topics to protect your messages from unauthorized and anonymous access. When a message is published to the encrypted topic, the messages are immediately encrypted using AES-GCM algorithm and a Customer Master Key issued with Key Management Service. The messages are decrypted as they are delivered to subscribing endpoints. Before deleting SNS topic, make sure all the subscriptions for the topic are deleted first. Using SNS, a message can be send to a variety of platforms such as iOS, Android, Java, Node Js, Python, .NET etc SDKs from AWS.

SNS is best suited for use cases such as event notification, publishing message to multiple subscribers, sending dedicated message to each of the SQS Queue.

3.1 SNS Message

Message pushed to SNS topic will have the following parts attached to it.

  • MessageId – Unique Id of the message
  • Timestamp – Message creation timestamp
  • TopicArn – Topic to which the message is published
  • Type – Type of delivery
  • UnsubscribeUrl – Link to unsubscribe from this topic
  • Message – Message Payload
  • Subject – Subject field
  • Signature – Base64 encoded signature of the message
  • Signature version – Version of SNS signature

3.2 SNS Limitations

  • The message from SNS may be delivered more than once
  • In case of subscriber failure, message may be delivered for a configured number of times.

4. Amazon MQ

Amazon MQ is a managed messaging service for Apache ActiveMQ. AWS
manages the provisioning, setup, patching, and maintenance of ActiveMQ. Existing
applications can be easily migrated and connect to Amazon MQ using standard APIs
and protocols such as JMS, NMS, AMQP, STOMP, MQTT and WebSocket.

Amazon MQ provided high availability and durability of messages. Latency in Amazon MQ is very less (single digit millisecond) between message producer and consumer. Messages placed in Amazon MQ are always stored redundantly across multiple Availability Zones. Amazon MQ supports distributed transactions. Broker name should be unique in an AWS account and there is a limit of 200GB storage capacity per broker.

A Lambda function can be triggered when a message is placed on Amazon MQ. In a hybrid cloud environment, an ActiveMQ in On-premises data center can be integrated with Amazon MQ on the cloud to exchange messages. Refer to this documentation for connecting your Java application with Amazon MQ and also for creating message producer and consumer.

AWS Messaging Services - Amazon MQ Integration
Amazon MQ Integration

4.1 Best Practices of Amazon MQ

  • Amazon MQ encrypts messages at rest and in transit using its securely managed encryption keys. It is highly recommended to use client encryption as well.
  • To optimize Amazon MQ for slow consumers, set the concurrentStoreAndDispatchQueues attribute to false.
  • Use right broker instance type for better throughput
  • In the case of multi-producer and multi-consumer applications, it can be highly inefficient to create a large number of connections on demand for multiple producers. Use connection pooling instead of using PooledConnectionFactory class. Don’t use PooledConnectionFactory on message consumers.

4.2 Configuring Connection pool (from AWS Documentation)

// Create a connection factory.
final ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(wireLevelEndpoint);

// Pass the username and password.

// Create a pooled connection factory.
final PooledConnectionFactory pooledConnectionFactory = new PooledConnectionFactory();

// Establish a connection for the producer.
final Connection producerConnection = pooledConnectionFactory.createConnection();

4.3 Amazon MQ Limitations

  • Number of brokers per broker instance type, per AWS account is limited to 20
  • Number of security groups limited to per broker is 5
  • Storage capacity per broker is limited to 200GB

5. Amazon Kinesis

Amazon Kinesis is a fully managed web service for processing big data in real time. Kinesis is helpful in collecting, processing and analyzing real-time streaming data to get insights into the information. Using Kinesis the data can be analyzed as it arrives and respond instantly. Using Kinesis the Video, audio, application logs, website click streams, and IoT telemetry can be analyzed. Amazon Kinesis is highly scalable and can handle any amount of streaming data and process data from hundreds and thousands of sources with little latency. Kinesis is capable of processing terabytes of data per hour from various data streams. Kinesis keeps the order of the messages.

AWS Messaging Services - Kinesis Integration
Kinesis Integration

Amazon Kinesis is best applicable in use cases such as collecting and analyzing sensor data such as temperature, humidity sensors etc. It is also useful in ingesting all market data reliably and build real-time analytics.

5.1 Kinesis Limitations

  • Data blob size is limited to 1 MB
  • By default message is deleted from the stream after 24 hrs, but can be extended up to 7 days

6. Amazon Pinpoint

Amazon Pinpoint is a fully managed marketing and analytics service
hosted on Amazon Web Services. Using Amazon Pinpoint marketing team can get
insights to strengthen campaigns and determine how to effectively communicate
with the customer base. Using Amazon Pinpoint, a marketer can define a campaign
schedule to send targeted messages to app end users.

Amazon Pinpoint can be integrated with Mobile and Web applications
to get insights into how the customers are interacting with your apps. It also
helps in sending targeted email, SMS, push notifications, and voice messages
and also track how the customers respond to your messages- i.e Number of
messages delivered, opened or clicked. This is useful for sending promotional
or educational content to engage and retain your users.

Some of the use cases where Amazon Pinpoint is useful are:

  • Aggregation of data from all Campaigns
  • Demographic charts that provide info on the
    devices on which your app is installed
  • Analytics about user engagement and campaign
  • Analytics about application usage time and customer
  • Revenue charts that provide details on customer
    purchases and revenue generated by it
  • User authentication analytics

7. Amazon IoT Broker

AWS IoT message broker is a managed high throughput publisher/subscriber
broker service that enables the sending and receiving of messages to and from
AWS IoT. Client sends a message to AWS IoT to a topic. This message broker
sends message to all the registered clients for that topic. Using AWS IoT, the
messages between all of your IoT devices can be transmitted securely with low
latency. Based on the message volume, the message broker can be scaled automatically.
AWS IoT supports one-to-one messaging or one-to-million broadcast notification
systems. More AWS IoT features are listed in AWS Documentation.

Pushpa Sekhara R Matli

Pushpa Sekhara R Matli is a Hands on Enterprise Architect with more than 15 years of IT experience in analyzing, planning, architecting, and implementing enterprise applications using a comprehensive approach. He has successfully architected, executed and implemented various highly scalable and highly available solutions for large complex projects in multiple domains such as Video, Finance and Logistics for Fortune 500 companies
Notify of

This site uses Akismet to reduce spam. Learn how your comment data is processed.

1 Comment
Newest Most Voted
Inline Feedbacks
View all comments
4 years ago

Hi ,

Nice Article but In your example 4. Mq you mentioned triggering a lambda but MQ dosent provide out of the box event based integration with lambda so you have to long poll which is not supported by lambda.

So have you tried running it on lambda or just local system ?


Back to top button