What is SQS?

• SQS stands for Simple Queue Service.
• SQS was the first service available in AWS.
• Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.
• Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component where a queue is a temporary repository for messages that are awaiting processing.
• With the help of SQS, you can send, store and receive messages between software components at any volume without losing messages.
• Using Amazon sqs, you can separate the components of an application so that they can run independently, easing message management between components.
• Any component of a distributed application can store the messages in the queue.
• Messages can contain up to 256 KB of text in any format such as json, xml, etc.
• Any component of an application can later retrieve the messages programmatically using the Amazon SQS API.
• The queue acts as a buffer between the component producing and saving data, and the component receives the data for processing. This means that the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer is only intermittently connected to the network.
• If you got two EC2 instances which are pulling the SQS Queue. You can configure the autoscaling group if a number of messages go over a certain limit. Suppose the number of messages exceeds 10, then you can add additional EC2 instance to process the job faster. In this way, SQS provides elasticity.

Let’s understand through an example.

Let’s look at a website that generates a Meme. Suppose the user wants to upload a photo and wants to convert into Meme. User uploads a photo on a website and website might store a photo in s3. As soon as it finished uploads, it triggers a Lambda function. Lambda analyzes the data about this particular image to SQS, and this data can be “what the top of the meme should say”, “what the bottom of the meme should say”, the location of the S3 bucket, etc. The data sits inside the SQS as a message. An EC2 instance looks at the message and performs its job. An EC2 instance creates a Meme and stores it in S3 bucket. Once the EC2 instance completed its job, it moves back to the SQS. The best thing is that if you lose your EC2 instance, then also you would not lose the job as the job sits inside the S3 bucket.

Let’s look at another example of SQS, i.e., Travel Website.

Suppose the user wants to look for a package holiday and wants to look at the best possible flight. AUser types a query in a browser, it then hits the EC2 instance. An EC2 instance looks “What the user is looking for?”, it then puts the message in a queue to the SQS. An EC2 instance pulls queue. An EC2 instance continuously pulling the queue and looking for the jobs to do. Once it gets the job, it then processes it. It interrogates the Airline service to get all the best possible flights. It sends the result to the web server, and the web server sends back the result to the user. A User then selects the best flight according to his or her budget.

If we didn’t have SQS, then what happened?

A web server passes the information to an application server and then application server queried an Airline service. If an Application server crashes, then a user loses its query. One of the great thing about SQS is that data is queued in the SQS even if the application server crashes, the message in the queue is marked as an invisible in a timeout interval window. When the timeout runs out, message reappears in the queue; then a new EC2 instance can use this message to perform its job. Therefore, we can say that SQS removes the application server dependency.

Queue Types

There are two types of Queue:

• Standard Queues (default)
• FIFO Queues (First-In-First-Out)

• Standard Queue

• SQS offers a standard queue as the default queue type.
• It allows you to have an unlimited number of transactions per second.
• It guarantees that a message is delivered at least once. However, sometime, more than one copy of a message might be delivered out of order.
• It provides best-effort ordering which ensures that messages are generally delivered in the same order as they are sent but it does not provide a guarantee.

• FIFO Queue

• The FIFO Queue complements the standard Queue.
• It guarantees ordering, i.e., the order in which they are sent is also received in the same order.
• The most important features of a queue are FIFO Queue and exactly-once processing, i.e., a message is delivered once and remains available until consumer processes and deletes it.
• FIFO Queue does not allow duplicates to be introduced into the Queue.
• It also supports message groups that allow multiple ordered message groups within a single Queue.
• FIFO Queues are limited to 300 transactions per second but have all the capabilities of standard queues.

SQS Visibility Timeout

• The visibility timeout is the amount of time that the message is invisible in the SQS Queue after a reader picks up that message.
• If the provided job is processed before the visibility time out expires, the message will then be deleted from the Queue. If the job is not processed within that time, the message will become visible again and another reader will process it. This could result in the same message being delivered twice.
• The Default Visibility Timeout is 30 seconds.
• Visibility Timeout can be increased if your task takes more than 30 seconds.
• The maximum Visibility Timeout is 12 hours.

Important points to remember:

• SQS is pull-based, not push-based.
• Messages are 256 KB in size.
• Messages are kept in a queue from 1 minute to 14 days.
• The default retention period is 4 days.
• It guarantees that your messages will be processed at least once.

Next Topic

SWF

← prev
next →