Using AWS SQS With Dead Letter Queues For Enhanced Reliability

Whenever a web application, say an e-commerce platform, experiences a spike in traffic, the servers get overwhelmed. Now, they have a lot of things in their bucket to process.

The servers must manage incoming traffic, process customers' cart information and suggest relevant items, process payments, place orders, communicate them to the sellers, send confirmation emails to customers, and so on.

Messaging queues help with asynchronous order processing, where orders are queued and processed by scalable worker instances. It also uses delayed queues for inventory updates, ensuring accurate stock levels without database overload.

Payment failures are managed through retries, and persistent issues can be moved to a Dead Letter Queue (DLQ) for later investigation. This setup ensures the system remains reliable, preventing service disruptions and allowing developers to focus on critical issues without immediate pressure.

This is why businesses that strive for high availability should consider queues for communication.

What Is AWS SQS?

AWS SQS (Simple Queue Service) is Amazon's proprietary messaging service. In this service, there are two parties, namely producers and consumers, and the message queue communicates between them.

There are two types of SQS queues: Standard queues and FIFO queues.

FIFO Queue

Assume there are ten messages, say message one, message two,..., message ten, in a FIFO queue. Now, the queue ensures that unless message one is consumed, the rest of the messages will remain in the queue only.

Standard Queue

While FIFO queues allow you to send messages in a specific sequence, standard queues are ideal when the order of messages doesn’t matter. Multiple consumers can listen for messages, and each message is guaranteed to be delivered to at least one consumer before its visibility timeout expires.

If a consumer exceeds the visibility timeout while processing a message, AWS SQS will re-queue the message, allowing another consumer to pick it up and process it.

Capabilities Of AWS SQS

Visibility Timeout
It temporarily hides a message once a consumer retrieves it to prevent concurrent processing by other consumers.

Batch Processing
It allows messages to be sent, received, and deleted in batches, which helps reduce costs and improve efficiency.

Scalability
AWS SQS automatically adjusts to handle any volume of messages, ranging from a few per day to millions per second.

Durability & Reliability
It ensures messages are delivered at least once and maintains multiple redundant copies for durability.

Dead Letter Queues (DLQs)
AWS SQS captures messages that can’t be processed successfully after a specified number of attempts, facilitating troubleshooting and error handling.

Delayed Messaging With AWS SQS

Choose The Right Queue Type

The FIFO queue is ideal when your application requires strict message ordering and ‘exactly-once’ processing, such as in financial transactions.

The standard queue is ideal for applications that don’t require message ordering or ‘exactly-once’ processing. It is highly recommended when your application demands high throughput.

Optimize Message Size And Batch Processing

Though the maximum message size is 256 KB, it is recommended to keep it as small as possible.

Batch operations such as send, receive, and delete helps improve throughput and reduce cost. Developers can batch up to ten messages through one request.

Configure Dead Letter Queues (DLQs)

Despite retrying to process some messages, they go unprocessed for various reasons. Setting up a DLQ helps handle messages that aren’t processed successfully after a specified number of attempts. Developers can also monitor DLQs to identify and resolve issues with message processing.

Set Appropriate Visibility Timeout

Since multiple consumers can pick up a message, setting appropriate visibility timeouts for messages helps prevent this error from happening.

When you set the expected processing time for a message longer than usual, it will not be processed prematurely. However, you can use the ‘ChangeMessageVisibility’ feature to adjust the visibility timeout for long-running tasks dynamically.

Use Long Polling

Long polling reduces the number of empty responses, and specifying a wait time for ReceiveMessage calls lowers costs.

Implement Idempotency

Processing the same message more than once produces the same result. Idempotency ensures that your message is processed only once to handle potential duplicate messages.

Secure Your SQS Queues

IAM policies help you control access to your SQS queues.
Server-side encryption helps you protect the contents of messages.
VPC endpoints connect you to SQS queues from within a VPC without using the internet.

Monitor And Scale Your Queues

Use CloudWatch to monitor SQS metrics, such as ‘ApproximateAgeOfOldestMessage,’ ‘NumberOfMessagesSent,’ and ‘NumberOfMessagesReceived,’ to ensure queues are performing at their optimum level. Developers can also use CloudWatch Alarms and Auto Scaling to adjust the number of consumers based on the queue length.

Handle Message Duplication And Loss

Enabling content-based deduplication avoids processing duplicate messages in a FIFO queue. When handling errors, implement retries with exponential backoff to deal with transient failures.

Optimize Costs

Developers can avoid unnecessary storage costs by configuring the message retention periods based on their needs. Also, it is recommended that you are aware of the costs associated with SQS usage, such as request costs and data transfer costs.

Steps To Configure Dead Letter Queue

1. Create The Dead Letter Queue

• Sign in to your AWS Management Console and navigate the Amazon SQS service.
• Click on "Create queue."
• Choose either a Standard Queue or FIFO Queue, and provide a name for your DLQ.
• Set other configurations as needed (e.g., visibility timeout, message retention period, etc.).
• Click "Create Queue" to finalize the creation of your DLQ.

2. Create The Main Queue

• Similar to the previous step, create the main queue that will process the messages.
• Choose the appropriate queue type (Standard or FIFO) and name it.
• Set configurations such as visibility timeout, message retention period, and delivery delay per your requirements.

3. Configure Redrive Policy

• Navigate to the main queue you created and choose "Edit" from the "Queue Actions" menu.
• Scroll down to the "Redrive policy" section.
• Check the box to enable the Redrive policy.
  
• Specify DLQ details:
  • Dead-letter queue: Select the DLQ you created from the dropdown list.
    
  • Maximum receives: Set the maximum number of times a message can be received before it is moved to the DLQ. This is important to prevent messages from getting stuck in the main queue indefinitely.

4. Save Changes

• After configuring the Redrive policy, save the changes to the main queue settings.

5. Test The Configuration

• Send a test message to the main queue.
• Use your consumer application to process the message. If it fails each time, ensure that the message is retried up to the maximum receive count and then moved to the DLQ.
• Check the DLQ to confirm that the message has been moved there after the maximum receive attempts.

Use Cases Where AWS SQS Might Not Be Ideal

Real-Time Communication

AWS SQS is intended for asynchronous message processing, and latency may creep in, which is a barrier to building efficient real-time applications. If you’re building real-time applications, consider using WebSockets, Amazon Kinesis, or Amazon SNS.

Large Message Payloads

The maximum message size in an SQS queue can be 256 KB, which can be a hindrance for applications needing to send larger messages. You can use Amazon S3 to store large payloads and send a reference through SQS.
Transactional Messaging

SQS does not support multi-message transactions or atomicity across multiple queues. Developers can overcome this by using a database with transaction support or Amazon MQ for full message broker features.

Complex Routing And Filtering

Since SQS lacks advanced message routing and filtering capabilities, developers can use Amazon SNS with message filtering or Amazon EventBridge for better event routing.

Guaranteed Message Order In High Throughput Scenarios

When using FIFO queues, which offer order guarantees, it is imperative to get lower throughput. For ordered and high-throughput message streams, you can use a combination of Amazon Kinesis or Kafka.

Long-Term Message Storage

A message's maximum retention period in an SQS queue is 14 days, as it is designed for short-term message queuing. Developers can use Amazon S3 or Amazon Glacier for long-term storage.

Conclusion

AWS SQS enhances system scalability and reliability by managing asynchronous message processing efficiently. It prevents server overloads, ensures prompt updates, and handles failures using delayed queues and dead letter queues. 

Best practices include optimizing message size, configuring DLQs, and optimizing operations. Despite limitations in real-time communication and handling large payloads, AWS SQS remains essential for businesses aiming for high availability and efficient backend processing.

Schedule a meeting with our experts to understand how AWS SQS can improve the efficiency of your backend applications.