Amazon goes Queuing

A colleague of mine passed on a surprise announcement from Amazon: They’ve just launched the v1.0 beta of their Simple Queue Service. It is an online queuing service with some very simple SOAP and REST-style APIs for message queuing.

For the beta, they’ve limited message size to 4KB, and a maximum of 4000 messages across all queues for a single subscription ID. According to the FAQ, messages are kept for up to 30 days.

The API is quite simple – methods for creating, updating, deleting and querying queues, as well as three methods for queue interaction: Enqueue, Read, and Dequeue.

The messaging read process is somewhat intriguing. A client application issues a ‘read’ request against the queue. Pending message(s) are returned (you can request more than one message in a read request) in response to the request, and a lock is placed on those messages. This means that successive read requests within the lock period will not get the same series of messages. The client application then explicitly dequeues the messages by issuing a ‘dequeue’ request against the queue, supplying the QueueEntryId for each message to remove.

One interesting caveat is mentioned in the documentation for the Read operation:

A message may be returned by the Read operation even though it has already been dequeued, and concurrent Read calls may return the same message to multiple readers. This behavior is a result of prioritizing reliable data storage (even in the face of hardware failures), and we expect such events to occur very infrequently.

It seems that SQS is optimized for reliability, but not for once-and-only-once delivery.

Applications utilizing SQS would need to ensure that they handle duplicate scenarios. The documentation basically implies that the appropriate solution is to ensure that messages are idempotent; retrieving the same message twice should have the same effect as receiving it once.

Currently security is pretty lax. One only needs to know the Subscriber ID to be able to retrieve a list of queues, and from there it would be possible to retrieve messages from those queues. I’m sure that this is something that Amazon would address as they further enhance the service. A simple password-based mechanism would take the security up a level, and, in the future, certificate-based mechanisms could be employed.

With this foray into more ‘Enterprise Level’ services, it does make me wonder where Amazon is headed. They’ve certainly mastered the eCommerce world, and have provided some interesting Web Services focused on that domain. Now it seems that they’re venturing out into some more horizontally focused services, and exposing some of their platform-level capabilities. I wonder what’s coming next…