Comet Best Practices: Subscribing and Initial State Load

Many complications arise when building a real-time web application that employs "comet" or "reverse Ajax" to push data from the server. This is mostly due to the fact that real-time web applications require multi-threading, either directly or indirectly, and many of the same issues that arise in threading scenarios directly parallel the issues present in comet-based websites.

A frequently found issue has to do with the timing of subscribing to a comet channel versus the initial state load of the web page.

For example, consider a simple chat application. When the page loads up, the user expects to see some recent chat messages, as well as start receiving any newly published messages. The initial state load is a distinct operation from the subscription to receive new messages. As such, one must be executed before the other. There are only two possible options:

1. Load state, then subscribe.
2. Subscribe, then load state.

The first option results in the possibility of missing messages. It's feasible (and likely under load) for a user to publish a message after the state has loaded, but before the subscription has completed. This message would not be included in the state load, and it would not be delivered to the user over comet since the subscription was not present at the time of delivery.

The second option results in the possibility of duplicated messages. It's feasible (and again likely under load) for a user to publish a message after the subscription has completed, but before the state has loaded. This message would be delivered to the user over comet since the subscription was present at the time of delivery, and it would be included in the state load.

It has been suggested that the initial state load should be delivered in the response to the subscription request. This is absolutely possible with WebSync, but it does not remove the problem - it just narrows the gap in which the problem can occur. The scenario where messages go missing or are duplicated becomes less likely, but we find ourselves in a dangerous situation when we try to hide bugs instead of fix them. Testing becomes more difficult and we run the risk of unleashing a bug that is more difficult to track down.

So what do we do?

In general, the best thing to do is not lose messages. (As always, there are exceptions, but this is the general case we have found.) So, subscribe first, and load state data afterwards, usually through an Ajax request.

• Page loads and a loading indicator is shown.
• WebSync client is created and subscribes to the channel.
• In the subscribe onSuccess handler, POST a request for initial state with Ajax.
• In the Ajax onSuccess handler, populate the UI and remove the loading indicator.

To get rid of the duplicate messages, set a loaded flag to true once the UI has been populated. If the WebSync client receives a message and the loaded flag is false, push it onto a list. Once the loaded flag has been set to true, read through the list and process new messages, discarding duplicates. Duplicates can be identified by a server-generated ID that is attached to every message publication.