Queues

From StatusNet

Captured from "laconica-dev : Moving to a messaging system"

So, one of the key changes I want to make in the Laconica codebase is moving us from a CMS model to a messaging model. That means that we stop treating each notice as a document that is accessed from one location over and over, and more like a message, which gets duplicated and moved around the system. (One such place, of course, is permanent storage.)

There are a couple of key elements to this:

[edit] Queues

1) Queues. Our entry points for notices (Web, email, sms, IM, API) need to stop doing all the routing for messages. Instead, they should push the notice into a primary queue, and other processes will copy it into further queues for further processing. I've got a boxes-and-lines diagram here, with boxes representing queues and circles representing processes that:

Enabling tool here is going to be RabbitMQ, Apache Qpid, or ActiveMQ. Or another Open Source queue manager -- Twitter apparently uses Starling, but they've been bumping into some upper limits on that tool, apparently.

[edit] Stream storage

2) Stream storage. This would be optimizing the main data structure of StatusNet -- notices in streams -- into an easy-to-access blob. For example, "Replies to evan" would be a stream containing all the reply messages to me.

Enabling tech here is some kind of cloud database, like CloudDB or Hypertable.

I very much want to keep the tool "scalable" from a commodity Web hosting system with just PHP/MySQL up to a big system with lots of servers, like identi.ca. So we'd need to make this stuff optional. I think we've done OK with queues enabled/disabled so far, but the storage is going to be trickier.

-Evan