Queues

Captured from &quot;laconica-dev : Moving to a messaging system&quot;

So, one of the key changes I want to make in the Laconica codebase is moving us from a CMS model to a messaging model. That means that we stop treating each notice as a document that is accessed from one location over and over, and more like a message, which gets duplicated and moved around the system. (One such place, of course, is permanent storage.)

There are a couple of key elements to this:

Queues
1) Queues. Our entry points for notices (Web, email, sms, IM, API) need to stop doing all the routing for messages. Instead, they should push the notice into a primary queue, and other processes will copy it into further queues for further processing. I've got a boxes-and-lines diagram here, with boxes representing queues and circles representing processes that:



Enabling tool here is going to be RabbitMQ, Apache Qpid, or ActiveMQ. Or another Open Source queue manager -- Twitter apparently uses Starling, but they've been bumping into some upper limits on that tool, apparently.

Stream storage
2) Stream storage. This would be optimizing the main data structure of StatusNet -- notices in streams -- into an easy-to-access blob. For example, &quot;Replies to evan&quot; would be a stream containing all the reply messages to me.

Enabling tech here is some kind of cloud database, like CloudDB or Hypertable.

I very much want to keep the tool &quot;scalable&quot; from a commodity Web hosting system with just PHP/MySQL up to a big system with lots of servers, like identi.ca. So we'd need to make this stuff optional. I think we've done OK with queues enabled/disabled so far, but the storage is going to be trickier.

-Evan