Daemon redesign

From StatusNet

In the long term we'll want to more aggressively refactor message and event routing along the lines of what's described at queues.

In the 0.9 timeframe, though, we just want to rearrange how our queue-handling daemons run into something that will actually scale to the deployment needs of StatusNet's hosting service.

Contents 1 Current model 2 Proposed new model 3 Execution model summary 4 Statistics and logging 4.1 Process state monitoring example 5 XMPP notes 5.1 Input 5.2 Output 5.3 Queue item structure 5.4 Component notes

[edit] Current model

We currently use long-running PHP processes as daemons to handle outgoing notice queues, XMPP i/o, and Twitter status polling:

PHP is pretty flaky for long-running daemons; memory leaks are very common, configuration may not be reloaded, and of course there's just no native threading.

More seriously, having about a dozen long-running PHP processes per site does not scale well to a farm running thousands of instances... OSs don't like dealing with tens of thousands of processes, and even if they did you really don't want them each eating up tens to hundreds of megs of memory each!

[edit] Proposed new model

Proposed model is to have a single long-running daemon, a lightweight über-queue master written in a more robust language. The master will maintain long-running network connections such as the ActiveMQ queue subscriptions and communications with the XMPP/Jabber server:

Updates Jan 2010:

• PHP startup costs were too great to do short-lived processes, so instead these are running within a medium-to-long-running PHP process... in with child processes that are respawned by the master process. Otherwise the model remains largely as conceived below.

Individual events coming in from the queue, XMPP i/o, or polling timeouts will trigger a short-lived subprocess, shelling out to the PHP-based queue handlers to deal with a single item.

Key advantages:

• Lighter weight, more robust master daemon should be more stable
• Short-running PHP processes avoid memory leaks
  • Instead using a soft memory limit, with automated graceful shutdown+respawning when we find we're getting up there.
• Short-running PHP processes ensure that StatusNet configuration and code are consistent for each event
  • Configurations are reloaded for each event; code updates still need to be handled more gracefully.
• Processor and memory load will be based on actual usage, not count of sites; an additional StatusNet instance will only spawn new processes when events occur.

Issues:

• Need to know cleanly which scripts to launch for which event queues [extend getvaliddaemons.php]
• Need to know which sites exist
  • Can pull from status_network but this is harder than it should be.
• Not 100% sure yet how to handle the XMPP connection handoff, need to research further
  • This is pretty much solved now for output threads; input thread not yet integrated.
• Any issues of event processing ordering? Should each queue/site be processed in sequence?
  • nope
• Consider "graceful" restart and shutdown (a la "apache2ctl graceful") that lets all events currently being processed complete cleanly
  • Needed on command for code updates.
  • Alternatively, a way to update the config while we're running to update the site & queue subscriptions
• Any logging/monitoring needs? [log file, ganglia instrumentation?]
  • (ejabberd's stats aren't working; even just basic counts of messages going in/out would be useful)
  • Initial monitoring with a UDP-based server is being worked on; plan to also output info over SNMP to integrate with other tools.

[edit] Execution model summary

Execution model of current queue handler & daemon processes:

	Function		Event model
Class	Input	Output	One-off	Queue-driven	Polling-driven	Connection
EnjitQueueHandler		X		X
FacebookQueueHandler		X		X
OmbQueueHandler		X		X
PingQueueHandler		X		X
PluginQueueHandler		X		X
SmsQueueHandler		X		X
Mail
MailDaemon	X		X
Twitter
TwitterQueueHandler		X		X
TwitterStatusFetcher	X				X
SyncTwitterFriendsDaemon	X				X
XMPP
JabberQueueHandler	~	X		X		X
PublicQueueHandler	~	X		X		X
XmppConfirmHandler	~	X		X**		X
XmppDaemon	X					X

JabberQueueHandler, PublicQueueHandler, and XmppConfirmHandler primarily are queue-driven and send output to XMPP, however they also accept input from XMPP which they forward back via the XMPP connection to XmppDaemon.

XmppConfirmHandler doesn't use the main queue (for items), but pulls things from the confirm_address table on its own queue model. @fixme notes the event handling should be redone... may be time to create a master queue which can handle things that aren't notices.

XmppDaemon only handles i/o on the XMPP channel, and doesn't pay any attention to SN-internal queueing.

[edit] Statistics and logging

• Aggregate: overall, per site, per queue type, per site/queue
  • rate of incoming events
  • rate of outgoing events
  • number of queued items
  • processing lag (time processed - time enqueued)
  • [consider XMPP connections to be a queue for this purpose]
• Current #s available, and historical graphs (rrd-style)?
  • rrdtool hates me, temporarily rolling this with sqlite & a JS graphing lib
• Push stats to Ganglia [per-server]
  • SNMP output will allow ops to integrate most of the monitoring info with other tools, not yet implemented

[edit] Process state monitoring example

[edit] XMPP notes

First implementation rolled out January 2010 kept the original model of a single XMPP input/command handler plus multiple client connections handling XMPP output, but switched from one per queue type per site (user notices, public notices, address confirmations) to one per queue processing thread per site (thread 1, thread 2, thread 3...). This unfornately turned up two major problems:

1. increasing the number of client threads for our bot puts a lot of extra strain on ejabberd (starting testing on prosody to see if it behaves better)
2. some clients have very unpleasant behavior when you're spitting a lot of messages out through different connections, which is much more visible now
   • psi, gajim, kopete reported to be problematic (separate chat windows for each resource)
   • adium, pidgin known to be ok (yes there's more resources but you don't really care)

Additionally we're not too sure how well this would behave when moved to a component model, as we need to ensure that we don't get duplicate processing etc.

My next planned step is to move all the actual XMPP communications to the xmppdaemon, so we only have to worry about one connection to the XMPP server (which can in future also be on a multi-site model).

The formatted notices themselves can run through another queue... something like this:

[edit] Input

updated brion 21:24, 18 January 2010 (UTC)

Incoming messages from clients will go to xmppdaemon.php as before -- but instead of processing and saving them direct to the database, we'll just shove the incoming XMPP stanza into an XMPP-in queue in ActiveMQ.

Queue processing threads in queuedaemon.php can grab them out of the queue for full processing. This reduces the amount of work that needs to be done in the XMPP communication thread to handle input, database connections etc, letting us maximize its throughput.

[It looks like it may be easier to queue message events rather than raw XMPP stanzas here. Incoming is harder to deal with than outgoing...]

[edit] Output

Update 2009-01-21: This part is now implemented in 0.9.x.

Output path is where the real fun happens: instead of each queue processing thread connecting to the XMPP server directly, they can format their outgoing messages and save them into a specialized queue in ActiveMQ for XMPP-bound output.

xmppdaemon.php will also listen to that queue, and send formatted messages on to their destinations.

This lets us handle the formatting and delivery target lookups from the notice queue threads in queuedaemon.php while consolidating all actual XMPP communication to a single thread.

Advantages:

1. no more multiple resource problems
2. faster connection startup/shutdown
3. less connections to stress the server
4. easier next-step switch to component model in place of client connections, only xmppdaemon will need changing in the next round

[edit] Queue item structure

Our existing queues are for notices and simply record a notice ID. For an XMPP output queue, the most flexible model looks like we need to record these bits of info:

• target address
• message text
• optional additional payload (for notice metadata; not used for address confirmations)

My inclination is to encode this as a JSON object on the queue rather than making up another XML format. ;)

Couldn't we just store it as an XMPP stanza, and then have the xmppdaemon push that raw? --Evan 20:25, 18 January 2010 (UTC)

Alternatively we could push out a complete XMPP stanza with the message. With XMPPHP this looks like it would be implemented most easily by overriding the I/O portions of the XMLStream / XMPPHP class so connect(), send() etc push the final stanzas to the queue instead of doing a real XMPP connection.

[edit] Component notes

(For next step after this...)

• should be able to switch to a single connection covering multiple sites
  • any limitation on target domains, etc that we cover?
    • one domain name per connection...?
  • adapt to switch site configs internally based on target address when handling xmpp in
• must include the 'from' when generating stanzas, currently seems to be left out and filled in by the connection

Protocol notes [1] [2]

• use different XML namespace (needs a subclass to alter protected vars)
• use handshake authentication instead of sasl auth (subclass)
• need to handle presence notifications on our own
• need to handle subscriptions on our own