Delayed queue

work in progress

Requirements

• handle retries for PuSH, IM sends etc without pegging CPU to 100% until it goes through
  • (currently, a queue item that requires retrying will get run over and over until it's either completed or discarded as doomable)
• work reasonably sanely with large numbers of sites set up with stomp queues

API

• $qm = DelayedQueueManager::get();
• $qm->enqueue($data, 'queuename', 15); // push this item back into the queues in 15 seconds
  • ?

Item properties

• transport & payload (same as queue data; may also need to list target site if shared)
• TTL/retry count
• approximate time to run

Table indexing:

• index by target time
• (can be a primary index on id as well, not really needed for loading though)

Delay control

Exponential backoff is the traditional way of controlling how long the delay is without having to manually specify. 1, 2, 4, 8, 16.... blah blah eventually if necessary into hours. :) A max TTL per data type is probably sensible; we can drop IMs pretty quickly but probably want to retry on PuSH for a while, so if your server is down for a day you'll get your updates eventually.

Some use cases actually could benefit from scheduling to specific times, though -- this could be used to schedule future posts.

Implementation

Simplest:

• within each queue daemon, keep failed items and their next retry time in an array
  • this avoids need for extra storage, which is good
  • my main worry is that if the queue daemon fails unexpectedly, we may have lost the item permanently.
    • appropriate claim cleanup might help here for db
    • for stomp, I'm not sure we can keep the item open beyond a transaction boundary, but should look into it

Naive:

• store in a table, indexed by time
• run a polling interval in the queue daemon
  • every N seconds check if there's something that's been scheduled for any time up to and including the current second
  • if so, dump them back into the main queue

Slightly less naive:

• maintain 'next scheduled item' state in the daemons
• when pushing something into the delayed queue, ping out an update to the 'next scheduled item'
  • daemons compare that against their last-known state, and if it's newer they save that new date
  • polling check in the queue daemons then doesn't have to check the database if there's nothing new to do, making it nearly free.

Other thoughts:

• Might be contention issues with multiple daemon threads; we've also got like the queuedaemon and the imdaemon, etc. Maybe have a dedicated single-thread daemon to handle these.
• Definitely want to be able to do this with a shared table for multi-site system.
  • If using raw DB, make sure that it's not going to get too much traffic that slows things down. Delayed tries should be a rarity, not the norm.