October 2011

Scheduled downtime 30 Oct 2011 22:00 EDT

Submitted by Evan Prodromou on Sun, 10/30/2011 - 06:03

I'm scheduling some downtime for this evening to fix up our database server situation.

2011-10-30 22:28 EDT: took identi.ca down. I need to set up replication between our master server and a slave server, which requires stopping all writes to the database.

2011-10-30 23:04 EDT: started transferring cold backup of master database server to new slave. Looks like about an hour to get that done.

2011-10-31 00:04 EDT: 5 more minutes.

Identi.ca down

Submitted by Evan Prodromou on Sat, 10/29/2011 - 15:51

The identi.ca master database server went off-line earlier today. I'm trying to figure out how to revive it.

It seems to be back up.

Big loss of data on Identi.ca

Submitted by Evan Prodromou on Fri, 10/28/2011 - 19:00

We've had a big loss of data on identi.ca. All data created between about 18:00 UTC 22 Oct 2011 and about 17:00 UTC 27 Oct 2011 is missing from our databases. Data

Cause of the data loss isn't clear at this time. I changed our master/slave configuration last weekend, which might explain the problem if data wasn't being written, but it wouldn't explain why it's apparently working again.

I'm investigating the problem and hopefully we'll be able to recover that historical data.

Identi.ca and other sites down

Submitted by Evan Prodromou on Tue, 10/18/2011 - 07:58

We've had an outage on the routing server that does our load-balancing on the StatusNet cloud; identi.ca and most older sites are down.

2011-10-12 08:45 EDT: I briefly had the site back up, but there was data inconsistency between master-slave servers. I'm revising now.

2011-10-12 09:45 EDT: I think we've fixed the data inconsistencies.

outage