Identi.ca down [UPDATE: not any more]

Evan Prodromou's picture

 Identi.ca is currently down; I just got off an airplane. We had a flood of new users and posters over the last 24 hours, and the extra traffic seems to have sunk our database server.

I'm trying to revive the server (it's currently re-starting), but I might take the opportunity to move to our new data centre. If we're down, maybe it's time to do the move. Watch this space for details.

Update 14:50 PM PST: I can't seem to revive the main database server, so I'm copying data to another server in the same data centre to see if it wants to wake up.

Update 23:40 PM PST: The main and backup database servers are down, so I've started the process of moving to the new data centre. It should take about 10-12 more hours.

Update 01:30 AM PST: The import from backup is going nicely. It looks like we're going to lose a few hours of posts from Monday night, but I might be able to recover them from the old servers. I've consequently changed the DNS for identi.ca to point to the new data centre; that's going to take a few hours to propagate, maybe up to 24 hours.

Update 03:40 AM PST: The import stopped because I did something dumb (had the database dump uncompressed on the same drive as /var/lib/mysql); I need to clean up the problem, and import the rest of the data. Good news is that it's not too difficult to do; bad news is that it will take some time to get everything moved around again.

Update 09:50 AM PST: Looks like I'm getting the last of the data imported (needed to move hundreds of gigs of data from one server to another, munge it with Perl, then move part of it back again). If things go well, we should see identi.ca back up within a few hours. DNS is still propagating for the new data centre, though; some people might see the old "server down" message for a while.

Update 17:00 PST: We've been back up for a few hours. DNS is still propagating; it may take a while for things to get out. Those of you who are comfy editing /etc/hosts may want to map "identi.ca" to 50.57.205.237 there. Otherwise, things will gradually come online soon.

Comments

As to silencing IPs, please

As to silencing IPs, please make sure you're not doing that to any IPs belonging to large access providers using a small pool of IPs ina round-robing way; you could be silencing/blocking a whole bunch of innocents.
link building

Banned

I have been banned also and I have never had a account . When I try to register a page comes up that say a banned user is at this address - WTF no one has ever registered on this site at my location- so how am I banned wen I never had a account.

identi.ca

It appears that I have been caught up in a ban, no way to register :( However, I will say that a good way to improve identi.ca is to charge that $5 fee. Announce the purge / reboot, allow people to download their data, then wipe the db clean and enter the new identi.ca allowing people to import that data given they pay the $5 fee.

Now I wait to be unbanned.

Search REST API still down..

http://identi.ca/api/search.json is currently returning the following:

client version is higher than daemon version (client is v.1.25, daemon is v.1.22)

Cannot register

Hello,

I cannot seem to register. Whenever i try to register using openid, a message saying "Internal Server Error: A banned user has registered from this address" . But this is the first time i'm trying to register. What do i do? I'm trying to post in the " http://identi.ca/group/statusnetsoc " page regarding the Google Summer of Code 2012.

thanking you

hey! why can't register in

hey! why can't register in this server? always put me the Too many registrations. Take a break and try again later.

Bots

The last time I was on the public timeline, just before the sytem died, I noticed a number of dents that looked like they were from bots.

Could it be that this was a planned attack on Identi.ca?

can't register a new account,

can't register a new account, Captcha does not match! always appers

can't register a new account,

can't register a new account, always Captcha does not match! error appears

Gracias

Mucho trabajo,

que debería ser compensado con donaciones tal vez así no pasa esto en un futuro.

Thanks!

...for your efforts! Appreciated!

Good luck with the server move, and thanks!

Identi.ca rocks; it's become one of my favorite communities on the Web. Good luck with the server move, and thanks for all the effort you put into keeping the site running.

Thank you

Thank you for all your work, Evan. Please also make sure you give yourself time to enjoy the holiday with your family :)

$5

Identi.ca and Evan are awesome. New signups should have to pay $5 for that ♥. Serves three purposes: resource spike throttle, spam user creation killer (sure, federated spammers, but that's another problem), $. Like I said in https://identi.ca/notice/84809804 ... works for Metafilter, and who doesn't think Metafilter is cool?

TELL US THE IP!

DNS records didn't update for some? I tried several servers, I don't see identi.ca running. It's a "new server" but down everywhere

Keep up the good work

Thanks for the hard work and can't wait to have that up and running again!

Appreciate your hard work and dedication getting Identi.ca fixed

Evan Prodromou, thanks for your hard work and dedication to fixing this problem. Do you ever sleep? I wish the best for your company, for you and yours, and Merry Christmas! Sincerely, George Hostler

Thanks and good luck

Thanks for all the hard work you guys put into identi.ca and good luck on getting it back up and running.

Thank you!

Thank you Evan for the hard work. Good luck!

Ooooh, shiny server solstice?

A new bi-yearly holiday, perhaps, for moving servers?

Glad it's going well

Good news that the migration is going well.
I run a hosting business so know how messy it can get!

Keep up the good work Evan ;)

Hello..

Is resume? Is there a new design?

Good Luck

I'm a web dev, and run my own instance, and I feel for ya. It's a shame spammers are the current cost of success. :(

You do good work, and all of us that use Identi.ca and the Status.net software owe you big time. I hope this doesn't impact your holidays too much.

Good luck with the data center move!

Three boos for spammers!

Hope the recovery gStoes well!

I've been meaning to ask: how do I do a cron-based backup of my stuff? Delicious, for example, lets you use the API to do an XML backup with a single wget/curl call - just point the program at it and you have a backup. Since StatusNet already has the backup tool, how does one go about backing stuff up with that, in accordance with all of the rules of the art?

How long will it take

Hi there Evan, I am sure you are at work and trying very hard to sort out the increase in traffic. Please do bring us back out identica. Thank you.

Thanks Evan

We all appreciate your work. Hope this clears up for you and that you have a GREAT holiday with your family.

Ops!

We expect your site to be opened as soon as possible..

Re: Identi.ca down

Things like this happens... Well, I hope the new data centre will be fit enough to keep up with the growing number of Identi.ca users ! And by the way, Merry Christmas !

Thanks

Thank you for laconica. Thanks for identica. Thanks for statusnet sites. Thanks for omb, and threaded convos, and ostatus. Thank you, thank you, thank you for your incredible patience, sense of humor, and hard work for so many years.

Best wishes to you and your family for the holiday season.

What a chance !

That's a chance we are the shorter day of the year ;)

one

one down one month ;) get up ;)

By Idea

Yes identica, useful for me and all netter thanks ..

Telling people what is going on.

Telling us what is occurring is very much appreciated.

Thank you!

Thank you very much!

And another idea...

...would be to simply throttle signups coming from the same IP. That would not block innocents from the large access providers but would give you (or software) time to look for behavioral patterns - and maybe block on that basis.

Thank you for all your hard

Thank you for all your hard work and good luck.

+1

+1

Cheers

Cheers for the update Evan!

How annoying!

I hope the move goes well, Evan! Not a good time of the for this kid of job!

As to silencing IPs, please make sure you're not doing that to any IPs belonging to large access providers using a small pool of IPs ina round-robing way; you could be silencing/blocking a whole bunch of innocents.

Same caution for blocking by IP. Better do a DNS lookup to see if it's a known zombie machine and/or check for anonymous proxies (IF there are multiple signups usnig the same pattern - not the same proxy - so it's not someone trying to get past their regime's blocks...).

Any kind of filtering by IP is always tricky business, though blocking zombie machines is pretty safe.

Thank you

thanks Evan for all of your hard work. wish this database migration solves all of the problems :)

thank you!

thank you evan for all your hard work and putting your heart into it. this really is appreciated by us.

may i ask that you please add a flattr button [drupal.org/project/Flattr] to this blog and claim @statusnet@twitter.com at flattr, too?

Flattr

I agree with @march, add some Flattr buttons. ;)

Try not to have too much fun @evan.

Happy Holidays!

FUUUUUUUUUUUUUUUUUUUUUuuuuuuu

FUUUUUUUUUUUUUUUUUUUUUuuuuuuuuuuuuuuuuu~

Yeah

I second that

Good luck

Good luck Evan! Thanks for all your hard work... and looking forward to resuming activity when it's all shiny and fixed!

Waiting patiently for the migration and move progress

I hope it doesn't take too long and also the flood of new users is likely to be spammers or those moving from twitter so I suggest maybe checking the most recent new accounts for spammy behavior such as product links and bogus website links squatting tag spam and bogus links just to flood the timeline and grab traffic or to rip people off. Damn spammers are annoying.

Good luck! Waiting for it to go back online so I can post too! :)

do you know how many users

do you know how many users move to other social?

No

It's not easy to measure that. I can probably get some numbers on the drop in activity on identi.ca over the last few month, but... not tonight.

Yikes

I certainly do hope things wake up. If we have to have an "Inbox Zero" situation, the clean start could prove very interesting. Thank you for all the hard work you do each and every day.

One more MySQL victim

I wonder why people choose to suffer MySQL when PostgreSQL is available.

Oh Yeah!

That. +100 :)

Will you revive freelish.us,

Will you revive freelish.us, too?

Yes

Freelish.us will rise again.

Hello, Thank you very much

Hello,

Thank you very much for the great job you do with identi.ca. Keep up the good work and good luck with this patch.

> I can't seem to revive the

> I can't seem to revive the main database server

heart attack? have you tried defibrillation? :)

You needed this like the proverbial second hole in the head

Good luck with the repairs, Evan!

Volunteer sysadmin work

I've mentioned on a couple of occasions that I am willing to help administrate the SN cloud on a strictly volunteer basis, but haven't seen a response from you either way. I can send you a CV and references, if you like. I want to help, but if you're not interested, at least let me know, so I can quit bugging you. :)

Sorry about that

I haven't been bringing anyone on because I don't want to spend the time showing anyone the ropes on the old data centre when we're moving anyways.

I'd very much like your help with the new data centre.

Spammers

Your service is getting flooded by spammers. I wish you had a report abuse button to make it easy to report these assclowns.

There's a "flag" button on

There's a "flag" button on each user's profile page that I assume is something like a "report abuse" button.

Whenever I block a spammer, it seems they're auto-flagged anyway.

keep them batteries topped

keep them batteries topped up.

IP Bans

I recall @speeddefrost saying that he was pretty sure that all of the offending acounts from before the crash were coming from the same IP, maybe you should give mods the ability to block based on that instead of just per-username.

Side Note

If you need more mods I can give it a shot. I'm on identica like 14 hours a day anyway.

IPs

Actually, if you silence someone, it silences everyone who registered from that same IP address.

It was just a guess.. Though

It was just a guess.. Though the accounts in question weren't advertising anything and it wasn't SEO-related. Just an endless flood of new accounts posting nonsense based off a template.

re: IP bans

ah, I guess that wouldn't help if they can still register(and just can't say anything.) I'm not entirely sure what a silence would entail

Update...

Hello, and thanks for the update. Look forward to identi.ca getting up and running again!

باشه لیوان جان. فقط بگو کی

باشه لیوان جان. فقط بگو کی برمی‌گرده؟

خوب اومدی

لیوان رو خوب اومدی

Just a quick note to thank

Just a quick note to thank you for the update and work so far. Appreciated :)

What about *freelishus*?

Did I lost all my links?

No

Still there.

Where?

Where?
I would really appreciate to get my links back.

Be strong dentmaster

Sorry to hear about the trouble. Those of us on our own instances were gritting our teeth last night as we watched Identi.ca get pounded.

I can't imagine why someone would want to attack you. Thanks for doing what you do. I'm so glad to be able to have control of my own social networking experience. :)