kaigou: Edward, losing it. (1 Edward conniption)
[personal profile] kaigou
I manage/design/dev an NFP's site for their annual festival. Last year the traffic was MASSIVE, and we nearly shut down the servers. I told the NFP we'd need to switch to a REAL hosting service, one with mirrored servers who could handle a million hits in the course of eight hours.

Well, they waffled and put it off, and were like, oh, yeah, but still. I ran into issues with the service in October, November, and December, and said, no, we really need to switch. They finally got around to dealing with it early January. After some deliberation, I suggested Dreamhost to them -- should've gone with my own isp, 1and1, even if it's more expensive & doesn't have NFP breaks, WHATEVER -- and it was hell doing the step-by-step process to switch hosting.

EXCEPT. The domain reg company WOULDN'T TRANSFER THE DOMAIN. Even when the domain owner paid $15 and faxed in info to update her email (whut!?), and talked to them ON THE PHONE and when we'd put the request through from Dreamhost, it'd get denied. WHUT!? We tried twice, and finally the NFP was all, oh, we can't deal with this right now, it'll be fine.

Hello, it's 830am on the day of the festival, DO YOU KNOW WHERE YOUR SITE IS.

It's down. Crashed. Gone. Unobtainable. Error establishing a database connection KTHXBAI.

I try all the tricks -- testing the connection, checking the sub-folders (the vendors stuff I built), even logging into the database directly. Works. Page still not up. I reinstall parts of WP, in case that's it. I check the wp-config file. I remove the plugins. I strip out the RSS feed. I do ALL the things. Nothing works.

I call the ISP. THEY'RE CLOSED ON WEEKENDS, WTF OVER. I call the emergency number. I wait 15 minutes, because apparently the guy was in the shower. He asks to call me back once he's dressed. I wait another 20 minutes. I guess he got breakfast, too. He calls me back, fiddles with the site, and apparently the connection info has changed. HOW. I HAVE NO IDEA. WTF OVER. He fixes it, warning me the site may crash AGAIN because now it'll be letting in all the connections.

And the site's up! Then it's down. WAY down.

Now he's got to drive to the server farm itself. I wait another 20 minutes. He calls me, asking if the site's up. No, now I can't even get to the database directly (to copy the festival's schedule, faq, and directions so I can at least post them on Facebook so they're SOMEWHERE). So now I'm waiting AGAIN, and the festival started almost an hour ago.


I told them, and told them, and told them, this hosting service would NOT be able to handle any more traffic than last year -- and they've been promoting like crazy, between twitter and facebook. OF COURSE our traffic is going to jump. Well, it would've, IF WE HAD A GODDAMN SITE. We even got moved on Thursday to a single server (not shared) in prep for the traffic -- and apparently somewhere in there, despite the updated info being put in, it got changed back to the wrong info, which doesn't make sense to me, since he's not touched it since Thursday and I was in Thursday evening tweaking things, and it worked fine all day Friday and Saturday.

Honestly, I warned them. I told them. I explained all the risks. I told them I saw this coming, and knew it would happen unless WE SWITCHED TO A REAL ISP. But nooooo, they just didn't see it as a priority, and now I'm the one person dealing with the endless complaints on Facebook about how the site is crashed.


I'm not even sure who I'm angry at. The domain reg company that refused to listen to the site owner and transfer the domain, and thus threw a spanner into getting us onto a solid and stable mirrored server? The local ISP that doesn't have the time, money or inclination to be a real ISP for a site and FREAKING TAKES WEEKENDS OFF when they KNOW our site is going to have MASSIVE TRAFFIC on a Sunday? The NFP itself for not taking my warnings seriously? Or just all the people who are sitting at their computers constantly hitting refresh and overloading the connections?


ETA: guy messages me on facebook.

Guy: Hi, I'm a manager at [largest server-farm hosting company in city]. Looks like you're having trouble with your hosting. Get in touch with me, we'd love to help out!


(no, actually, I was nicer than that but the general ALL CONNECTIONS BUSY PLEASE TRY AGAIN IN A FREAKING WEEK was probably pretty clear.)

Date: 3 Mar 2013 05:07 pm (UTC)
From: (Anonymous)
I think you can be enraged at all of the above, and have nothing but sympathy. Because /my god/. *just kind of boggles over the whole clusterfuck*

Date: 3 Mar 2013 05:08 pm (UTC)
branchandroot: oak against sky (Default)
From: [personal profile] branchandroot
...that was me. Apparently DW logged me out on my way to your entry. *scratches head*

Date: 3 Mar 2013 05:14 pm (UTC)
law_nerd: Lion with one paw over it's face: face!palm (FacePalm)
From: [personal profile] law_nerd
I'm not even sure who I'm angry at.
I vote: All of the above.

Sending virtual tea, alcohol, and cookies; and entirely real empathy.

Date: 3 Mar 2013 05:20 pm (UTC)
hollyberries: (Default)
From: [personal profile] hollyberries
Hang in there! *sends virtual coffee*

(Stab them tomorrow.)

Date: 3 Mar 2013 07:13 pm (UTC)
sharibet: (Default)
From: [personal profile] sharibet
Man, oh man. Total sympathy here, and rage completely justified.

Also, having working in technical/IT production support at various point in my life, the tech support guy's Sorry your site is down, can I call you back after I've taken a shower? made my jaw drop in disbelief.

Sending over a bottle of virtual absinthe to help numb the pain.

Date: 3 Mar 2013 07:45 pm (UTC)
annotated_em: a branch of a Japanese maple, with bright red leaves (Default)
From: [personal profile] annotated_em
Oh my god. I'm so sorry.

Date: 3 Mar 2013 09:15 pm (UTC)
kathmandu: Close-up of pussywillow catkins. (Default)
From: [personal profile] kathmandu
I think I would blame the domain registration company. It sounds like they had the simplest part of the whole thing, and stopped the whole thing in its tracks.

Date: 3 Mar 2013 09:47 pm (UTC)
From: (Anonymous)
No good deed, etc.

+ CP

(PS: WTF, no more OpenId login with LJ account?)

Date: 3 Mar 2013 10:34 pm (UTC)
wordweaverlynn: (Default)
From: [personal profile] wordweaverlynn
What a nightmare. And I am crossing myself and praying and sacrificing used modems, because next weekend I have FOGcon, and I run the site.

Have a lovely day tomorrow.

Date: 3 Mar 2013 10:45 pm (UTC)
nagasvoice: lj default (Default)
From: [personal profile] nagasvoice
*Sends inadequate virtual tea, hugs, and cookies*
Good luck on doing non-code things tomorrow?

Date: 4 Mar 2013 05:25 am (UTC)
nagasvoice: lj default (Default)
From: [personal profile] nagasvoice
*hastily grabs up the virtual chocolate cookies for you instead*

Date: 4 Mar 2013 12:25 am (UTC)
majoline: picture of Majoline, mother of Bon Mucho in Loco Roco 2 (Default)
From: [personal profile] majoline
Oh yikes. *offers hugs*

Date: 4 Mar 2013 03:50 am (UTC)
majoline: picture of Majoline, mother of Bon Mucho in Loco Roco 2 (Default)
From: [personal profile] majoline
Yeah. That's some serious fuckery right there. I cannot believe they did that.

Date: 4 Mar 2013 06:34 am (UTC)
brainwane: My smiling face, including a small gold bindi (Default)
From: [personal profile] brainwane
Wow. I don't even. Srs raeg.

I send you chocolate.