The roof, the roof, the roof is on fire...

...we don't need no b3ta let the motherfucker burn.

There's a temporary forum here, for anybody who's noticed the current
site is down since thier datacentre caught fire.

http://forum.robmanuel.com/viewforum.php?f=3

 

so much for The Planet's SLAs. What's the point in having 5 physically
seperate datacentres if you can't mirror customer data across them?

 

I suspect some of this might be a case of what the customer thought
they'd got vs what they'd actually paid for - systems redundancy
doesn't come free, after all.

It's a monumental **** up none the less.

 

We (not The Planet) have several DCs scattered around Europe, allowing
us to provide data replication and DR capacity for customers who are
prepared to pay for it. Many (most) don't.

 

One of my mates site's was down yesterday as a result of this. He was
back on-line at ~ 10 am today. Talk about crap DR contingency.

 

Oh FFS. <surrenders to the apostrophe police>

 

"We can't re-write our app, and don't want to pay for a second
datacentre..but can you give us 99.999% uptime anyway?"

 

I've never worked anywhere that had decent DR/BCP provision. Pretending and
hoping seems to be the norm.

 

good point. I haven't seen *any* "oh, The Planet lost a datacentre but
our website's still up" claims.

WTF were the generators *inside* the datacentre?

 

good point.

 

ours seems to work. Amazingly.

 

I've spent most of the last 6 months working on DR stuff for a punter
running a large non-olympic international sporting event which will
remain nameless. It works almost perfectly.

 

From what I've read they were in an electrical plant room not the
datacentre. Quite how an 'electrical short' can blow out 3 walls
though I'm not sure.

 

Oh, OK. From what I'd read I'd thought the generators were in the same
building as the servers.

 

Heh. Ours is pretty good, even to the point where our backup
is on a different tectonic plate to the main system, just in
case like.

In my previous place, it was a bit of a joke. We had to have
"battle boxes" that consisted of everything we needed should
we have to do remote working. I just put a pack of pencils
and some paper pads in so we could play noughts and crosses
until the fuss died down.

The BCP guy didn't take kindly to us pointing out all the
flaws in his plans. The best one was the "we need a satellite
phone in case the mobile network goes down." Ok, so you have
one satellite phone - now who are you going to call? Then
there was the "getting quotes for gas masks for all employees
to have on their desks" debacle, shortly followed by the "how
much bottled water do we need to keep all the staff alive in
the office for a day" which got extrapolated to the point
where he was about to draw up a list of who was more important
and therefore got water first. It all ended in tears when we
rigged the Secret Santa one xmas and bought him the Zombie
Survival Guide book so he could get some tips.

Oh how I miss people like that. Sometimes.

 

I could tell you stories about other parts of your organisation.

 

I'm sure. And there is one seriously major flaw in our system, which
despite my trying and trying to get fixed over the years, looks like
remaining in place (for political, rather than technical reasons).

 

Except you named them last week.

 

he did, didn't he?

 

Bit of an own goal, really..