Short outtage today 4/3/2008
April 3rd, 2008 at 2:58 pm   starstarstarstarstar      

Hello.  Some of you may have noticed a short outtage today.

 

While an electrician worked on a power panel he accidentially shorted a circuit that disrupted power to many network switches.

 

Since the main network connections were interrupted, redundant servers couldn't help this particular mishap.

 

P.S.  The electrician was not injured :)

 

Thanks, Shane Merem

www.websiteforge.com

John Casey says:
April 3rd, 2008 at 3:11 pm   starstarstarstarstar      
I'm glad no one was injured in the accident. However my query would be, surely some of the servers are on a back-up platform or there is no guarantee this incident would not be repeated.
Much like we're told we should back up files off the main HD and have them elsewhere or use a UPC device.

I think what I'm getting at is
were peoples websites (who you host )down during this outage or just WF's own site?
Shane Merem says:
April 3rd, 2008 at 3:38 pm   starstarstarstarstar      

Hi John.  Thats an excellent question.  We have MORE than a UPC backup device.  We have diesel generators as well.

 

Our web site runs on the exact same platform and engine as our clients.. So they were all affected.

 

This was a very rare situation.  No matter how much redundancy .. if there is an electical fault like this inserted manually on the INSIDE of the redundant power sources.. It could happen.

 

It's like if you have dual gas tanks on a truck -- If a mechanic makes a mistake and cuts the lines AFTER the point where the two tanks come together then it would still stop gas flow to the engine.

 

The redundant systems are designed to overcome failures of many components of the entire system. 

 

Today's particular issue is rare but not easy to "foresee".  It's like buying a car known for reliability and saying "How come it broke when the guy hit it with the sledge hammer?"  :)

 

I hope that makes sense.  It doesn't make it any better.  The good news is that things were back to normal very quickly.  Thats why we have tech people onsite 24x7 to handle hardware issues.

 

On another note.  The data storage systems are a SAN system.  Very reliable and fault tolerant.  Plus we backup the data every 4 hours to another device.

 

So in terms of a small interruption like we had -- it can happen.  It would be much harder to somehow kill the file system.  The file system is made up of many hard drives spread among many "computers" that make up a SAN array.  It is designed to overcome component failures without losing data.

 

And finally YES.  ALWAYS backup files any time you can.  You can never have enough backups.

 

Shane Merem

www.websiteforge.com

Jose Medina says:
April 3rd, 2008 at 4:01 pm   starstarstarstarstar      

Always good thoughts on the back ups.  Glad he was not injured. He wasn't drinking any alcohol was he?  If so why were we not invited?  LOL

 

Glad its up and running and I love the system. Thanks again for the head ups

 

Jose Medina 

Jon Scott says:
April 3rd, 2008 at 9:54 pm   starstarstarstarstar      

Shane:

 

I am amazed that this is the first time you have had any downtime, my competitors go down often (no joke intended) there will always be unavoidable situations regardless of how hard you try, and too many things ouside of a single person or company's control, there are no absolutes in life, as I have pointed out on my page explaining guarantee's.

 

The difference is how any entity reacts in such occurrences. If all the major N.O.C.'s went offline, I doubt that would be your fault, as unlikely as that senario may be.  I am glad the electrical worker did not get hurt, but stuff happens and there are no absolute guarantee's in this world, except maybe the motto of Website Forge, "The Last Website you will ever buy." Your Company ethic and the value we get from Website Forge far makes up for any unavoidable inturuptions that are possible.

 

While no company likes downtime, anyone that has a complaint with regard to this occurrence has no concept of the magnitude of your opperation or your failsafes.  The inturuption was minor but the prompt reaction and correction is what makes WF special.

 

Jon Scott, Trustee

Trust-The-Entertainer.com

Justin says:
April 7th, 2008 at 12:11 pm   starstarstarstarstar      

Glad it was a short outage.  Was this the same issue that caused down time this morning (April 7th)?

Shane Merem says:
April 7th, 2008 at 1:36 pm   starstarstarstarstar      

Hi Justin.  No it wasn't the same.  This issue was network related.  Just coincidental.

 

Thanks, Shane

Name * 
Email * 
Rate This Post  
Spam Protection 
27, thirty seven, twenty nine and 22: the 2nd number is?
Send to Kindle
Archives