Hello.

 

As many of you are aware.  DDOS attacks on the K2 Gaming network recently caused some service outtages.  Since then, new attack detection systems are in place.  Throughout the week the network was attacked a couple times with ZERO service interruption.  This morning at around 11:18 am EST an attack caused a very brief service interruption.  The service was restored in about 15 minutes.  This includes the drive time of high-level engineers.  Without the detection system finding the attack source and destination took much more time.

 

The attack detection system is now configured to mitigate similar attacks without human intervention.  The attack detection system can and will be adjusted to mitigate new types of attacks -- usually without downtime.  Additional hardware and routing upgrades will further secure the networks from such issues.

 

I will keep communicating to you with updates. 

 

Thanks,

 

Shane Merem
www.websiteforge.com

I want to share a communication from the CEO of Intelenet.  Intelenet is basically an ISP's ISP.  Meaning a large provider of service to many large companies including ours.  I'm sharing this to help you understand the resources that have been involved in assuring the recent issues are resolved.  

 

Please understand this attack was not directed to Website Forge or our customers.  However the ripple effect of such an attack can cause other equipment within an associated network to become sluggish.  So the named security measures will help reduce the chance of an attack affecting the speed of our network connectivity. 

 

Here is an excerpt from an email sent to me today:

 

------------------------------------------------------------------
Starting Friday, March 30th, InteleNet experienced a number of very
sophisticated Internet-wide DDOS attacks, including TCP SYN, HTTP flood,
UDP flood, etc., to some of our network infrastructure. The attacks were
directed to our different infrastructure devices such as routers, DNS
server, Mail servers, etc.  Also, some of our customers were directly
targeted.  As our engineers worked to mitigate and block each attack,
the type, magnitude, and the destination of each new attack changed so
additional countermeasures had to be put in place.  

 

Over the last few days, InteleNet engineers in conjunction with our
network vendors such as Cisco and some of our IPS providers we were able
to block the attacks and start working the in infrastructure changes to
allow better protection of potential vulnerable public services.  
InteleNet, its engineering staff, and security consultants have been
working around the clock to rectify these problems and have been able to
stabilize the network.   You can be certain that this attack was
immediately escalated to InteleNet?s highest level and that it availed
itself of every possible resource to solve these serious attacks.

 

In going forward, we have also been working with our security and
network vendors to quickly deploy the latest technology in preventing
DDOS attacks as well as a major upgrade of our network infrastructure
with the latest Cisco equipment.  InteleNet is reviewing additional
infrastructure and security solutions that will ensure that any future
attacks do not compromise our network.  You will receive additional
communications in the next few weeks on when you can except these major
network upgrades.  

 

On a personal note, I know these issues have had an impact on your
business and I am very sensitive to the nature of that impact.  I want
to give you my personal assurance that I, and InteleNet, take these
issues extremely seriously and it is our absolute highest priority to
increase the resilience of our network so issues like this are kept to
an absolute minimum.

 

Sincerely,

 

Mark E. Towfiq
President & CEO
InteleNet Communications, Inc.

 

------------------------------------------------------------------

 

Thank You

 

Shane Merem
www.websiteforge.com

I regret to report that some of you experienced service issues yesterday because of another DDOS (Denial of Service attack) by hackers.  (click here for previous post about the same)

 

Website Forge nor Website Forge customers were the target of this attack.  However the target is located in the same data center. This is why your service was affected.

 

I cannot name the companies -- however there are many large companies that house their servers at the same location we do.  Hackers are known to pick large targets when they attack for the maximum effect.

 

What are we doing about it?  Well we can only rely in the expertise in our data center.  They are relocating and re-routing the "target" servers to parts of the data center that will have the least effect on other companies like ours.

 

There is immense pressure to get this resolved very quickly -- A large cyber-attack like this one is something that you can only react to and learn from. 

 

I will keep everyone posted as we find the final resolution of this issue. 

 

FYI.  The DOS attack happened again this morning as I was writing this BLOG post and was resolved by our data center in about 3 minutes.  So this indicates we are on track for a full resolution. 

 

Thanks for your patience.

 

Shane Merem
www.websiteforge.com

Hello everyone.  It has been a rough couple of days for all of us.  It's not an easy job taking responsibility for hundreds of online businesses.  Usually it is terrific and sometimes we pay our dues.  This is one of those "bumps in the road" that are never fun but definitely can build character. 

 

You should know that we replace and repair servers all the time with ZERO downtime.  This particular issue took some unexpected turns.

 

For those that are interested, I have a pretty detailed explanation and some answers to a few of your questions.

 

The technical details:

 

Wednesday: For unknown reasons our primary database server rebooted and self recovered. We decided at that time to allow it to recover instead of diverting to the backup database server.  (If we use the backup server the servers must be resynched later and that causes a slow down in performance so we avoid that if we can.)

 

Thursday: In the morning the same thing happened.  So we immediately cut over to the backup database server and took down the primary server to diagnose the hardware.  All servers and email were up and running. 

 

When the primary server rebooted and joined the cluster the SCSI channel card for unexplained reasons re-arranged it's physical channels and mounted it's partitions incorrectly causing a myriad of issues too long to explain here.  The entire cluster was disabled while we corrected this issue.  Why this happened is still under investigation.  Every IT professional we have discussed this with so far has no idea why this hardware misbehaved in this fashion.  It could not be predicted nor avoided.  So the down time was primarily caused by this unexpected issue.

 

Once recovered the primary database server had to synchronize with the backup so that they would both contain the most current db transactions.  This is what caused the speed to decrease for about 12 hours. 
 

 

Friday:  The database servers finished synchronizing.  The primary database server was  put back into the cluster and things were reported to be 100% again. 

 

UNFORTUNATELY the primary server crashed again with the exact symptom that it had Wednesday and Thursday.  We again replaced the hardware without any downtime (the way it should of happened the first time) and began to troubleshoot software issues that may be the cause. 

 

We had to restart the server hardware and software several times throughout the day causing very short but annoying email and web site interruptions.  Near the end of the day we determined the issue was due to a Linux kernel bug.  I won't bore you with the details, however the team was able to mitigate the issue. 

 

We also performed a database server upgrade on the backup server and will complete the upgrade to the primary database server at 3:00 am Saturday morning.  This will result in a very short downtime and server reboot.

 

Currently things are running 100% and running at the normal speed.

 

Answers to common questions:

 

Question: I thought redundant systems never fail.  Why isn't the system running 100% of the time?

 

Answer: It sounds great, however the fact is nobody can create a system that is up 100% of the time.  No matter how much money or time you have, these things can and may happen.  I did some research and found that a similar issue happened to Google's Gmail system recently.  In fact over the last few years you can find most large companies have a similar story (even Oracle systems which purport to "never fail").  I'll add that the tradeoff for reliability is speed.  You can't have both. 

 

Question:  What have you done to make sure this doesn't happen in the future?

 

Answer:  We have written scripts for the servers to detect any changes in the SCSI addressing upon boot.  This way if the drives mount incorrectly, the server services will not start.  This will avoid the same issue happening in the future.  Once we get word from the hardware manufacturer we may make additional adjustments.

 

Question:  Why didn't your tech support team give us this detailed information?

 

Answer:  Primarily because the team worked feverishly and expected to have it resolved much faster.  The tech support guys did not have this detail.  Only the server administrators and myself knew the play-by-play.  I apologize if any of you felt uninformed.  We did try to contact anyone with specific questions to respond.

 

 Please understand that the team worked non-stop until this issue was resolved.  Some team members didn't sleep for the better part of 30 hours.  We take ALL issues seriously and care about each and every customer.  I cannot guarantee 100% uptime.  However I can guarantee we will always work as hard as possible to provide the best possible service to our Website Forge customers.

 

Shane Merem
http://www.websiteforge.com/
Website Design and E-commerce

 

Hi everyone.  As some of you may have noticed we had a temporary outtage today.

 

One of our primary Database servers crashed for one reason or another.  Generally when that happens the cluster (other servers),  will physically reboot the server and it will quickly recover itself.  The server DID recover,  however it detected issues with the file system and database that it repaired before putting itself back online.

 

If this same scenario happened with any other server in the cluster, you would not have noticed any downtime.  However this one particular server is the only one that can potentially create a short down time -- and it did.

 

The good news is this should be the worst case scenario.  The most critical server crashed for whatever reason.  It was rebooted without human intervention.  It repaired itself and brought itself back online.  That's a good thing.

 

And for your peace of mind -- IF the server was damaged beyond repair, we do have a hot spare that can take it's place in minutes. 

 

Thanks,

 

Shane Merem
www.websiteforge.com
Website Design and E-commerce