Jump to content

Explosion at ThePlanet takes down 9000 websites


Tarun

Recommended Posts

  • Administrator

On the night of May 31st, an explosion occurred at ThePlanet's H1 data center. Markus Langenfeld updated his blog about the issue, including the e-mail notification that customers received.

"I got an interesting email tonight, apparently there was an electrical fire at ThePlanet’s H1 data center in Dallas Texas. According to some sources there was an actual explosion powerful enough that it knocked down walls in an equipment room. So far there are no reports of damage to servers or networking equipment but power is offline at the facility and approximately 9000 websites are affected. This will surely be all over the tech news websites by morning."

There have been several updates since the initial announcement. Repairs are taking much longer than estimated.

Todd Mitchell has posted several updates, including the following:

Update 1:

"Our UNIX and development team continue to work to restore service to both ServerCommand and EV1 DNS. Based on current information, 4 of the 8 DNS servers are in service and we expect the remaining DNS servers to come online within the next 180 minutes. The same approx. time line holds true for ServerCommand. The server farm has been relocated to another data center and development is currently working on bringing the services back online.

In terms of the facility, we do not have a firm ETR at the moment. Facilities continues to work with our on-site vendors to acquire replacement equipment and get it installed to bring service back online.

We do not have an Estimated Time to Repair at present. Our staff and management continue to work through the night and we will continue to provide hourly updates."

Update 2:

"We are continuing to work through various issues this morning. We will have additional contractors on-site this morning starting at approx. 7 AM. Some will hand-off from contractors who worked overnight and others will start the recovery/installation of new electrical gear to power the data center.

We are still working through the EV1 DNS and ServerCommand items. We are making progress on both items and expect to have both functional within the next 120 minutes.

In addition to the above, the network engineering group worked overnight to prepare the network for the recovery of H1. We expect the reconvergence of the network to go smooth once H1 comes back online.

We do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning-- we will continue to provide hourly updates."

Update 3:

"The team here at The Planet continues to work through the various issues that we continue to encounter. We are still making progress on the previous items that I mentioned in my last post. DNS infrastructure has been migrated to another data center and propagation has begun. We are working through some database issues with ServerCommand and fully expect those to be resolved within the next hour.

I’d also like to address the idea of migrating from one data center to another. During the early stages of the H1 data center we opportunistically relocated some customers to another data center. However, due to network and data center (power/cooling) constraints, this option is no longer available and requests for migration cannot be honored. Please rest assured that our teams are working diligently to return service to all affected customers.

At this time we do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning-- we will continue to provide hourly updates."

For more updates you can read their official forum post.

Source: Markus Langenfeld's Blog

Link: Lunarsoft

Link to comment
Share on other sites

9000 websites, wow thats a lot.

An explosion, thats an excusable explanation for downtime.

It sucks with incidents of downtime due to someone trip over cords, spill coffee, or some random stranger walk into the server room and walks away with the server on his back.

Dynamite - the best way to knock a UNIX server offline!

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...