Archive for the ‘Outage’ Category

*Resolved* Maidenhead Connectivity Problems

Monday, February 6th, 2012

Update 16:48: The problems were caused by a large scale DDoS attack aimed at a specific customer. We have taken some additional steps to ensure any further attacks are not as service affecting, however the sheer volume of traffic inbound was extremely high. Apologies for any inconvenience caused. The issue is now resolved.

Update: Connectivity has been restored

Dear Customers

We are aware of a problem affecting Maidenhead customers transit and interconnects. This is causing intermittent packet loss problems at this time. We have identified the problem and are working with the customer involved. Further updates will follow.

 

 

***Resolved*** Greenwich issue

Saturday, February 4th, 2012

Greenwich are experiencing very high temperatures and it has been reported that a site evacuation has been performed. We have contacted and reported this to Greenwich who have escalated accordingly.

We will provide further updates as soon as Greenwich respond.

Update 13/02/2012

A full RFO is available here

Update 04/02/2012 – 02:17

We have received unconfirmed reports that there is an air conditioning concern is server room 1. Greenwich have called in all possible emergency contacts. There is no ETA regarding an official response but we shall do our best to provide updates as soon as we hear more.

Update 04/02/2012 – 02:24

Greenwich have confirmed that there has been an air conditioning failure in server room 1. The site has been fully evacuated and the London Fire Brigade are on site. Greenwich haven’t reported any outbreaks of fire, though have confirmed that temperatures are very high and some machines have powered themselves down.

Update 04/02/2012 – 02:46

We appear to have lost connectivity to Greenwich. We’ll have pulled in as many contacts as possible to resolve. Greenwich have been informed to check all racks and power on as soon as possible. Please check here for updates.

Update 04/02/2012 – 03:00

Greenwich have isolated power to server room one. We are pushing for power to be restored but as of yet there is no ETA.

Update 04/02/2012 – 03:14

We have heard that air conditioning and electrical engineers are due to arrive in 15 minutes.

Update 04/02/2012 – 03:20

Greenwich have advised customers not to travel to site as this could impede works currently being carried out by Greenwich staff and third parties. We have been assigned a dedicated on site engineer and first priority to attend to each rack and power on any device that is off as soon as the all clear to enter has been given.

Update 04/02/2012 – 03:25

For the safety of all customers, please do not attend site until Greenwich advise otherwise. Greenwich are very aware of our concerns and shall respond to instructions we provide as soon as they are able to do so. We are handling a number of incoming calls and may not be able to respond immediately. We shall return each call, but if you could also log tickets at support.c4l.co.uk we shall provide updates as soon as possible.

Update 04/02/2012 – 03:36

Greenwich report that the air conditioning and electrical engineers are now on site.

Update 04/02/2012 – 03:50

Greenwich are planning to chill the room in around 25 minutes time. This process will last about 35 minutes after which our assigned dedicated engineer will attend each device in each rack and perform full power cycles.

Update 04/02/2012 – 04:20

Room temperature has now been reduced. Greenwich are now working on the restoration of power. We shall provide updates as soon as Greenwich respond.

Update 04/02/2012 – 05:26

Connectivity has been restored to our routing devices.

Update 04/02/2012 – 05:34

Greenwich are in the process of checking each device to ensure power and connectivity. Please contact support.c4l.co.uk if you are still experiencing connectivity issues.

Update 04/02/2012 – 6:08

Greenwich have released an interim Reason For Outage which will be sent to all affected customers.

Update 07/02/2012 – 9:56

Please see the latest update from BIS Greenwich:

“BIS are currently collating reports and information from M&E contractors for the Greenwich Data Centre in order to produce a definitive RFO report.  As soon as this work is complete the report will be sent out to customers.”


Link issues between Manchester and London

Thursday, January 26th, 2012

Dear Customers,

Update: 26/01/2012 – 12:36

Connectivity has been restored, and an RFO requested. This will be passed to all affected customers.

We are currently experiencing link issues connecting our devices between London and Manchester.

We are actively liaising with our privider in order to restore the full operatibility of the link as soon as possible.

We apologise affected C4L customers for the inconvenience caused and aim to update you as soon as the situation has been resolved.

***Resolved*** Synergy House Network Down

Friday, January 20th, 2012

Dear Customers,

Update 15:37: Connectivity has been restored, and an RFO requested. This will be passed to all affected customers.

We have been advised by Synergy House in Manchester that our connectivity to there is currently suffering from network problems. All customers based there on the C4L network will be currently offline. Remedial work is ongoing and we will post further updates here as we get them.

C4L Support Team

Storage Controller Flap on SAN

Wednesday, January 4th, 2012

Update 13:50
Dell have identified an issue that may have caused the controller crashes pertaining to VMware ESXi’s handling of iSCSI hardware acceleration. There is no resolution for the bug in the most recent EqualLogic firmware, but Dell have identified a workaround that we can implement on our servers. This is presently being enacted; a further update will be given when this has been completed

Update 12:33
For now we have stabilised the system by disabling the redundant port. Customers should still consider themselves at risk at this time whilst we continue to investigate the matter with our supplier.

Dear Customers

We have observed a problem with our SAN storage in Bournemouth where by our controller had begun failing over its primary 10Gbit ports. This started at 11:43 and persisted until 12:00. The system then became stable and we have checked error logs etc. Upon inspection the controller has logged no reason for it doing this so we have immediately raised a support ticket with our supplier.

The system has now again just started failing over its ports at 12:15.

This will be causing disruption to VDS customers hosted on this infrastructure as well as some storage customers.

We will update this post as soon as we have further information.