Archive for the ‘Emergency Work’ Category

*** Resolved *** Telehouse Router Reboot

Thursday, May 26th, 2011

Update 01:30: After investigating all possibilities including power, slot compatabilty and chassis capability, we re-inserted the card once again and powered the line cards up one by one. This brought the new 48 port copper gigabit line card online without issues this time. On investigation, the power available in the chassis remains at acceptable levels.

During the below maintenance, our Cisco 6506 chassis disabled power to all line cards. We re-enabled power and all has returned to normal. This would have affected all customers directly connected to this device and any interconnect customers going through Telehouse North.

We are continuing to invesitigate the cause of this.

*** Completed *** Switch Reload Telecity Williams ‘Splinter’ April 19th 23:00

Monday, April 18th, 2011

The following notification is to inform you of a scheduled maintenance on the C4L Network:

Start: 2011-04-19 23:00 BST
End: 2011-04-19 23:05 BST

Affected Devices:
Telecity Williams: Splinter

Affected Services:
All services for customers plugged into device ‘Splinter’. Due to the nature and scope of this work, all Manchester customers should consider themselves to be AT RISK during this work.

Service Impact:
Network device ‘Splinter’ will have a change made to the configuration, which requires a reboot of the device. All customers here will see an outage on their ports, lasting no more than 5 minutes, whilst this work is undertaken.

Outage Reason:
After Friday night’s work, unexpectedly this device is not performing optimally, and needs a configuration change made to bring this back to previous performance levels. The cause of this had been tracked to the change in network topology in Manchester and once peak traffic returned to the ring, this device started to software switch a lot of traffic.

Contact:
Should customers have any queries, please raise a support ticket on the support site, https://support.c4l.co.uk or give the technical team a call on 01202 299799 ext. 3

Kind Regards
C4L Technical Team

*** Completed *** Emergency Works Global Switch 1 April 15th 22:00 – 23:00 BST

Thursday, April 14th, 2011

The below work has been completed successfully.

The following notification is to inform you of a scheduled maintenance on the C4L Network:

Start: 2011-04-15 22:00 BST
End: 2011-04-15 23:00 BST

Affected Devices:
Global Switch 1: Jess & Krypton

Affected Services:
Layer 2 services transitioning through GlobalSwitch 1

Service Impact:
Customers may notice a brief outage while circuits are re-routed around the ring whilst this work is undertaken.

Outage Reason:
We had an outage on one of our links between two devices at Global Switch 1 at 15:00 today. This caused an amount of distruption around the C4L network which affected transit feeds and interconnect customers. As a precaution, and to avoid this happening again, we are swapping out the optics and the patch leads at this location.

Contact:
Should customers have any queries, please raise a support ticket on the support site, https://support.c4l.co.uk or give the technical team a call on 01202 299799 ext. 3

Kind Regards
C4L Technical Team

*** Resolved *** Docklands Fibre Down

Monday, April 4th, 2011

Dear Customers,

Update April 7th 23:00: All traffic has now been moved back to this link. This issue has now been resolved.

Update April 7th 10:00: The link has performed without errors or further drops, and we will be moving all layer 2 traffic back over it at 11pm this evening.

Update April 5th 17:51: We have swapped over to the new fibre now and have returned IP traffic over it. We will monitor for 24 hours and bring all traffic back over this link in 48 hours, should the link remain stable over this course.

Update April 5th 17:10: We are preparing to swap over to the new fibre shortly.

Update April 5th 09:15: All ip traffic has been placed back on the link, however interconnect customers remain off this link while we perform further testing. Interconnect customers should still consider themselves at risk of a second fibre break on the London Docklands ring at this time.

We have seen an increasing error rate and port drops on one of our Fibre ports in the Docklands area after some internal work this morning in one of the datacentres by external contractors. We have closed this down as to avoid any further distruptions to customer traffic. All traffic would have been re-routed automatically around the ring. However, at this point, please consider yourselves to be AT RISK should a further fibre break occur.

We are working with the datacentre to find the cause of this and will issue a further update shortly.

Kind Regards
C4L Technical Team

**Resolved** Goswell Road to Sovereign House Fibre

Thursday, September 16th, 2010

17:30:We have now moved all customer traffic back over this link.

Update 18th September 2010 11:10: The link has been up since yesterday afternoon without passing customer traffic whilst we monitor the signal levels. We are happy the link is now stable and will be moving all traffic back over this at 17:30 BST today.

Dear Customers,

We have taken down the redundant link from Goswell Road to Soverign House as we observed a reduction in signal level between these two datacentres. We have engineers on route to investigate using an OTDR and replacement optics and patch leads in case components need to be swapped out.

Once we have found the root cause of the problem, we will monitor the signal levels for a period of time before bringing the leg back up again out of hours to avoid any further distruption.

All circuits have been re-routed, however customers at Goswell Road, Telehouse Metro, City Lifeline, and Interxion should consider themselves to be at risk from a further fibre break at this time.

Further updates to follow.

Kind Regards,
C4L Technical Team