Archive for the ‘Backup Platform’ Category

*** Complete *** Offsite backup service Maintenance on 13/11/2010 10:00 UTC

Thursday, November 11th, 2010

Update Monday 15th @ 12:54pm
There was a delay in returning to full service after the end of the maintenance window due to a license activation error occurring within the backup server software when it was started.

The vendor was contacted immediately and the software began servicing requests again just before 12:45. All users should now find they are able to run backups as normal. In the event any problems are encountered now, please create a support ticket through the usual mechanism.

The maintenance was successful for 3 out of our 4 back-end filesystems, but the remaining filesystem was not ready for use by the end of the available window, and so is still being presented from the previous storage. A further maintenance will be planned in and announced during the upcoming weeks to complete this last migration.

Original Post

In follow up to the unplanned maintenance that was announced on October 27th relating to one of the servers providing our storage and backup platforms, we have been closely monitoring the operation of the server in question.

Our observations have lead us to decide to migrate our backup platform to one of our other storage servers. Based on current loads and performance, this is intended to ensure a good service can continue to be delivered both to backup customers, and to storage customers using the original server.

Due to the large user base of our backup service, this migration will involve a transfer of many terabytes of data between two of our sites. During the same maintenance, it is also our intention to perform a complete filesystem check across all filesystems for preventative maintenance.

We have examined schedule trends and determined that the minimum of disruption to customers will be caused by performing this operation during a weekend.

Starting on Saturday, November 13th at 10:00 UTC, all C4L backup services will be unavailable. A maintenance window of 48 hours has been allotted, during which any backup jobs scheduled will not run. This is a conservative estimate, and service may be resumed earlier depending on data transfer rates.

The maintenance window ends Monday, November 15th at 10:00 UTC, after which all scheduled backup jobs should continue to start and run normally.

Storage services are expected to be unaffected during this time.

Posted in Backup Platform, Storage Platform | Comments Off

storage4.c4l.co.uk – iSCSI + Backup

Wednesday, October 27th, 2010

Update: 28/10/2010 09:05
Last night we re-enabled the service after completing the checks, all systems appear to be working correctly. If you have any problems please get in touch with the team via the support site.

Update: 17:30
The checks we are running are due to finish before 8pm this evening. As soon as they have finished we shall once again enable the service.

Update: 16:09
We have experienced another kernel problem on storage4 and rebooted the machine. We have currently suspended service on this server whilst we run some checks.

Update: 09:57 The storage server has been rebooted and has come back online, the iSCSI services on this server are back online. The Backup service is also back online

Dear Customers

We are aware of a server issue with storage4.c4l.co.uk. Customers who take iSCSI services on this server will presently be down. In addition any backup jobs scheduled to start after 4:30am will have failed to run.

We have looked at the remote console and discovered a kernel panic and are now in the process of arranging remote hands to reboot the device asap.

More information to follow

Posted in Backup Platform, Outage, Storage Platform | Comments Off

*Completed* Storage and backup services maintenance – Interruptions on July 10th and July 17th 00:30 UTC

Friday, July 2nd, 2010

Update: 11th July 2010 03:40 UTC+1
Work on storage4 completed. Normal service resumed.

Update: 17th July 2010 02:40 UTC+1
Work on storage2 completed. Normal service resumed.

Type: Scheduled Maintenance Notification

Provider: C4L
Services Affected: Storage and Backup Solutions.

Maintenance Window 1 : Saturday 10th July 00:30 UTC – 04:30 UTC:
Maintenance Window 2 : Saturday 17th July 00:30 UTC – 04:30 UTC:

Further Information:

As a precursor to implementing new health and performance monitoring and response features across our storage platform, we will be performing operating system upgrades on servers on Saturday, July 10th and Saturday, July 17th.

The maintenance on July 10th will commence at 00:30 UTC (01:30 BST/GMT+1) and will effect customers with iSCSI services on storage4.c4l.co.uk. Additionally, the offsite backup service on backup1.c4l.co.uk will be suspended during the maintenance, as it depends on the previously-mentioned storage server.

The maintenance on July 17th will commence at 00:30 UTC (01:30 BST/GMT+1) and will effect customers with iSCSI services on storage2.c4l.co.uk.

In both cases, the maintenance is expected to run for 4 hours until 04:30 UTC (05:30 BST/GMT+1). Although the service may become accessible before this time, customers are advised not to attempt to make use of the service, as availability is expected to be intermittent until the end of the maintenance window.

The upgrade will be to Ubuntu 10.04 LTS, using iETd 1.4.19 to deliver iSCSI targets. The functionality presented to iSCSI initiators will not change, so no reconfiguration at the client side is expected to be needed.

It may be necessary to reconnect or restart your iSCSI client software and to remount volumes at completion of the maintenance. It is recommended that before the commencement of the maintenance, customers gracefully unmount filesystems located on any affected volumes.

Posted in Backup Platform, Planned Work, Storage Platform | Comments Off