Server outage problem this morning

[whole diary]

[grahame]

I was also a night owl. Woke at around 0430 couldnt get back to sleep so was going to browse some old threads that may be interesting from before my time began on here.  ...

I know ... and you were on very quickly once we were back up.   I've just checked the access pattern out of interest ... Google Analytics reports up to 80 different visitors per hour during the day, and that drops during the night to just a handful - but it's hardly ever zero (certainly hasn't ben for the last week!).   There's always a handful of people signing up in the middle of the night from places where it's daytime, hoping to be able to add "forumspam", but it's rare for an hour to go by without seeing any members at all.

[grahame]

One of our servers (not the main one that hosts the Coffee Shop) suffered a catastrophic failure last night, and I'm taking the opportunity of Easter to take a little longer to restore it than I normally would - applying updates and upgrades before it goes back fully live.

On the "Coffee Shop", you'll find that the current feed of journeys is broken, and that images I've posted and my avatar will give you broken image symbols - there may be a couple of other minor effects too.  I'm anticipating that the server will progressively return to service over the next 12 hours ... will update you here

[grahame]

... will update you here

Mostly back ... QR^▸ (QR Code - Quick Response code) codes, images and the current running map which are fed by that server are now back in operation.

[grahame]

I am getting unconfirmed reports of issues and there's a chance the server may go down.   Please note the following URL on another server for status.

http://www.wellho.net/share/status.html

[grahame]

Our server went offline at around 12:30 yesterday (12th June) and is now back for testing at 11:00 (13th June).  We're on a different server with a new operating system and new versions of databases.

[grahame]

Our server went offline at around 12:30 yesterday (12th June) and is now back for testing at 11:00 (13th June).  We're on a different server with a new operating system and new versions of databases.

Well ... that posted OK 

We may have lost any contributions made between 11:30 and 12:30 yesterday (most recent backup to system failure) - sorry about that, chaps and chapesses, and any post you make this afternoon are not guaranteed to remain (not that they ever are, but I'm just testing at the moment on here)

To login, you may need to get an email reminder sent and reset your password.   Your previous login cookie will certainly be invalid.  And when you come to re-login, please use the top left boxes.


I am aware of a few issues and am looking at some of them.   And I will come back and add a more complete as to what hit us yesterday.    Prime concentration is on getting up and running again and letting people know that we're still here 

[Andrew1939 from West Oxon]

Am I going blind? I have recently had news about progress on the new Hanborough Car park and there has been an ongoing heading for parking at Charlbury and Hanborough stations but I can't seem to see it since yesterday's problems.

[grahame]

Am I going blind? I have recently had news about progress on the new Hanborough Car park and there has been an ongoing heading for parking at Charlbury and Hanborough stations but I can't seem to see it since yesterday's problems.

See http://www.firstgreatwestern.info/coffeeshop/index.php?topic=12535.msg134428#msg134428

As noted in this thread too, I am aware of these issues.  It's been quite a traumatic 36 hours; I do have the data but I am not rushing to put it up in case I manage to screw it up royally. Will probably fall asleep early this evening, do a complete backup as we stand and then load the extra stuff (including some extra parking at Charlbury) in the early hours.

[Chris from Nailsea]

Many thanks for your obviously very strenuous efforts to get the Coffee Shop forum 'up and running again', grahame. 

[grahame]

Posts / threads back (all, I think) including the popular Reading thread and the prize quiz. Next - a similar thing to sort out personal messages, within the next 24 hours. 

[Red Squirrel]

Your hard work is very much appreciated!

If anyone reading this hasn't managed to reset their password yet, please do check your spam or junk e-mail folder - that's where I found my reset notification.

[grahame]

Follow up and updates at http://www.firstgreatwestern.info/coffeeshop/index.php?topic=12538.0 - similar topics in two threads so I'm locking this one (and testing in the process!)

[grahame]

Sorry about the brief outage within the last half hour.   Issues were a networking problem at the London data centre which looks after our server.

[grahame]

My apologies about the outage before dawn this morning (if anyone was up   ) - we ran out of space during a big server backup.

[grahame]

We had some server issues early this morning - this is an interim report as testing and investigation is still under way.    The effect was a loss of service on the Coffee Shop.

I have restored the system back to how it was at 01:30 this morning - so that means that any "night owl" posts and messages will have been lost.  I hope there weren't too many, but on usual form for a Saturday morning I would be very surprised ...

Because the issue is still under investigation, I cannot rule out the possibility of further posts and messages being deleted if we have to restore again, though I would put such a need in the 'unlikely' basket.  I suggest you keep a local copy of your posts just in case




Edit note: One minor typo corrected, for clarity - 'night owl', rather than 'night own'. Thanks, grahame. CfN.