Server Outage - during the night until about 08:00, 18.1.2024

[whole diary]

[grahame]

Server Outage - during the night until about 08:00, 18.1.2024

Don't know what happened (yet) - server was offline, now back on.  Appears running OK but checking and looking back.  More to follow.

[grahame]

Server Outage - during the night until about 08:00, 18.1.2024

Don't know what happened (yet) - server was offline, now back on.  Appears running OK but checking and looking back.  More to follow.

Indications are that the issue was a problem at our web service provider rather than on our server, which appears to have been running all the time but only able to talk to itself.   That is an educated guess just at the moment - I'm checking around, ensuring backups systems are clean in case I'me just looking at a symptom.

Serving Coffee again ;-)

[Red Squirrel]

Double espresso please!

[GBM]

Double espresso please!

LARGE skinny latte please!

[grahame]

It is unusual (!) for the Coffee Shop to be down like it was when I woke this morning - so perhaps a good time to remind members that http://status.firstgreatwestern.info will give you a status report and is the first place I go to flag any issues that are so dramatic that the normal server can't be reached.

Should I anticipate a long outage (more that an hour or two from when I am alerted to the problem), I am able to switch normal Coffee Shop URLs to point to this status page. Because elements of this redirection are cached, the switch across and back will take time to be fully effective and so this switching is not something to be done for just a few minutes!

[infoman]

What would the earliest you can rang to notify you of a problem?

Okay to ring on the land line or would your prefer a text/call to your mobile number?

[grahame]

What would the earliest you can rang to notify you of a problem?

Okay to ring on the land line or would your prefer a text/call to your mobile number?

I was getting emails from the backup server from 04:24 (it does a heartbeat check on the live one every 15 minutes) - just that I was sleeping deeply - so I was straight onto it when I woke.   

If the server is down, yes,  by all means, members welcome to call.  But please note that admins / moderators are in touch and it's unlikely the outage will be too long anyway.  PLEASE check the status server first though ... because if I know about the problem already, I'll be very busy fixing it rather than having time to chat much as I will normally talk the hind legs off a donkey!

[grahame]

What would the earliest you can rang to notify you of a problem?

Okay to ring on the land line or would your prefer a text/call to your mobile number?

I was getting emails from the backup server from 04:24 (it does a heartbeat check on the live one every 15 minutes) - just that I was sleeping deeply - so I was straight onto it when I woke.   

If the server is down, yes,  by all means, members welcome to call.  But please note that admins / moderators are in touch and it's unlikely the outage will be too long anyway.  PLEASE check the status server first though ... because if I know about the problem already, I'll be very busy fixing it rather than having time to chat much as I will normally talk the hind legs off a donkey!

Server was out during the night - for just under 3 hour from soon after midnight.  When I woke to the issue, it was  something that had happened before (a once in 5 year event) and it just took 5 minutes to fix so no point in updating the status page which - however - I have now done at http://status.firstgreatwestern.info to log that we had a problem and to remind members of that news route.

Sorry for the outage - like so many accidents it's caused (I believe) by a number of very rare occurrences on the server coming together at exactly the same time.   As it can be fixed quickly, best approach is to do that, then note it in case it becomes more common, but not to change systems to try and avoid it again.  Damage done minimal - just my "we're run 24/7" pride hurt.