144 replies to this topic

[Jogger]

its pretty rare that coolrunning has an outage, but it happened today.
we did an upgrade to our database system approx 4.30am this morning (sydney time).
some people reported problems this morning but it was fine for me this morning.
this afternoon we had some database issue - It seemed to affect mysql across the whole of coolrunning as our hompage, wiki, calendar, forums were all bung.
then our isp had a power failure approx 4pm - this is very very rare.
this caused cr to go down the tubes completely and obviously everything else on our server (all other domain names), but it came back pretty quick - total outage was maybe 15mins. this is approx the first time in 2yrs I reckon.
when it did come back everything else was sorted except the cr forums. there was one table that had crashed. however I had the kids and dog to the vet. when I got back it was still buggered (I had left emails for people that might have been able to help).
I then googled and tried and repaired the cactus database. I was impressed with this. All back approx 30mins ago.

[funrunner63]

JoggerKev, on Jul 25 2009, 05:34 AM, said:

its pretty rare that coolrunning has an outage, but it happened today.
we did an upgrade to our database system approx 4.30am this morning (sydney time).
some people reported problems this morning but it was fine for me this morning.
this afternoon we had some database issue - It seemed to affect mysql across the whole of coolrunning as our hompage, wiki, calendar, forums were all bung.
then our isp had a power failure approx 4pm - this is very very rare.
this caused cr to go down the tubes completely and obviously everything else on our server (all other domain names), but it came back pretty quick - total outage was maybe 15mins. this is approx the first time in 2yrs I reckon.
when it did come back everything else was sorted except the cr forums. there was one table that had crashed. however I had the kids and dog to the vet. when I got back it was still buggered (I had left emails for people that might have been able to help).
I then googled and tried and repaired the cactus database. I was impressed with this. All back approx 30mins ago.

Thanks Kev,

Hope the vet had as much luck fixing the kids (and dog),

Funrunner aka Craig

[AndyP]

Not good enough.  I want my money back.

Seriously though, I know the dramas with being a volunteer administrator of a forum, so I can completely understand the forum being down sometimes.

[Whippet Man]

I just thought Rob Hall had caused an overload from his hyperposting on the You Yangs thread.  

[Mick]

I had problems getting in today.

IE8 couldn't resolve the address.

Being a geek, did an nslookup and it timed out. Tried it from web based nslookup and got the IP for the domain, hit this and got the CR Control Pane. Presume needed header to get to real site, so added hosts file entry and can now post this.

So seems there is a problem with not all DNS servers being able to resolve. I'm currently using Telstra NextG.

Or maybe it's just Windows7, but I can get to everything else ...

[Trick]

I just thought that you had finally tired of all the personal bashing you seem to be getting on different threads and just went.... you know what,  Feck You all.....   and shut down the site.

Thankfully this is not the case.  Good on you Kev!    

edit. spelling

Edited by Trick, 26 July 2009 - 08:50 AM.

[sook54]

There have been ongoing intermittent problems. Hopefully as of 32 minutes ago all is ok again. Please report issues to info@coolrunning.com.au.

[running sweetlee]

Whippet Man, on Jul 25 2009, 10:10 PM, said:

I just thought Rob Hall had caused an overload from his hyperposting on the You Yangs thread.  

Funny you should say that because that was exactly where I was on the forum at the time CR crashed!  I kept trying to get back in to finish reading thread and just couldn't.  So happy when all was well again  

No problems for me so far this morning

[funrunner63]

Trick, on Jul 25 2009, 05:49 PM, said:

I just thought that you had finally tired of all the personal bashing you seem to be getting on different threads and just went.... you know what,  Feck You all.....   and shut down the site.

Thankfully this is not the case.  Good on you Kev!    

edit. spelling

Is FECK the correct spelling???????

All these years I've had it wrong!!

[Bellthorpe]

Judging from the number of feckless posters we have from time to time, I'd say it was correct.

[Huff]

JoggerKev,

I accept your apology!  
The outages made it that little bit harder to get through my night-shift last night!

[Peterhorse]

i read this thread as "outrage" and just... yep, it's happened again  

[CoolRunning Admin]

For those wondering what happened - we had a disk fail on our server.
From now on we will list any outages here:
http://docs.google.c...u...24&hl=en_GB

so tech geeks can read the whole sorry saga.
Unfortunately we may have lost much of yesterday's posts.
I haven't had time to check yet.
We may be able to recover some.

Kevin

[CoolRunning Admin]

It looks like anything between 1.10pm yesterday and approx 5.15-5.30pm last night (when the site went down) is likely to be lost, ie just over 4hrs data gone awol.

they are looking to see if we can recover but its looks doubtful.

Kevin

[TRAVY]

I just thought it was me ....so I did not worry about it.... I thought O well I will come back tomorrow....and every thing turned out o.k. ...well done

[flyingemu]

CoolRunning Admin, on Jul 28 2009, 06:45 AM, said:

For those wondering what happened - we had a disk fail on our server.
From now on we will list any outages here:
http://docs.google.c...u...24&hl=en_GB

so tech geeks can read the whole sorry saga.

So basically, the whole whing was FSCKed. Gotcha.

[brettsrun]

Some would suggest it was sabotage of the Inaugural You Yangs 5050 Kev,
With such a great event on the calendar to rival 6foot the system was taken down to reflect all the feedback that may topple the 6foot as one of the most iconic events on the calendar or are you punishing me for chasing my 2009 singlet relentlessy
- I still havent got it - any chance it may surface in time for the summer
just incase the smileys dont serve the intended purpose, best I add - just kidding, oh except for my singlet, I still want it

[FakePlasticTrees]

We've lost my great post from the Sydney Grand Slam thread.

Quote

Today the server went off again requiring another FSCK. This can be due to data problems on the disk, or a problem with the disk itself.

Sounds like a crock to me. Saturday outage probably screwed up the file system. Generally UNIX systems don't reboot just because the filesystem requires an fsck. Certain things might stop working if the drive dies but not normally a reboot.

[funrunner63]

FakePlasticTrees, on Jul 27 2009, 05:47 PM, said:

We've lost my great post from the Sydney Grand Slam thread.



Sounds like a crock to me. Saturday outage probably screwed up the file system. Generally UNIX systems don't reboot just because the filesystem requires an fsck. Certain things might stop working if the drive dies but not normally a reboot.

What the f_ck is an fsck?? or has someone just got the spelling wrong?????

Funrunner aka Cyber-Illiterate

[errorfilenotfound]

filesystem check ?

Edited by soyboy, 28 July 2009 - 01:17 PM.

[Brick]

funrunner63, on Jul 28 2009, 12:40 PM, said:

What the f_ck is an fsck?? or has someone just got the spelling wrong?????

Funrunner aka Cyber-Illiterate

"file system check"

[funrunner63]

ta

[CoolRunning Admin]

What can I say - other than I am glad that CR is back online and we are terribly sorry for an inconvenience caused.

We had another disk fail, and have lost 44hrs since 2pm Weds when the last backup was successful. The site was wonky ever since. Not only have we got a new disk but we have 2 setup in a raid-array so sort of backed-up/mirrored.

All the gory details here

Please bookmark that link as we will use it for any updates for future outages!

Kevin

[Colin]

...although it seems like the 'last posts' are old and everything from last few days (since outages) are missing  

last post seems to be 1:58pm 19th August

Edited by Colin, 21 August 2009 - 10:23 AM.

[BostonCalling]

In the last 3 hours, I haven't been so productive at work since joining CR.  I have even got all my tax stuff together, which I have been procrastinating on.

Good to have CR back though!

[blair]

Colin, on Aug 21 2009, 10:20 AM, said:

...although it seems like the 'last posts' are old and everything from last few days (since outages) are missing  

last post seems to be 1:58pm 19th August

Yes. That's what Kevin said:

CoolRunning Admin, on Aug 21 2009, 10:17 AM, said:

We had another disk fail, and have lost 44hrs since 2pm Weds when the last backuop was successful.

[BEN-HUR]

Thank you Kevin & admin team for the service you provide, also for the time & effort of maintaining it...  

[Chelbub]

Thanks to those who worked hard to get it back up...we appreciate the work you do..

Great to have it back!

Rachael

[Mick]

Thanks for all the hard work by Vurt, Kevin, etc in getting it back up and running.

Good to see all that technical babble by those IT geeks has been removed from this thread. Those guys seriously need to get a life.

[bennie]

Hey...

CoolRunning Admin, on Aug 21 2009, 10:17 AM, said:

What can I say - other than I am glad that CR is back online and we are terribly sorry for an inconvenience caused.

Hey Kevin!!! Thanks a million mate! I haven't said it before (and I should have) but you guys put such a big effort into the site and at no cost to the user! I work in IT myself and can understand how frustrating it's been for you!!!!

When you provide such an fantastic free and valuable website like this, any outage is NOT an inconvenience simply an 'outage'. You don't appreciated what you've got until it's not there! :-)

Your time, effort and dedication is very very much appreciated!

THANK YOU ! THANK YOU!

Cheers.
benjamin

ps. Where can I donate towards the site???

IronMan001, on Aug 21 2009, 10:19 AM, said:

About time! I've been trying to access CoolRunning after my run yesterday. Hopefully that's the end of all these outages!

'About time' you say???? really... gezzz chris... Instead, why not say 'thank you' to the people that have been up all day/all night trying to fix these issues.

[Pelagie]

Thank you, your hard work is appreciated.  Sure missed it!

[Koala1]

THanks to everyone for bring CR back up on line - awesome to have you back again!

[Colin]

balri, on Aug 21 2009, 10:25 AM, said:

Yes. That's what Kevin said:

CoolRunning Admin, on Aug 21 2009, 10:17 AM, said:

We had another disk fail, and have lost 44hrs since 2pm Weds when the last backup was successful.

Thanks...thought he referred to CR being 'off' for 44hrs ...not losing the posts ...which actually were there as late as yesterday.

...or perhaps they just tried to kill some threads off...now there's an idea.   

Edited by Colin, 21 August 2009 - 11:11 AM.

[paigebaby]

I add my thanks. It must be a thankless job sometimes, but you efforts are hugely appreciated.
CR withdrawals now subsiding.

[redbackrun]

yes i miss Cr when it down but very happy when its back on. must not be easy getting these damn computers to work sometimes. A lot of time & hard effort done by Admin - thanks

[Vurt]

There will be some rolling outages today as we move hardware or try and recover data.

http://docs.google.c...u...24&hl=en_GB for details

edit: we are all working day (real) jobs at the moment. I have been posting updates on twitter as well - @benscarf

[Luckylegs]

So much time consuming work goes on behind the scenes of Cool Running and all for the benefit for us....the often too   complacent troops!  I sincerely thank all who worked through the 'dark hours' to bring Cool Running online this morning.

I want to especially thank Ellie80 who so graciously answered my email late last  night explaining what was happening & who then  followed it up later with an update email.  I don't know of too many other people who would have done the same thing.  Thank you Ellie80 & all who got Cool Running back to us today.   Wonderfully generous people...proud to be a fellow member!     LL

[bobbys]

Vurt, on Aug 21 2009, 11:21 AM, said:

There will be some rolling outages today as we move hardware or try and recover data.

http://docs.google.c...u...24&hl=en_GB for details

edit: we are all woring day (real) jobs at the moment. I have been posting updates on twitter as well - @benscarf

Thank you to all in getting the site back. My work however doesn't thank you as my productivity has just dropped off now that CR is back up again.

BobbyS

[Bandanna]

Thanks to Vurt, Kevin and the other IT boffins for getting CR up and running again.  Much appreciate your hard work.

It's been tough, but I think I have my twitch under control now.

[BennyMid]

IronMan001, on Aug 21 2009, 10:19 AM, said:

About time! I've been trying to access CoolRunning after my run yesterday. Hopefully that's the end of all these outages!

You gotta be kidding IK ?

How about " Thanks to the team at CR for getting things back "

Ungratefull ....................................


b

[courtlylove]

Adding my thanks to the folks who do all the hard work in keeping this site up and running. It's really appreciated. This site is a great resource and sorely missed when glitches get into the work. Thanks for the de-glitching.

CL

[StellaBella]

Thank you to all that have been working round the clock to get stuff sorted out AND keeping us updated

[Didge]

IronMan001, on Aug 21 2009, 10:19 AM, said:

About time! I've been trying to access CoolRunning after my run yesterday. Hopefully that's the end of all these outages!

Pull your head in IM. I'm sure these guys have been working their ar*** off to get it fixed - just remember, they do it for nothing, so they don't need that sort of attitude.


On  a more positive note - well done guys on getting things up & running again. The majority of us ( as seen by many other posts ) really appreciate your efforts.  

[Katz]

Sooo glad to see CR back up and running.

Well done guys and thank you!!  

Hopefully you can all relax now and get back to having a real life.

[Ellie80]

Yes good work you techie sorts

LuckyLegs I am glad I could give you an update - it was about all I could do as I am utterly useless in the technical department.

There is definitely something about wanting things you can't have isn't there?

[blkbox]

Didge, on Aug 21 2009, 11:59 AM, said:

Pull your head in IM. I'm sure these guys have been working their ar*** off to get it fixed - just remember, they do it for nothing, so they don't need that sort of attitude.

For god's sake he is a 15 year old kid who probably doesn't have the same command of the English language that the rest of us do.

He doesn't need you giving him grief about his "poor choice" of words or sentence building.

Maybe if we helped and supported him it would be far more productive than continually taking a giant dump on him from a great height.

*Rant over*

Back on topic - well done Kev and Vurt getting everything back in working order/


edit for spelling

Edited by blkbox, 21 August 2009 - 12:17 PM.

[HillsAths1]

bennie, on Aug 20 2009, 08:01 PM, said:

When you provide such an fantastic free and valuable website like this, any outage is NOT an inconvenience simply an 'outage'. You don't appreciated what you've got until it's not there! :-)

Outage, surely you mean Outrage!!

Just kidding thanks to Kev and the whole team, for getting everything back up and running(pardon the pun).

I was beginning to miss everyone.

[bennie]

HillsAths1, on Aug 21 2009, 12:27 PM, said:

Outage, surely you mean Outrage!!

hehehehehe.

I went to bed last night around 11pm and did one last check on the computer to see if CR was back! Nothing like a 'Severity #1' that keeps you on your toes....

HillsAths1, on Aug 21 2009, 12:27 PM, said:

I was beginning to miss everyone.

I know exactly what you mean HillsAths1 !!! I've missed my CR buddies terribly !!! :-) It's nice to have them back!!!

*big hug to everyone!* :-)

now: let's get out there running!!!

[Gadfly]

Another vote of thanks to those involved getting things up and running...and keeping it going.   Top effort, especially on top of your 'normal' jobs.    

[Didge]

blkbox, on Aug 21 2009, 12:17 PM, said:

For god's sake he is a 15 year old kid who probably doesn't have the same command of the English language that the rest of us do.

He doesn't need you giving him grief about his "poor choice" of words or sentence building.

Maybe if we helped and supported him it would be far more productive than continually taking a giant dump on him from a great height.

So why do you single me out from others who have also said similar?

P.S. I don't need to comment about the amount of help & support IM receives on here.......so that point is irrelevent.

*My rant over*

Back to topic.....

Edited by Didge, 21 August 2009 - 12:49 PM.