Server Restored

[Brandon Martus]

And, we're back.

I don't think there was any data loss .. let me know if you find anything weird happening by responding to this thread.

I haven't tuned mysql/apache completely, but will do so before things get too hectic.

[coalhot]

Wooooo! Yay for the OP!

[Whippet]

Yay! I can waste time at school again!

Congrats on getting the server back up!

[JohnBoucher]

Brandon, Thanks.

[dag0620]

Thanks Brandon! Glad to have it back up. You realize quickly how much you are on Chief Delphi when you don't have it.

[George1902]

This is probably a great opportunity to say thanks to you, Brandon. Not only for getting CD back up so promptly, but for all the work you've done over the years and continue to do for FIRST, VEX, and the STEM education community as a whole.

Thank you, sir.

[Steven Donow]

Day 3...after the downtime, I believe I have survived the withdrawal..

[PayneTrain]

OP delivered.

Thanks!

[techhelpbb]

At least it was November not January 10th :eek:

Great job!

[Starke]

Thanks to Brandon for your continued hard work with the site. It is truly appreciated!

[pmangels17]

Thanks for being so quick about it. Love the site.

[Brandon_L]

I survived solely on 610's facebook posts/youtube videos (https://www.facebook.com/Team610) these past few days.


Thanks for a speedy restoration & all the time you put in!

[jgustafson6156]

YAY! THANKS!!!

[Chris Endres]

Great to have the site up and running, thanks for the hard work you put in for the site.

[mechanical_robot]

What happened anyways, was it just a server issue? Though thankgod it is back.

[magnets]

Thanks for getting the forum back up quickly!

Out of curiosity, where/how is the website hosted, and what sort of server is it?

[Brandon Martus]

The server was on SoftLayer's CloudLayer system. There was some data corruption (they say), the server crashed and wouldn't boot back up.

I ended up moving everything over to RackSpace's Cloud Servers, since I'm more familiar with their setup (use them at work and for other projects) and it's a little bit cheaper for the computing power you get, I believe.

We're currently sitting on a cloud server with 15GB RAM and 6 vCPUs.

[yash101]

Yay! I can waste time at school again!

Congrats on getting the server back up!

Same Here! LOL!

[yash101]

So, was there RAID? If there was, what a coincident to have all the disks fail at the same time! I think this culd be partially caused by bad server maintenance! Also, do you use SSH to fix things or do you go to the hosting provider? For me, I do both (Server located in my room!)

[GaryVoshol]

And spam is back up again too. Sigh. :( :(

[Roger]

I think the clock is off a bit -- CD sent me the automatic birthday greetings this evening at 7:01 pm local time, and there's a birthday cake next to my name, but it's not until tomorrow the 13th. (And the "Today's Birthdays" list doesn't have my name... yet?) What time zone does the server think it's in???

And thanks, Brandon, for the restoration!

[Brandon Martus]

I think the clock is off a bit
Yep, timezone was set incorrectly (UTC).

Should be fixed now.

[magnets]

And, we're back.

I don't think there was any data loss .. let me know if you find anything weird happening by responding to this thread.

I haven't tuned mysql/apache completely, but will do so before things get too hectic.

This is really a minor detail, but the stats page at http://www.chiefdelphi.com/stats/
is missing past march 2012.

[Roger]

Should be fixed now.Getting closer! Got another b'day email at 8:18 pm. :) (But lost the cake :( ) I like this; can you just keep jiggling the clock every hour or so, so I keep getting these emails?

[Brandon Martus]

This is really a minor detail, but the stats page at http://www.chiefdelphi.com/stats/
is missing past march 2012.
Yeah, that's me just being lazy. :)

[Invictus3593]

I literally had to look at my entire life while it was down. SO glad its back!

[Domenic Rodriguez]

This is probably a great opportunity to say thanks to you, Brandon. Not only for getting CD back up so promptly, but for all the work you've done over the years and continue to do for FIRST, VEX, and the STEM education community as a whole.

+1

Thank you Brandon!

[yash101]

+1

Thank you Brandon!

Yeah, also, thank you for giving me some time for homework! I owe you a big one! :D :D :D :D :D

[EricH]

Not sure if this was planned/preexisting or not... but a number of the gray box links near CD-Media and CD-Spy have disappeared (like the direct link for UFH--though that might have disappeared earlier and I just didn't notice).

[Brandon Martus]

Not sure if this was planned/preexisting or not... but a number of the gray box links near CD-Media and CD-Spy have disappeared (like the direct link for UFH--though that might have disappeared earlier and I just didn't notice).

Some of them have been gone for a while, as part of the changes mentioned in this thread (http://www.chiefdelphi.com/forums/showthread.php?t=118256). I removed UFH last night when I was looking over things -- all of the previous UFH honorees profiles have been moved to their discussion thread in the UFH sub-forum (http://www.chiefdelphi.com/forums/forumdisplay.php?f=110).

[Joe Ross]

This isn't a new bug, but it isn't possible to sponsor through paypal. It reports no subscriptions available.

[mechanical_robot]

And spam is back up again too. Sigh. :( :(

Yea when the site went down I was hoping they were making improvements to the spam filter. Well O well, I hope they get it fixed by the time of FRC 2014 kickoff or who know the person spamming will increase their spamming.

[Invictus3593]

I saw a small amount of time on 11/13 at around 10:11PM that the server was down, even checked with isitdownrightnow.com and downforeveryoneorjustme.com and they both saw it down as well. Was this just a small update? or was there another failure of some sort? or was it just me?

[Brandon Martus]

Nothing went on that I know of .. the monitoring I was using was every 30 min. I just switched over to a new monitor to check every 5 min., in case it happens again.

[Invictus3593]

Like i said, most likely my own user error, though I do wonder why those two sites saw it as "down" as well :eek:

[yash101]

This is off topic, but do you have any backup service like CloudFlare? What about a backup server, or even a node system, where users can install an app on their computer which will allow their computers to be a server (very low load), especially during peak times?

[yash101]

Like i said, most likely my own user error, though I do wonder why those two sites saw it as "down" as well :eek:

Maybe the internet connection just had a "hiccup." This means that the connection will be lost for a few seconds to a few minutes and then reconnect.

[Joe Ross]

Yep, timezone was set incorrectly (UTC).

Should be fixed now.

I saw a backup occur at 11 est, rather then later. Seems like another Utc bug.

[Brandon Martus]

I saw a backup occur at 11 est, rather then later. Seems like another Utc bug.
I may not have restarted Apache after fixing the timezone.

If I remember, I'll check tonight at 11 to see if it's still happening. (Should be happening at 4am eastern).

[yash101]

There was a backup at 12 today. You may want to change that to a weekly or monthly backup instead because for us, in the west coast, this is the time that we finish our homework and actually get active online

[Brandon Martus]

There was a backup at 12 today. You may want to change that to a weekly or monthly backup instead because for us, in the west coast, this is the time that we finish our homework and actually get active online
This should happen at 4am Eastern .. I've made another change to the schedule.

Let me know if it comes up again tonight.

[yash101]

I wonder who surfs at 4AM? Also, why do you go by daily backups? I back up my website every three months. Also, if anything seems to be brewing, I have the ability to switch hard drives and move the data flawlessly.

As a matter of fact, I haven't backed up my server yet! I should maybe have a shadow copy of the database in the eMMC, but it doesn't really matter because the hard drive is only an year old

[MichaelBick]

I wonder who surfs at 4AM?

4AM eastern is 1AM pacific. I know that I am sometimes on at midnight. Also I would think CD is used quite a bit more frequently than your personal website.

[yash101]

I agree with that. That is why I don't like the idea of having daily backups, where the entire site is shut down!

[MichaelBick]

The backups are quick and if it is what is needed to potentially save a month worth of data then it is completely with it IMO.

[yash101]

Yes, the backups are pretty fast. However, this is a big site, so I believe it takes up to an hour normally, which can get a little annoying!

[MichaelBick]

For some reason I remember them being max 15 minutes long(I could be wrong though so please correct me if I am). You only have to back up new information, so I can't fathom it being that long. Very few people are on from 1-4AM anyways(except for the people in Israel), so I think most people prefer the quick backup every day that almost always doesn't affect them over a much longer backup every month that would likely bring the sight down for a significant amount of time. The daily 15 min backups would turn into a 7.5 hr backup.

[Gregor]

4AM eastern is 1AM pacific. I know that I am sometimes on at midnight. Also I would think CD is used quite a bit more frequently than your personal website.

Sometimes I'm still up at 4AM eastern. :rolleyes:

[Brandon_L]

Sometimes I'm up at 4AM eastern. :rolleyes:

Yeah, my "Bed time" is when CD starts to back up.

[Joe Ross]

I've been noticing a long delay (10s of seconds) when reporting a post. Has anyone else seen that?

[Steven Donow]

I've been noticing a long delay (10s of seconds) when reporting a post. Has anyone else seen that?

Yeah, I've noticed it too. If I'm not mistaken, was reporting a post previously considered making a post? Because I'd always get the "one post in 47 seconds" message, up until recently, which I'm guessing that change came with the delay...

[yash101]

I thought that was just a bug. Now I think that the mail server could be acting up and causing a delay

[Navid Shafa]

I've been noticing a long delay (10s of seconds) when reporting a post. Has anyone else seen that?

Yes, It's been like this for a long time. I sometimes worry if it goes through, especially when reporting from my phone.

[Brandon Martus]

I was just able to report a post and it took 1-2 seconds to do .. not sure what would cause the delay you're seeing.

I've been noticing a long delay (10s of seconds) when reporting a post. Has anyone else seen that?


Also -- I've added some additional IP filtering. Any IP originating in certain countries will be blocked, and given instructions on how to unblock themselves (e-mail me). I have initially blocked out the countries with IPs that have been causing a lot of the spam recently (India and Bangladesh).

[z_beeblebrox]

I was just able to report a post and it took 1-2 seconds to do .. not sure what would cause the delay you're seeing.




Also -- I've added some additional IP filtering. Any IP originating in certain countries will be blocked, and given instructions on how to unblock themselves (e-mail me). I have initially blocked out the countries with IPs that have been causing a lot of the spam recently (India and Bangladesh).

Doesn't that mean you'll be getting hundreds of spam emails from India and Bangladesh?

[Brandon Martus]

Most likely the spam bots will just move onto another vBulletin site.

[Joe Ross]

I was just able to report a post and it took 1-2 seconds to do .. not sure what would cause the delay you're seeing.

It just worked for me in about 3 seconds, much faster then before.

[Brandon Martus]

Next time it takes 10sec+, send me an email with the approximate time, so I can look through some monitoring I have set up.

[Joe Ross]

Next time it takes 10sec+, send me an email with the approximate time, so I can look through some monitoring I have set up.

It may be related to the number of moderators it is sending an email to. I just reported a thread in general forum (with 35 moderators) at 1:50 PST, and it took almost 30 seconds. Our previous tests would have only sent emails to a few moderators.

[DonRotolo]

It happened to me just 30 minutes ago, at 4:22 when reporting this thread http://www.chiefdelphi.com/forums/showthread.php?p=1317493#post1317493

I explained my concerns and clicked Submit, then waited maybe 10 seconds before it responded.

[yash101]

Most likely the spam bots will just move onto another vBulletin site.

That SFS Plugin should prevent that, not only on vBulletin, but on other platforms like SMF too.

[Steven Donow]

Next time it takes 10sec+, send me an email with the approximate time, so I can look through some monitoring I have set up.

I just experienced this problem (just to make aware that this is still an issue of some sorts) reporting a post in the general forum at around 11:01 PM EST. Wrote about a sentence in the explanation box, then took at least ten seconds for the report to go through

[Joe Ross]

This is really a minor detail, but the stats page at http://www.chiefdelphi.com/stats/
is missing past march 2012.Yeah, that's me just being lazy. :)

It appears that awstats.pl is no longer executable, and that stats.chiefdelphi.com no longer exists, so it's more then just not adding links

[yash101]

Is is just me, or is this site slowing to a crawl every time someone presses the [Submit Reply] button?

[Whippet]

Is is just me, or is this site slowing to a crawl every time someone presses the [Submit Reply] button?

The website works well for me.