server crashes?

[Denman]

has anyone else except me noticed that the server keeps dying in the mornings (uk time, so very early for you.) EG this morning at9 am uk, it was down , and still down at like 12am.... it was down also yesterday, but came up not that long afterwards.
Is there a problem at the moment, and is there anything we can do if there is?
Stephen

[Brandon Martus]

named keeps dying and not restarting like it should. I'm looking into it right now actually.

If you're talking about the nightly backup, I apologize. You're probably seeing it at early morning, and theres nothing we can do about that (for now) since we have a majority of the people in the US on the site we have to do it at that time.

[Brandon Martus]

Originally Posted by Brandon Martus

named keeps dying and not restarting like it should. I'm looking into it right now actually.

If you're talking about the nightly backup, I apologize. You're probably seeing it at early morning, and theres nothing we can do about that (for now) since we have a majority of the people in the US on the site we have to do it at that time.

So, pretty much we're running out of memory during the database backup, and the kernel starts killing off the processes taking the most memory .. so the machine won't die. named (dns) and mysqldump (backup) are the first two that get axed, causing the backup to fail, and the machine to be floating in Internet la-la land only accessable via IP, and not 'chiefdelphi.com'.

So. Once we get a better server with more memory, we'll be ok.

Until then, I guess I'll write some sort of sanity check script to make sure named is running every 2-3 hours.

[Denman]

hmm. if you mean the one at 5:47, i see that every day at 1047... so i don't mind that.

Maybe we should go to the donate to CD area

[Joe Ross]

Can you add a swap file (I'm assuming it's hard if not impossible to increase the swap partition)? Obviously it will kill performance when it's needed, but if it's only used during the backup, it shouldn't be as bad having the OOM killer active.

[Ryan M.]

I had been wondering about that... every morning at 6-7ish for me.

Maybe you should fire someone to sit there and type "named" <enter> over and over again as fast as they can...

Why can't you just increase the swap?

[Randy]

Brandon,
Why not use external DNS? Also, look below for how to add SWAP ... My offer for resources still stands

Joe,
Increasing the size of the SWAP partition really wouldn't be an option, but you could add SWAP space. We had to do this on a client's server that literally blewup (traffic wise).



Create the file that we're going to turn into SWAP space
Replace xxxxxx with the size of the SWAP space you'd like to add.

# dd if=/dev/zero of=/dev/SwapFile bs=1024 count=xxxxxx

chmod the SWAP file for security purposes

# chmod 600 /dev/SwapFile

Make is SWAP

# mkswap /dev/SwapFile

Turn it on

# swapon /dev/SwapFile

Check amount of SWAP

# free -m

And ... if you want it to mount on boot, add this to /etc/fstab

/dev/SwapFile swap swap defaults 0 0

[Mike AA]

Originally Posted by Randy

Brandon,
... My offer for resources still stands ...

so does mine

-Mike