Monday, February 9, 2009

Out of space! Out of space! Out of...

The box died a couple weeks ago, just completely stopped responding. I rebooted the thing and managed to pull up the MythWeb site and checked its status, and it was reading 0 bytes free. Well, that might have something to do with it. I shelled into it (since the web site again stopped responding), and I found an older video file and just deleted the thing, along with its "pretty" link. That should do, right? At least, it should get the box up and running again to check on the schedules and see what's filling up the drive, maybe set a few things to auto-expire that weren't before.

Except it didn't work. It was still unresponsive. Why? Shelled back into it and ran the df command. There was some space on the /myth partition, so that was fine, but... the ROOT partition was full as well! What?

I solicited some help in getting a command to find what directory is hogging my disk space. (I could do this in Windows with WinDirStat. Heck, I could do this on this box with KDirStat, except I wanted to do this from the comfort of my laptop shelled remotely with a text terminal instead of standing in front of my TV.) The command du -sk * | sort -nr | head gave me the top 10 directories, and by progressively drilling into each one, I was able to track down the offender to /var/log/mythtv, whose most recent log file was over two gigabytes.

For kicks and giggles, I took a peek at the file. It was a couple gigs' worth of:

<date & time> TFW, Error: safe_write(): File I/O errcnt: 1
eno: No space left on device (28)
<date & time> TFW: safe_write(): funky usleep

So, apparently, /myth ran out of space, and in response, mythtv filled up the log on root complaining about it until that ran out of space, too.

Well, now all schedules are set to auto-expire, so hopefully that will help to alleviate the problem with disk space filling up. At least, unless and until I get a larger hard drive for the thing.

No comments: