Pay attention

I remember chatting with one of the engineers some years ago when I worked at the ferries. He was telling me a story about somebody who'd bumped into something and knocked off a bolt or some such, and caused a few thousand dollars worth of damage. The moral of the story was, "If you touch anything accidentally, look around and make sure you didn't fuck something up." Sage advice.

Fast forward to late 2008. I was editing my Samba configs, and was trying to figure something out, so I cranked up the logging to full debug (see where this is going yet?). Once I was satisfied, I closed smb.conf and walked away, leaving the log level at 9. Tsk.

This evening I was doing some PHP on the old box via Samba (why use vim when you have EditPlus). For some reason it was taking a long time - 2 or 3 seconds on occasion - to open the files. WTF. Top didn't show anything too untoward, except... hold on, the swapping daemon seemed to be working a lot harder than it should have been. What the hell is swapping? I restarted a few services, but nothing really made a difference.

I started poking through logs, and eventually found out the Samba logs were approaching 500MB in size. So of course Samba itself is the culprit, because not only is it logging like mad, it's appending that text onto the end of a HUGE file. So I cranked the logging level back down to 2, rotated the logs, and whaddaya know - files are opening quickly again.

Seriously - if you touch anything accidentally, look around and make sure you didn't fuck something up.