So, some days ago I found this comic:
http://www.commitstrip.com/en/2017/01/16/nice-try/
So, yes, now something embarrassing: one of my servers nearly could be one of these servers. The background: Recently I switched from a file based logging for my applications, including MediaWiki, to a structured logging in elasticsearch. Anything went fine, and for the time of transitioning, I logged anything twice, once still on the server as a file and once in logstash/elasticsearch. Unfortunately, it seems I really messed up with log rotate (shame on me), so the log files took really a lot of space in the disk.
Fortunately, the system automatically tracks the disk space and the untaken parts of the space, so I got notified via E-mail and could fix the problem. Redoing the logrotate rules for MediaWiki, deleting old logs and anything works now fine again. And: I can be sure, that this will not happen again.
One task is still left: turning off the file based logging for all applications, which are also logged in logstash, completely, as it makes no sense anymore to save them twice.
So, the comic brings the following feeling in me: yes, totally right and funny, but it’s a bit embarrassing that it happened me, too 😛