Since I’ve switched from manual configuring my servers to using puppet as a configuration management tool I had one bigger problem (at least one I still have). I use monit to monitor my servers and get notified when something goes wrong and needs my manual assistance. This worked very well until I moved the configuration and management of monit to puppet.
In fact, I simply moved the _existing_ configuration of monit (it’s a generally simple syntax and format) from my existing installations to puppet manifests. So if I have to install a new server, the whole monit installation is done automatically and configured automatically, too. After I finished it, there was no problem. However, normally, monit sends you an information mail, if the instance (PID) changed, so you know, that the service or the server was restarted. This can be normal, if I do something with the server or whatever. However, I noticed, that monit doesn’t send this information anymore. After digging a bit, I found out, that monit doesn’t send any emails anymore, which is pretty bad for a monitoring tool.
In the logs, monit mentioned the following line:
Sendmail: Cannot switch to SSLOK, that doesn’t make much sense. As sending E-Mails from multiple applications (MediaWiki, Gerrit, …) And the server itself works without any problems, my guess was, that the problem was related to monit.
After searching in the internet along time I gave up and started a thread on serverfault.com, as there were no information about the problem in the internet. It seemed I was the only/first person with this problem (always bad, as I’ve the feeling then, that the problem is related to me and I did something wrong :P).
As I had no answer on my question on serverfault, I recently started working at this problem again (I doesn’t remember anymore, why). So, after spending again a lot of time, I found out, that Ubuntu 14.04 comes with a really outdated version of monit (5.6), so I tried upgrading it. And ta-da: with monit 5.20 it sends the emails again like a charm.
Now I get notified again, when something went replay wrong on my servers. Looking backwards, it took me too long to try an upgrade of monit outside the official apt repository, but ok, now I’ve learned it for the next time!