~#nixy

What is daemontools?

From the daemontools website:

daemontools is a collection of tools for managing UNIX services.

supervise monitors a service. It starts the service and restarts the service if it dies. Setting up a new service is easy: all supervise needs is a directory with a run script that runs the service.

At first when I was introduced to this tool at work, I thought “What a typical Linux-admin. FreeBSD’s rc. system is superior.” Despite my personal preferences, whatever software is used at work is what I have to use and learn to use, too. After getting a little more familiar with supervise, and installing it on a FreeBSD server, I was finally convinced that this may also have a place on FreeBSD machines.

Have you ever needed to know that a process is 100% sure to be running no matter what? Well, some of our applications need that extra little safety net, and you might too. Just right of the bat I can mention things like httpd, sshd, denyhosts, and syslog(-ng). While the theoretical risk of these applications crashing randomly and still being able to run again without any direct editing of some configuration file seems to be very low, in a production environment where loads are extremely high and all processes are pushed into a stage where their theoretical load-handling capacity is on the edge with what has practically been tested, this may happen to you – and you can’t afford the service being down until you figure out a way to fix it permanently.

Either way, if some application crashes in a recoverable manner, it’s most likely that either 1) supervise is still running and will try to revive the process or 2) your box is so broken, it doesn’t even matter that supervise is still running. It’s all about that extra little factor of reliability.

Convinced? Here’s the walkthrough:
install sysutils/daemontools. When the configure page pops up, make sure to let it install the manpages, but leave out SIGQ12 if you don’t know what it is or don’t need it. After installing, edit the file /usr/local/etc/rc.d/svscan.sh. For some odd reason, daemontools doesn’t have a separate configuration file, so you’ll have to edit all options right here. The comments say that you should edit /etc/rc.conf.d/svscan, but that is probably some oddball Linux-specific thing, and no such directory exists in FreeBSD. Have a quick look through the first lines of the file, and edit any settings that are specific to your setup. Either way, uncomment the (MIN|MAX)* lines, and the ulimits. The next important thing to note, is the svscan_servicedir variable. Unless you have a really good reason to edit this, don’t.

Before starting svscan, we need to create the services dir and a service for it to start. If you didn’t edit the servicedir variable, you should just mkdir /var/service. Okay, what now?
Imagine a scenario where you have a development server with two NICs. One internal, one external. On the external NIC, you want to be running with “PermitRootLogin no”, because any sane sysadmin wouldn’t allow root logins from random people on the internet. On the other hand, you have twenty developers who need to share this server – And you want to let them create their own accounts on the server as they wish, without disturbing anybody else, so you also want to run PermitRootLogin set to “yes” on the internal NIC.
This might sound crazy, because obviously, it’s impossible to have the two at the same time. You have to only pick one configuration or make a very dirty hack, right? Incorrect. We can do this cleanly with supervise.

This one is one of the more complicated setups, just so you’ll have an idea of which conveniences supervise can provide, other than auto-starting crashed daemons.

cd /var/service && mkdir sshdint sshdext; \
cp /etc/ssh/sshd_config sshint/; cp /etc/ssh/sshd_config sshext/
What we’ve done now, is to create to service directories for each instance of the ssh daemon we want to run, and copy over the system-wide configuration file. Now go edit the sshd_config in sshdint to allow root logins and listen on the internal IP, and the one in sshdext to deny them and listen on the external IP.

sshdint/sshd_config:

# You can only have one sshd instance listening on any one given IP.
ListenAddress [internal ip]

# For the internal IP, enable root logins.
PermitRootLogin yes

# It's important to have a different pid file for each daemon.
# Otherwise the universe will collapse.
PidFile /var/run/sshdint.pid

sshdext/sshd_config:
ListenAddress [external ip]
PermitRootLogin no
PidFile /var/run/sshdext.pid

Edit other configuration settings as you wish. For example, it could maybe be a good idea to have the external sshd listen to a non-default port if you don’t want to run DenyHosts. But that’s beyond the scope of this article. You’ll be able to experiment with that by yourself once you’re done with this article.

Now, create the run scripts for each sshd.

Create /var/service/(sshdint|sshdext)/run, and edit them.

sshdint/run:
#!/bin/sh
/usr/sbin/sshd -Df /var/service/sshdint/sshd_config

If you don’t specify the -D flag, sshd will detach and supervise will think that it’s down – so it’ll try to start the service again, for eternity.
The other run file should contain the exact same content, except that “sshdint” should be “sshdext” in the path to the configuration file. Now, just chmod u+x run in each directory.

We’re almost done! For this next part, make sure you have direct access in the form of a screen and keyboard, as it’s very likely that you did a misconfiguration somewhere and that your sshd’s will all die.
- Disable sshd in /etc/rc.conf
- Enable svscan in /etc/rc.conf (svscan_enable=”YES”)

Warning: Before proceeding, quadruple-double-triple-check that you’ve written everything correctly, that all the hard-coded paths exist, and that every little configuration is correct. You should even parse every script that you know will soon run in your head three times, before this next step. I can’t stress this enough. If you don’t have direct access to the box you’re playing with, either through a keyboard and screen or through a KVM swith, you’re 90% likely to lose all possible ways of interacting with this server once you execute the next step. Consider yourself warned. I don’t want any complaints in my comments about misconfigurations on your part.

- Run /usr/local/etc/rc.d/svscan start
If it starts correctly, it will now try to spawn two instances of sshd, but they’ll both fail at starting up because you’ve got one system-wide sshd running, which is bound to all network interfaces on your system by default. If everything seems to be right, you should now see svscan running (ps ax|grep svscan) and you should get a lot of errors in /var/log/messages about sshd’s not being able to bind to any address. Now you can:
- Stop sshd. /etc/rc.d/sshd stop.

You should now have two working sshd’s each listening on their own port. If you still get error messages in your messages log, you can either killall -9 sshd or just cleanly reboot the system.

Right now it’s late and my girlfriend is getting grumpy, so I’ll just leave this article open-ended (for now), but following things are to be aware of:
- I think the sshd’s are trying to generate new certificates each time they are restarted (Although I’m absolutely not sure). This can easily be fixed by assigning each sshd it’s own set of certificates and making the appropriate changes in each sshd configuration file.
- I wrote this article as I was doing the dual-sshd setup for the first time in my life, not even knowing if it will eventually work, so a lot of errors may occur, and I probably forgot to write a lot of important notes and things to consider.
- You will not be able to ssh to localhost after this, because you no longer have a sshd listening on the loopback interface. Optionally, you could run an extra instance of sshd if you really need to ssh to localhost.
- The dual-sshd setup is also achievable without daemontools, but not as cleanly as here.

Enjoy your new supervise setup :)