Anyway, on a VPS you should do anything you can to minimize resource usage – because it’s just nicer that way. This was a good chance for me to scrap apache httpd, which in all honesty is a bit like the Hummer of web servers. Figure out the perfect analogy yourself, apache is heavy.

There are, as I understand, two primary choices. nginx and lighttpd.

One of them is needlessly inflexible and complex, the other is simple and flexible but is being ruined by community-related issues. Which should I pick? How does one pick between semantical hell and a school yard?

I’m currently trying to figure out how to install additional software on ESXi. This is a bit of a tough one for now.

My findings are as follows:

• There are two vmfs file systems which contain a set of .tgz files. The contents are identical, but with the hash sums do not match
• One of these file system trees contain a checksum.md5 file. The other does not – I suppose that the one with the checksums is what ESXi considers as being the “original” files.
• When I replaced /lib/libc.so.6 on the server, it immediately dropped all my ssh sessions, and denied access to the console. Once I power cycled the server, all the changes I made to the userland and base of the OS were reverted (I’m guessing they came from the “recovery” file system)
• ESXi complains if the libraries in use do not have a VMWARE_SOMETHING version string (And thus the previously described scenario occurs)
• Most of the userland consists of busybox and dropbear sshd. There’s also gpg.

I’m not big on building software, the usage of gcc, or anything related to linux for that matter, so help from some linux guru on this matter would be greatly appreciated.

But not anymore! In one of my general fits of boredom, I went into /boot/ and had a look through the files. Always remember to familiarize yourself with your operating system, even if you’ve been running it for ages.

In /boot/beastie.4th, there’s some ascii art. One looks like a colored beastie, the other like a monotone beastie, and a third is the ASCII art text. Now, shouldn’t there be some configuration switch to change this behaviour as it fits us? Sure thing!

Just grep /boot/defaults/loader.conf after “logo”. grep logo /boot/defaults/loader.conf, and voilá!

[alive@p0rnteddy /boot]$ grep logo defaults/loader.conf
#loader_logo="fbsdbw"           # Desired logo: fbsdbw, beastiebw, beastie, none

See, you can even remove the logo altogether, or hack the beastie.4th file to include your own. Just remember to make sure it fits on the screen – I don’t even know what would happen if it didn’t; Haven’t been stupid enough to test.

Anyway, the colored beastie, while a nice feature, requires tweaking – as it looks ugly to my eyes. Open up your /boot/loader.conf and write loader_logo="beastiebw".

Reboot your system and enjoy the sight.

The origin of the bug for me is unknown. Naturally, I’ve had to research this issue as it became apparent to me when I had to set up two network interfaces in bonding (Thus calling them bond0) and then putting this new virtual interface into work with two 8021q vlan’s (Thus calling them bond0.100 and bond0.101). Now, I do realize that having a 9-character long interface name is a bit on the side of the unusual, but rarely you’d spot any software which supports any less than 255 characters, right? I mean, this is just common sense these days. Pretending to hunt for bits of RAM by shaving off something like 248 bytes just SO belongs back in 1980’s computing. Not fixing this bug from version 1.0.0 to version 1.0.4 of your software belongs back at Microsoft. Not having a part of your website dedicated to bug reports belongs back in FAIL.

Anyway, this is not an opinion piece.

After spending a good part of a weeks worth of office time hunting down this bug, I finally decided to trace back my steps and see if there was something I could do to solve this problem.

When creating vlans on an interface in Linux, you usually use a tool called vconfig. Turns out that vconfig had the option to provide a custom naming scheme. Just run vconfig set_name_type VLAN_PLUS_VID_NO_PAD and you’re golden. Now your interface name is called vlan100 or something similar.

Isn’t this awesome? Now I also have ads, so that I can some time move this blog out of my closet and into a co-location datacenter.

• Set Hz to 100/250/1000
• Try newer kernel
• Add –directisa option to hwclock (Would only fix a symptom of the problem, and not touch the problem itself at all)

The symptoms:
You get a lot of messages like this:
rtc: lost some interrupts at 2048Hz
This varies (in my case) between 1024Hz, 2048Hz and 4096Hz. The message is printed to the terminal as fast as it can handle, and is written to syslog at a rate of 3500/sec – Though some times less.
If you run “hwclock” on the system, it spits out an error message:
select() to /dev/rtc to wait for clock tick timed out.

Chris Snyder suggests to avoid the problem instead of fixing it, by disabling precise timekeeping in VMware (Which is what was my main problem). What did fix the problem, however, was to compile a new kernel with HPET_EMULATE_RTC. This is more or less a battle in itself, as HPET_EMULATE_RTC depends on other stuff on its own. I’m to exhausted to give a proper walkthrough on this, as I’ve done about 50 kernel recompiles in the past week or so, and just want to forget this issue altogether ASAP.

Why HPET_EMULATE_RTC isn’t on by default is beyond my scope of comprehension, as there’s absolutely no good reason to not have it enabled unless you’re running an extremely small embedded kernel which doesn’t need it in the first place, in which case you’ll disabe it forcefully anyway.

Update: After notifying Chris about this blog post, he suggested that I update it with the kernel config options that I enabled to make HPET RTC emulation work. As opposed to what I wrote earlier, there’s not really hundreds of different .config options to edit. Only a few if you’re editing .config directly and only one if you’re using menuconf.

In menuconf, go Device Drivers ---> Character Devices ---> Enchanced Real Time Clock Support. You’ll see that it is built as a module. What we really need is to make it built-in. Just press Y. Now you can type / (slash) and search for HPET. Go to the bottom of the search results and you should hopefully see HPET_EMULATE_RTC=y.
For all the others, here’s the diff for the Debian 2.6.18 kernel:

147d146
< CONFIG_HPET_EMULATE_RTC=y
2140c2139,2141
< CONFIG_RTC=y
---
> CONFIG_RTC=m
> CONFIG_GEN_RTC=m
> CONFIG_GEN_RTC_X=y

Please note, CONFIG_HPET_EMULATE_RTC depends on CONFIG_RTC=y.

From the daemontools website:

daemontools is a collection of tools for managing UNIX services.

supervise monitors a service. It starts the service and restarts the service if it dies. Setting up a new service is easy: all supervise needs is a directory with a run script that runs the service.

At first when I was introduced to this tool at work, I thought “What a typical Linux-admin. FreeBSD’s rc. system is superior.” Despite my personal preferences, whatever software is used at work is what I have to use and learn to use, too. After getting a little more familiar with supervise, and installing it on a FreeBSD server, I was finally convinced that this may also have a place on FreeBSD machines.

Have you ever needed to know that a process is 100% sure to be running no matter what? Well, some of our applications need that extra little safety net, and you might too. Just right of the bat I can mention things like httpd, sshd, denyhosts, and syslog(-ng). While the theoretical risk of these applications crashing randomly and still being able to run again without any direct editing of some configuration file seems to be very low, in a production environment where loads are extremely high and all processes are pushed into a stage where their theoretical load-handling capacity is on the edge with what has practically been tested, this may happen to you – and you can’t afford the service being down until you figure out a way to fix it permanently.

Either way, if some application crashes in a recoverable manner, it’s most likely that either 1) supervise is still running and will try to revive the process or 2) your box is so broken, it doesn’t even matter that supervise is still running. It’s all about that extra little factor of reliability.

Convinced? Here’s the walkthrough:
install sysutils/daemontools. When the configure page pops up, make sure to let it install the manpages, but leave out SIGQ12 if you don’t know what it is or don’t need it. After installing, edit the file /usr/local/etc/rc.d/svscan.sh. For some odd reason, daemontools doesn’t have a separate configuration file, so you’ll have to edit all options right here. The comments say that you should edit /etc/rc.conf.d/svscan, but that is probably some oddball Linux-specific thing, and no such directory exists in FreeBSD. Have a quick look through the first lines of the file, and edit any settings that are specific to your setup. Either way, uncomment the (MIN|MAX)* lines, and the ulimits. The next important thing to note, is the svscan_servicedir variable. Unless you have a really good reason to edit this, don’t.

Before starting svscan, we need to create the services dir and a service for it to start. If you didn’t edit the servicedir variable, you should just mkdir /var/service. Okay, what now?
Imagine a scenario where you have a development server with two NICs. One internal, one external. On the external NIC, you want to be running with “PermitRootLogin no”, because any sane sysadmin wouldn’t allow root logins from random people on the internet. On the other hand, you have twenty developers who need to share this server – And you want to let them create their own accounts on the server as they wish, without disturbing anybody else, so you also want to run PermitRootLogin set to “yes” on the internal NIC.
This might sound crazy, because obviously, it’s impossible to have the two at the same time. You have to only pick one configuration or make a very dirty hack, right? Incorrect. We can do this cleanly with supervise.

This one is one of the more complicated setups, just so you’ll have an idea of which conveniences supervise can provide, other than auto-starting crashed daemons.

cd /var/service && mkdir sshdint sshdext; \
cp /etc/ssh/sshd_config sshint/; cp /etc/ssh/sshd_config sshext/
What we’ve done now, is to create to service directories for each instance of the ssh daemon we want to run, and copy over the system-wide configuration file. Now go edit the sshd_config in sshdint to allow root logins and listen on the internal IP, and the one in sshdext to deny them and listen on the external IP.

sshdint/sshd_config:

# You can only have one sshd instance listening on any one given IP.
ListenAddress [internal ip]

# For the internal IP, enable root logins.
PermitRootLogin yes

# It's important to have a different pid file for each daemon.
# Otherwise the universe will collapse.
PidFile /var/run/sshdint.pid

sshdext/sshd_config:
ListenAddress [external ip]
PermitRootLogin no
PidFile /var/run/sshdext.pid

Edit other configuration settings as you wish. For example, it could maybe be a good idea to have the external sshd listen to a non-default port if you don’t want to run DenyHosts. But that’s beyond the scope of this article. You’ll be able to experiment with that by yourself once you’re done with this article.

Now, create the run scripts for each sshd.

Create /var/service/(sshdint|sshdext)/run, and edit them.

sshdint/run:
#!/bin/sh
/usr/sbin/sshd -Df /var/service/sshdint/sshd_config

If you don’t specify the -D flag, sshd will detach and supervise will think that it’s down – so it’ll try to start the service again, for eternity.
The other run file should contain the exact same content, except that “sshdint” should be “sshdext” in the path to the configuration file. Now, just chmod u+x run in each directory.

We’re almost done! For this next part, make sure you have direct access in the form of a screen and keyboard, as it’s very likely that you did a misconfiguration somewhere and that your sshd’s will all die.
- Disable sshd in /etc/rc.conf
- Enable svscan in /etc/rc.conf (svscan_enable=”YES”)

Warning: Before proceeding, quadruple-double-triple-check that you’ve written everything correctly, that all the hard-coded paths exist, and that every little configuration is correct. You should even parse every script that you know will soon run in your head three times, before this next step. I can’t stress this enough. If you don’t have direct access to the box you’re playing with, either through a keyboard and screen or through a KVM swith, you’re 90% likely to lose all possible ways of interacting with this server once you execute the next step. Consider yourself warned. I don’t want any complaints in my comments about misconfigurations on your part.

- Run /usr/local/etc/rc.d/svscan start
If it starts correctly, it will now try to spawn two instances of sshd, but they’ll both fail at starting up because you’ve got one system-wide sshd running, which is bound to all network interfaces on your system by default. If everything seems to be right, you should now see svscan running (ps ax|grep svscan) and you should get a lot of errors in /var/log/messages about sshd’s not being able to bind to any address. Now you can:
- Stop sshd. /etc/rc.d/sshd stop.

You should now have two working sshd’s each listening on their own port. If you still get error messages in your messages log, you can either killall -9 sshd or just cleanly reboot the system.

Right now it’s late and my girlfriend is getting grumpy, so I’ll just leave this article open-ended (for now), but following things are to be aware of:
- I think the sshd’s are trying to generate new certificates each time they are restarted (Although I’m absolutely not sure). This can easily be fixed by assigning each sshd it’s own set of certificates and making the appropriate changes in each sshd configuration file.
- I wrote this article as I was doing the dual-sshd setup for the first time in my life, not even knowing if it will eventually work, so a lot of errors may occur, and I probably forgot to write a lot of important notes and things to consider.
- You will not be able to ssh to localhost after this, because you no longer have a sshd listening on the loopback interface. Optionally, you could run an extra instance of sshd if you really need to ssh to localhost.
- The dual-sshd setup is also achievable without daemontools, but not as cleanly as here.

Enjoy your new supervise setup :)

Nonetheless, taking just an extra measure of security is a good idea, and this is where DenyHosts comes into the picture. DenyHosts is a small Python script which makes password-guessing on your OpenSSH deployments virtually impossible, by allowing only a limited number of login attempts to your sshd. After a set number of tries, DenyHosts simply denies the given IP further attempts. What’s even cooler about DenyHosts, is that the most recent version (2.0) allows you to benefit from over 23.400 other peoples ban lists, thus meaning you’re saving yourself a lot of worrying about those pesky login attempts. An added bonus is that you’ll save yourself a few kB’s of network traffic and a few CPU cycles by straight-out denying any previous offenders a connection to your server. :)

Other people might just want to save themselves the trouble, and move their sshd to a different port. I believe that this solution is just security through obscurity, and is dangerous in the way that it gives a false sense of security to the admin. I disagree with that kind of solution, as you might notice if you look at the comments the article in question has generated. (Please notice that even though I disagree with his method of avoiding sshd bots, I deeply respect the work that Dan is putting into The FreeBSD Diary.)

DenyHosts depends on Python, which you have to make sure you want to have on your server. Python isn’t such a bad thing to have – but porting DenyHosts to sh would certainly reduce the amount of CPU cycles needed, and of course, remove Python as a dependency for those who don’t want to bloat their server.

So let’s get started, shall we?

First, make sure that you’ve got the updated ports tree (Use portsnap) and install the security/denyhosts port. If you’ve already got Python, this should only take a few seconds, as nothing needs to be compiled.

Once installed, DenyHosts gives you a lot of info about how to configure it, but I’ll repeat it just in case.

Open /etc/rc.conf, add the lines
denyhosts_enable="YES"
syslogd_flags="-c"
The first line ensures that DenyHosts starts at boot time. The second line makes sure that syslogd does not group repeated log messages, so that they appear as “Last line repeated 23234 times” – This makes it possible for DenyHosts to actually count how many login attempts any given hosts has made.

Now edit /etc/hosts.allow and add the lines
sshd : /etc/hosts.deniedssh : deny
sshd : ALL : allow
This will ensure that the hosts.allow mechanism in FreeBSD denies connections to your sshd from any of the hosts listed in hosts.deniedssh, and allows all other connections. The last line parsed is authoritative, so in theory if you add an IP to the block list in hosts.deniedssh, and allow the same IP later in the parsing process, the host will be allowed to connect.

Before we go further, we need to make sure that the hosts.deniedssh file exists and has the correct permissions:
touch hosts.deniedssh
chmod 644 hosts.deniedssh
chown root:wheel hosts.deniedssh

Almost there. For good measure, we should probably edit the /usr/local/etc/denyhosts.conf. If it’s not there, copy over denyhosts.conf-dist to denyhosts.conf, and fire up your favorite editor.
Make sure to read every single line of the sample configuration file, and adjust the settings to whatever fits your system and preferences. The most vital options here are the:

• SECURE_LOG, which should be /var/log/auth.log, just in case the “Mandrake, FreeBSD or OpenBSD” comment right above it confused you
• HOSTS_DENY, should be set to /etc/hosts.deniedssh just in case you forgot which file you were creating a second ago

All of the timing and recurrency options are quite sane defaults, so I’d recommend you not to touch them unless you either know what you’re doing, or if you’re very picky about these kinds of things.

Now you should have a nicely working DenyHosts installation, but wait, remember the 23.400 people who are contributing to the DenyHosts lists? Why not be a part of this amazing network? All it requires, is that you enable the synchronization feature. Just uncomment the SYNC_SERVER variable, and synchronization will work fine.

The reason why synchronization isn’t enabled by default even though it’s the optimal setting for the vast majority of installations, is because synchronization poses a theoretical security risk, and a theoretical privacy risk.

The security risk is – as I said – purely theoretical, as if some evil hacker was to be pissed off by all these people who are not responding to his ssh-bots, and should decide to hack the denyhosts synchronization server, he could send malicious code to all synchronizing DenyHosts clients. Remember, though, that this risk is as improbable as the sky falling down on our heads, and that anybody can buy a license to use the synchronization server for use in their private network. You could also reverse engineer the synchronization protocol that DenyHosts uses and make your own synchronization server – But then again, this entire paragraph consists solely of theoretical possibilities.

All that’s left to done is to restart the system (To make sure syslogd starts with the -c flag) and to make sure DenyHosts has started at boot time.
ps -ax|grep denyhosts
If you enabled synchronization, you can also watch as DenyHosts fills up the hosts.deniedssh file with lots and lots of persistent sshd-brute forcing zombies. Dare to cat it?

Enjoy your new non-obscurely secured sshd!

I use Windows XP on my desktop workstations at home (Because I’m lazy) and at work (Because I have to), so my primary way of interfacing with Unix servers is through PuTTy and it’s tools. I have at least one terminal window open at any time of the day, so you may say that PuTTy is my most frequently used application ever. During the course of using PuTTy, I’ve been fed up with having to double-click on the putty icon, type the server name, then type my user name and then password. There has to be a more efficient way, I thought.

After experimenting a bit, I realized that putty takes roughly the same command line arguments as your run-of-the-mill ssh client on *nix, so I set it up to be as fast as when invoking ssh in a command line. Here is how you take advantage of that.

Walkthrough:

First, we’re going to set up your $PATH in Windows so that you can exec binaries from the PuTTY install directory.

1. Left-click on My Computer
2. Select “Properties”
3. Go to the “Advanced” tab
4. Click “Environment Variables” button
5. Scroll down to “Path” variable in “System Variables” field
6. Double-click “Path” variable
7. Append your PuTTy install directory to the “Variable value” field, for example “;C:\Program Files\PuTTy” to the end of the line. Remember the colon first – It’s important. There’s probably a better way of doing this, but I’ll correct that later.
8. Apply settings, get back to desktop.

Now we’re done with all of the configuration. What we’ve done, is to add the putty install directory to the places where Windows searches for executable files when you invoke them by name alone. This principle is exactly the same as in any *nix, except that Microsoft somehow decided a GUI was a good way to do this.

The whole point of this article is that the way I launch putty on my system, is type WIN+R and write “putty username@server.tld”. Now all that is needed for me, is to type my password – or gain instant access if I’m using key-based authentication (More about that in a later post).

Enjoy!

Nixy.dk will be a blog about general Unix tips, and to some degree, FreeBSD-specific stuff.

First nixy post ETA 30min.