I’m currently trying to figure out how to install additional software on ESXi. This is a bit of a tough one for now.

My findings are as follows:

• There are two vmfs file systems which contain a set of .tgz files. The contents are identical, but with the hash sums do not match
• One of these file system trees contain a checksum.md5 file. The other does not – I suppose that the one with the checksums is what ESXi considers as being the “original” files.
• When I replaced /lib/libc.so.6 on the server, it immediately dropped all my ssh sessions, and denied access to the console. Once I power cycled the server, all the changes I made to the userland and base of the OS were reverted (I’m guessing they came from the “recovery” file system)
• ESXi complains if the libraries in use do not have a VMWARE_SOMETHING version string (And thus the previously described scenario occurs)
• Most of the userland consists of busybox and dropbear sshd. There’s also gpg.

I’m not big on building software, the usage of gcc, or anything related to linux for that matter, so help from some linux guru on this matter would be greatly appreciated.

The origin of the bug for me is unknown. Naturally, I’ve had to research this issue as it became apparent to me when I had to set up two network interfaces in bonding (Thus calling them bond0) and then putting this new virtual interface into work with two 8021q vlan’s (Thus calling them bond0.100 and bond0.101). Now, I do realize that having a 9-character long interface name is a bit on the side of the unusual, but rarely you’d spot any software which supports any less than 255 characters, right? I mean, this is just common sense these days. Pretending to hunt for bits of RAM by shaving off something like 248 bytes just SO belongs back in 1980′s computing. Not fixing this bug from version 1.0.0 to version 1.0.4 of your software belongs back at Microsoft. Not having a part of your website dedicated to bug reports belongs back in FAIL.

Anyway, this is not an opinion piece.

After spending a good part of a weeks worth of office time hunting down this bug, I finally decided to trace back my steps and see if there was something I could do to solve this problem.

When creating vlans on an interface in Linux, you usually use a tool called vconfig. Turns out that vconfig had the option to provide a custom naming scheme. Just run vconfig set_name_type VLAN_PLUS_VID_NO_PAD and you’re golden. Now your interface name is called vlan100 or something similar.

• Set Hz to 100/250/1000
• Try newer kernel
• Add –directisa option to hwclock (Would only fix a symptom of the problem, and not touch the problem itself at all)

The symptoms:
You get a lot of messages like this:
rtc: lost some interrupts at 2048Hz
This varies (in my case) between 1024Hz, 2048Hz and 4096Hz. The message is printed to the terminal as fast as it can handle, and is written to syslog at a rate of 3500/sec – Though some times less.
If you run “hwclock” on the system, it spits out an error message:
select() to /dev/rtc to wait for clock tick timed out.

Chris Snyder suggests to avoid the problem instead of fixing it, by disabling precise timekeeping in VMware (Which is what was my main problem). What did fix the problem, however, was to compile a new kernel with HPET_EMULATE_RTC. This is more or less a battle in itself, as HPET_EMULATE_RTC depends on other stuff on its own. I’m to exhausted to give a proper walkthrough on this, as I’ve done about 50 kernel recompiles in the past week or so, and just want to forget this issue altogether ASAP.

Why HPET_EMULATE_RTC isn’t on by default is beyond my scope of comprehension, as there’s absolutely no good reason to not have it enabled unless you’re running an extremely small embedded kernel which doesn’t need it in the first place, in which case you’ll disabe it forcefully anyway.

Update: After notifying Chris about this blog post, he suggested that I update it with the kernel config options that I enabled to make HPET RTC emulation work. As opposed to what I wrote earlier, there’s not really hundreds of different .config options to edit. Only a few if you’re editing .config directly and only one if you’re using menuconf.

In menuconf, go Device Drivers ---> Character Devices ---> Enchanced Real Time Clock Support. You’ll see that it is built as a module. What we really need is to make it built-in. Just press Y. Now you can type / (slash) and search for HPET. Go to the bottom of the search results and you should hopefully see HPET_EMULATE_RTC=y.
For all the others, here’s the diff for the Debian 2.6.18 kernel:

147d146
< CONFIG_HPET_EMULATE_RTC=y
2140c2139,2141
< CONFIG_RTC=y
---
> CONFIG_RTC=m
> CONFIG_GEN_RTC=m
> CONFIG_GEN_RTC_X=y

Please note, CONFIG_HPET_EMULATE_RTC depends on CONFIG_RTC=y.