UniFi Access Points, error http code: 000, ntp issues

TL; DR: Scroll down for the simple solution. Read on for the full discovery of how intricately I shot myself in the foot with solid network security practices.

So I had this very niche error with the UniFi access points. Any time I tried to upgrade the access point in the UniFi controller web interface, it would fail silently.

Further investigation showed that upgrading from the console, using the upgrade command would write the following in /var/log/messages of the AP-AC-Pro-Gen2

Jan 1 01:01:03 hostname user.notice dak: Upgrade Firmware Download:
Jan 1 01:01:03 hostname user.notice dak: error http code: 000

Why was this? Well I don’t know of any HTTP error code numbered ‘000’ so running a manual file transfer in the terminal would provide further clues and a more verbose error message:

BZ.v3.8.6# curl https://dl.ubnt.com/unifi/firmware/U7PG2/3.8.14.6780/BZ.qca956x.v3.8.14.6780.170915.2223.bin
curl: (60) SSL certificate problem: certificate is not yet valid

If you look at the earlier logs I got from running the upgrade command, you’ll see a clue as to why curl thinks the certificate is not yet valid. The time on the server was set to the UNIX Epoch. Why was the time not correct? a ‘ps w’ showed the ntp client running just fine! The log did not show any ntp errors!
Running the ntp client manually, finally revealed the culprit behind my missing firmware upgrades.

BZ.v3.8.6# ntpd -nd -p dk.pool.ntp.org
ntpd: resolved peer dk.pool.ntp.org to 193.200.91.90
ntpd: sent query to 193.200.91.90
ntpd: timed out waiting for 193.200.91.90, reach 0x00, next query in 1s

OK. So why in the world would ntpd ever time out waiting for a server? Two options, either the server is offline or I’m an idiot. How likely is a public ntp server to be offline? How likely am I to be an idiot?

I run a tight ship. I have strict security policies. The access points management network is not the same as the network for the wifi clients. The wifi clients run on a separate VLAN. The management network is locked down tighter than a security consultants asshole during audit season. The management network only allowed HTTP(s) to public networks.

The solution

The solution is to ensure that all NTP queries are allowed to reach the public internet from the unifi access points.

Face, meet hand.

Dovecot: Unknown protocol: imaps

Came across this weird bug when upgrading a system through Weezy -> Jessie -> Stretch today. A server was running dovecot with imaps. After the upgrade, it was no longer responding on port 993.

After some digging around, i found out that the issue was located in 10-ssl.conf:

ssl = no

You can go ahead and change that setting to “ssl = required” and have another look at the various settings in the file.

REPOST: RTC problems in Debian on Dell PowerEdge servers

I’ve had some problems with RTC on Dell PowerEdge servers with Debian 4.0 (etch), which were unfixable by all of the following:

  • Set Hz to 100/250/1000
  • Try newer kernel
  • Add –directisa option to hwclock (Would only fix a symptom of the problem, and not touch the problem itself at all)

The symptoms:
You get a lot of messages like this:
rtc: lost some interrupts at 2048Hz
This varies (in my case) between 1024Hz, 2048Hz and 4096Hz. The message is printed to the terminal as fast as it can handle, and is written to syslog at a rate of 3500/sec – Though some times less.
If you run “hwclock” on the system, it spits out an error message:
select() to /dev/rtc to wait for clock tick timed out.

Chris Snyder suggests to avoid the problem instead of fixing it, by disabling precise timekeeping in VMware (Which is what was my main problem). What did fix the problem, however, was to compile a new kernel with HPET_EMULATE_RTC. This is more or less a battle in itself, as HPET_EMULATE_RTC depends on other stuff on its own. I’m to exhausted to give a proper walkthrough on this, as I’ve done about 50 kernel recompiles in the past week or so, and just want to forget this issue altogether ASAP.

Why HPET_EMULATE_RTC isn’t on by default is beyond my scope of comprehension, as there’s absolutely no good reason to not have it enabled unless you’re running an extremely small embedded kernel which doesn’t need it in the first place, in which case you’ll disabe it forcefully anyway.

Update: After notifying Chris about this blog post, he suggested that I update it with the kernel config options that I enabled to make HPET RTC emulation work. As opposed to what I wrote earlier, there’s not really hundreds of different .config options to edit. Only a few if you’re editing .config directly and only one if you’re using menuconf.

In menuconf, go Device Drivers ---> Character Devices ---> Enchanced Real Time Clock Support. You’ll see that it is built as a module. What we really need is to make it built-in. Just press Y. Now you can type / (slash) and search for HPET. Go to the bottom of the search results and you should hopefully see HPET_EMULATE_RTC=y.
For all the others, here’s the diff for the Debian 2.6.18 kernel:

147d146
< CONFIG_HPET_EMULATE_RTC=y
2140c2139,2141
< CONFIG_RTC=y
---
> CONFIG_RTC=m
> CONFIG_GEN_RTC=m
> CONFIG_GEN_RTC_X=y

Please note, CONFIG_HPET_EMULATE_RTC depends on CONFIG_RTC=y.