Debian’s stupid ntpd
Lately I came to notice that my computer’s time began to drift irrationally, although I have ntpd running. I have the default ntpd.conf, and in particular with this servers list:
server 0.debian.pool.ntp.org iburst server 1.debian.pool.ntp.org iburst server 2.debian.pool.ntp.org iburst server 3.debian.pool.ntp.org iburst
So why in the world would my time drift? This configuration has been working fine for years. I began to check the synchronization using ntpdc:
ntpdc -c peers
And there, I saw only one line, telling me that I peered with 172.17.0.2. For instance, my network is 192.168.42.0/24 and runs no ntp server, so there is a bit of a problem here. In fact, I went to someplace where the DHCP announces a ntp server, and the configuration was kept ever since.
Thus, a very simple solution exists:
sudo rm -f /var/lib/ntp/ntp.conf.dhcp service ntpd restart
And now everything is well:
17:57 remy@magi ~ % ntpdc -c peers
remote local st poll reach delay offset disp
=======================================================================
*2a01:e0b:1:88:2 2a01:d35:213d:8 2 64 377 0.02591 -0.028799 0.03157
=thor.netservice 192.168.42.129 2 64 377 0.14030 -0.028539 0.03108
=utility-lax.rac 192.168.42.129 3 64 377 0.19328 -0.016656 0.03149
=isaachayes.khre 192.168.42.129 2 64 377 0.18999 -0.020809 0.03113
We’ll note that it furiously looks like Debian’s #569775 bug.
KVM-ized Debian on LVM
The question of storing virtual machines datas has a lot of answers, but only a few seems to be as flexible as what you’ll find on a plain old physical machine.
First try
For instance, what I looked for was to be able to have a storage pool inside of which I can extend partitions sizes as needed, because I host several services with relatively unpredicted expansion.
Usually when you need to do that, you can use LVM, and hot resize you partitions. However, the virtual world is not so simple, and this can’t be done directly. My first try was to have my standard LVM pool in which I would create the logical volumes I need, and then add them as independent disks.
This appoach works, however the Debian installer requires that you create a partition table, which makes it pretty difficult to use from the host, and moreover is almost impossible to resize.
A few commands
Imagine that the volume storing your virtual hard disk is the LVM logical volume /dev/mapper/myvm-root. First you have to tell the kernel to map the devices from the partition table:
kpartx -a /dev/mapper/myvm-root ls /dev/mapper/myvm-root*
This shows you that your partitions appeared in the mapper. You can now use them as usual
mount /dev/mapper/myvm-root1 /mnt
But now if you want to resize the hard disk, you have to resize your LVM volume, but also to make the change in your partition table. As nobody has ever heard of size-changing hard disks, I don’t know any tool that does this, so unless I missed an option somewhere, it’s pretty much fucked up for you.
Better solution
In fact, you must create the partitions beforehand, so the installer sees them and can use them without creating a brand new partition table. And you can see the partition directly from the host.
A few other commands
Once the logical volume is created, you can format it the standard way:
mkfs.ext4 /dev/mapper/myvm-root
Then when you launch the Debian installer you’ll see the partition as already existing, and won’t be prompted to create a partition table:
Once your Debian is installed, you can just mount the partition(s):
mount -o ro /dev/mapper/myvm-root /mnt
Or if your VM is down, you can even resize the volume
lvresize -L +1G /dev/mapper/myvm-root e2fsck -f /dev/mapper/myvm-root resize2fs /dev/mapper/myvm-root
Then when you reboot your VM it will have got a 1 gigabyte bigger hard disk.
Other Approaches
This is certainly not the only way to achieve the goal of be able to dynamically resize your hard drive. For instance, you could probably manage something with qcow2 disks growing as needed, or maybe create several disks that you aggregate as physical volumes, or who knows.
But the solution I found here is what I found to be the more practical to use, without loosing too much performances. Still, you can’t do hot resize, and this might be a problem for some people, but I don’t see how you could hot-resize virtual disks. I guess it’s a feature to come!
Anyway, if you know a better way to get this done, I’m quite interested in it ;)
OVH Survival Guide : Update
I had some reactions from OVH (1, 2, 3, 4, 5) since the previous post, and it looks like that some things were updated since I last used some features.
Rescue Mode
In order to be able to access quickly your server when rebooted into rescue mode, there is now the possibility to put your SSH key, that will automatically be installed into the rescue server. And it looks like that the mail is now pretty quick to arrive. I did not crash my server for the test however.
There is also something that I forgot to mention in my previous post, it’s the fact that there is a certain amount of time to keep between forceful reboots. This means that when leaving rescue mode, you have to be pretty sure of what you’re doing, as rebooting is not “free”.
Backup
I’m also told that the FTP server are improved, but I don’t really know how. From what I see, they pushed up the number of simultaneous connection from 2 to 3. The banner of the server says that IPv6 is accepted, but the domain name do not give me any AAAA entry. Anyway, duplicity stays a quite useful option for backup.
OVH Release
Apparently, a Release 3 is in preparation. Let’s hope it will be be easier to maintain than the Release 2, like having a working package manager. Wait and see :)
Phone Music
Hell yeah, actually it’s changed! I’d prefer the nyan cat, but I think that the current music is as fine as a hotline music can be. Again, phone is not made for music.
Reboots When Not Responsive
I was a bit confused when talking about automatic reboots because of monitoring, and forgot half of the process (it’s been a long time that I did not crash my servers!). So please forget what I said earlier, and replace it by the following:
When your server do not ping, they send a technician to check out what’s been happening. He will try to reboot the machine by doing some minimal maintenance, and if it fails he will reboot it into rescue mode.
So far that’s good, but in fact there’s a terrible flaw: when the technician is called, you cannot do any reboot on your own, and you have to wait for its intervention (say 15 minutes). If it’s the first time that the server crashes, it’s all right, but when you are debugging the server and that you did something wrong, the server will go into maintenance before you can do anything, and then instead of waiting 5 minutes for the rescue mode, you wait 20 minutes for the technician.
In short, you can leave the monitoring activated under normal circumstances, but when debugging a crashed server, think to turn it off.
Conclusion (again)
This time I think that I did not forget nor confuse anything, but who knows :)
Soon featured on hyperthese: why are hotline music comparable to absolute evil? Stay tuned!
OVH Dedicated Server Survival Guide
I have been client of OVH for years now, and I never was disappointed: they won’t sell you the moon (for instance, cheap crappy servers), but you get what you paid for, including a very good support. However, with time I discovered some issues that might be noteworthy to mention.
EDIT: I posted an update to this post.
Installing the server
The first thing you get to do with your server is to install an OS. Here you usually choose whatever you prefer. If in doubt, choose Debian, since it is the best OS ever you can’t be deceived by it, but you are free to have bad tastes. Ahem. Whatever, there is a golden rule here: do not ever under any pretext choose the “OVH Release 2″. Never ever. It’s kinda a transgenic Gentoo, so heavily modified that you can’t do anything on it. In short, the only way to administrate it is to use the embedded Webmin. And forget about updates. Urgh.
If I recall correctly, when you install your server for the first time, you don’t have much choice about partitioning. Or maybe is it because I did that long ago ? Anyway, if you want custom partitioning, in the worst case you can do it by re-installing the server, the process is quite fast and straightforward. However as far as I am concerned, I find the web interface a bit too restricted regarding LVM configuration, so I generally simply install a standard OVH Debian, create my LVM layout, and debootstrap the thing.
Kernel Updates
There is something else that I find embarrassing, it’s the fact that the kernel is supposed to be loaded by netboot. Their rationale is this: they give you an always up-to-date kernel patched with grsecurity. Why not, this should help you not worrying about the kernel. But in fact, you don’t want the last kernel, you just want a secure kernel. Indeed, you can achieve this by using the OVH kernel, but that would imply to reboot each time there’s a new version out, because you can’t really sort out which versions include a security fix, and which ones do not.
What I prefer to do instead, it’s to install a standard Debian kernel, and read debian-security-announces. This way I only have to reboot when my kernel is flawed.
Server Monitoring
OVH is able to perform some basic monitoring on your server, which can be useful, but to some extent might lead you to serious troubles. Not that monitoring is bad, but by default if your server do not ping for 5 minutes or so, it will automatically be rebooted into rescue mode. Big mistake. Because the day when your server will go through a long fsck while booting, it will never be able to start and it might take you a long time to figure out why (true story, and remember you don’t see the screen).
What I always do is to disable this auto-reboot, and do manual reboots when needed. It won’t cause longer downtime, because the auto-reboot puts the server in rescue mode, and then no services are served. There’s quite fewer disadvantages to manually reboot the server. But of course, it’s also up to you to decide of an action plan to detect when your server(s) go down and what to do then.
Rescue Mode
Talking about rescue mode, what is it? It’s a special kernel/distro you can boot through your manager, that embeds a web interface and some basic tools to help you repair your server in case of problems. This is a very, very useful feature! Nothing bad to mention about it, just wanted to bring it up in case you wouldn’t know it.
Oh I’ll forgot. There is a slight problem with it, actually. When you reboot into rescue mode, the root password will be mailed to you. It is fine, but the mail might take quite a while to reach its destination (anywhere between 5 and 20 minutes).
Backup
I think that nowdays, each and every OVH server comes with at least 100Gio of FTP backup. This is really great, but their FTP is quite paranoid and limited. Not that I don’t understand the reasons behind this, but this means that you’ll have to deal with it. A few years ago, I created a script based on incremental tar snapshots to do the job, but recently I discovered duplicity, who fills the duty pretty well and is quite simple to use.
Support
OVH support is great: they are not just reading procedures, they know what they talk about. And when you have a hardware problem, you won’t have any trouble to get it changed. For instance, I had a flawed hard drive, and after a simple email showing a SMART report, I had the replacement planned. And they kept the old drive connected, so I could copy the data. I also had several issues of failing power, which were well detected and immediately replaced. Big up for OVH support !
One dark point however: the phone support is just awful. Not because of the people behind it, but because of the utterly atrocious music (seriously, did anyone tell the people making hotlines about Shannon’s Theorem? You CAN’T put high-pitched music on the phone), and because of the long waiting. If you want something done, just use the email support, it will save you a lot of time and money and headaches.
Mail history
The OVH manager is quite a piece of software, with a lot of surprises awaiting you. While I seriously doubt the mental sanity of people coding this stuff, you can’t avoid it to administrate your server and associated services. There is a quite unknown and half-hidden feature that however can be very useful: the emails history. You can find the whole history of emails that OVH sent to you. Including sometimes those that you did not receive (yet). Might boost the time to access the Rescue Mode (or not).
To access it, in French: from the Manager v3, go to “Administration”, “Mes paramètres” and then choose “Historique des emails”. I don’t know the English translation, probably “Administration” -> “My parameters” -> “History of emails”.
Conclusion
Of course, everything isn’t nice and shiny, but I never had any serious problem with OVH on the areas that matter. Whatever people say, the network is quite good, there is IPv6, I never had a power outage, and the support is just fine. I clearly won’t recommend OVH for specific applications, say e-commerce, but for general purpose hosting with a good overall quality of service, I’m quite satisfied, once I’ve applied my little workarounds. I have been around for at least 4 years, and I am still alive :)
