Debian’s stupid ntpd
Lately I came to notice that my computer’s time began to drift irrationally, although I have ntpd running. I have the default ntpd.conf, and in particular with this servers list:
server 0.debian.pool.ntp.org iburst server 1.debian.pool.ntp.org iburst server 2.debian.pool.ntp.org iburst server 3.debian.pool.ntp.org iburst
So why in the world would my time drift? This configuration has been working fine for years. I began to check the synchronization using ntpdc:
ntpdc -c peers
And there, I saw only one line, telling me that I peered with 172.17.0.2. For instance, my network is 192.168.42.0/24 and runs no ntp server, so there is a bit of a problem here. In fact, I went to someplace where the DHCP announces a ntp server, and the configuration was kept ever since.
Thus, a very simple solution exists:
sudo rm -f /var/lib/ntp/ntp.conf.dhcp service ntpd restart
And now everything is well:
17:57 remy@magi ~ % ntpdc -c peers
remote local st poll reach delay offset disp
=======================================================================
*2a01:e0b:1:88:2 2a01:d35:213d:8 2 64 377 0.02591 -0.028799 0.03157
=thor.netservice 192.168.42.129 2 64 377 0.14030 -0.028539 0.03108
=utility-lax.rac 192.168.42.129 3 64 377 0.19328 -0.016656 0.03149
=isaachayes.khre 192.168.42.129 2 64 377 0.18999 -0.020809 0.03113
We’ll note that it furiously looks like Debian’s #569775 bug.
Howto not crash Cisco’s NAT
Maiz is the dormitory of my school, and it hosts around 170 people. As a school of network & telecom, it is only natural that us students manage our own network, and so do we.
However, we suffer from severe bandwidth limitations. Indeed, because of many factors, we can only have a few ADSL lines. Still, we try to give our comrades a decent Internet connection, but with only lousy Netgears modems (see the previous post), it’s quite difficult to get the network to scale up.
So the current challenge is to find modems that will not crash after a few hours/minutes. To give you an order of idea, there is approximately 20 connections opened each second. We have to reboot the Netgear every 20 minutes in order to keep it alive…
Then we tried a Cisco 1700 with a modem card. It goes fine, but however crashed after a few hours. Reason ? Too many NAT entries and no RAM left. And indeed, Cisco’s timeouts are really, really conservative: the default TCP timeout is 24 hours. After setting those to more reasonable values, it seems to work better!
ip nat translation timeout 3 ip nat translation tcp-timeout 600 ip nat translation udp-timeout 120 ip nat translation finrst-timeout 3 ip nat translation syn-timeout 10 ip nat translation dns-timeout 3 ip nat translation icmp-timeout 3 ip nat translation max-entries 30000
You’ll note the last line setting the max-entries. Its purpose is not to limit the users, but to have the router to refuse connections before it gets out of memory.
I can’t wait for IPv6 and for this f*cking NAT to be dead! #optimistic
Why is Microsoft going to fuck everyone’s back?
Before anything, I’m far, far, very far to be a Microsoft fanboy. However, I lately came to (re-)discover things made by Microsoft that make me think that they are not the club of incompetent idiots that everybody tends to take them for.
By example, it’s of common knowledge that Microsoft did never invent anything and did copy all its products on its competitors. Really ? We’ll have a quick tour :
- Peedy the Talking Parrot (1995). Have you ever heard of him ? Well, it’s a parrot able to understand natural language to execute various tasks. In the video demos, you can see how you could control your music player using natural language sentences, and get a human-like answer (including expressions). Doesn’t it quite remember Apple’s Siri ?
- The Briefcase. You can put documents in it, and then synchronize them between computers. It appeared at least in Windows 95, and seriously reminds of Dropbox.
- Microsoft’s vision of Internet in… Windows 98 installer. Remember those texts that the installer displays in order to make you wait by teasing you on how awesome Windows is. If you read them with today’s eye, they mostly describe a vision of Internet where you can do e-shopping, have a virtual identity, etc. This vision is being realized, but its accomplishment is not even finished!
- Microsoft Bob. All right, this one is even worse than an EPIC fail, because they actually tried to make the computer look like all other common objects with the technology they had back in the time. You can’t fit a house in a screen. But you’ll fit all sorts of electronic devices, corresponding to Bob’s applications, inside your house. That’s exactly what ubiquitous computing is about! We’ll note that ubiquitous computing opposes itself to virtual reality, but I explained the relation I see between the two.
So, why a few 20-years old epic fails would mean that Microsoft is not staying in his grave? After all, as we do all know, they splendidly missed the Web 2.0 thing, and that’s what puts them in a “bad” position. But is it really a fault? Why did they need so long to get Internet Explorer back on tracks? Firefox probably stole 20% of their market share before they woke up. Don’t tell me that one of the largest company in the world can’t get a web browser right. Ironically enough, we’ll note that the whole Ajax/Web 2.0 hype relies on one and single thing: the XHR object, that was introduced by Microsoft.
But maybe that they see on the longer term. If no Web 3.0 is coming out (shut up you troll in the back), it’s probably because the Web 2.0 is merely a transition to something else, that is, once again, ubiquitous computing. See the smartphone/tablets delirium that’s floating around? It’s only the beginning.
And for 20 years, Microsoft did not make a move on anything. They just polished their Office pack, killed GPU for Aero to work and other superficial stuffs like that. Fundamentally, Windows 7 does nothing more than Windows 3.11, if you make abstraction of the technical side of things. But for 20 years, Microsoft Research was not shut down, and you sometimes see them surfacing. Like Surface, or all the thing’s they’ve been doing on robotics. Those things are still in the labs, because they’ve learned of their mistakes and now they’ll wait for the market to be ready before to take a product out.
The first strike: the Kinect of course. What’s next? Who knows… My belief is that they’ll use a strategy similar to Apple’s. When a new product starts to take off (like smartphones by example), they commercialize a new product more advanced than all of the others (at least, marketingly speaking).
In other words, if you’re going to do new products, watch your back, because Microsoft probably has a better one in its labs!
There is however another possiblity: Microsoft guys are indeed plain idiots, and thus Microsoft Research did never manage to get anything working, which means that in 10 years I’ll re-read this post and roll on the floor laughing to be so wrong. Time will tell :)
“Modem” (= bridge) mode on Netgear DG632
Recently I configured a Netgear DG632 in a bridge mode. Initially I was expecting the modem to translate from PPPoE on the LAN port to PPPoA on the WAN port. Wrong :)
In short, the configuration :
---- Machine 1
((Internet))----[ DG632 ]----[ Gateway ]----|---- Machine 2
---- Machine N
Actually, it dials the ISP and identifies itself just as in the router mode, but the difference is that it expects to find a computer with the public IP on the LAN port. However the router that you gateway must use is not the one given by IPCP, and rather the management IP of the modem. Here’s the relevant part of my FreeBSD’s rc.conf:
ifconfig_re1="inet 193.251.78.150 netmask 255.255.255.0" static_routes="adsl" route_adsl="-host 192.168.1.1 -iface re1"
KVM-ized Debian on LVM
The question of storing virtual machines datas has a lot of answers, but only a few seems to be as flexible as what you’ll find on a plain old physical machine.
First try
For instance, what I looked for was to be able to have a storage pool inside of which I can extend partitions sizes as needed, because I host several services with relatively unpredicted expansion.
Usually when you need to do that, you can use LVM, and hot resize you partitions. However, the virtual world is not so simple, and this can’t be done directly. My first try was to have my standard LVM pool in which I would create the logical volumes I need, and then add them as independent disks.
This appoach works, however the Debian installer requires that you create a partition table, which makes it pretty difficult to use from the host, and moreover is almost impossible to resize.
A few commands
Imagine that the volume storing your virtual hard disk is the LVM logical volume /dev/mapper/myvm-root. First you have to tell the kernel to map the devices from the partition table:
kpartx -a /dev/mapper/myvm-root ls /dev/mapper/myvm-root*
This shows you that your partitions appeared in the mapper. You can now use them as usual
mount /dev/mapper/myvm-root1 /mnt
But now if you want to resize the hard disk, you have to resize your LVM volume, but also to make the change in your partition table. As nobody has ever heard of size-changing hard disks, I don’t know any tool that does this, so unless I missed an option somewhere, it’s pretty much fucked up for you.
Better solution
In fact, you must create the partitions beforehand, so the installer sees them and can use them without creating a brand new partition table. And you can see the partition directly from the host.
A few other commands
Once the logical volume is created, you can format it the standard way:
mkfs.ext4 /dev/mapper/myvm-root
Then when you launch the Debian installer you’ll see the partition as already existing, and won’t be prompted to create a partition table:
Once your Debian is installed, you can just mount the partition(s):
mount -o ro /dev/mapper/myvm-root /mnt
Or if your VM is down, you can even resize the volume
lvresize -L +1G /dev/mapper/myvm-root e2fsck -f /dev/mapper/myvm-root resize2fs /dev/mapper/myvm-root
Then when you reboot your VM it will have got a 1 gigabyte bigger hard disk.
Other Approaches
This is certainly not the only way to achieve the goal of be able to dynamically resize your hard drive. For instance, you could probably manage something with qcow2 disks growing as needed, or maybe create several disks that you aggregate as physical volumes, or who knows.
But the solution I found here is what I found to be the more practical to use, without loosing too much performances. Still, you can’t do hot resize, and this might be a problem for some people, but I don’t see how you could hot-resize virtual disks. I guess it’s a feature to come!
Anyway, if you know a better way to get this done, I’m quite interested in it ;)
New Project: Karte
Nothing ground-breaking today, I’m just starting a new project. Once again it’s university-related, and this time it’s going to have a real-life application. Indeed, the goal is to create a payment card for all pupils. It’s going to allow electronic micro-payments in various places and occasions (parties, cafeteria, …).
There will be a client, used on the points of sale, and a server, probably backed-up by a PostgreSQL database. The client is going to use Qt, and the server will use the very good Twisted framework, that is a Python framework that allows you to easily create and deploy network applications.
For now the only thing more or less advanced is the data model, created using the Django ORM and Modelviz. We probably won’t use the ORM at all, but it helped to design the database without loosing too much time with poor database design tools. And it generates both the graph and the SQL!
Regarding the “card” technology in itself, it’s probably going to be RFID/NFC. Not because it’s hype, but because it’s cheap and the student card is a NFC anyway, so we’ll try to use it! And by the way, NFC is quite the same thing as RFID, it’s just that RFID designates a wider range of technologies than NFC. In case you are being confused, it’s just marketing, not technology.
Anyway, since I’m being evaluated on this project, I’m pretty sure I will finish it, or at least greatly advance it. So stay tuned to see how it turns out :)
Armadeus, Wiimotes and Lightsabers
Those last weeks I had lessons talking about embedded systems, and in order to concretise what we learned, we had to make a project using embedded boards. So we had two Armadeus APF27 equipped with a touchscreen. When the teacher asked about ideas for a project, we thought about a game where you can fight someone else using a Wiimote as a Lightsaber.
What is great about Armadeus boards, it’s that Qt is included in the default buildroot, so it’s easy to make graphical applications. We also used Cwiid as the Wiimote library, that is included as well. The application works with 2 boards connected through ethernet, plus bluetooth dongles on the USB port to connect the Wiimotes.
Currently the application is quite basic and buggy, because we had few time to get something really stable. The network protocol lacks of reliability mechanisms (it’s built over UDP), there is sever performance issues, mainly because of the too high sampling level of the sensors, and there is segfaults when not doing things in the right order.
Still, we could enjoy having a good lightsaber fight ! To anyone who’d like to try/read the code, see you on github :) .
OVH Survival Guide : Update
I had some reactions from OVH (1, 2, 3, 4, 5) since the previous post, and it looks like that some things were updated since I last used some features.
Rescue Mode
In order to be able to access quickly your server when rebooted into rescue mode, there is now the possibility to put your SSH key, that will automatically be installed into the rescue server. And it looks like that the mail is now pretty quick to arrive. I did not crash my server for the test however.
There is also something that I forgot to mention in my previous post, it’s the fact that there is a certain amount of time to keep between forceful reboots. This means that when leaving rescue mode, you have to be pretty sure of what you’re doing, as rebooting is not “free”.
Backup
I’m also told that the FTP server are improved, but I don’t really know how. From what I see, they pushed up the number of simultaneous connection from 2 to 3. The banner of the server says that IPv6 is accepted, but the domain name do not give me any AAAA entry. Anyway, duplicity stays a quite useful option for backup.
OVH Release
Apparently, a Release 3 is in preparation. Let’s hope it will be be easier to maintain than the Release 2, like having a working package manager. Wait and see :)
Phone Music
Hell yeah, actually it’s changed! I’d prefer the nyan cat, but I think that the current music is as fine as a hotline music can be. Again, phone is not made for music.
Reboots When Not Responsive
I was a bit confused when talking about automatic reboots because of monitoring, and forgot half of the process (it’s been a long time that I did not crash my servers!). So please forget what I said earlier, and replace it by the following:
When your server do not ping, they send a technician to check out what’s been happening. He will try to reboot the machine by doing some minimal maintenance, and if it fails he will reboot it into rescue mode.
So far that’s good, but in fact there’s a terrible flaw: when the technician is called, you cannot do any reboot on your own, and you have to wait for its intervention (say 15 minutes). If it’s the first time that the server crashes, it’s all right, but when you are debugging the server and that you did something wrong, the server will go into maintenance before you can do anything, and then instead of waiting 5 minutes for the rescue mode, you wait 20 minutes for the technician.
In short, you can leave the monitoring activated under normal circumstances, but when debugging a crashed server, think to turn it off.
Conclusion (again)
This time I think that I did not forget nor confuse anything, but who knows :)
Soon featured on hyperthese: why are hotline music comparable to absolute evil? Stay tuned!
OVH Dedicated Server Survival Guide
I have been client of OVH for years now, and I never was disappointed: they won’t sell you the moon (for instance, cheap crappy servers), but you get what you paid for, including a very good support. However, with time I discovered some issues that might be noteworthy to mention.
EDIT: I posted an update to this post.
Installing the server
The first thing you get to do with your server is to install an OS. Here you usually choose whatever you prefer. If in doubt, choose Debian, since it is the best OS ever you can’t be deceived by it, but you are free to have bad tastes. Ahem. Whatever, there is a golden rule here: do not ever under any pretext choose the “OVH Release 2″. Never ever. It’s kinda a transgenic Gentoo, so heavily modified that you can’t do anything on it. In short, the only way to administrate it is to use the embedded Webmin. And forget about updates. Urgh.
If I recall correctly, when you install your server for the first time, you don’t have much choice about partitioning. Or maybe is it because I did that long ago ? Anyway, if you want custom partitioning, in the worst case you can do it by re-installing the server, the process is quite fast and straightforward. However as far as I am concerned, I find the web interface a bit too restricted regarding LVM configuration, so I generally simply install a standard OVH Debian, create my LVM layout, and debootstrap the thing.
Kernel Updates
There is something else that I find embarrassing, it’s the fact that the kernel is supposed to be loaded by netboot. Their rationale is this: they give you an always up-to-date kernel patched with grsecurity. Why not, this should help you not worrying about the kernel. But in fact, you don’t want the last kernel, you just want a secure kernel. Indeed, you can achieve this by using the OVH kernel, but that would imply to reboot each time there’s a new version out, because you can’t really sort out which versions include a security fix, and which ones do not.
What I prefer to do instead, it’s to install a standard Debian kernel, and read debian-security-announces. This way I only have to reboot when my kernel is flawed.
Server Monitoring
OVH is able to perform some basic monitoring on your server, which can be useful, but to some extent might lead you to serious troubles. Not that monitoring is bad, but by default if your server do not ping for 5 minutes or so, it will automatically be rebooted into rescue mode. Big mistake. Because the day when your server will go through a long fsck while booting, it will never be able to start and it might take you a long time to figure out why (true story, and remember you don’t see the screen).
What I always do is to disable this auto-reboot, and do manual reboots when needed. It won’t cause longer downtime, because the auto-reboot puts the server in rescue mode, and then no services are served. There’s quite fewer disadvantages to manually reboot the server. But of course, it’s also up to you to decide of an action plan to detect when your server(s) go down and what to do then.
Rescue Mode
Talking about rescue mode, what is it? It’s a special kernel/distro you can boot through your manager, that embeds a web interface and some basic tools to help you repair your server in case of problems. This is a very, very useful feature! Nothing bad to mention about it, just wanted to bring it up in case you wouldn’t know it.
Oh I’ll forgot. There is a slight problem with it, actually. When you reboot into rescue mode, the root password will be mailed to you. It is fine, but the mail might take quite a while to reach its destination (anywhere between 5 and 20 minutes).
Backup
I think that nowdays, each and every OVH server comes with at least 100Gio of FTP backup. This is really great, but their FTP is quite paranoid and limited. Not that I don’t understand the reasons behind this, but this means that you’ll have to deal with it. A few years ago, I created a script based on incremental tar snapshots to do the job, but recently I discovered duplicity, who fills the duty pretty well and is quite simple to use.
Support
OVH support is great: they are not just reading procedures, they know what they talk about. And when you have a hardware problem, you won’t have any trouble to get it changed. For instance, I had a flawed hard drive, and after a simple email showing a SMART report, I had the replacement planned. And they kept the old drive connected, so I could copy the data. I also had several issues of failing power, which were well detected and immediately replaced. Big up for OVH support !
One dark point however: the phone support is just awful. Not because of the people behind it, but because of the utterly atrocious music (seriously, did anyone tell the people making hotlines about Shannon’s Theorem? You CAN’T put high-pitched music on the phone), and because of the long waiting. If you want something done, just use the email support, it will save you a lot of time and money and headaches.
Mail history
The OVH manager is quite a piece of software, with a lot of surprises awaiting you. While I seriously doubt the mental sanity of people coding this stuff, you can’t avoid it to administrate your server and associated services. There is a quite unknown and half-hidden feature that however can be very useful: the emails history. You can find the whole history of emails that OVH sent to you. Including sometimes those that you did not receive (yet). Might boost the time to access the Rescue Mode (or not).
To access it, in French: from the Manager v3, go to “Administration”, “Mes paramètres” and then choose “Historique des emails”. I don’t know the English translation, probably “Administration” -> “My parameters” -> “History of emails”.
Conclusion
Of course, everything isn’t nice and shiny, but I never had any serious problem with OVH on the areas that matter. Whatever people say, the network is quite good, there is IPv6, I never had a power outage, and the support is just fine. I clearly won’t recommend OVH for specific applications, say e-commerce, but for general purpose hosting with a good overall quality of service, I’m quite satisfied, once I’ve applied my little workarounds. I have been around for at least 4 years, and I am still alive :)
Old Blog
To those nostalgic of the old blog, it can still be found at http://old.hyperthese.net/.
