This is how anyone’s first French lesson should start! 
General Approach
Over the past week, I have been doing benchmarking on KVM performance. The reason for this is that I want to use KVM on my new server and need to know what the impact of KVM is on performance and how significant the effect of certain optimizations is.
Googling about KVM performance, I found out that a number of optimizations can be useful for this:
The tests will include specific benchmarks focused on disk and network, s well as more general benchmarks such as unixbench and a kernel compilation..
After running quite a few tests with different configurations and doing also some manual testing, I am finding out that more or less the main factor in performance of KVM versus native is the disk IO. In particular, the schedulilng on both host and guest seems to have a significant effect. What it looks like is even that, contrary to wisdom and rationale on the internet, it is not more efficient to use noop as scheduler for IO in the guest.
Unfortunately though I had to fix an issue in the VMs related to disk partitioning and I am now not 100% sure anymore of the settings that I used to run the tests. So I need to rerun tests and it is also time for a more systematic approach now that I have preliminaray results. In fact, I will now automate the entire testing even further so I can run multiple configurations sequentially and also verify a larger number of combinations.
To be continued.
I have started to do benchmarking with virtual machines under opensuse 11.3 with KVM and had setup a number of different domains to run the same OS under different settings. Just one minor issue occurred which was that some domains with device emulation for the disk wouldn’t startup anymore. Very strange since I tested this before, starting from a working setup with disk emulation and then switching to the virtio drivers.
After some troubleshooting it appears it has to do with the drivers that are loaded at boot time through the init ram disk. Apparently, when creating the init ram disk under an OS with virtio drivers, it does not automatically load the drivers for IDE and SCSI, so disk emulation does not work.
The problem is easily illustrated by the output of mkinitrd on the guest using IDE emulation:
Kernel image: /boot/vmlinuz-2.6.34.7-0.5-default
Initrd image: /boot/initrd-2.6.34.7-0.5-default
Kernel Modules: thermal_sys thermal scsi_mod libata ata_piix ata_generic processor fan virtio virtio_pci virtio_ring virtio_net virtio_blk dm-mod dm-snapshot crc16 jbd2 ext4 pata_sl82c105 pata_hpt3x2n sata_mv pata_sch pata_netcell pata_acpi pata_sc1200 pata_it8213 sata_vsc pata_serverworks sata_via pata_ns87415 ahci pcmcia_core pcmcia pata_pcmcia pata_mpiix pata_jmicron pata_piccolo sata_svw sata_inic162x pdc_adma pata_atp867x pata_ali pata_hpt3x3 pata_efar pata_marvell pata_sil680 pata_cs5530 pata_pdc202xx_old sata_sil pata_it821x sata_sil24 pata_cypress pata_opti sata_promise sata_nv pata_optidma pata_sis pata_hpt37x pata_cmd640 pata_artop pata_amd sata_qstor sata_uli pata_cs5520 sata_sis pata_radisys pata_rz1000 sata_sx4 pata_cmd64x pata_ns87410 pata_triflex pata_hpt366 pata_ninja32 pata_via pata_rdc pata_atiixp pata_pdc2027x pata_oldpiix sd_mod usbcore mmc_core ssb ohci-hcd ehci-hcd uhci-hcd usbhid linear
Features: dm block usb lvm2 resume.userspace resume.kernel
Bootsplash: openSUSE (800×600)
compared to the output on the same guest using virtio drivers:
Kernel image: /boot/vmlinuz-2.6.34.7-0.5-default
Initrd image: /boot/initrd-2.6.34.7-0.5-default
Kernel Modules: thermal_sys thermal scsi_mod libata ata_piix ata_generic processor fan virtio virtio_pci virtio_ring virtio_net virtio_blk dm-mod dm-snapshot crc16 jbd2 ext4 usbcore pcmcia_core pcmcia mmc_core ssb ohci-hcd ehci-hcd uhci-hcd usbhid linear
Features: dm block usb lvm2 resume.userspace resume.kernel
Bootsplash: openSUSE (800×600)
I have configured the system to always include the virtio drivers by adding “virtio_blk” and “virtio_net” to /etc/sysconfig/kernel, so the initrd created on the guest with IDE emulation also works on the guest with virtio but not the other way around.
The solution/workaround here is quite simply to create a so-called monster initrd (mkinitrd -A) that has all available drivers available in the ram disk. Perhaps it doesn’t seem that nice, but the initrd is still only 20MB large compared to 6MB for a ‘normal’ initrd. So I created a monster!
As preparation for the new server I am getting, I am preparing for that by at least getting the software to a more recent baseline. The idea is to first upgrade the software to the more recent versions while still having a 32bit OS. Then a separate step is to go to 64bit on the new server.That step will then be following by running the server in its separate virtual domain. After that, splitting off the mythtv functionality in a separate ‘box’. And then even more stuff (it’s a hobby).
The main problem as usual is the upgrade of mythtv. I have been running a quite old version 0.21 and needed to upgrade to 0.24. Instead of doing this directly on the server however, I used a backup of the server on a separate hard disk and used KVM to upgrade that backup in a separate VM. That was what I did last week and I am happy I did because there were many many issues. I have a list of at least 20 points that I had to solve. Some were quite difficult and caused me hours of searching.
There were really subtle issues with mythtv and mysql requiring it to use /var/run/mysql instead of /var/lib/mysql. Also IPV6 issues where localhost gave a first match of the IPV6 form ::1 with 127.0.0.1 (IPV4) second. That caused problems for php. Then there were PHP settings for date and time. Samba losing all user accounts, viewvc getting uninstalled at the upgrade, and device names changing so smartd drive monitoring no longer worked. There was also a preload package intended to optimize startup of the server but in fact causing very high CPU load preventing server startup. That one caused me quite a head ache. Of course, a first upgrade attempt today failed because the DVD rom I burned was damaged, so instead I installed from NFS (from the laptop).
The good thing is that the upgrade itself was quite relaxed because I knew all of the issues beforehand. Even tested the backup I did on the VM before to verify that my TV card was still recognized. Really happy I tried it out first, and… getting more impressed by KVM.
But… it is done now. Finally.
For the new server I want to use a virtualized setup and would like to split up the server into several independent parts. At least one VM dedicated to mythtv and I stil have to think about the rest. Anyway, I started out using XEN because that was the main approach pushed by SuSE but ran into a lot of problems. In particular, if you run XEN then you have to take special steps to get a graphicall display running and it turns out there are conflicts between the NVIDIA driver and XEN resulting in crashes.
In particular, I tried to upgrade a backup of my server to opensuse 11.3 in a VM and that kept on crashing half way. So, I started looking for alternatives. One of the reasons for usin XEN was that it was going to be part of the kernel, but this is still not the case. It appears however that there is very little chance it will be because it was refused by the kernel developers. Luckily there is an alternative called KVM which is a standard part of the kernel and does not even require a special kernel and boot mechanism.
Trying out KVM was a real pleasant surprise because it just worked. In one evening I got VMs running with NAT and with bridged networking setup without all headaches that XEN has caused me in the past. Of course, this is also due to the good documentation that KVM provides. Over the past weekend I have used KVM intensively to try out the upgrade of my server to a newer version on VMs, and in the end did several upgrades and did not encounter any stability problems. Also it seemed much faster.
Migrating from Xen to KVM is also a breeze since the information you have to specify is the same. Even the tooling is almost the same. Conversely, going back to Xen would also not be that hard if I would want to do that. Main problem that I see now with Xen is that it is an add-on to the kernel and therefore not actively maintained at every kernel release. I would not dare running any server with Xen on it because of the stability problems.
So with one last changes of the heatsink to a Supermicro SNK-P0038P (bigger than the Intel) and of the chassis to the more silent SC743 TQ-865B-SQ, the server is now complete and I have placed the order. Now the waiting starts because I know that some parts are not in stock.
Incredible how much you learn while looking for the correct parts for a server. Initially I started out with an Intel S5500HCVR motherboard based on the 5500 chipset and an Antec Take 4 650 case and a Noctua NH-U12DX 1366 fan.
Then I started comparing the compatibility of the fan and the case and it turned out the fan didn’t fit, so I changed to a Noctua NH-U9DX 1366 fan with the help of Noctua support by the way.
Secondly, I looked at the motherboard and found out it used the SSI-EEB form factor and apparently that did not fit inside the Antec Case. As a result I started looking at other silent cases that would fit this motherboard but didn’t really find any. So I started looking at other motherboards and found one by Supermicro (the X8TDi-F). Meanwhile I also switched to a motherboard using the 5520 chipset because of better performance with the 5600 series Xeon processors. Fortunately, supermicro provides a really good website and they even had a silent case (the SC743T-665B). Additionally, I could verify the compatibility with the memory that I chose. So all is well, better even, as the new motherboard has 2 old style 32bit PCI slots which can come in handy.
Then while checking the dimensions of the case to see if it would really fit in my rack, I was reading the manual and found out that the case has a so-called air-shroud. This one is basically a ducting solution that connects the fans in the front with those in the back and provides an efficient cooling solution. Of course, the fan that I chose wouldn’t fit anymore but Intel provides a nice passive cooling solution (the STS100P) that fits very well in this case (reducing costs at the same time).
So, all in all a very good collection of parts that I have now. A really good silent rackmount case, passive processor cooling, and a good low power cooling solution. Meanwhile I have also found one shop that can provide all the stuff in one order. And that really helps, in particular in case of problems.
It is flue season again, but this time it is not a regular flue but more like a server virus. I have had this virus before and it usually ends up in me spending a lot of money on very interesting hardware. The last times I had the virus, I bought:
Don’t know where I caught this virus but perhaps it was already there and my resistance was weakened. Anyway, just spent the past evenings looking at specifications for this new server. Tonight this included a big search for a suitable CPU fan for which I also needed to dig in to some Intel motherboard specification to see whether it would fit or not. Since today I know what Universal Retention System and Unified Backplate Assembly mean.
I now have a good setup for what my server ‘could’ look like (supposing I would get it of course):
Interesting stuff….
Edit: Reusing of course the RAID card, TV card, and eSATA bracket from my current setup.
In 2006 I got myself a new server. The old one was already 7 years old and was at its limit. Memory was tight and the setup was not ideal. For instance, I had 2 internal hard disks with a RAID card and it was just generating a lot of heat and required a lot of cooling. Amazing that it held on for so long.
So in 2006 I got myself a new server. Using a Core Duo processor (not Core 2 Duo) at 1666MHz, one of the first dual core mobile processors. This was of course to reduce power consumption and heat production. To reduce heat production further, I used external hard disks that were put into a separate enclosure. After some time in operation I got myself a 19” rack (of the kind used by musicians) and replaced the external storage by a 19” storage rack. Also, I got a 19” UPS to protect the hardware some more. Another improvement was the use of an Areca ARC-1200 instead of a highpoint rocketraid to get true hardware RAID. Other improvements at that time were the use of logical volume managment and the use of postfix and cyrus instead of sendmail and Universtiy of Washington IMAP.
Over the past weeks I have been looking into managability and backup/restore a bit more. The reason is that this server has been up for already 4 years continuously so failure is becoming more likely every day. To deal with this, I improved my backup strategy so I will lose at most one week of data in case of a problem (or one month in case my house burns down). In any case, I have tested this procedure and have a documented method to simply boot from the backup and continue running the server. In addition I have setup hard disk monitoring so I should have a bigger chance of knowing beforehand when a drive is going to fail.
But having this worked out this changes the whole picture again. Now, instead of having to replace the server by a newer one as a precaution, I can just as well run the server until it breaks. But still, some things are still bothering me. For one, it would be really really cool to be able to:
In particular, one of my aims is to have a hardware independent server that can just be easily replaced by another one in case of problems. Doing this would require a storage area network (e.g. based on iSCSI) and using hardware virtualization (Intel VT). Paravirtualization is not an option as it increases maintenance problems because it ties host and guest together more (dependency between host and guest versions). A problem that I have currently is that although my current processor supports Intel VT, the motherboard blocks it. Also, I do not have enough memory for such a virtualized setup
So in summary, I am now in a situation that there is basically no need to replace my server. Also, it would feel like a waste to ditch a server which is in perfect working order. However, getting a new server would allow me to do al kinds of new interesting stuff with it and to take the next step to a hardware independent server.Another reason to replace the server by a more silent one. In particular, a low power Xeon processor can be cooled passively and there are also good fanless powersupplies that would enable me to get a really silent system. What should I do? Get a new server or not?