Why do developers write instead of reuse?

I am frequently amazed at the amount of software that is being written instead of simply looking around and reusing what’s already available. In practice I have seen a lot of reasons for this:

  • Our problems are unique: The misconception that “our problems are unique”. I really can’t recall how many times I have seen this but this is really occurring a lot.
  • Not looking for similar solutions: Simply forgetting to look for similar solutions on the internet to see what’s available (if only as an inspiration on how to best solve the problem). This is often also a side effect of thinking that this is a unique problem.
  • Underestimation of the problem: The misconception that it’s easy to write it yourself. In most cases, it is easy to come up with a first (half) working version that does approximately what you need. However, the work involved in making the same solution maintainable and with the correct feature set will make it much more expensive (the 80-20% rule).
  • Limited scope: A developer specialized in platform X (e.g. X = java) will typically only look for solutions in that area, whereas looking broader will reveal more solutions.
  • Coolness factor: It is cool to develop it yourself. Perhaps it involves an opportunity to do something cool with clustering or another chance to use one of your favorite frameworks. Perhaps you could use one of those cloud databases?
  • Overestimation of oneself: The idea that we can do something better in a few weeks time than what the industry or open source community has come up with using man years of development.
  • The desire for fame by writing reusable software: Paradoxically, the desire for reusable software can stimulate to roll your own. The problem is that writing reusable software (or calling it reusable) provides you with fame (even if it’s only in your local department). The reality is however that reuse can only exist through the willingness of people to use other people’s software. If there is one developer writing a reusable piece of software and 20 others using it, then clearly the willingness to use other’s software far outweighs writing it yourself.

I have seen these problems in companies of all sizes.

Posted in Uncategorized | 2 Comments

Moving countdown


Yes folks! The countdown timer has been started again. This time it is counting down to the time when the move really starts and first boxes will be loaded onto a truck towards my new home.


Really looking forward to it…

Posted in Uncategorized | Leave a comment

Moving

The last time I moved to a different city was 13 years ago. And before that time I had been moving every two years or so. So when I finally settled in 1998, I decided that I was going to stay in one place for a much longer time. It is time now however to move again, I got a new job in a new location and it makes a lot of sense to move. For one, I will have much better house (buying a house in the middle of the credit crunch), with a very nice garden, and it will reduce my traveling time to and from work considerably. Also, the environment is quite nice because my favorite mountainbiking locations are closer and there are also many more opportunities for mountainbiking close by.

One of the most important things when moving is of course…. my server. Of course, I am depending a lot on it. For one it is running my mail server and it also handles a number of mailing lists. It runs 4 web sites, and it is also my VCR (mythtv).

Therefore, it is important to me to minimize downtime of the server during the move. Luckily, I am already prepared for this since I am running the server as a virtual machine already. So as part of the move I will run this virtual machine on my laptop, which gives me plenty of time to disassemble the server rack and set it all up again at my new location. In fact as I am writing this, I am already running the server from my laptop. It is easy for me to do this because my regular server backups are bootable, see here.

Because of this setup, I can minimize the total down time of my web sites to the order of minutes and minimize mail down time to less than possibly one day in total (but no-one will notice that because mail servers retry sending mail).

Interestingly, I had quite a fight today to get things working again with my TVIX M-6500 which allows me to play movies hosted on the server (through NFS) on my TV. As it turns out there are subtle issues with network bridges on linux dropping UDP packages in some cases, see here.  As it turns out, the TVIX uses UDP for NFS, which can give problems with bridged network interfaces on virtual machines in some cases. Luckily, I managed to solve this by replacing the virtio network model on the machine by device emulation of a RTL8139 chipset. Anyway, all is  good now. The server VM is now fully functional again and I can watch movies, send/receive mail and all my websites are up. The only thing I cannot do is record at this time, but ok, this is only for the next 10 days or so. On the 16th of February I hope to be able to start the server again at its new location.

Posted in Uncategorized | 1 Comment

Nested Logical Volume Management for VMs

As I blogged earlier, I have replaced the server setup that I originally had with a virtualized server setup. This introduces the concept of “hardware independent server” and makes it easy to run the server on any hardware without modification. More concretely, it allows me to run until the hardware fails. Previously I used to replace the server hardware before it really broke, but in this setup I can run it until it breaks. Should I have a serious hardware failure I can simply run the server(s) from any other hardware such as a laptop. This is because I have “bootable backups”. I.e. if the server breaks, I can either run a replacement server based on the same data or simply use a laptop and run the backup in a virtualized manner.

As part of the original migration from running native to virtualized I used the identical setup, which meant passing physical hardware partitions to the virtual machine. The virtual machine then used Linux Logical Volume Management based on these hardware partitions. For new virtual machines I used another approach which was allocating a disk logical volume on the host, and then partitioning this on the guest and using LVM again to manage storage within the guest. This in fact results in nested logical volume management and as I have seen from one of the new virtual machines works like a charm. It provides a nice separation of concerns where the host simply assigns storage to guests and the guests decide how to use this storage.

However, there was still one virtual machine (the original hardware based server) that was still being passed physical disk partitions. This introduced the problem of both the host and virtual machine seeing the same logical volumes and thus the chances for administrative error and data corruption when multiple OSes would concurrently access the same logical volumes.

To remedy this, I used the following procedure:

  • Allocate a physical volume on the host and a “disk” logical volume on it big enough to contain all logical volumes from the VM
  • Stop the VM
  • Add this virtual disk to the VM.
  • Start the VM
  • Partition the new disk on the VM and extend existing volume groups to use physical partitions on this disk.
  • Use pvmove to move data to the disk and remove the old unused physical partitions from the volume groups afterwards.
  • Stop the VM
  • Remove old physical partitions from the VM, leaving only the new “disk” logical volume
  • Start the VM

In executing this procedure I ran into the basic problem that I did not have enough storage. To solve this I used a separate disk that was connected temporarily to the server. Now, after executing this procedure, all physical storage on the existing logical volumes (RAID array) was unused, so I extended the logical volume for the disk with that from the RAID array on the host. Then again using pvmove to move data to the RAID array from the temporary disk. And afterwards removing the unused physical volumes on the temporary disk from the volume group. Of course, all done while the virtual machine was up and running (no-one likes downtime).

The new setup reduces the chance of administrative error considerably and allows me to move storage for virtual machines to other locations without even having to shutdown a virtual machine. It also nicely separates the allocation of storage to VMs on the host from how each VM uses its allocated storage.

Posted in Server/LAN | Leave a comment

Improvements to Snapshot Backup Scripts

The snapshot scripts that I blogged about earlier have undergone a number of important changes. I had been having a lot of problems with the cleanup of snapshot volumes and with the deletion of the old backup logical volumes. This was all related to this bug. After applying the workarounds there, the backup procedure is completely robust again.

Also, I have added improved logging together with scripts to check the result of a backup.
Additionally, the software is now also available on an RPM repository (works at least on opensuse 11.3).

For more information, have a look at the snapshot website.

Posted in Uncategorized | Leave a comment