Two worlds meet (1): Automated creation of Yum Repos with Maven, Nexus, and Hudson

This is the first of a series of blogs titled ‘Two worlds meet’ talking about how two technologies can be used together to solve a problem. Mostly one world will be linux or more specifically a virtualized linux setup using kernel virtual machine, with the other world being java.

In this blog I will be looking at automated creation of a yum repository using maven, nexus, and hudson. First however, some background is needed. Some time ago I bought a new server with more than sufficient resources to run multiple virtual machines. The aim there was to do some separation of concerns having virtual machines with different purposes and also be able to run conflicting software. Doing that introduces a whole new problem of maintaining the software on these virtual machines. Of course, I use a standard linux distribution such as opensuse but I still have some custom scripts that I need and want to have available on all VMs.

Using the standard linux tooling an abvious method is to just create my own Yum repository to publish my own RPMs in and then add that Yum repo as a channel in all of my VMs. Of course, the challenge is then to easily and automatically create such a YUM repository. Fortunately, since I am working quite a lot with Java and Maven (earning a living with it basically), there is a quite easy solution with a nice separation of concerns.

The ingredients of this solution are:

  • maven for building the rpm using the maven rpm plugin
  • the maven release plugin for tagging a released version of the RPM, stepping its version, and publishing it into a maven repository
  • a maven repository such as nexus for publishing RPMs into
  • hudson for detecting changes and automatically updating/building the Yum repository upon changes to the RPMs.

In addition, some basic infrastructure is needed such as:

  • a version control system such as subversion
  • apache for providing access to subversion and for serving the Yum repo to all VMs
  • an application server such as glassfish for running hudson and nexus

This may seem like a lot of infrastructure, but before I started I already had most of this except for the nexus maven repository, so all in all the effort for this solution was quite limited.

The main new ingredient of the solution is the script to create the Yum repository from the nexus repository. This script exploits the fact that nexus stores its repositories in a standard maven directory structure (an approach using REST web services is also possible):

#!/bin/bash

REPO=/usr/java/nexus/nexus/storage/rpms

# Create the repo directory
YUM=$HOME/yum
rm -rf $YUM
mkdir -p $YUM
mkdir -p $YUM/noarch

# Find the RPMs in the nexus repository and use hardlinks
@ to preserve modification times and avoid the overhead of
# copying
for rpm in $( find $REPO -name '*.rpm' )
do
  echo "RPM $rpm"
  ln $rpm $YUM/noarch
done

# createrepo is a standard command available on opensuse
# to create a Yum repository
createrepo $YUM

# sign it
gpg -a --detach-sign $YUM/repodata/repomd.xml
gpg -a --export F0ABC836 > $YUM/repodata/repomd.xml.key

# sync the results to their final destination to make them
# available
rsync --delete -avz $YUM/ /data/www/http.wamblee.org_yum/public

Using this approach it is really easy to update an RPM and make it available on all my VMs. The procedure is basically as follows:

  • edit the source of the RPMs and check in
  • Now tag it and step the versions using:
    mvn release:prepare
  • Deploy the just tagged version to the nexus repository:
    mvn release:perform
  • Some time later the Yum repository has been automatically updated by hudson based on the contents of the nexus repository
  • On a specific VM simply use a standard update using
    zypper up

    Note that this may require an explicit

    zypper refresh

    to make sure that zypper sees the latest versions of all RPMs. Autorefresh will also work but might require some more time before zypper sees the latest versions.

Therefore, in the end a really simply procedure to quickly make RPMs available on all VMs and also make sure each version is properly tagged in subversion. The only issue is that hudson will always run on every SCM change, so not only when an RPM is released but I consider that a minor issue.

The YUM repo is here.

An example pom is below:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd ">
    
    <parent>
        <groupId>org.wamblee.server</groupId>
        <artifactId>root</artifactId>
        <version>1.0.1</version>
    </parent>
    
    <modelVersion>4.0.0</modelVersion>
    
    <packaging>rpm</packaging>
    <groupId>org.wamblee.server</groupId>
    <artifactId>kvmguest</artifactId>
    <version>1.0.3-SNAPSHOT</version>
    <name>kvmguest</name>
    <description>KVM guest support</description>
    <organization>
        <name>org.wamblee</name>
    </organization>
    
    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>rpm-maven-plugin</artifactId>
                <version>2.0.1</version>
                <extensions>true</extensions>
                <configuration>
                    <changelogFile>CHANGELOG</changelogFile>
                    <copyright>Apache License 2.0, 2010</copyright>
                    <group>org.wamblee.server</group>
                    <packager>Erik Brakkee</packager>
                    
                    <mappings>
                        <mapping>
                            <directory>/usr</directory>
                            <filemode>755</filemode>
                            <username>root</username>
                            <groupname>root</groupname>
                            <sources>
                                <source>
                                    <location>files/usr</location>
                                </source>
                            </sources>
                        </mapping>
						<mapping>
                            <directory>/etc/kvmguest.d</directory>
                            <filemode>755</filemode>
                            <username>root</username>
                            <groupname>root</groupname>
                            <sources>
                                <source>
                                    <location>files/etc/kvmguest.d</location>
                                </source>
                            </sources>
                        </mapping>
						<mapping>
                            <directory>/usr/share/doc/packages</directory>
                            <filemode>444</filemode>
                            <username>root</username>
                            <groupname>root</groupname>
                            <sources>
                                <source>
                                    <location>files/usr/share/doc/packages</location>
                                </source>
                            </sources>
                        </mapping>
                    </mappings>
                    <provides>
                       <provide>kvmguest</provide>
                    </provides>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

This entry was posted in Devops/Linux, Java, Software. Bookmark the permalink.

15 Responses to Two worlds meet (1): Automated creation of Yum Repos with Maven, Nexus, and Hudson

  1. Pingback: KVM Setup Overview | Nonsense and other useful things

  2. Daniel Serodio says:

    Do you publish RPMs to Nexus? Do you have a sample POM you can share? Thanks.

  3. Erik Brakkee says:

    It is very simple, just use packaging ‘rpm’ in your pom. Alternatively, you can use the maven build-helper plugin to attach artifacts to your build.

    I have updated the post to include an example pom.xml.

  4. drumeng says:

    Maybe I misunderstand, but if you just want to share scripts, why not set up
    an NFS or Samba share and just mount it on your virts ?

  5. Erik Brakkee says:

    It is not just about scripts but also about some third party (open source software) that I am repackaging for myself such as a pre-packaged and configured glassfish application server.

    Another thing is that I want to have version control/configuration management for my scripts and not all software should be available everywhere. Also, an NFS solution would couple all VMs to one other VM which is hosting the scripts. Right now, each VM works standalone and there is only a dependency on the VM hosting the RPM repo for the duration of the installation. This approach keeps each VM self-contained. Also, this approach is consistent with the general linux way of managing software.

  6. We do the same stuff at IS24, except that we have a nexus-yum-plugin that automatically builds the yum repo into the maven repo. Maybe this is useful to you: http://code.google.com/p/nexus-yum-plugin/

  7. Jamila says:

    To Sebastian Herold

    I’m using the plugin but nothing happens … the repository is not recognise as a yum plugin

    Did you do something special?

  8. Generally I do not read post on blogs, however I wish to say that this write-up very forced me to try and do it! Your writing style has been amazed me. Thanks, quite nice article.

  9. I used to be suggested this blog by way of my cousin. I’m now not positive whether or not this submit is
    written via him as nobody else know such distinctive approximately my trouble.
    You are incredible! Thank you!

  10. Hey there, I think your website might be having browser compatibility issues. When I look at your blog in Chrome, it looks fine but when opening in Internet Explorer, it has some overlapping. I just wanted to give you a quick heads up! Other then that, fantastic blog!

  11. I’m impressed, I have to admit. Seldom do I encounter
    a blog that’s equally educative and entertaining,
    and without a doubt, you’ve hit the nail on the head.
    The problem is something which too few men and women are speaking intelligently about.

    Now i’m very happy I found this during my search for something
    concerning this.

  12. Twicsy says:

    Just wish to say your article is as amazing. The clearness in your put
    up is just spectacular and that i can think you are an expert on this subject.
    Well together with your permission allow me to snatch your feed to
    stay updated with drawing close post. Thank you a million and
    please keep up the enjoyable work.

  13. Gbo Slot says:

    You’ve made sߋme good points there. I looked on the
    net to learn m᧐re about thе issue and foսnd most people wipl go along witһ
    your views on thiѕ site.

    my web blog; Gbo Slot

  14. Login Gbosky says:

    Τhank you for sharing your thoughtѕ. I really aρpreciate youг efforts and I will bee waiting for your
    further post tһanks once again.

    Aⅼso visitt mʏ web pagе: Login Gbosky

  15. login gbo338 says:

    Ρгetty! Tһis ԝas a really wonderful post. Thanmk you for supplying these details.

    Review mʏ web site … login gbo338

Leave a Reply to Erik Brakkee Cancel reply

Your email address will not be published. Required fields are marked *