Monday, September 19, 2011

Yum 5.x Local repository and the new Continuous Release Repo

A few months back, possibly a year ago I started running my own local mirror of the CentOS YUM repository. The main benefit is speed in updating all of my servers. I also was able to setup a local repo of just tools that we use but aren't available in the Base CentOS repos. (I've since started adding the EPEL repository to servers that needed it.) For those curious I mirror the os, updates, and now the cr repos.

A few weeks ago the CentOS announce list and twitter feed introduced yet another repo the Continuous Release repo or cr for short. This new repo contains security and bug fixes from the upstream 5.7 release and I imagine will continue with further releases (5.8 & 5.9). CentOS has also promised a 6.0 Continuous Release repo, but have yet to deliver on that.

First off create the directories that will become the repositories.
mkdir -pv /Storage/yumRepo/{5,5.4,5.5/{os,updates,cr}/i386
mkdir -pv /Storage/yumRepo/local/el5/i386

So with out further ado, lets get into the guts of actually getting everything pulled down, synced and a repository created.

Below is the script I use to keep my mirror in sync with the closest mirror that offers rsync:

      1 #!/bin/sh
      3 rsync="rsync -avrt --bwlimit=796"
      5 mirror=rsync://
      7 verlist="5 5.5 5.6 5.7"
      8 archlist="i386"
      9 baselist="os updates cr"
     10 local=/Storage/yumRepo/centos
     12 for ver in $verlist
     13 do
     14  for arch in $archlist
     15  do
     16   for base in $baselist
     17   do
     18     remote=$mirror/$ver/$base/$arch/
     19     $rsync $remote $local/$ver/$base/$arch/
     20     createrepo -v --update $local/$ver/$base/$arch
     21   done
     22  done
     23 done
     24 hardlink /Storage/yumRepo/centos

Now, lets break this script down line by line.  Line one is obvious, it tells the program loader what interpreter to use when running this file. Line three sets up the rsync command I'll be using.  In my case I need to limit the bandwidth usage to something sane, even though I run it off peak hours. Line five sets the mirror I'm running against. Line seven sets the versions of CentOS I'm interested in.  I really could trim this back to just 5 and 5.7 now, but I'm going to leave it in there for now. Line eight tells me I'm only going to sync i386 architect and not the x64 architect. I'm doing this because as of right now we only run x86. Line nine gives me which repositories in the i386 architect I'm going to mirror. Line ten sets up where on the local server I'm going to store everything.

Lines twelve through twenty three run rsync against all of the variations of repository and architect type for each version. In my case it would be twelve different syncs. If I where also running x64 architect it would double to twenty four. Line nineteen is the actual rsync command while line twenty creates the necessary repository files for yum to use.

Line twenty four is rather new to this script and requires you have the hardlink command installed (yum install hardlink). The hardlink command will run through everyfile in the directory specified and compare it to every other file in the directory.  It is looking for the same contents with the same permissions, but it can have a different filename.  If it finds a match it will hardlink one of them to the other, thus reducing the space required to store two copies. My only wish is that I had some hard stats for how much disk space this is saving me on the repository alone.

Now create the repository file for Yum.
#vi /etc/yum.repos.d/MyRepo.repo
This file contains:
name=CentOS-$releasever - Base$releasever/os/$basearch/

name=CentOS-$releasever - Updates

name=CentOS-$releasever - Local

We need to also tell Apache about these directories and how to serve them out over HTTP.
I changed the DocumentRoot to
DocumentRoot "/Storage/yumRepo"
then I changed the Default directory to

Options +Indexes +FollowSymLinks
AllowOverride All
Allow from all

Restart apache.
#service httpd restart

Populate Company Local repo with custom RPMs.
On every server we have installed Webmin. In the past this was installed either by hand or via our install script, but that had to be updated by hand with each update. So I will move the latest into our Company Local repo.
#mv /Storage/rpmDownloads/webmin-1.510.rpm /Storage/yumRepo/local/el5/i386/

You can move as many files into here as you want, including built from source RPMs that you may have made. After you move in any custom RPMs you have to update the Repository.
#createrepo -v --update /Storage/yumRepo/local/el5/i386/

No comments: