Month: January 2011

Setting up vmware fusion, puppet and gitolite

The goal today is to start a rather ‘complete’ local dev environment on my laptop, from scratch, suitable for playing with continuous integration and deployment tools. My host is a mac. I’ll use VMWare Fusion for virtualization, running CentOS 5.5 guests. I’ll be setting up poor man’s vm cloning, and after that gitolite for version control, puppet for configuration management.

The main goal of all this is actually playing around with configuration management, so I’m not going to bother with backups or redundancy or any high availability config, however I will be deploying some basic java webapps and some basic PHP frontends to exercise the config management put in place.

Unfortunately, bootstrapping such a setup is a lot of work so I’ll write down a detailed installation log. This way, it may be less work next time, as per my own advice on bootstrapping. Maybe it helps you, too 🙂

1. Virtual Machine Setup

All these things will be deployed in virtual machines so as to match the production environment as much as possible. So the first step is to make it easy to create identical virtual machines that can be spun up and down on demand. Probably the best way is to use cobbler, but here’s how I did things:

1.1 Create a base vanilla VM

  • make a new or get a base vm image installation with CentOS 5.5 on it, name it vanilla-sandbox.
  • If starting from a downloaded base VM (I didn’t), verify it, then set a new root password by typing passwd
  • ensure correct contents of /etc/resolv.conf, /etc/sysconfig/network, /etc/hosts (ensure DHCP, set hostname to vanilla-sandbox, check the right DNSes were found through DHCP)
  • add any custom RPM repositories:
    rpm --import https://fedoraproject.org/static/217521F6.txt
    rpm -ivh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
    rpm --import http://repo.webtatic.com/yum/RPM-GPG-KEY-webtatic-andy
    rpm -ivh http://repo.webtatic.com/yum/centos/5/`uname -i`/webtatic-release-5-1.noarch.rpm
    vi /etc/yum.repos.d/webtatic.repo (enable the release repo)
  • run yum update
  • install (but do not configure) puppet: yum install puppet
  • install vmware tools
  • shrink disk(s), run vmware-toolbox-cmd disk shrink /
  • shutdown -h now

1.2 Create a setup to be able to clone VMs

As part of an effort to learn a bit about vmware, I wrote a simple script, that goes into ~/Documents/Virtual Machines:

#!/usr/bin/env python
# make-sandbox.py: clones virtual machines

import sys
import re
import os
import shutil

def usage():
    print "./make-sandbox.py [name]"
    print "  name should match ^[a-z][A-Z0-9]{1,15}$"

def fail(msg):
    print msg
    sys.exit(1)

def isVmConfigFile(name):
    return re.match("^.*?vanilla-sandbox\.(vmdk|vmsd|vmx|vmxf)$",
            name)

def writeNewConfig(src, dst, renamer):
    s = open(src, 'r').read()
    s = renamer(s)
    print "writing custom", dst
    f = open(dst, 'w')
    f.write(s)
    f.close()

def sandboxCopyTree(src, dst, renamer):
    names = os.listdir(src)

    print "mkdir", dst
    os.makedirs(dst)
    errors = []
    for name in names:
        srcname = os.path.join(src, name)
        dstname = renamer(os.path.join(dst, name))
        
        try:
            if os.path.islink(srcname):
                linkto = os.readlink(srcname)
                os.symlink(linkto, dstname)
                print "ln -s", srcname, dstname
            elif os.path.isdir(srcname):
                sandboxCopyTree(srcname, dstname, renamer)
            elif srcname.endswith(".log"):
                continue
            elif isVmConfigFile(srcname):
                writeNewConfig(srcname, dstname, renamer)
            else:
                print "cp", srcname, dstname
                shutil.copy2(srcname, dstname)
        except (IOError, os.error), why:
            errors.append((srcname, dstname, str(why)))
        # catch the Error from the recursive copytree so
        # that we can continue with other files
        except shutil.Error, err:
            errors.extend(err.args[0])
    try:
        shutil.copystat(src, dst)
    except OSError, why:
        errors.extend((src, dst, str(why)))
    if errors:
        raise shutil.Error(errors)

def sandboxRenameMaker(name):
    def renamer(dst):
        return dst.replace("vanilla", name)
    return renamer

if __name__ == "__main__":
    if len(sys.argv) != 2:
        usage()
        sys.exit(1)
    
    name = sys.argv[1]
    if not re.match('^[a-z][a-z0-9]{1,15}$', name):
        usage()
        sys.exit(1)

    basename = "%s-sandbox" % (name,)
    if os.path.exists(basename):
        fail("%s already exists?" % (basename,))
    
    sandboxCopyTree("vanilla-sandbox", basename,
            sandboxRenameMaker(name))

The script is pretty basic and hardcodes some stuff it probably shouldn’t, but it works ok for me. You can find various other imperfect scripts that do similar things on the vmware fusion forums. This script expects a directory ./vanilla-sandbox containing a vm named vanilla-sandbox, and the intended name of the new virtual machine as an argument.

1.3 Create some VMs

Invoke the new script like so:

cd ~/Documents/Virtual\ Machines && ./make-sandbox.py puppet

Which results in a virtual machine named puppet-sandbox. The virtual machine is not ready for use yet. Additional steps:

  • In VMWare Fusion, select File > Open..., then open the new VM.
  • Start the VM. VMWare will ask whether you moved or copied the VM. Select “I copied…”
  • log in as root
  • vi /etc/sysconfig/network, change hostname to match vm name (i.e. puppet.sandbox)
  • vi /etc/hosts, change hostname to match vm name
  • hostname [hostname], change hostname for the running system

Yes, this manual edit of the network settings is kind of icky, but I looked at how cobbler integrates vmware and koan and it just seemed a bit too much work for me right now. Perhaps I’ll look at that later.

Specifically for puppet, I want the machine running it to have a static IP, so I can put a static entry into /etc/hosts on the guest OS and have that always work. So, for the puppet machine, network config gets an extra step:

cp /etc/sysconfig/network-scripts/ifcfg-eth0
        /etc/sysconfig/network-scripts/ifcfg-eth0.bak
cat >/etc/sysconfig/network-scripts/ifcfg-eth0 <<END
# Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
DEVICE=eth0
BOOTPROTO=static
IPADDR=172.16.64.3
NETMASK=255.255.255.0
NETWORK=172.16.64.0
BROADCAST=172.16.64.255
ONBOOT=yes
GATEWAY=172.16.64.2
END
vi /etc/resolv.conf (172.16.64.2 is DNS)
service network restart

Then, on the host, echo "172.16.64.3 puppet.sandbox puppet" >> /etc/hosts.

Note 172.16.64.x is the default network used by vmware fusion NAT, vmnet8. You can find these details in /Library/Application Support/VMware Fusion/vmnet8/dhcpd.conf, which I believe really ought to learn about this, too:

host puppet {
    hardware ethernet 00:0C:29:3E:FC:56;
    fixed-address 172.16.64.3;
}

where that ethernet address is generated by vmware fusion, and you can find it with cat ~/Documents/Virtual Machines/puppet-sandbox/puppet-sandbox.vmx | grep generatedAddress. Restart vmware’s networking fanciness with /Library/Application Support/VMware Fusion/boot.sh --restart. Now restart networking on the puppet guest and check it’s working ok:

/etc/init.d/network restart
ping -c 1 www.google.com

2. Bootstrapping services

Once you have yourselves some base VMs one way or another, the next step is to get the combo of puppet and gitolite up properly. Normally these really need dedicated machines but I’m trying to conserve RAM so they’ll have to fit together on one VM for now.

2.1 Basic puppet server install

This bit is very easy:

yum install puppet-server ruby-shadow
mkdir -p /etc/puppet/manifests
puppetmaster --genconfig > /etc/puppet/puppet.conf
cat >/etc/puppet/manifests/site.pp <<END

file { "/etc/sudoers":
  owner => root, group => root, mode => 440
}

END
service puppetmaster start
puppetd --test

2.2 Basic gitolite install

We’ll actually use this puppet to install gitolite for us, then move the puppet config into gitolite once it’s set up.

Note that gitolite currently needs git 1.6.2+, so you need to get that from somewhere, it isn’t currently in CentOS 5 or EPEL. For this reason, I added the webtatic yum repo config earlier. Probably not a good idea for a production environment!

Here’s all the bits and pieces to get gitolite set up:

yum install git

# move /etc/sudoers resource to a module
cd /etc/puppet
mkdir -p /etc/puppet/modules/sudo/manifests
cat > /etc/puppet/modules/sudo/manifests/init.pp <<END
class sudo {
    file { "/etc/sudoers":
      owner => root, group => root, mode => 440
    }
}
END

# create a gitolite module
mkdir -p /etc/puppet/modules/gitolite/{manifests,files}

cat > /etc/puppet/modules/gitolite/files/install-gitolite.sh <<END
#!/usr/bin/env bash
# initial system install of gitotis. Run as root.

set -e

cd /home/git

if [[ ! -d "gitolite-source" ]]; then
    git clone git://github.com/sitaramc/gitolite gitolite-source
fi
cd gitolite-source
git checkout v1.5.8
mkdir -p /usr/local/share/gitolite/conf
        /usr/local/share/gitolite/hooks
src/gl-system-install /usr/local/bin
        /usr/local/share/gitolite/conf
        /usr/local/share/gitolite/hooks
END

cat > /etc/puppet/modules/gitolite/files/setup-gitolite.sh <<END
#!/usr/bin/env bash
# initial for-gitolite-user setup of gitolite. Run as gitolite.

set -e

/usr/local/bin/gl-setup /home/git/lsimons.pub
END

# note: truncated ssh key for blog post
cat > /etc/puppet/modules/gitolite/files/lsimons.pub <<END
ssh-rsa AAAA...== lsimons@...
END

cat > /etc/puppet/modules/gitolite/files/gitolite-rc <<END
\$GL_PACKAGE_CONF = '/usr/local/share/gitolite/conf';
\$GL_PACKAGE_HOOKS = '/usr/local/share/gitolite/hooks';
\$REPO_BASE="repositories";
\$REPO_UMASK = 0077;         # gets you 'rwx------'
\$PROJECTS_LIST = \$ENV{HOME} . "/projects.list";
\$GL_ADMINDIR=\$ENV{HOME} . "/.gitolite";
\$GL_LOGT="\$GL_ADMINDIR/logs/gitolite-%y-%m.log";
\$GL_CONF="\$GL_ADMINDIR/conf/gitolite.conf";
\$GL_KEYDIR="\$GL_ADMINDIR/keydir";
\$GL_CONF_COMPILED="\$GL_ADMINDIR/conf/gitolite.conf-compiled.pm";
\$GIT_PATH="";
\$GL_BIG_CONFIG = 0;
\$GL_NO_DAEMON_NO_GITWEB = 0;
\$GL_NO_CREATE_REPOS = 0;
\$GL_NO_SETUP_AUTHKEYS = 0;
\$GL_GITCONFIG_KEYS = "";
\$HTPASSWD_FILE = "";
\$RSYNC_BASE = "";
\$SVNSERVE = "";
\$GL_WILDREPOS = 0;
\$GL_WILDREPOS_PERM_CATS = "READERS WRITERS";
1;
END

cat > /etc/puppet/modules/gitolite/manifests/init.pp <<END
class gitolite {
    package { git:
        ensure => latest
    }
    
    group { git:
        ensure => present,
        gid => 802
    }
    
    user { git:
        ensure => present,
        gid => 802,
        uid => 802,
        home => "/home/git",
        shell => "/bin/bash",
        require => Group["git"]
    }
    
    file {
        "/home/git":
            ensure => directory,
            mode => 0750,
            owner => git,
            group => git,
            require => [User["git"], Group["git"]];

        "/home/git/install-gitolite.sh":
            ensure => present,
            mode => 0770,
            owner => git,
            group => git,
            require => File["/home/git"],
            source => "puppet:///modules/gitolite/install-gitolite.sh";
            
        "/home/git/setup-gitolite.sh":
            ensure => present,
            mode => 0770,
            owner => git,
            group => git,
            require => File["/home/git"],
            source => "puppet:///modules/gitolite/setup-gitolite.sh";
            
        "/home/git/lsimons.pub":
            ensure => present,
            mode => 0660,
            owner => git,
            group => git,
            require => File["/home/git"],
            source => "puppet:///modules/gitolite/lsimons.pub";
        
        "/home/git/.gitolite.rc":
            ensure => present,
            mode => 0660,
            owner => git,
            group => git,
            require => File["/home/git"],
            source => "puppet:///modules/gitolite/gitolite-rc";
    }
    
    exec {
        "/home/git/install-gitolite.sh":
            cwd => "/home/git",
            user => root,
            require => [
                        File["/home/git/install-gitolite.sh"],
                        Package["git"]
                ];

        "/home/git/setup-gitolite.sh":
            cwd => "/home/git",
            user => git,
            environment => "HOME=/home/git",
            require => [
                    Exec["/home/git/install-gitolite.sh"],
                    File["/home/git/setup-gitolite.sh"],
                    User["git"]
                ]
    }
}
END

# update the site config to use the sudo and gitolite modules
cat > /etc/puppet/manifests/modules.pp <<END
import "sudo"
import "gitolite"
END

cat > /etc/puppet/manifests/nodes.pp <<END
node basenode {
    include sudo
}

node "puppet.sandbox" inherits basenode {
    include gitolite
}
END

cat > /etc/puppet/manifests/site.pp <<END
import "modules"
import "nodes"
END

# invoke puppet once to apply the new config
puppetd -v --test

So now we have gitolite installed on the server. So far so good.

2.3 Creating an scm git repo

I now need a repository in which to put the puppet configs. I was originally planning to have a repo ‘scm’ and a directory ‘puppet’ within it, so that I could have /etc/scm, with /etc/puppet a symlink to /etc/scm/puppet. It turns out puppet doesn’t support symlinks for /etc/puppet, so I ended up fiddling about a bit…

# on client
git clone git@puppet:gitolite-admin
cd gitolite-admin
cat >>conf/gitolite.conf <<END

repo    scm
        RW+     =   lsimons

END
git add conf/gitolite.conf 
git commit -m "Adding 'scm' repo"
git push origin master
cd ..

2.4 Making puppet use the config from the scm repo

First we need to get the existing config into version control:

mkdir scm
cd scm
git init
cat >> .git/config <<END
[remote "origin"]
    url = git@puppet:scm
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
    remote = origin
    merge = refs/heads/master
END
scp -r root@puppet:/etc/puppet/* .
git add *
git commit -m "Check in initial puppet config"
git push --all

Next, get the config out of version control and underneath puppet, and automate this process:

# install from-git puppet config on server

cd /etc
mv puppet puppet.bak
mkdir puppet
chown git puppet
cd puppet
sudo -u git git clone file:///home/git/repositories/scm.git puppet

# install post-receive hook to update /etc/puppet after a push

su - git
cat > /home/git/repositories/scm.git/hooks/post-receive <<END
#!/usr/bin/env bash
(cd /etc/puppet; env -i git pull -q origin master)
END
chmod u+x /home/git/repositories/scm.git/hooks/post-receive
exit

That little bit of env -i git inside that hook had me baffled for a bit. It turns out that I needed to empty the environment before invoking git from inside of a hook, because otherwise it’ll pick up the GIT_DIR variable. D’oh!

With this config re-set up, there should be effectively 0 changes when we run puppet. Let’s check:

puppet /etc/puppet/manifests/site.pp

2.5 On (not) putting all the puppet.sandbox config in puppet

Note that installing the post-receive hook from the previous step is not puppeted. The reason for this is one of synchronization. For example, if puppet somehow creates that post-receive file before the scm repository exists, gitolite will complain. It seems easier to have puppet not touch things managed by gitolite and vice versa.

Similarly, the installation of puppet itself is not puppeted, leaving the configuration of puppet.sandbox not something that can be completely automatically rebuilt.

Instead, rebuilding this box should be done by first re-following the instructions above, and then restoring the contents of the git@puppet:gitolite-admin and git@puppet:scm repositories from their current state (or latest backup). For my current purposes, that’s absolutely fine.

3. Setting up puppet dashboard

I also had a look at installing puppet dashboard. Because I know ruby and rails and gems can be a big dependency hell I figured I didn’t even want to try it in a VM, and instead I “just” got it running on my mac.

3.1 MySQL, ruby and mac os x

Puppet dashboard is built using ruby on rails and suggests using mysql for persistence (in retrospect I should not have listened and used sqlite :-)). Ruby on Rails apparently accesses MySQL through the mysql gem. The MySQL gem has to link against both the native mysql library and the native ruby library. Fortunately, I’m aware enough of the potential pain that I tend to carefully install the most compatible version of systems like this:

$ file `which ruby`
/usr/local/bin/ruby: Mach-O executable i386
$ file `which mysql`
/usr/local/mysql/bin/mysql: Mach-O executable i386
$ ruby --version
ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin9.7.0]
$ mysql --version
mysql  Ver 14.14 Distrib 5.1.31,
        for apple-darwin9.5.0 (i386) using readline 5.1

You’d hope that the mysql gem build picks up on all this ok, but that’s not quite the case. Instead, you really have to be quite explicit:

vi /usr/local/lib/ruby/1.8/i686-darwin9.7.0/rbconfig.rb
# change to CONFIG["ARCH_FLAG"] = "-arch i386"

sudo gem uninstall mysql
sudo env ARCHFLAGS="-arch i386" gem install --verbose \
    --no-rdoc --no-ri mysql \
    -- --with-mysql-dir=/usr/local \
    --with-mysql-config=/usr/local/mysql/bin/mysql_config
cd /usr/local/mysql/lib/
sudo ln -s mysql .
cd .

3.2 Look, ma, it’s a rails app

After this, fortunately it’s easy again.

tar zxf ...puppet-dashboard...
mv ... ~/puppet-dashboard
cd ~/puppet-dashboard
rake RAILS_ENV=production db:create
rake RAILS_ENV=production db:migrate
./script/server -e production
open http://localhost:3000/

Works like a charm. To get some data to see requires tweaking the puppet VM:

cd /Users/lsimons/puppet-dashboard/ext/puppet
vi puppet_dashboard.rb
# change HOST to 172.16.64.1
ssh root@puppet mkdir -p /var/lib/puppet/lib/puppet/reports
scp puppet_dashboard.rb root@puppet:/var/lib/puppet/lib/puppet/reports/
exit

cd ~/dev/scm/
vi puppet/puppet.conf
# report = true for [puppetd]
# reports = puppet_dashboard for [puppetmasterd]
git add puppet/puppet.conf
git commit -m "Enable reporting"
git push

4. Recap

So now we have:

  • A working, documented, repeatable process for creating new VMs
  • A working, documented, repeatable process for bootstrapping puppet
  • A neat version-controlled way of changing the puppet config
  • An installation of a puppet master that serves up the latest config
  • A puppeted installation of gitosis
  • A not-so-great but working installation of puppet dashboard
  • A few more VMs to configure