Better Code

Swimming in Someone Else's Pool

How I Ruby, Part 2a: Deployment (Ruby)

| Comments

In this post I’m going to show you how use ruby-install to build a Debian package of MRI itself, and put together a trivial apt repository so you can serve it on your local network. Why do this? First, you aren’t dependent on ftp.ruby-lang.org being available when you want to deploy. Second, that you don’t waste time during your deployment rebuilding a ruby binary. I’ve seen builds of ruby 2.0.0 take 10 minutes or so, and it’s no fun waiting for that when you’ve got an urgent redeploy waiting.

You should note that the packages we’ll build here are the simplest possible packages to get a ruby binary installed on a Debian system. They are the base minimum required to get that one job done. They bear only a passing resemblance to the sort of quality of package you’ll find either in Debian itself, or in any of the other public repositories offering Ruby packages (like John Leach’s excellent Ubuntu packages, over here). You should use these strictly internally, since we cut several corners to build the simplest packages that can possibly work.

As before, I’m assuming you’re building on and deploying to Debian Stable, but the same instructions should work on Ubuntu. You’ll need ruby-install installed, along with the dpkg-dev package.

Here’s the script which does the bulk of the work. I’d suggest pasting this into an executable file called ruby-install-deb somewhere handy on your $PATH:

ruby-install-deb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
#!/usr/bin/env ruby

require 'fileutils'


def run( argv )
  ruby_version=argv.shift or fail "Need an MRI version number"

  making_dirs( ruby_version ) do
    make_makefile( ruby_version )
    make_debian
    add_depends
  end
end


def making_dirs( ruby_version )
  proj_name = "ruby-install-ruby"
  FileUtils.mkdir_p proj_name

  Dir.chdir proj_name do
    pack_name = "#{proj_name}#{ruby_version.tr("-", '')}-1"
    FileUtils.mkdir_p pack_name
    Dir.chdir pack_name do
      yield
    end
  end
end


def makefile( ruby_version )
  makefile=<<-MAKEFILE
DESTDIR?=/
RUBY_VERSION?=#{ruby_version}
RUBY_ROOT=/opt/rubies/ruby-$(RUBY_VERSION)
RUBY_PATH=$(RUBY_ROOT)/bin
RUBY_BIN=$(RUBY_PATH)/ruby

all: root/$(RUBY_BIN)

root/$(RUBY_BIN):
 mkdir -p root
 DESTDIR="$$(cd root/; pwd)" ruby-install -i "$(RUBY_ROOT)" ruby $(RUBY_VERSION)

install:
 mkdir -p $(DESTDIR)
 cp -a root/* $(DESTDIR)

clean:
 rm -rf root/

.PHONY: all install
  MAKEFILE
end


def make_makefile( ruby_version )
  File.open("Makefile", "wb") do |f|
    f.puts( makefile( ruby_version ) )
  end
end


def add_depends
  dependencies = `grep apt: /usr/share/ruby-install/ruby/dependencies.txt | sed 's/apt: //'`.strip.split(/\s+/)
  dependencies.delete "build-essential"
  dep_string = dependencies.join(", ")
  cmd = "sed -i '/^Depends:/s|$|, #{dep_string}|' debian/control"
  system cmd
end


def make_debian
  system "/bin/bash -c 'echo | dh_make --copyright gpl2 --native --single'"
  system "mkdir -p debian.keep"
  system "/bin/bash -c 'cp debian/{rules,control,compat,changelog} debian.keep/'"
  system "rm -rf debian"
  system "mv debian.keep debian"
end


if $0 == __FILE__
  run ARGV
end

You run it like this, specifying the MRI version you want to package on the command-line:

””
1
$ ruby-install-deb 2.0.0-p195

This will build a .deb file at ruby-install-deb/ruby-install-ruby2.0.0p195_1_amd64.deb. If you just want to see how to host this package inside your network and don’t want to bother with the details of how the above script works, skip to “Building a repository” below.

Broken down into its component parts, there isn’t much to the script above. We make a directory to build ruby in, construct a Makefile which knows how to use ruby-install, strap in the few parts that define a Debian package using that Makefile, and finally add the runtime dependencies.

I’ll run through the individual parts to highlight some of the details.

1: Top-level sequence

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#!/usr/bin/env ruby

require 'fileutils'


def run( argv )
  ruby_version=argv.shift or fail "Need an MRI version number"

  making_dirs( ruby_version ) do
    make_makefile( ruby_version )
    make_debian
    add_depends
  end
end

...


if $0 == __FILE__
  run ARGV
end

Hopefully this is self-explanatory. It’s a literal transliteration of the previous paragraph into Ruby. The one thing worth noting here is that both the directory name and the Makefile depend on the ruby version we’re building. This is due to the vagaries of Debian packaging, which I’ll point out more of as we go along.

2: Directory Structure

1
2
3
4
5
6
7
8
9
10
11
12
def making_dirs( ruby_version )
  proj_name = "ruby-install-ruby"
  FileUtils.mkdir_p proj_name

  Dir.chdir proj_name do
    pack_name = "#{proj_name}#{ruby_version.tr("-", '')}-1"
    FileUtils.mkdir_p pack_name
    Dir.chdir pack_name do
      yield
    end
  end
end

This might look a little odd, but it’s mostly dictated by how building a Debian package works. First, we create a top-level directory to do all our work in. The name of this directory isn’t important.

Next, we construct a nested directory where we’re going to keep the actual package metadata. The name of this directory is important, since the Debian tooling is going to use it to pick up both the package name and the package version. The scheme I’ve chosen here will give us packages which can, in principle, allow you to have more than one ruby version installed at once. Here’s an example.

Say we generate a package for ruby 2.0.0-p247. The name of the working directory we’ll generate will be ruby-install-ruby2.0.0p247-1. When we generate our debian package metadata in a moment, the tool we use to do it will pick out ruby-install-ruby2.0.0p247 as the package name, and 1 as the package version. If we hadn’t dropped the hyphens with #tr(), and instead gone for ruby-install-ruby-2.0.0-p247, the package we generated would be called ruby-install-ruby, and the package version would be 2.0.0-p247. That would stop us from having more than one ruby version installed at a time, but it would also complicate deploying an updated package since as soon as we uploaded a later ruby version, an apt-get update on any host we’d previously installed to would grab the later version.

3: Makefile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
def makefile( ruby_version )
  makefile=<<-MAKEFILE
DESTDIR?=/
RUBY_VERSION?=#{ruby_version}
RUBY_ROOT=/opt/rubies/ruby-$(RUBY_VERSION)
RUBY_PATH=$(RUBY_ROOT)/bin
RUBY_BIN=$(RUBY_PATH)/ruby

all: root/$(RUBY_BIN)

root/$(RUBY_BIN):
 mkdir -p root
 DESTDIR="$$(cd root/; pwd)" ruby-install -i "$(RUBY_ROOT)" ruby $(RUBY_VERSION)

install:
 mkdir -p $(DESTDIR)
 cp -a root/* $(DESTDIR)

clean:
 rm -rf root/

.PHONY: all install
  MAKEFILE
end

def make_makefile( ruby_version )
  File.open("Makefile", "wb") do |f|
    f.puts( makefile( ruby_version ) )
  end
end

The #makefile() method generates a Makefile which defines a make all task that calls ruby-install to build ruby. It also contains a make install task to copy the built binaries into place.

When we generate the package metadata, it’ll rely on those two tasks.

The variables at the top of the generated Makefile are worth looking at briefly.

DESTDIR is used by the Debian package building system so the make install task doesn’t actually install the package we’re building to the system we’re building it on. DESTDIR is specified in the GNU coding standards, and all Debian package Makefiles should support it.

Note that we set a different DESTDIR variable to pass to ruby-install in the root/$(RUBY_BIN) task. That DESTDIR is passed to ruby’s own Makefile, so that it doesn’t have the location it’s built at hardwired in.

I’m using the RUBY_* variables to point at the location where ruby-install will drop our compiled ruby binary, so that successive make invocations won’t redo unnecessary work.

4: Dependencies

1
2
3
4
5
6
7
def add_depends
  dependencies = `grep apt: /usr/share/ruby-install/ruby/dependencies.txt | sed 's/apt: //'`.strip.split(/\s+/)
  dependencies.delete "build-essential"
  dep_string = dependencies.join(", ")
  cmd = "sed -i '/^Depends:/s|$|, #{dep_string}|' debian/control"
  system cmd
end

ruby-install helpfully includes a list of the system packages that ruby depends on, in a format we can almost just drop into our Debian package definition. The gotcha is that it lists build dependencies, when we actually want runtime dependencies. Since we don’t want a compiler and associated gubbins installed when we apt-get install our shiny new ruby package, we have to drop the build-essential dependency. Fortunately for us, in the case of all the other dependencies, the runtime dependency is listed as a dependency of the build dependency so that, for instance, where /usr/share/ruby-install/ruby/dependencies.txt lists libreadline-dev, libreadline-dev will depend on libreadline6.

There is a small catch to listing dependencies like this. Once we’ve built the packages, they have to be installed with apt-get install --no-install-recommends. This is because one of the -dev packages recommends g++, which we’re trying not to include. If you prefer not to rely on --no-install-recommends, you could replace the first two lines of #add_depends with:

1
  dependencies = %w{zlib1g libyaml-0-2 libssl1.0.0 libgdbm3 libreadline6 libncurses5 libffi5}

I don’t like hardcoded lists like that, but it’s a workable approach.

5: Debianise

The next step is to generate the actual package metadata to turn our humble Makefile into a buildable package.

The files involved are a little fiddly, so we cheat. All we want is a package that’s buildable, rather than one that we could hope to submit upstream, so we can lightly abuse the Debian packaging toolkit to make our lives easier.

1
2
3
4
5
6
7
def make_debian
  system "/bin/bash -c 'echo | dh_make --copyright gpl2 --native --single'"
  system "mkdir -p debian.keep"
  system "/bin/bash -c 'cp debian/{rules,control,compat,changelog} debian.keep/'"
  system "rm -rf debian"
  system "mv debian.keep debian"
end

The critical line is the first, the rest is housekeeping. dh_make is a rather nice little debhelper tool - install it with apt-get install dh-make - which exists to set up the boilerplate you need for a Debian package.

It’s worth taking a look at the generated files. They’ll be in, for instance, the ruby-install-ruby/ruby-install-ruby2.0.0p195/debian directory. control and changelog in specific are worth your time, as they both contain fields you can edit to add useful cosmetic information to the final package.

The remainder of this method is concerned with cleaning out a whole load of example configuration files that dh_make generates which we simply don’t need.

Build it

I’ve got the above script saved to ~/bin/ruby-install-deb. I’m going to build a package for ruby 2.0.0-p195. Here’s what I do, and the console output:

1
2
3
4
5
6
7
8
9
10
$ ruby-install-deb 2.0.0-p195
Maintainer name  : Alex Young
Email-Address    : alex@blackkettle.org
Date             : Mon, 26 Aug 2013 16:19:47 +0100
Package Name     : ruby-install-ruby2.0.0p195
Version          : 1
License          : gpl2
Type of Package  : Single
Hit <enter> to confirm: Done. Please edit the files in the debian/ subdirectory now. You should also
check that the ruby-install-ruby2.0.0p195 Makefiles install into $DESTDIR and not in / .

You can safely ignore the Please edit... message, since we know the Makefile we’ve generated follows the rules.

1
2
3
4
5
6
7
$ ls
ruby-install-ruby
$ cd ruby-install-ruby
$ ls
ruby-install-ruby2.0.0p195-1
$ cd ruby-install-ruby2.0.0p195-1
$ dpkg-buildpackage -us -uc

This dpkg-buildpackage command is what triggers our ruby-install build. The -us -uc flags are to do with package signing, saying, in short, don’t do it. This is another place packages for public consumption would differ from these that we’re building here.

Since ruby-install calls sudo, I need to authenticate; then after a while, I get the output:

1
2
3
4
5
6
7
8
9
...
   dh_builddeb
dpkg-deb: building package `ruby-install-ruby2.0.0p195' in `../ruby-install-ruby2.0.0p195_1_amd64.deb'.
 dpkg-genchanges  >../ruby-install-ruby2.0.0p195_1_amd64.changes
dpkg-genchanges: including full source code in upload
 dpkg-source --after-build ruby-install-ruby2.0.0p195-1
dpkg-buildpackage: full upload; Debian-native package (full source is included)

$

And it’s built! Now, it’s dropped packages in the directory above us, so let’s take a look at what it did:

1
2
3
4
5
6
7
$ cd ..
$ ls -1
ruby-install-ruby2.0.0p195-1
ruby-install-ruby2.0.0p195_1_amd64.changes
ruby-install-ruby2.0.0p195_1_amd64.deb
ruby-install-ruby2.0.0p195_1.dsc
ruby-install-ruby2.0.0p195_1.tar.gz

The .deb is the only file we strictly need.

Building a repository

Having packages is only half the battle. To make the packages easy to install at deploy-time, and to make dependency resolution work for us, we need to set up an apt repository. While we’re at it, we’ll build a couple more ruby packages, to see how they fit together.

Back in the our root directory, make a rubies file to list which ruby binaries we want to build. Mine looks like this:

1
2
3
4
5
$ cat rubies
1.9.3-p429
2.0.0-p195
2.0.0-p247
$

Now, we can set up a debian directory for each of these like so:

1
2
3
4
5
6
7
$ cat rubies | xargs -n1 bin/ruby-install-deb
...
$ ls -1 ruby-install-ruby
ruby-install-ruby1.9.3p429-1
ruby-install-ruby2.0.0p195-1
ruby-install-ruby2.0.0p247-1
$

So that’s given us three package definitions to play with. Here’s a small script to build them all, which I’ve got saved to ~/bin/ruby-install-deb-build:

ruby-install-deb-build
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/usr/bin/env ruby

debdirs = Dir['ruby-install-ruby/*/debian'].
  select{|d| File.directory?( d )}.
  map{|d| File.dirname( d )}

failures = debdirs.each_with_object([]) do |debdir,failures|
  success = Dir.chdir( debdir ) do
    system "dpkg-buildpackage -us -uc"
  end
  failures << debdir unless success
end

unless failures.empty?
  $stderr.puts failures.map {|debdir| "Building #{debdir} failed."}
  exit 1
end

This simply iterates over the debian directories it finds, and issues a dpkg-buildpackage for each one. Unfortunately these can’t run in parallel, because ruby-install calls apt-get install and that’s guaranteed to fail if you run it more than once at a time.

Nevertheless, once it’s finished, you can see what it has built:

ruby-install-deb-build
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ ls -1 ruby-install-ruby
ruby-install-ruby1.9.3p429-1
ruby-install-ruby1.9.3p429_1_amd64.changes
ruby-install-ruby1.9.3p429_1_amd64.deb
ruby-install-ruby1.9.3p429_1.dsc
ruby-install-ruby1.9.3p429_1.tar.gz
ruby-install-ruby2.0.0p195-1
ruby-install-ruby2.0.0p195_1_amd64.changes
ruby-install-ruby2.0.0p195_1_amd64.deb
ruby-install-ruby2.0.0p195_1.dsc
ruby-install-ruby2.0.0p195_1.tar.gz
ruby-install-ruby2.0.0p247-1
ruby-install-ruby2.0.0p247_1_amd64.changes
ruby-install-ruby2.0.0p247_1_amd64.deb
ruby-install-ruby2.0.0p247_1.dsc
ruby-install-ruby2.0.0p247_1.tar.gz

So now we’ve got a bunch of packages. We could stop here. If we did, to install the packages we’ve just built, we’d need to copy the .deb to our new host, and use a tool like gdebi to install the .deb and its dependencies.

Let’s not do that. It’s better to put together an apt repository to act as a local mirror, and to let apt-get install our dependencies as normal.

To put together a minimal repository, all we need to do is gather our files together, build an index, and upload the files and index to a web server. The repository can be served by a purely static server, and doesn’t need any dynamic server support at all.

What we’re building here is what Debian calls a “trivial” repository. Technically they’re deprecated because they don’t support some apt features, but they’re fine for our purposes.

Here’s how we gather the files and build the index:

ruby-install-deb-build
1
2
3
4
$ mkdir repo
$ mv ruby-install-ruby/*.deb repo/
$ cd repo
$ dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

The dpkg-scanpackages step will give you a couple of warnings. You can safely ignore them.

I’ve explicitly ignored the .dsc, .changes and .tar.gz files here. Again, they’re needed for specific apt features, and if all we want to do is build a repository to let us install ruby onto a server of our choice, we don’t need them.

All that’s left now is to upload the contents of repo/ to a convenient webserver. A fresh Debian stable VM with Apache or nginx apt-get installed will do fine here:

ruby-install-deb-build
1
2
3
4
# assuming you've made the `/var/www/ruby-install` directory
# and can upload to it...
$ rsync -az repo/* webserver:/var/www/ruby-install/
$ ssh webserver chown -R www-data: /var/www/ruby-install

Installing from the repository

Ok. So we’ve built packages, made a repository, and made it available. Now we’re ready to actually use the package in deployment. Assuming your new machine is called new-vm, here’s how you do that by hand.

ruby-install-deb-build
1
2
3
$ ssh -t root@new-vm
root@new-vm:~# echo "deb http://webserver/ruby-install ./" > /etc/apt/sources.list.d/ruby-install.list
root@new-vm:~# apt-get update

At this point, new-vm should know about the packages we’ve built. Let’s check:

ruby-install-deb-build
1
2
3
4
root@new-vm:~# apt-cache search ruby-install-ruby --names-only
ruby-install-ruby1.9.3p429 - <insert up to 60 chars description>
ruby-install-ruby2.0.0p195 - <insert up to 60 chars description>
ruby-install-ruby2.0.0p247 - <insert up to 60 chars description>

If you don’t like having the filler info there as the package description, you can edit it in debian/control before building the packages.

Anyway, now apt knows about the packages, and we should be able to install them. We will be asked if they’re safe to install without verification; since they (and the repository) are unsigned, just say “yes”.

ruby-install-deb-build
1
2
3
4
5
6
7
8
9
10
11
root@new-vm:~# apt-get install ruby-install-ruby1.9.3p429 ruby-install-ruby2.0.0p195 --no-install-recommends
...
Setting up ruby-install-ruby1.9.3p429 (1) ...
Setting up ruby-install-ruby2.0.0p195 (1) ...
root@new-vm:~# ls /opt/rubies
ruby-1.9.3-p429
ruby-2.0.0-p195
root@new-vm:~# /opt/rubies/ruby-1.9.3-p429/bin/ruby -v
ruby 1.9.3p429 (2013-05-15 revision 40747) [x86_64-linux]
root@new-vm:~# /opt/rubies/ruby-2.0.0-p195/bin/ruby -v
ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-linux]

Hooray! For me, on a fresh local Wheezy VM with 256MB of RAM, apt-get installing a single ruby-install-ruby package, including dependencies, takes a minute. That’s better than compiling from source every time, by quite a long way.

ruby

Comments