Using Ksplice to Patch Head Nodes

1  The problem, and Ksplice

It’s often a problem to reboot HPC head nodes¹ to apply kernel security patches (or, possibly, important bug fixes). There are often processes that can’t be killed without disrupting running jobs,² and you don’t want to waste resources draining the queues concerned. In some circumstances, it might be appropriate to patch compute nodes too without wanting to take them down, but typically they’ll just be rebooted after the current job ends, and re-provisioned then if they’re stateful. (At least traditionally, kernel patches were the only thing that required a reboot on sane systems—of which Solaris isn’t one—but these days there seem to be more things like udev that are heavily involved with the kernel and could also cause trouble, but I’m not sure if they’re a real problem.)

Possible problem areas for reboots of HPC heads include some MPI implementations, kinds of job steering that require something on the head to communicate with the compute nodes (which admins don’t necessarily know about!), and Globus (sigh) GRAM processes. Those have all bitten us, though thankfully, we’re not going to be saddled with the problematic MPI (SCore). Some such processes may be checkpoint/restartable through a reboot, but in general they’re not.

Technology has recently appeared to address this by hot-patching³ systems running Linux, called Ksplice. It doesn’t seem to be as widely known as it perhaps should be, hence this article. It doesn’t have any problems I’ve come across, assuming you can derive the source patch to apply, but note that it only works on Linux versions 2.6.n, i.e. not on RedHat 3, for instance. Here are some notes on using Ksplice, with recent examples of using it on our systems. Corrections, clarifications, and additions are welcome, of course.

2  Necessary components

How easy it is to use Ksplice depends on whether you’re using a GNU/Linux distribution with a patch for the relevant problem already available. See §3.

Apart from a patch, you’ll need to install the Linux source package for your distribution (linux-source on Debian-ish systems and SuSE, or the unpacked SRPM for RedHat-ish ones), whatever you need to rebuild Linux, and ksplice itself. There are ksplice packages for recent Ubuntu, Debian (at least in testing), and Fedora. On RedHat/CentOS and OpenSuSE, I adapted the .spec from the Fedora 11 SRPM,⁴ and subsequently trivially updated for the current source tarball from the Ksplice distribution site.

The necessary changes to use the Fedora 11 ksplice.spec file with the latest Ksplice tarball on OpenSuSE 10.3 or RHEL 5.3 were just these (which probably won’t apply by cut-and-paste, due to tabs getting tab-mangling):

--- ksplice.spec	2009-11-09 15:14:04.000000000 +0000
+++ ../SPECS/ksplice.spec	2009-11-06 14:28:00.000000000 +0000
@@ -1,5 +1,5 @@
 Name:		ksplice
-Version:	0.9.7
+Version:	0.9.9
 Release:	2%{?dist}
 Summary:	Patching a Linux kernel without reboot
 
@@ -27,7 +27,7 @@
 Requires:	make >= 3.78
 Requires:	sh-utils, m4,
 Requires:	findutils, diffutils, tar, gzip, bzip2, perl, gawk
-Requires:	util-linux-ng
+Requires:	util-linux
 
 %description
 Ksplice allows system administrators to apply security patches to the 

RHEL 4 is apparently missing (the equivalent of?) the binutils-devel package for the BFD bits but, regardless, I built the package with whatever development stuff we have on our RHEL 4 system by additionally removing the binutils-devel from BuildRequires: in ksplice.spec.

3  Obtaining or creating a patch

Getting the right patch to use with Ksplice depends on whether or not one has been released for your distribution. If so, it’s probably referenced in the relevant security alert and/or in a bug database, perhaps more conveniently than in the relevant Linux source package. Otherwise—e.g. before a fix is released, or if you have to keep an obsolete distribution running—look for a kernel.org patch in the security alert. For example, consider CVE-2009-3547. The RedHat erratum is at https://rhn.redhat.com/rhn/errata/details/Details.do?eid=9301, with a reference to a kernel.org patch, which is in the kernel SRPM as linux-2.6-fs-fix-pipe-null-pointer-dereference.patch.

If you don’t have a patch appropriate for your current Linux source, things are trickier. Maybe the appropriate kernel.org one applies anyway. (After you’ve tested it, back it out from the source tree with patch -R or by recovering the .orig files patch created if it failed.) Otherwise you need to look for one elsewhere that will apply, or adapt the closest you can find to your source tree, assuming it’s sufficiently obvious how to do that, as it has been to me so far. The Ksplice paper has historical data on applying/adapting patches.

In the case of our unfortunately-now-obsolete OpenSuSE 10.3—thanks, dear vendor—the CVE-2009-3547 patch applied directly. For CVE-2009-2908, from the same RedHat erratum notice, it didn’t, but it’s clear that what’s needed is just to wrap the body of ecryptfs_unlink in dget...dput.⁵ (I edited ecryptfs/inode.c, got a patch against the Emacs backup file with M-x diff-backup, and reverted the change with C-u C-c C-a in the diff-mode buffer.) Ksplice expects a patch with the top-level Linux directory at the head of the path by default, so it can be applied in the tree with patch -p1, so probably fix that if necessary. For example, the ecryptfs.diff patch I ended up with in this case is:

diff -u /usr/src/linux-2.6.22.19-0.4/fs/ecryptfs/inode.c\~ /usr/src/linux-2.6.22.19-0.4/fs/ecryptfs/inode.c
--- linux-2.6.22.19-0.4/fs/ecryptfs/inode.c~	2009-11-06 17:21:58.000000000 +0000
+++ linux-2.6.22.19-0.4/fs/ecryptfs/inode.c	2009-11-09 10:39:34.000000000 +0000
@@ -456,6 +456,7 @@
 	struct vfsmount *lower_mnt = ecryptfs_dentry_to_lower_mnt(dentry);
 	struct inode *lower_dir_inode = ecryptfs_inode_to_lower(dir);
 
+ 	dget(lower_dentry);
 	lock_parent(lower_dentry);
 	rc = vfs_unlink(lower_dir_inode, lower_dentry, lower_mnt);
 	if (rc) {
@@ -468,6 +469,7 @@
 	dentry->d_inode->i_ctime = dir->i_ctime;
 out_unlock:
 	unlock_parent(lower_dentry);
+ 	dput(lower_dentry);
 	return rc;
 }
 

4  Applying patches

Given a suitable patch, Linux source, and ksplice installation, you can now apply the patch along the lines of the documented example, though linking the headers into the Linux tree probably isn’t necessary if you have the relevant kernel development package installed. You may want to use the –id or (undocumented) –description option of ksplice-create(1) so that you can make sense better of the applied patches with ksplice-view(1), or of the created tarballs.

If you have an updated Linux binary package, you’ll want to install that for the next reboot. If you don’t, add a ksplice-apply(1) in the appropriate init script. For RedHat and Debian that’s probably /etc/rc.local, and /etc/init.d/boot.local on SuSE, assuming you don’t worry about a short vulnerability window during init. (The –partial option of ksplice-apply(1) may be useful.) When you have an updated vendor Linux package, remove the relevant lines from the init script before you boot into it; otherwise, you’ll likely get an obscure error

No kid at /usr/sbin/ksplice-apply line 64, <CONTENTS> line 3.

though it’s harmless as far as I know.

5  An easier life

If you run one of the distributions they support, you can pay the Ksplice author’s company to deal with the updates for you with their Uptrack service (a commercial service using their free software). It’s gratis for Ubuntu 9.04 and 9.10 if you happen to be running one of them. I have it on my desktop but haven’t checked out the recently-advertised trial for RedHat, in particular what it costs.