Boot environments with OpenSolaris domUs on xVM

By Alasdair Lumsden on 23 Aug 2010

I was having a nightmare with an OpenSolaris domU xen guest I had broken. It had multiple boot environments, all I needed was to boot one of them – the default rpool/ROOT/opensolaris one would do. However when booting the guest, no grub menu was presented by pygrub, it just goes straight to booting the kernel:

root ~ (xvm03.everycity): xm start guest -c
v3.4.2-xvm chgset 'Wed Feb 17 02:28:56 2010 -0800 19674:ba65a89eded4'
SunOS Release 5.11 Version snv_147 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
/platform/i86xpv/kernel/drv/amd64/rootnex: undefined symbol 'iommulib_nexdma_unmapobject'
WARNING: mod_load: cannot load module 'rootnex'

panic[cpu0]/thread=fffffffffbc609e0: BAD TRAP: type=e (#pf Page fault) rp=fffffffffbcb4ec
 addr=20 occurred in module "genunix" due to a NULL pointer dereference
skipping system dump - no dump device configured

I spent ages trying to get the grub menu to display, but couldn’t work out how to, so decided to take a different approach.

First, I needed a list of boot environments out of the guest. If you already know which boot environment you want to boot (such as the original rpool/ROOT/opensolaris boot environment) then you can skip this bit. I include it as it’s quite a nifty method – thanks to a friend from IRC, he provided me with this clever script:

import sys
import pygrub
import fsimage

disk = sys.argv[1]
file = sys.argv[2]

fs =, pygrub.get_fs_offset(disk))
f = fs.open_file(file)


Pop this into a file called This script steals some libraries from pygrub, so you’ll need to copy this to the same working directory. The usage is as follows:

# cp /usr/lib/xen/bin/pygrub ./
# ./ /dev/zvol/dsk/rpool/xvm/guest /boot/grub/menu.lst

This will then spit back the grub menu list at you:

# python2.4 /dev/zvol/dsk/data/xvm/guest/disk0 /boot/grub/menu.lst
timeout 5
default 3
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title OpenSolaris Development snv_134
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/opensolaris
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive

From this list, the information we want is the “bootfs” argument from the boot environment you want to boot. So for example rpool/ROOT/opensolaris. We can then shove this into the xvm xml config file for the guest:

<domain type='xen'>
 <bootloader_args>--args="-B zfs-bootfs=rpool/ROOT/opensolaris"</bootloader_args> 

The key line above there is the bootloader_args argument – this passes the correct bootfs to the kernel telling it which ZFS filesystem to use for the boot. Hey presto, your guest should now boot :)