By Alasdair Lumsden on 23 Aug 2010
I was having a nightmare with an OpenSolaris domU xen guest I had broken. It had multiple boot environments, all I needed was to boot one of them – the default rpool/ROOT/opensolaris one would do. However when booting the guest, no grub menu was presented by pygrub, it just goes straight to booting the kernel:
root ~ (xvm03.everycity): xm start guest -c v3.4.2-xvm chgset 'Wed Feb 17 02:28:56 2010 -0800 19674:ba65a89eded4' SunOS Release 5.11 Version snv_147 64-bit Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved. /platform/i86xpv/kernel/drv/amd64/rootnex: undefined symbol 'iommulib_nexdma_unmapobject' WARNING: mod_load: cannot load module 'rootnex' panic[cpu0]/thread=fffffffffbc609e0: BAD TRAP: type=e (#pf Page fault) rp=fffffffffbcb4ec addr=20 occurred in module "genunix" due to a NULL pointer dereference ... skipping system dump - no dump device configured rebooting...
I spent ages trying to get the grub menu to display, but couldn’t work out how to, so decided to take a different approach.
First, I needed a list of boot environments out of the guest. If you already know which boot environment you want to boot (such as the original rpool/ROOT/opensolaris boot environment) then you can skip this bit. I include it as it’s quite a nifty method – thanks to a friend from IRC, he provided me with this clever script:
#!/usr/bin/python2.4 import sys import pygrub import fsimage disk = sys.argv file = sys.argv fs = fsimage.open(disk, pygrub.get_fs_offset(disk)) f = fs.open_file(file) print f.read();
Pop this into a file called readfile.py. This script steals some libraries from pygrub, so you’ll need to copy this to the same working directory. The usage is as follows:
# cp /usr/lib/xen/bin/pygrub ./pygrub.py # ./readfile.py /dev/zvol/dsk/rpool/xvm/guest /boot/grub/menu.lst
This will then spit back the grub menu list at you:
# python2.4 readfile.py /dev/zvol/dsk/data/xvm/guest/disk0 /boot/grub/menu.lst timeout 5 default 3 #---------- ADDED BY BOOTADM - DO NOT EDIT ---------- title OpenSolaris Development snv_134 findroot (pool_rpool,0,a) bootfs rpool/ROOT/opensolaris kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS module$ /platform/i86pc/$ISADIR/boot_archive ...
From this list, the information we want is the “bootfs” argument from the boot environment you want to boot. So for example rpool/ROOT/opensolaris. We can then shove this into the xvm xml config file for the guest:
<domain type='xen'> <name>guest</name <memory>4194304</memory> <currentMemory>4194304</currentMemory> <vcpu>8</vcpu> <bootloader>/usr/lib/xen/bin/pygrub</bootloader> <bootloader_args>--args="-B zfs-bootfs=rpool/ROOT/opensolaris"</bootloader_args> <os> <type>linux</type> </os> ...
The key line above there is the bootloader_args argument – this passes the correct bootfs to the kernel telling it which ZFS filesystem to use for the boot. Hey presto, your guest should now boot :)