PXE Booting SmartOS from SmartOS zone

Motivation

We’ve bought a new Supermicro server – chassis and four blades. The provider installed Ubuntu on one of them, and from this I have already set up SmartOS on three other blades. As you know, host machine running on SmartOS boots from PXE server. But I don’t need a separate blade running on Linux, so to ensure safety I decided that each blade could be used as a loader for the rest of them. It was possible to deploy Linux on each host in KVM, but I found a better solution – to deploy PXE server in native SmartOS zone. Isn’t that wonderful when SmartOS can boot SmartOS? 

Here's how to set up a simple PXE server in a SmartOS zone that will serve up SmartOS

imgadm update

imgadm import 8639203c-d515-11e3-9571-5bf3a74f354f

create pxe-server.json with following:
 

Zone Configuration

{
  "alias": "pxe-server",
  "hostname": "pxe-server",
  "brand": "joyent",
  "max_physical_memory": 64,
  "quota": 2,
  "image_uuid": "8639203c-d515-11e3-9571-5bf3a74f354f",
  "resolvers": [
  "8.8.8.8",
  "8.8.4.4"],

 "nics": [

    {
      "nic_tag": "admin",
      "ip": "192.168.0.2",
      "netmask": "255.255.255.0",
      "gateway": "192.168.0.1",
      "dhcp_server": "1"
    }
  ]
}

vmadm create -f pxe-server.json

Setting up TFTP

Use zlogin to log into the zone:

zlogin <uuid>

In the zone:

pkgin -y install tftp-hpa

mkdir /tftpboot

echo "tftp dgram udp wait root /opt/local/sbin/in.tftpd in.tftpd -s /tftpboot" > /tmp/tftp.inetd

svcadm enable inetd

inetconv -i /tmp/tftp.inetd -o /tmp

svccfg import /tmp/tftp-udp.xml

svcadm restart tftp/udp

Setting up DHCP (using Dnsmasq)

pkgin -y install dnsmasq

 
Edit /opt/local/etc/dnsmasq.conf

dhcp-range=192.168.0.200,192.168.0.220,2h
dhcp-match=set:gpxe,175
dhcp-boot=tag:!gpxe,undionly.kpxe
dhcp-boot=smartos.ipxe
dhcp-leasefile=/etc/dnsmasq.leases

svcadm enable dnsmasq

Setting up the tftpboot directory

Ben Rockwood provides a version of undionly.kpxe on his site. Run the following to get the PXE chainload binaries in place:

cd /tftpboot

curl http://cuddletech.com/IPXE-100612_undionly.kpxe > undionly.kpxe

At this point a generic PXE boot server is complete. iPXE will still expect smartos.ipxe, but that can be created with whatever content is needed. For those interested in booting SmartOS, what follows are the steps to provide SmartOS boot services on this server.

Providing SmartOS PXE Boot Services

A template iPXE config is useful both upfront and when updating to new platform releases. Create /tftpboot/smartos.ipxe.tpl with the following content (-B smartos=true is essential, otherwise logins will fail):

#!ipxe
# /var/lib/tftpboot/smartos.ipxe.tpl
kernel /smartos/$release/platform/i86pc/kernel/amd64/unix -B smartos=true
initrd /smartos/$release/platform/i86pc/amd64/boot_archive
boot

cd /tftpboot

mkdir smartos

Deploy/Update to the latest SmartOS platform release

The steps in this section work for both initial deployment and upgrades as Joyent releases them.

Next get the latest SmartOS platform and massage it into a workable shape for our iPXE config:

cd /tftpboot/smartos

curl https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/platform-latest.tgz > /var/tmp/platform-latest.tgz
(Just now URL https://download.joyent.com/pub/iso/platform-latest.tgz is invalid, 404… )

cat /var/tmp/platform-latest.tgz | tar xz

directory=`ls | grep platform- | sort | tail -n1`

release=${directory:9}

mv $directory $release

cd $release

mkdir platform

mv i86pc platform

cd /tftpboot

cat smartos.ipxe.tpl | sed -e"s/\$release/$release/g" > smartos.ipxe

Make sure PXE boot is enabled and that it is the first in the boot sequence. 

Thanks

Thanks to Alain O'Dea for his notes about his experience in setting up Ubuntu Server 12.04.1 LTS as a PXE server to boot SmartOS and big thanks to Ben Rockwood for creating and maintaining the PXE Booting SmartOS wiki page. Without their instructions I would not have done it.

Enjoy and stay tuned!