Table of contents

1. Intruduction

Intel Preboot eXecution Environment (PXE) allows you to boot up a system through network - get network configuration from DHCP during boot time, download kernel and boot it along with userland or system installation. PXE does not need it's own server - it uses bootstrap protocol (BOOTP). You can either use bootpd or dhcpd and tftpd. This mini-howto covers this second option. I have considered using MFS (memory filesystems), but due to some limitations, decided to end up with NFS.

The only software, that is not a part of FreeBSD base system, which will be used is ISC DHCP server.

For my own purposes, I have configured all above services for /28 network - you can of course change those settings to fit to your requirements. Also, because of performance and stability issues, I have decided to put client's distribution on mirrored ZFS filesystem with snapshots, but this matter won't be described here. I am using separate network interface to serve system to client's hosts - in regard of portability, I have renamed it to pxeserver).

My configuration:

2. Preparing host system

First thigs first (...):

hosts.allow / firewall - DO UZUPELNIENIA

As mentioned above, I have renamed my interface:

ifconfig bge0 name pxeserver
ifconfig pxeserver inet add 10.1.248.241/28
echo 'ifconfig_bge0_name="pxeserver"'>>/etc/rc.conf
echo 'ifconfig_pxeserver="inet 10.1.248.241 network 255.255.255.240"'>>/etc/rc.conf

3. Preparing system distribution for diskless clients

This part is pretty much the same, as creating new jailed system from scratch, with one difference: you have to install kernel along with world and kernel have to be compiled with "LOADER_TFTP_SUPPORT" option.

echo 'LOADER_TFTP_SUPPORT="yes"'>>/etc/make.conf
cd /usr/src
make buildworld
make buildkernel
make installworld DESTDIR=/usr/FreeBSD-client
make installkernel DESTDIR=/usr/FreeBSD-client
make distribution DESTDIR=/usr/FreeBSD-client

However, you might want to build system for different architecture, than the one running on server. Simply, you do it just like with jails, but, again, with separate kernel, not just the main kernel with COMPAT options. Example here is i384 client system architecture, running on amd64 host:

echo 'LOADER_TFTP_SUPPORT="yes"'>>/etc/make.conf
cd /usr/src
make TARGET_ARCH=i386 TARGET=i386 buildworld
make TARGET_ARCH=i386 TARGET=i386 buildkernel
make installworld TARGET_ARCH=i386 TARGET=i386 DESTDIR=/usr/FreeBSD-client
make installkernel TARGET_ARCH=i386 TARGET=i386 DESTDIR=/usr/FreeBSD-client
make distribution DESTDIR=/usr/FreeBSD-client
cd sys/boot
make clean&&make TARGET_ARCH=i386 TARGET=i386 depend
make TARGET_ARCH=i386 TARGET=i386
mv /usr/FreeBSD-client/boot/pxeboot /usr/FreeBSD-client/boot/pxeboot-old
cp i386/pxeldr/pxeboot /usr/FreeBSD-client/boot/
diff /usr/FreeBSD-client/boot/pxeboot-old /usr/FreeBSD-client/boot/pxeboot

Of course i386 kernel config have to differ from amd64 one by few things:
< cpu HAMMER
> cpu I686_CPU
> device apic
< options COMPAT_IA32
< options COMPAT_FREEBSD32
< options KDTRACE_HOOKS
etc...

4. Installing and configuring dhcpd

At the time of writing this, the latest isc-dhcp in ports is version 3.1.3. You can of course use any newer one, however configuration might be slightly different.

cd /usr/ports/net/isc-dhcp31-server/
make config install clean
vi /usr/local/etc/dhcpd.conf

/usr/local/etc/dhcpd.conf:

use-host-decl-names on;
option domain-name-servers 217.17.34.10,194.204.159.1;
default-lease-time 7200;
max-lease-time 28800;
ddns-update-style ad-hoc;
log-facility local7;
#option next-server 10.1.248.241;
subnet 10.1.248.240 netmask 255.255.255.240 {
  range 10.1.248.242 10.1.248.254;
  option routers 10.1.248.241;
  option broadcast-address 10.1.248.255;
  filename "system/boot/pxeboot";
  option root-path "10.1.248.241:/usr/FreeBSD-client/system";
}

In this example, I have used most useful (fast) DNS servers in my country - of course you should change it to fit to your ISP/connection/location.

5. Preparing tftpd, rpcbind, mountd and nfsd

We need to add tftp, rpcbind, mountd and nfsd to system startup, as well as the NFS export to /etc/exports file.

echo 'tftp dgram udp wait root /usr/libexec/tftpd tftpd \
  -l -s /usr/FreeBSD-client'>>/etc/inetd.conf
echo 'inetd_enable="YES"'>>/etc/rc.conf
echo 'nfs_server_enable="YES"'>>/etc/rc.conf
echo 'mountd_enable="YES"'>>/etc/rc.conf
echo 'mountd_flags="-h 10.1.248.241 -l"'>>/etc/rc.conf
echo 'rpcbind_enable="YES"'>>/etc/rc.conf
echo '/usr/FreeBSD-client/ -network 10.1.248.240 -mask 255.255.255.240'>>/etc/exports

We need to prepare client's loader.conf for NFS booting. As mentioned above, I am using ZFS filesystem under the /usr/FreeBSD-client/ directory. If you don't, you can skip the first line.

echo 'zfs_load="YES"'>>/usr/FreeBSD-client/boot/loader.conf
mv /usr/FreeBSD-client/boot/loader.conf /usr/FreeBSD-client/boot/_old_loader.conf
echo 'boot.nfsroot.server="10.1.248.241"'>>/usr/FreeBSD-client/boot/loader.conf
echo 'boot.nfsroot.path="/usr/FreeBSD-client/"'>>/usr/FreeBSD-client/boot/loader.conf
echo 'vfs.root.mountfrom="nfs"'>>/usr/FreeBSD-client/boot/loader.conf
echo 'vfs.root.mountfrom.options=rw'>>/usr/FreeBSD-client/boot/loader.conf

Now, let's check, if everything is well configured, up and working

/etc/rc.d/rpcbind restart
/etc/rc.d/nfsd restart
/etc/rc.d/mountd restart
/usr/bin/showmount -e 10.1.248.241

6. Possible problems, errors explained

Server side:

tftpd[xxxx]: Prevent NAK storm
tftpd[xxxx]: Got ERROR packet: TFTP Aborted

This indicates, that you have a problem with NFS communication - nfsd is not running, not registered with RPC, blocked, etc. At this point PXE client can freez after obtaining network configuration and pxeboot file, or after booting kernel, when it tries to mount filesystem over NFS.

mountd: NFS server is not available

NFS server is not running, or not registered with rpcbind

/etc/rc.d/nfsserver: WARNING: Unable to load kernel module nfsserver

Proper NFS options are not compiled into kernel, nor nfsserver.ko module can be found and/or loaded

Client side:

BTX loader 1.00 BTX version is 1.02

If client's loader freezes at this point, this may indicate, that pxeboot file is compiled for another architecture

PXE-T01: File not found
PXE-E3B: TFTP Error - file Not found
PXE-M0F: Exiting Intel PXE ROM.
FATAL: Could not read from the boot medium! System halted.

This error indicates, that the pxeboot file ('filename' option from dhcpd.conf) cannot be found and/or accessed.

pxe_open: server addr: 10.1.248.241
pxe_open: server path: /usr/FreeBSD-client
pxe_open: gateway ip: 10.1.248.241
\
start not found

If the loader freezes at this point, or even without this last line, it most likely means, that you have built system for another architecture, than the client-computer. I have tested this situation only with amd64 served system on one physical i386 laptop and on i386 "FreeBSD" version under Oracle VirtualBox, when 64-bit FreeBSD was working well.

pxe_open: server addr: 10.1.248.241
pxe_open: server path: /usr/FreeBSD-client
pxe_open: gateway ip: 10.1.248.241
\
can't load 'kernel'

Type '?' for a list of commands, 'help' for more detailed help
OK ls
open '/' failed: no such file od directory
OK

This situation indicates, that DHCP and tftpd are working ok, but there is a problem with nfs/bootp.

Trying to mount root from nfs
nfs_diskless: no interface
ROOT MOUNT ERROR:
If you have invalid mount options, reboot, and first try the following from
the loader prompt:

        set vfs.root.mountfrom.options=rw

and then remove invalid mount options from /etc/fstab.

I have found this problem persisting on an old-old computer with (fxp) Intel Server Adapter PRO/100 NIC (the same card is working fine on other machine in that matter...) and with Oracle VirtualBoX PXE emulation based on (em) Intel 82578DC Gigabit Ethernet PHY. Even if both modules - fxp and em, are compiled into kernel, and even booted before kernel, with loader.conf problem persists. I have found some information, that problem may be motherboard-ACPI-related: http://forums.freebsd.org/showthread.php?t=8324. Problem appeared on both tested systems: FreeBSD 8.1-RELEASE (801000) and 9.0-CURRENT (900014).