ZFS-Only FreeBSD

There are plenty of guides on how to use ZFS without UFS on FreeBSD 8, but that doesn't mean I can't make one as well.

Much of this information comes from http://wiki.freebsd.org/ZFSOnRootWithZFSboot and http://blogs.freebsdish.org/lulf/2008/12/16/setting-up-a-zfs-only-system/. The specific goal of this guide is to provide a single-OS system (FreeBSD, obviously) with simple disk redundancy and no RAID hardware; this is accomplished with a "mirror" (RAID-1) zpool and also swapping to either a GEOM_MIRROR device or to a ZFS volume.

Side notes about swapping:

It's been said recently (late September, 2009) that swapping to zvol can be problematic. The current recommendation is still to use a non-ZFS device for swap.
If you hope to obtain a kernel crash dump, be aware that /etc/rc.d/dumpon runs before /etc/rc.d/zfs, so you cannot send crash dumps to a ZFS volume without hacking the rc startup order or manually running /etc/rc.d/dumpon after the system starts.

You will use the "fixit" command shell rather than sysinstall(8), so I've also provided steps for several setup tasks that sysinstall(8) would normally take care of.

The shell commands in this guide are formatted and variable-ized for easy copying/pasting. Remember that a colon at the beginning of a command line or immediately after a semicolon is considered by sh(1) to be a comment, so you can freely paste or pipe large command blocks into your fixit command shell without removing the inline comments.

Boot the "livefs" CD or the DVD. If booting the CD, either have a second drive for the "disc1" CD or have disc1's contents available via NFS.
Access "Fixit" mode and set the following variables at the shell prompt. They will be used through step 8. $zfspart will be set to a new value in step 4 if you swap to a GPT partition.
```
disk0=ad4
disk1=ad6
zfspart=2
```

Run the following commands:

gpart create -s GPT $disk0
gpart create -s GPT $disk1
gpart add -s 128 -t freebsd-boot $disk0
gpart add -s 128 -t freebsd-boot $disk1

If you do not wish to swap to ZFS (zvol), create a swap partition. <swapsize> may be expressed as an integer suffixed with "M" or "G" (or, if you insist, no suffix, implying 512-byte blocks).
```
zfspart=3 ; : Since this will place swap on ${disk0}p2, respecify where ZFS will go
gpart add -s <swapsize> -t freebsd-swap $disk0
gpart add -s <swapsize> -t freebsd-swap $disk1
```
If you did not "load geom_mirror" at a loader prompt, load it now:
```
kldload /mnt2/boot/kernel/geom_mirror.ko
```
Create the mirror:
```
gmirror label swap0 /dev/${disk0}p2 /dev/${disk1}p2
```

If you did not "load zfs" at a loader prompt, load it now:

kldload /mnt2/boot/kernel/opensolaris.ko
kldload /mnt2/boot/kernel/zfs.ko

Allocate space for ZFS. Add "-s" and a size before "$disk0" and "$disk1" to specify a size if you don't want to use the remainder of the disk.
```
gpart add -t freebsd-zfs $disk0
gpart add -t freebsd-zfs $disk1
```

Make disks bootable:

gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 $disk0
gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 $disk1

Create pool and root filesystem; make bootable. $poolname will be referenced throughout the remainder of the guide and should be exported to the environment for its use in subshells a couple times.
```
export poolname=z0 ; : change as desired

zpool create ${poolname} mirror /dev/${disk0}p${zfspart} /dev/${disk1}p${zfspart}
zfs set mountpoint=none ${poolname}
zfs create -o mountpoint=/${poolname} ${poolname}/root
zpool set bootfs=${poolname}/root ${poolname}
```
Others may use ${poolname} as the root filesystem, but using (for example) ${poolname}/root allows you to set default properties for descendents of ${poolname} without affecting the root filesystem's own properties.

Create and mount other filesystems:

: chmod before creating separate /tmp so it's right if /tmp isn't mounted.
: install will take care of mode on ${poolname}/tmp filesystem.
: repeat for /${poolname}/var/tmp if creating a separate filesystem for that.
mkdir /${poolname}/tmp
chmod 1777 /${poolname}/tmp

: feel free to change/add/remove options, quotas, filesystems/mountpoints, etc.
zfs create -o mountpoint=/${poolname}/tmp -o setuid=off -o compression=on ${poolname}/tmp
zfs create -o mountpoint=/${poolname}/usr ${poolname}/usr
zfs create -o mountpoint=/${poolname}/var ${poolname}/var
zfs create -o mountpoint=/${poolname}/var/crash -o exec=off -o setuid=off -o compression=on -o quota=5120M ${poolname}/crash
zfs create -o mountpoint=/${poolname}/usr/obj -o atime=off ${poolname}/obj
zfs create -o mountpoint=/${poolname}/usr/ports -o atime=off -o compression=gzip-9 ${poolname}/ports
zfs create -o mountpoint=/${poolname}/usr/src -o atime=off -o compression=on ${poolname}/src
zfs create -o mountpoint=/${poolname}/usr/local ${poolname}/usrlocal
zfs create -o mountpoint=/${poolname}/usr/local/homes ${poolname}/homes
: etc...

If you wish to swap to ZFS rather than a swap partition, create a ZFS swap volume. <swapsize> may be expressed as a number with a 'K', 'M', or 'G' suffix (8G, 2560M, etc.).
```
zfs create -V <swapsize> ${poolname}/swap
zfs set org.freebsd:swap=on ${poolname}/swap
zfs set checksum=off ${poolname}/swap
```
If using two CDs or CD+NFS instead of the DVD, mount it as appropriate. Use /mnt2/rescue/mount_cd9660 if you have a second CD-ROM drive installed, as /sbin/mount in the fixit environment will not find mount_cd9660. (You could also run (cd /;ln -s mnt2/rescue) if you want to use mount -t cd9660 ....)
Start a new shell (sh -V or sh -E as desired) to make for an easy environment cleanup after the next steps.
Enter cd /dist/<OSVersion> (DVD) or cd /mnt/<OSVersion> (two CDs or CD+NFS, "disc1" mounted under /mnt)
Install the base and man pages. Not installing man pages here causes an error in step 16's "make install".
```
export DESTDIR=/${poolname}
for a in base *pages; do
  (cd $a && ./install.sh)
done
```
install.sh will prompt you to overwrite any installation in /${poolname}. Make sure it warns you about the right directory—if not, something went wrong with the "export" line above.
Continue installation with OS source and the GENERIC kernel.
```
(cd src && ./install.sh all)
(cd kernels && ./install.sh generic)
cd /${poolname}/boot
rmdir kernel
mv GENERIC kernel
exit
```

Create OS configuration files:

: /etc/src.conf
: THIS IS A MUST-HAVE!

echo 'LOADER_ZFS_SUPPORT=true' > /${poolname}/etc/src.conf
: add other entries as desired. edit etc/make.conf if needed.

: --------------------

: /boot/loader.conf
cat << EOH > /${poolname}/boot/loader.conf
# skip/delete/comment out next line if swapping to zvol:
geom_mirror_load=YES

zfs_load=YES
vfs.root.mountfrom="zfs:${poolname}/root"
# if "sysctl vm.kmem_size" is very different from size of physical RAM,
# set it to size of physical RAM
#vm.kmem_size=8G

# add anything else you know you'll need/want
EOH

: --------------------

: /etc/fstab
: uncomment swap entry if you followed step 4; zvol swap is activated
: automatically and so does not need an fstab entry
cat << EOH > /${poolname}/etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
proc                    /proc           procfs  rw              0       0

#/dev/mirror/swap0       none            swap    sw              0       0

# uncomment after installing Linux ports
#linprocfs               /compat/linux/proc linprocfs rw,late    0       0
EOH

: --------------------

: /etc/rc.conf
cat << EOH > /${poolname}/etc/rc.conf
zfs_enable=YES

# fill in values as desired
hostname=
ifconfig_xxxX=
defaultrouter=

# $dumpdev is now "NO" by default!
#dumpdev=/dev/mirror/swap0

#sshd_enable=YES
#ntpdate_enable=YES
#ntpdate_flags=-b
#ntpd_enable=YES
# etc...
EOH

: /etc/resolv.conf
cat << EOH > /${poolname}/etc/resolv.conf
nameserver x.y.z.w
search example.com example.org
EOH

cp /${poolname}/usr/share/zoneinfo/<YourTimeZone> /${poolname}/etc/localtime

: uncomment next line if system clock is not set to UTC:
: touch /etc/wall_cmos_clock

Also see etc/dhclient.conf, etc/ntp.conf, etc/ssh/sshd_config, and others as needed.

In a chrooted shell, build/install ZFS-aware loader and set root password. Be sure you created etc/src.conf as directed in the previous step.
```
mount -t devfs devfs /${poolname}/dev
chroot /${poolname}
cd /usr/src/sys/boot
make obj && make depend && make
cd i386/loader
make install

passwd
```
Set your root password, and exit the chroot environment:
```
exit
umount /${poolname}/dev
```

Create zpool.cache so ZFS knows after reboot that this pool is not from a foreign system:

mkdir /boot/zfs
zpool export ${poolname} && zpool import ${poolname}
cp -p /boot/zfs/zpool.cache /${poolname}/boot/zfs/zpool.cache

Unmount all ZFS filesystems and set post-install mount points.

zfs unmount -a
zfs set mountpoint=legacy ${poolname}/root
zfs set mountpoint=/usr ${poolname}/usr
zfs set mountpoint=/var ${poolname}/var
: etc...

Exit the fixit shell and the installer. You should boot right into a UFS-free system! Once you do boot the installed system, it may be advisable to recompile the operating system. Under FreeBSD 8.0 or 7.2 and earlier, do not run freebsd-update(8), as it may overwrite your boot loader with one that is not ZFS-aware!

Back

Jeff Blank <jfb@mr-happy.com>