ZFS-Only FreeBSD

There are plenty of guides on how to use ZFS without UFS on FreeBSD 8, but that doesn't mean I can't make one as well.

Much of this information comes from http://wiki.freebsd.org/ZFSOnRootWithZFSboot and http://blogs.freebsdish.org/lulf/2008/12/16/setting-up-a-zfs-only-system/. The specific goal of this guide is to provide a single-OS system (FreeBSD, obviously) with simple disk redundancy and no RAID hardware; this is accomplished with a "mirror" (RAID-1) zpool and also swapping to either a GEOM_MIRROR device or to a ZFS volume.

Side notes about swapping:

You will use the "fixit" command shell rather than sysinstall(8), so I've also provided steps for several setup tasks that sysinstall(8) would normally take care of.

The shell commands in this guide are formatted and variable-ized for easy copying/pasting. Remember that a colon at the beginning of a command line or immediately after a semicolon is considered by sh(1) to be a comment, so you can freely paste or pipe large command blocks into your fixit command shell without removing the inline comments.

  1. Boot the "livefs" CD or the DVD. If booting the CD, either have a second drive for the "disc1" CD or have disc1's contents available via NFS.
  2. Access "Fixit" mode and set the following variables at the shell prompt. They will be used through step 8. $zfspart will be set to a new value in step 4 if you swap to a GPT partition.
  3. Run the following commands:
    gpart create -s GPT $disk0
    gpart create -s GPT $disk1
    gpart add -s 128 -t freebsd-boot $disk0
    gpart add -s 128 -t freebsd-boot $disk1
  4. If you do not wish to swap to ZFS (zvol), create a swap partition. <swapsize> may be expressed as an integer suffixed with "M" or "G" (or, if you insist, no suffix, implying 512-byte blocks).
    zfspart=3 ; : Since this will place swap on ${disk0}p2, respecify where ZFS will go
    gpart add -s <swapsize> -t freebsd-swap $disk0
    gpart add -s <swapsize> -t freebsd-swap $disk1
    If you did not "load geom_mirror" at a loader prompt, load it now:
    kldload /mnt2/boot/kernel/geom_mirror.ko
    Create the mirror:
    gmirror label swap0 /dev/${disk0}p2 /dev/${disk1}p2
  5. If you did not "load zfs" at a loader prompt, load it now:
    kldload /mnt2/boot/kernel/opensolaris.ko
    kldload /mnt2/boot/kernel/zfs.ko
  6. Allocate space for ZFS. Add "-s" and a size before "$disk0" and "$disk1" to specify a size if you don't want to use the remainder of the disk.
    gpart add -t freebsd-zfs $disk0
    gpart add -t freebsd-zfs $disk1
  7. Make disks bootable:
    gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 $disk0
    gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 $disk1
  8. Create pool and root filesystem; make bootable. $poolname will be referenced throughout the remainder of the guide and should be exported to the environment for its use in subshells a couple times.
    export poolname=z0 ; : change as desired
    zpool create ${poolname} mirror /dev/${disk0}p${zfspart} /dev/${disk1}p${zfspart}
    zfs set mountpoint=none ${poolname}
    zfs create -o mountpoint=/${poolname} ${poolname}/root
    zpool set bootfs=${poolname}/root ${poolname}
    Others may use ${poolname} as the root filesystem, but using (for example) ${poolname}/root allows you to set default properties for descendents of ${poolname} without affecting the root filesystem's own properties.
  9. Create and mount other filesystems:
    : chmod before creating separate /tmp so it's right if /tmp isn't mounted.
    : install will take care of mode on ${poolname}/tmp filesystem.
    : repeat for /${poolname}/var/tmp if creating a separate filesystem for that.
    mkdir /${poolname}/tmp
    chmod 1777 /${poolname}/tmp
    : feel free to change/add/remove options, quotas, filesystems/mountpoints, etc.
    zfs create -o mountpoint=/${poolname}/tmp -o setuid=off -o compression=on ${poolname}/tmp
    zfs create -o mountpoint=/${poolname}/usr ${poolname}/usr
    zfs create -o mountpoint=/${poolname}/var ${poolname}/var
    zfs create -o mountpoint=/${poolname}/var/crash -o exec=off -o setuid=off -o compression=on -o quota=5120M ${poolname}/crash
    zfs create -o mountpoint=/${poolname}/usr/obj -o atime=off ${poolname}/obj
    zfs create -o mountpoint=/${poolname}/usr/ports -o atime=off -o compression=gzip-9 ${poolname}/ports
    zfs create -o mountpoint=/${poolname}/usr/src -o atime=off -o compression=on ${poolname}/src
    zfs create -o mountpoint=/${poolname}/usr/local ${poolname}/usrlocal
    zfs create -o mountpoint=/${poolname}/usr/local/homes ${poolname}/homes
    : etc...
  10. If you wish to swap to ZFS rather than a swap partition, create a ZFS swap volume. <swapsize> may be expressed as a number with a 'K', 'M', or 'G' suffix (8G, 2560M, etc.).
    zfs create -V <swapsize> ${poolname}/swap
    zfs set org.freebsd:swap=on ${poolname}/swap
    zfs set checksum=off ${poolname}/swap
  11. If using two CDs or CD+NFS instead of the DVD, mount it as appropriate. Use /mnt2/rescue/mount_cd9660 if you have a second CD-ROM drive installed, as /sbin/mount in the fixit environment will not find mount_cd9660. (You could also run (cd /;ln -s mnt2/rescue) if you want to use mount -t cd9660 ....)
  12. Start a new shell (sh -V or sh -E as desired) to make for an easy environment cleanup after the next steps.
  13. Enter cd /dist/<OSVersion> (DVD) or cd /mnt/<OSVersion> (two CDs or CD+NFS, "disc1" mounted under /mnt)
  14. Install the base and man pages. Not installing man pages here causes an error in step 16's "make install".
    export DESTDIR=/${poolname}
    for a in base *pages; do
      (cd $a && ./install.sh)
    install.sh will prompt you to overwrite any installation in /${poolname}. Make sure it warns you about the right directory—if not, something went wrong with the "export" line above.

    Continue installation with OS source and the GENERIC kernel.

    (cd src && ./install.sh all)
    (cd kernels && ./install.sh generic)
    cd /${poolname}/boot
    rmdir kernel
    mv GENERIC kernel
  15. Create OS configuration files:
    : /etc/src.conf
    echo 'LOADER_ZFS_SUPPORT=true' > /${poolname}/etc/src.conf
    : add other entries as desired. edit etc/make.conf if needed.
    : --------------------
    : /boot/loader.conf
    cat << EOH > /${poolname}/boot/loader.conf
    # skip/delete/comment out next line if swapping to zvol:
    # if "sysctl vm.kmem_size" is very different from size of physical RAM,
    # set it to size of physical RAM
    # add anything else you know you'll need/want
    : --------------------
    : /etc/fstab
    : uncomment swap entry if you followed step 4; zvol swap is activated
    : automatically and so does not need an fstab entry
    cat << EOH > /${poolname}/etc/fstab
    # Device                Mountpoint      FStype  Options         Dump    Pass#
    proc                    /proc           procfs  rw              0       0
    #/dev/mirror/swap0       none            swap    sw              0       0
    # uncomment after installing Linux ports
    #linprocfs               /compat/linux/proc linprocfs rw,late    0       0
    : --------------------
    : /etc/rc.conf
    cat << EOH > /${poolname}/etc/rc.conf
    # fill in values as desired
    # $dumpdev is now "NO" by default!
    # etc...
    : /etc/resolv.conf
    cat << EOH > /${poolname}/etc/resolv.conf
    nameserver x.y.z.w
    search example.com example.org
    cp /${poolname}/usr/share/zoneinfo/<YourTimeZone> /${poolname}/etc/localtime
    : uncomment next line if system clock is not set to UTC:
    : touch /etc/wall_cmos_clock
    Also see etc/dhclient.conf, etc/ntp.conf, etc/ssh/sshd_config, and others as needed.
  16. In a chrooted shell, build/install ZFS-aware loader and set root password. Be sure you created etc/src.conf as directed in the previous step.
    mount -t devfs devfs /${poolname}/dev
    chroot /${poolname}
    cd /usr/src/sys/boot
    make obj && make depend && make
    cd i386/loader
    make install
    Set your root password, and exit the chroot environment:
    umount /${poolname}/dev
  17. Create zpool.cache so ZFS knows after reboot that this pool is not from a foreign system:
    mkdir /boot/zfs
    zpool export ${poolname} && zpool import ${poolname}
    cp -p /boot/zfs/zpool.cache /${poolname}/boot/zfs/zpool.cache
  18. Unmount all ZFS filesystems and set post-install mount points.
    zfs unmount -a
    zfs set mountpoint=legacy ${poolname}/root
    zfs set mountpoint=/usr ${poolname}/usr
    zfs set mountpoint=/var ${poolname}/var
    : etc...
  19. Exit the fixit shell and the installer. You should boot right into a UFS-free system! Once you do boot the installed system, it may be advisable to recompile the operating system. Under FreeBSD 8.0 or 7.2 and earlier, do not run freebsd-update(8), as it may overwrite your boot loader with one that is not ZFS-aware!


[Valid XHTML 1.0 Strict]
Jeff Blank <jfb@mr-happy.com>