FreeBSD on a Hetzner Cloud VPS

July 26, 2026

Hetzner has no FreeBSD image and their ISO library stops at 15.0. You can still get any release on there, unattended, in about an hour.

The box runs AthletOS, a training app I am building. Predefined programs, session logging and tracking.

— I picked FreeBSD for ZFS boot environments and jails, so a bad deploy is a rollback instead of an incident.

This is the whole procedure, in order, with the things that went wrong and what fixed them. I used FreeBSD’s own ISO rather than a prebuilt image, so the checksum can be verified against the project’s manifest.

1. Create the server

A CX23 gives 2 vCPU and 4 GB, which is plenty. Firewall first, then the box.

1-create-server.sh

hcloud firewall create --name athletos
for p in 22 80 443; do
  hcloud firewall add-rule athletos --direction in --protocol tcp \
    --port $p --source-ips 0.0.0.0/0 --source-ips ::/0
done

hcloud server create --name athletos --type cx23 --location hel1 \
  --image debian-12 --ssh-key my-key --firewall athletos

The image is a throwaway. It only exists so the server boots at all.

2. Boot into rescue

2-rescue.sh

hcloud server enable-rescue athletos --ssh-key my-key
hcloud server reset athletos

ssh root@<ip>          # linux rescue, ~1.9GB overlay in RAM

3. Fetch and verify the ISO

Take bootonly, not disc1. The rescue root is a small RAM overlay and a 1.3 GB ISO plus its extraction plus the rebuild will not fit.

3-fetch-iso.sh

B=https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/15.1
I=FreeBSD-15.1-RELEASE-amd64-bootonly.iso

curl -fsSL -o CHECKSUM $B/CHECKSUM.SHA256-FreeBSD-15.1-RELEASE-amd64
curl -fsSL -o $I $B/$I

E=$(grep -E "\($I\)" CHECKSUM | sed 's/.*= //')
A=$(sha256sum $I | cut -d' ' -f1)
[ "$E" = "$A" ] && echo "verified" || echo "STOP"

4. Remaster the ISO

Three edits: a serial console so we can watch it, the prompt that would otherwise block forever, and the network the scripted install never configures for itself.

4-remaster.sh

apt-get install -y xorriso qemu-system-x86

mkdir -p isowork isomnt
mount -o loop,ro $I isomnt && cp -a isomnt/. isowork/ && umount isomnt
chmod -R u+w isowork

# append, never replace: the original carries vfs.root.mountfrom
cat >> isowork/boot/loader.conf <<'EOF'
console="comconsole"
comconsole_speed="115200"
boot_serial="YES"
vfs.root.mountfrom="cd9660:/dev/iso9660/FREEBSD_INSTALL"
EOF

# the serial path asks for a terminal type with an unconditional read
sed -i 's/^\([[:space:]]*\)read TERM$/\1:/' \
    isowork/usr/libexec/bsdinstall/startbsdinstall

# a scripted install never configures the network
sed -i 's|^mkdir /tmp/bsdinstall_etc$|&\nifconfig vtnet0 inet 10.0.2.15/24 up\nroute add default 10.0.2.2\necho "nameserver 10.0.2.3" > /tmp/bsdinstall_etc/resolv.conf|' \
    isowork/etc/rc.local

What went wrong

I replaced loader.conf instead of appending, which threw away vfs.root.mountfrom and left it at a mountroot prompt.

Then it hung at Console type [vt100]: because that read runs before the script is ever looked for.

Then distfetch died on DNS, because the interactive path exports BSDINSTALL_CONFIGCURRENT and the scripted path does not.

5. Write the unattended install script

isowork/etc/installerconfig

PARTITIONS=DEFAULT
DISTRIBUTIONS="kernel.txz base.txz"
export ZFSBOOT_VDEV_TYPE=stripe
export ZFSBOOT_DISKS=vtbd0
export ZFSBOOT_SWAP_SIZE=2g
export ZFSBOOT_CONFIRM_LAYOUT=0
export nonInteractive=YES
export BSDINSTALL_DISTSITE=http://10.0.2.2:8000

#!/bin/sh
sysrc hostname="athletos"
sysrc ifconfig_vtnet0="DHCP"
sysrc sshd_enable="YES"
sysrc zfs_enable="YES"

# ARC takes half of RAM by default. On a 4GB box that is 2GB it does not
# need, and it turns up later looking like a database problem.
echo 'vfs.zfs.arc_max="512M"' >> /boot/loader.conf

mkdir -p /root/.ssh && chmod 700 /root/.ssh
echo 'ssh-ed25519 AAAA... you@host' > /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

10.0.2.2 is QEMU’s host address. Serving the distribution sets from rescue skips DNS entirely and is quicker than fetching them twice.

5-serve-dist.sh

mkdir -p dist && cd dist
B=https://download.freebsd.org/releases/amd64/15.1-RELEASE
for f in MANIFEST base.txz kernel.txz; do curl -fsSL -O $B/$f; done

while read -r name sha rest; do
  [ -f "$name" ] || continue
  [ "$(sha256sum "$name" | cut -d' ' -f1)" = "$sha" ] && echo "OK $name"
done < MANIFEST

setsid nohup python3 -m http.server 8000 >/dev/null 2>&1 &

Tip: leave that server running. The installer fetches from it in step 7, not now.

6. Build the ISO and wipe the disk

Delete the source ISO before building, or you run out of space in the overlay.

6-build-and-wipe.sh

rm -f $I
xorriso -as mkisofs -r -V FREEBSD_INSTALL -b boot/cdboot -no-emul-boot \
        -o athletos-install.iso isowork/
rm -rf isowork

# zfs labels sit at both ends of the device, so wipefs alone is not enough
SECTORS=$(blockdev --getsz /dev/sda)
MIB=$(( SECTORS / 2048 ))
wipefs -a /dev/sda
dd if=/dev/zero of=/dev/sda bs=1M count=64 conv=fsync
dd if=/dev/zero of=/dev/sda bs=1M seek=$(( MIB - 64 )) count=64 conv=fsync

What went wrong

An earlier attempt got far enough to create a zroot pool before failing.

The next run found it and stopped to ask for a different pool name, which is another dialog nobody is there to answer. Wipe both ends between attempts.

7. Run the installer under QEMU

Hetzner Cloud has no nested virtualisation, so there is no /dev/kvm and this runs under TCG emulation. It takes roughly half an hour. Nothing to do but watch the log.

7-install.sh

qemu-system-x86_64 -accel tcg,thread=multi -m 2048 -smp 2 \
  -drive file=/dev/sda,format=raw,if=virtio,cache=writeback \
  -cdrom athletos-install.iso -boot d \
  -netdev user,id=n0 -device virtio-net-pci,netdev=n0 \
  -display none -monitor none -serial file:/root/install.log -daemonize

grep -aoE "Running installation step: [a-z]+" /root/install.log | uniq
# zfsboot mount distfetch checksum distextract bootconfig config
# entropy umount script

Kill QEMU once the script step finishes, or it reboots straight back into the installer.

8. Boot it, and fix fstab

8-boot.sh

hcloud server disable-rescue athletos
hcloud server reset athletos

ssh root@<ip>

# the installer ran under qemu virtio (vtbd0), hetzner presents da0.
# the pool is fine because zfs scans devices. swap is not.
printf '/dev/gpt/swap0\tnone\tswap\tsw\t0\t0\n' > /etc/fstab
swapon -a

What went wrong

The generated fstab pointed at /dev/vtbd0p2, which does not exist on the real machine.

It booted with no swap and said nothing about it. Refer to it by GPT label and the device name stops mattering.

Result

athletos

FreeBSD athletos 15.1-RELEASE ... GENERIC amd64

zroot            658M / 34.2G      all pools are healthy
/dev/gpt/swap0   2.0G                     0% used
vfs.zfs.arc_max  536870912                (512 MiB)

1  gptboot0   512 KiB   A501  freebsd-boot
2  swap0      2.0 GiB   A502  freebsd-swap
3  zfs0      36.1 GiB   A504  freebsd-zfs

Result

15.1 on root-on-ZFS, a release the ISO library does not carry, from media you can verify.

Under load

I measured this from a separate Hetzner VPS in the same datacentre, hel1, against a control: an identical CX23 running Ubuntu 24.04, the same Caddy 2.11.4 serving byte-identical files, with an ECDSA P-256 certificate to match. vegeta, a fresh TCP and TLS connection per request, which is what a crawler does and what a browser never does.

9-load.sh

vegeta attack -targets targets.txt -rate 500 -duration 10s \
       -keepalive=false -timeout 20s | vegeta report

# freebsd   100/s  4.9ms   250/s  4.7ms   500/s  4.0ms   mean
# ubuntu    100/s  5.7ms   250/s  5.0ms   500/s  5.2ms   mean

At 100, 250 and 500 fresh connections a second both boxes delivered the full offered rate with 100% success, and the FreeBSD box was the faster of the two at every step.

The crypto is not the constraint either.

openssl speed ecdsap256

# sign/s
#  37920   this box (epyc, one core)
#  32614   ubuntu control (xeon, one core)

The EPYC signs 16% faster than the control’s Xeon, which is worth stating because it means the hardware favours the FreeBSD box rather than the other way round. Against 500 connections a second, 37,920 signatures is roughly 75x headroom. Whatever the ceiling is, it is not asymmetric crypto.

Two things I did not establish

Above 500/s the measurements were not reproducible. One clean run at 1000/s had FreeBSD at 497/s and the Ubuntu control at 999/s. A later run gave 179/s at 1000/s, and then 566/s at 1500/s — more offered load producing more throughput, which is not a property a server can have. It is the single load generator accumulating TIME_WAIT between runs. A real answer needs several generators.

An earlier attempt measured this from a Windows laptop over the public internet and got 50-100 connections a second. That was low by five to ten times, and the cause was ephemeral-port exhaustion on the client. The first measurement was wrong, and it was wrong in a way that produced a confident and completely incorrect explanation.

One hypothesis died on the way. FreeBSD’s kern.ipc.soacceptqueue is 128 and Ubuntu’s net.core.somaxconn is 4096, so the listen backlog differs by 32x, which looked like the obvious cause and survived the arithmetic. I set the control to 128, restarted Caddy so the listener picked the new value up, and it still delivered 999.9/s. Refuted.

That leaves pf, which redirects 443 to Caddy’s high port, as the untested suspect. Untested because that high port is not reachable from outside, so there was no way to bypass the redirect and compare against it.

The same steps work for any release. Change the version in step 3 and the rest follows.

Every command above ran against a real CX23 in hel1.