Building a custom OpenWrt image

K was poking around at Raspberry Pi stuff the other day, and the Pi3 she was using was pretty slow (and unable to run what the book was showing), so I replaced my Pi4 router with an AAEON UP-CHT01 machine that I had on hand. The specs of the CHT01 are roughly:

  • 1.4 GHz Atom SOC
  • 1.4 GB of DDR3
  • 16 GB eMMC storage
  • 4x USB2
  • 1x USB3 (with a funky OTG Micro-B connector that needs an adapter)
  • 1x Gigabit NIC (Realtek)
  • Cherry Trail chipset

It’s a functional machine, and certainly has more CPU grunt than the Raspberry Pi 4, though I’m not sure this application calls for it – the Pi4 seemed pretty happy as a router that moves a maximum of 500 Mbit/s down, and 50 Mbit/s up. However, it freed up the Pi4 for K to use, and it’s turned out to be a fun learning exercise in terms of building custom OpenWrt images. The RAM size is a bit odd, but it’s not like I need to use this machine as a VM host for a small lab.

As with the Pi4, the second network card is attached via USB3 (as USB2 caps the network to about 340 Mbit/s). The on-board NIC uses the r8169 driver, while the USB NIC uses the rtl8152 driver. An ancient USB Wi-Fi dongle rounds out the collection of network cards, driven from the rt2800 driver.

Image Builder

The OpenWrt project provides pre-built images that can be downloaded and used. The downside is that due to image size constraints (because OpenWrt is designed to work on machines with 16 MB of storage), all of the network drivers that you might want are not present in the image, plus the file systems are sized to fit lots of small devices. An alternative approach is to use the Image Builder tooling provided by the project, and set up a customized disk image that has the desired network drivers, a custom root file system size, as well as any customized configuration options. This is basically what wulfy’s image was (that I used to use) – though done with a full build from scratch, not a build from precompiled packages.

# Override the OpenWrt default root filesystem size; it's too darn small and upsets LXC
sed -i -e 's/CONFIG_TARGET_ROOTFS_PARTSIZE=.*/CONFIG_TARGET_ROOTFS_PARTSIZE=256/' .config

# Build the custom image, with some uci-defaults files (autoconfigures the OS), and some packages I always
# want available (and removed in the case of the last line).
make image\
 FILES="files"\
 PACKAGES="block-mount ddns-scripts-cloudflare hostapd fdisk f2fs-tools kmod-fs-f2fs\
 kmod-r8169 kmod-usb-net-rtl8152 kmod-rt2800-lib kmod-rt2800-usb kmod-rt2x00-lib kmod-rt2x00-usb kmod-wireguard kmod-veth\
 luci luci-app-wireguard luci-app-ddns luci-app-lxc luci-theme-material lxc-auto\
 lxc-start lxc-destroy lxc-attach lxc-autostart lxc-checkconfig lxc-hooks lxc-config lxc-configs\
 lxc-console lxc-device lxc-monitor lxc-stop lxc-cgroup lxc-checkconfig lxc-common lxc lxc-templates\
 r8169-firmware resize2fs usbutils wireguard-tools wget ca-certificates gnupg gnupg-utils\
 -kmod-e1000 -kmod-tg3 -kmod-bnx2 -kmod-amazon-ena -kmod-amd-xgbe -kmod-forcedeth -kmod-igb -kmod-ixgbe -kmod-e1000e"

The FILES=”files” reference is a pointer to where the uci-defaults files can be found; these are simple /bin/sh scripts that call the OpenWrt uci tool to write the persistent configuration files. Once they’ve been read and applied, they’re deleted from the running image.

As an example, here’s a snippet from the network configuration defaults – simple, verbose, and functional.

# VLAN 10 for Eir
VLAN=$(uci add network device)
VLAN_NAME='eir.vlan10'
uci set network.$VLAN.type='8021q'
uci set network.$VLAN.ifname='eth1'
uci set network.$VLAN.vid='10'
uci set network.$VLAN.name=$VLAN_NAME

WAN6=$(uci add network interface)
uci set network.$WAN6.type='interface'
uci set network.$WAN6.proto='dhcpv6'
uci set network.$WAN6.device=$VLAN_NAME
uci set network.$WAN6.reqaddress='try'
uci set network.$WAN6.reqprefix='auto'
uci set network.$WAN6.clientid='beefdead'
uci rename network.$WAN6="wan6"
uci add_list firewall.wan.network="wan6"

For testing, QEMU is pretty helpful; it can emulate x86_64 trivially, providing two network cards, and thus I have a nice little test environment where I can make sure changes I want to make to the production router actually work.

cd bin/targets/x86/64

# Uncompress the packaged image
gunzip openwrt-22.03.0-rc1-x86-64-generic-ext4-combined-efi.img.gz

# Boot qemu; provide two NICs to the VM since it's a router. User NIC driver means no sudo needed.
qemu-system-x86_64 -m 512 -hda ./openwrt-22.03.0-rc1-x86-64-generic-ext4-combined-efi.img -nic user -nic user -enable-kvm

Deployment

With the CHT01, I have two options

  • Write the OpenWrt image to a USB stick (which creates partitions), boot from the stick directly
  • Copy the image to a USB stick, boot from a Ventoy-enabled stick, start Finnix, write the OpenWrt image to the eMMC storage

I initially tried the second option, because it seemed sensible to use the onboard storage to run the router. However, it’s a right pain for doing upgrades, because the router has to be removed from the small rack-mount cabinet that it’s in, brought upstairs, HDMI cables have to be fiddled with, and I have to do a lot of typing.

The first option is much nicer – build the image, dd it to a small USB stick, put stick in computer, (re)boot computer. By the time I’ve walked upstairs, the router is back on the network. It’s a USB2 drive, but the performance of USB2 really doesn’t matter here; it’s fast enough, and it’s better than booting from a floppy disk.

Rough edges

A caveat here is that the QEMU system appears to provide an emulated e1000 by default, so the -kmod-e1000 argument for PACKAGES is not sensible. Alas the board I’m using uses kernel modules that QEMU can’t emulate, so my test image has slight discrepancies from my production image. However, it’s still valid enough to allow me to boot the image and make sure it gets configured correctly for things like Wireguard, DHCP etcetera.

Another annoyance is that the Image Builder writes out disk images that fdisk and the kernel both complain about. It’s easily fixed by opening the image with fdisk, using w to re-write the partition table, and then write the image to the USB stick, but it’s still annoying.

My previous LXC installation of UniFi can’t be ported across from the Pi4 installation to the x86_64 installation – binary format incompatibility. I might be able to boot the old OS via QEMU though, using the right emulation layer. Then I can export the configuration and reload it in a x86_64 LXC container. Maybe.