System image under btrfs

[serpent]

I’m not sure if this topic is fitting to Daily-driver tweaks for Reform, MNT Reform System Image V3 Beta, or other thread. Feel free to move if you find better place for it.

While discussing Community contributions, I mentioned that I was working on using btrfs as main filesystem on MNT Reform. Also @jackhill was trying to use it in cooperation with Guix. I managed to create image using ext4 for /boot and btrfs with subvolumes for / (root).

My script creates and mounts loop partitions (therefore requires root and cannot be run under ordinary user), prepares subvolumes, and then points mmdebstrap to it. I had to stop it from complaining about non-empty partition (option –skip=check/empty) and force it to activate usrmerge by not installing usrmerge package (option –hook-dir=/usr/share/mmdebstrap/hooks/merged-usr).

To allow for system start, /etc/fstab needs to point to appropriate subvolume; options set up by default image creation scripts (errors=remount-ro) are also not accepted by btrfs. Therefore root line in /etc/fstab should look like:

/dev/mmcblk1p2 / btrfs defaults,subvol=@root_a53 0 1

We also need to point kernel to appropriate subvolume, by adding rootflags=subvol=@root_a53 to option LINUX_KERNEL_CMDLINE_DEFAULTS in /etc/default/flash-kernel. Therefore appropriate line in calling mmdebstrap becomes:

--essential-hook='{ echo LINUX_KERNEL_CMDLINE=\"console=ttymxc0,115200 console=tty1\"; echo LINUX_KERNEL_CMDLINE_DEFAULTS=\"ro no_console_suspend cma=512M pci=nomsi rootflags=subvol=@root_a53\"; } > "$1"/etc/default/flash-kernel' \

Hope my description will help people who want to customise their image.

I also tried to set up encryption (directly during building of image, not afterwards as it’s done by scripts from reform-tools package), but haven’t succeeded yet.

[josch]

@serpent

My script creates and mounts loop partitions (therefore requires root and cannot be run under ordinary user),

I have never used btrfs but its man page states that it has the --rootdir option. Maybe you can try first creating a directory and then create the btrfs volume using --rootdir and thus avoid having to mount anything which needs superuser privileges?

@serpent

I had to stop it from complaining about non-empty partition (option –skip=check/empty)

If you just created the partition, why was it non-empty?

@serpent

and force it to activate usrmerge by not installing usrmerge package

What happens if you do not do that?

@serpent

Hope my description will help people who want to customise their image.

Have you codified all of this is a script? Maybe as a patch against the reform-system-image git? You could also file a merge request and maybe it would be useful to have an option that creates a system image using btrfs?

[serpent]

Sorry for long time to respond - got busy with Python/PostgreSQL packages before freeze
Full script is at the bottom: use at your own risk!

@josch

If you just created the partition, why was it non-empty?

As you can see in script (first half): partition is not empty, I created some subvolumes and directories there.

@josch

What happens if you do not do that?

After recent change in Debian’s init-system-helpers (which is Essential: yes), it depends on either usrmerge or usr-is-merged packages. When hook is not added to mmdebstrap, it tries to install usrmerge which gets run quite late in installation process, and gets confused when there are existing directories and subvolumes. So it fails, and entire mmdebstrap call ends with error. I guess this would not be a problem if I haven’t created all the subvolumes.

@josch

Maybe as a patch against the reform-system-image git? You could also file a merge request and maybe it would be useful to have an option that creates a system image using btrfs?

I’m not sure if it makes sense to propose patch or merge request to reform-system-image repository - at least not yet. Current script is work in progress (I’m not using system generated by it daily), and it is (more correctly - will be) customized for my needs. But I might be wrong here - if there is something you find interesting, feel free to use it.

At the same time, I learned a bit when I was working on it: what is needed to build minimal bootable image, what are the steps when booting… Your description from document boot options (#2) · Issues · Reform / reform-handbook · GitLab helped - thanks for it!

set -x
set -e

/usr/lib/apt/apt-helper download-file https://source.mnt.re/reform/reform-boundary-uboot/-/jobs/artifacts/v3/raw/flash.bin\?job\=build flash.bin

BOOTSIZE=488
ROOTSIZE=3072

dd if=/dev/zero of=boot.raw bs=1M count=$BOOTSIZE
dd if=/dev/zero of=root.raw bs=1M count=$ROOTSIZE

mkfs.ext4 boot.raw
mkfs.btrfs --checksum sha256 --label Debian -d single -m single root.raw

mount root.raw /media/usb -t btrfs -o loop
pushd /media/usb
btrfs subvolume create @root_a53
mkdir snapshots
pushd snapshots
mkdir home root
chmod 0700 root
chmod 1777 home
popd
pushd @root_a53
btrfs subvolume create boot
btrfs subvolume create etc
btrfs subvolume create usr
btrfs subvolume create var
chattr +C tmp
chmod 1777 tmp
pushd usr
btrfs subvolume create local
popd
popd
popd
umount /media/usb

mount root.raw /media/usb -t btrfs -o loop,subvol=@root_a53
mount boot.raw /media/usb/boot -t ext4 -o loop

PACKAGES_IMAGE="mmdebstrap klibc-utils arch-test cryptsetup"
PACKAGES_KERNEL="linux-image-arm64 linux-headers-arm64 flash-kernel cryptsetup-initramfs"
PACKAGES_DISK="u-boot-tools btrfs-progs gdisk"
PACKAGES_UTILS="isc-dhcp-client git zstd zutils mount binfmt-support reform-tools"

mmdebstrap \
	--architectures=arm64 \
	--components=main \
	--variant=important \
	--mode=auto \
	--verbose \
	--skip=check/empty \
	--include="$PACKAGES_IMAGE $PACKAGES_KERNEL $PACKAGES_DISK $PACKAGES_UTILS" \
	--hook-dir=/usr/share/mmdebstrap/hooks/merged-usr \
	${comment#setup apt preferences} \
	--setup-hook='{ echo "Package: *"; echo "Pin: release n=reform, l=reform"; echo "Pin-Priority: 990"; } > "$1"/etc/apt/preferences.d/reform.pref' \
	${comment#setup flash-kernel} \
	--essential-hook='mkdir -p "$1"/etc/flash-kernel/ubootenv.d' \
	--essential-hook='mkdir -p "$1"/etc/flash-kernel/preboot.d' \
	--essential-hook='echo "MNT Reform 2" > "$1"/etc/flash-kernel/machine' \
	--essential-hook='{ echo /dev/mmcblk1p2 / btrfs defaults,subvol=@root_a53 0 1; echo /dev/mmcblk1p1 /boot auto errors=remount-ro 0 1; } > "$1"/etc/fstab' \
	--essential-hook='{ echo LINUX_KERNEL_CMDLINE=\"console=ttymxc0,115200 console=tty1\"; echo LINUX_KERNEL_CMDLINE_DEFAULTS=\"ro no_console_suspend cma=512M pci=nomsi rootflags=subvol=@root_a53\"; } > "$1"/etc/default/flash-kernel' \
	${comment#select timezone} \
	--essential-hook='echo tzdata tzdata/Areas select Europe | chroot "$1" debconf-set-selections' \
	--essential-hook='echo tzdata tzdata/Zones/Europe select Berlin | chroot "$1" debconf-set-selections' \
	${comment#initramfs-tools} \
	--essential-hook='mkdir -p "$1"/etc/initramfs-tools' \
	--essential-hook='printf "pwm_imx27\nnwl-dsi\nti-sn65dsi86\nimx-dcss\npanel-edp\nmux-mmio\nmxsfb\nusbhid\nimx8mq-interconnect\n" > "$1"/etc/initramfs-tools/modules' \
	${comment#provide a copy of u-boot for (re)flashing} \
	--customize-hook='copy-in flash.bin /boot' \
	--customize-hook='chown -R root:root "$1"/boot/flash.bin' \
	${comment#populate /etc} \
	--customize-hook='echo reform > "$1"/etc/hostname' \
	--customize-hook='{ echo 127.0.0.1 localhost reform; echo ::1 localhost ip6-localhost ip6-loopback reform; echo ff02::1 ip6-allnodes; echo ff02::2 ip6-allrouters; } > "$1"/etc/hosts' \
	${comment#remove root password -- using `passwd -d root` produces unreproducible output} \
	--customize-hook='echo "root:root" | chroot "$1" chpasswd' \
	--customize-hook='chroot "$1" sed -i "s/^root:[^:]\+:/root::/" /etc/shadow' \
	unstable /media/usb http://deb.debian.org/debian "deb [trusted=yes] https://mntre.com/reform-debian-repo reform main"
#	--customize-hook='chroot "$1" reform-boot-config /dev/mapper/debian_crypt' \
#	--customize-hook='chroot "$1" update-initramfs -u' \

chattr +C /media/usb/var/tmp

umount /media/usb/boot
umount /media/usb

dd if=/dev/zero of=image.raw bs=1M count=4096
dd if=flash.bin of=image.raw conv=notrunc bs=1k seek=33
dd if=boot.raw of=image.raw conv=notrunc bs=1M seek=4
dd if=root.raw of=image.raw conv=notrunc bs=1M seek=$((BOOTSIZE+4))

sgdisk --zap-all image.raw

sgdisk --new=1:4M:+${BOOTSIZE}M image.raw
sgdisk --typecode=1:BC13C2FF-59E6-4262-A352-B275FD6F7172 image.raw
sgdisk --change-name=1:boot image.raw

sgdisk --new=2:0:0 image.raw
sgdisk --typecode=2:8305 image.raw
sgdisk --change-name=2:Debian image.raw

sgdisk --print image.raw

[josch]

@serpent

After recent change in Debian’s init-system-helpers (which is Essential: yes), it depends on either usrmerge or usr-is-merged packages. When hook is not added to mmdebstrap, it tries to install usrmerge which gets run quite late in installation process, and gets confused when there are existing directories and subvolumes. So it fails, and entire mmdebstrap call ends with error. I guess this would not be a problem if I haven’t created all the subvolumes.

That sounds like a bug in usrmerge then. Can you try first creating an unmerged system using --hook-dir=/usr/share/mmdebstrap/hooks/no-merged-usr and after that is done, remove /etc/unsupported-skip-usrmerge-conversion and then replace usr-is-merged with the usrmerge package which will attempt converting the system? If that fails, then usrmerge does not support the boot, etc, usr and var btrfs subvolumes which should definitely be fixed as this will also affect existing systems with such a setup of subvolumes.

Thank you for your script! I also learned new things.