Reform-system-image pipeline job fails with "genext2fs: couldn't allocate a block (no free space)"

MNT Reform
1

I attached a gitlab-runner to my fork of reform-system-image, but I am struggling to get past a “couldn’t allocate” error.

I added a df to indicate the amount of available free space and also turned by inotify up significantly, but still errors. Any tricks I am not aware of which we use to get the job to complete in the upstream repo?

[opc@k3s01 ~]$ sudo sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 524288
[opc@k3s01 ~]$ sudo sysctl fs.inotify.max_user_instances
fs.inotify.max_user_instances = 8192

+ mmtarfilter '--path-exclude=*' --path-include=/boot '--path-include=/boot/*' --strip-components=2
copying from tar archive -
+ dd if=boot.img of=reform-rescue-system.img seek=1 bs=4194304
122+0 records in
122+0 records out
511705088 bytes (512 MB, 488 MiB) copied, 0.594908 s, 860 MB/s
+ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          99G  3.5G   90G   4% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/sdb         99G  3.5G   90G   4% /cache
+ rm boot.img
+ mmtarfilter '--path-exclude=/boot/*'
+ genext2fs --block-size 1024 --size-in-blocks 2097152 --bytes-per-inode 16384 --tarball - root.img
copying from tar archive -
genext2fs: couldn't allocate a block (no free space)
Traceback (most recent call last):
  File "/usr/bin/mmtarfilter", line 244, in main
    out_tar.addfile(member, file)
  File "/usr/lib/python3.11/tarfile.py", line 2030, in addfile
    copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
  File "/usr/lib/python3.11/tarfile.py", line 251, in copyfileobj
    dst.write(buf)
  File "/usr/lib/python3.11/tarfile.py", line 437, in write
    self.__write(s)
  File "/usr/lib/python3.11/tarfile.py", line 445, in __write
    self.fileobj.write(self.buf[:self.bufsize])
BrokenPipeError: [Errno 32] Broken pipe

full job log build (#1653) · Jobs · Adam Chasen / reform-system-image · GitLab

2

This has nothing to do with inotify. What made you think it did?

Your df -h call is not helpful. This error message is not about missing free space on the file system on the outside but missing free space on the filesystem that you are filling with the chroot contents.

Maybe the packages to be installed into the chroot increased in size or their number or both since the last time the pipeline ran (one month ago)?

I see that you tried changing the pipeline to bookworm. I have been successfully doing so for reform.debian.net. Here are my bookworm changes to reform-system-image:

3

I found some references to inotify in other genext2fs issues in my research, but could have been related to a different symptom in the thread.

Ah, thanks! I think that may be a threat I can pull on. The df was in the script before the genext2fs. I will increase ROOTSIZE and try again.