Reform-tools for all distros

Hi distro maintainers,

in preparation of reform-tools landing in vanilla Debian after it passed the Debian NEW queue the new reform-tools release 1.64 will have its own Makefile which installs everything into $DESTDIR instead of this being done via Debian packaging scripts. This should make it easier for other distros to pick up the reform-tools package. I am thus pinging those who I think are maintaining the Reform platforms on other GNU/Linux distros. Please tell me if I forgot anybody:

Essentially, I’m looking for feedback on how I can make the reform-tools package more friendly for distros that are not Debian. You can see the current status of the next reform-tools release here:

There will still be some Debian-isms in the Makefile but to find them I need feedback from those who run make install on other distros or those who package it for other distros. :slight_smile:

So, how can I make your life packaging this easier?

11 Likes

I think the makefile implementation will be super easy for void. I’ll take a look as soon as I can and try to actually make a package for it.

2 Likes

The makefile approach will make it easy to adopt the version to an Arch (AUR) package.

Unfortunately the AUR package is quite outdated…i guess no one is currently running Arch on a Reform.

2 Likes

First, as always, thank you for your work @josch and for checking in with community members to enable different distributions and OSes to be easier to build and support. It’s really appreciated!

I’ve been doing a bit of work on Alpine & Chimera Linux for Pocket Reform as well as doing the Pocket work for NixOS recently, and was also involved in some of the later OpenBSD work, so I can comment from those three perspectives.

Firstly, I think that having Makefiles will be super helpful. I know this is a bit of a long shot, but packaging Makefiles which also work with bsd make would be very handy for portability (especially on BSD) when packaging reform-tools for other platforms. Most platforms also have gmake (GNU make) available so it’s not a show-stopper. Being able to specify DESTDIR= during install is extremely helpful as well, for all the use cases I mentioned as it is typically very easy to set this once to the correct output directory and know that the files will end up in the right place during package build. This avoids a lot of manual cp’ing and mv’ing.

Specifically for distros like NixOS (and GUIX as well, IIRC), they don’t conform to FHS for things like library, binary and header path locations, given the usage of a centralised store for packages/derivations. They do however do a good job of setting things like PATH, LD_LIBRARY_PATH and various XDG_ environment variables to help finding things like binaries, headers, libraries and data files when they are often under a /nix store directory. So, I would also suggest the where possible, assumptions around things being in /usr/lib, /usr/include etc. are checked - when these kinds of paths are hard-coded in scripts etc. it can mean manual patching is needed when building packages. One example of this would be the reform-handbook script, which opens the handbook (which is a really cool idea to package!) however the path to the handbook is hard-coded to /usr - that’s not where it will be on NixOS, for example. It might be better to find it in XDG_DATA_DIRS and then use that path as a more flexible approach which will accommodate things like NixOS or GUIX.

Other than that, it’s really helpful if the releases are in a git repository and tagged with semver version numbers. Most packaging tools deal really well with this pattern (especially Chimera Linux, and Alpine). It’s very helpful for NixOS too as it wants to build derivations for known/stable Git references to be able to ensure reproducibility. I’m fairly sure this is already your plan, though.

Thinking about udev rules, it would be also handy if the Makefile could install those, and ideally to an alternate path (not DESTDIR, necessarily) as some of the distributions mentioned use different udev implementations and the best directory to install these rules into may not be consistent between distributions or implementations (i.e. eudev vs udev vs systemd-udev). This one is very low priority, as I’m fairly sure that all these implementations at least have /etc/udev/rules.d in common so sticking to that might be an easy answer.

For building kernel modules, not every distribution uses DKMS to build things like reform2_lpc so it would be good if installing DKMS files could also be optional. If a distribution is not using DKMS I think it would be reasonable for a package to provide its own configuration for keeping the module updated with kernel changes/upgrades, but we should avoid cluttering packages with DKMS files which won’t be used by installing them by default in make install. Alternatively a variable to skip DKMS config install in make install could be handy.

initramfs-tools is another similar area that might require special handling between distributions. The initramfs-tools package is fairly Debian specific (but I do know of other distributions which use it), other distributions might be using things like dracut or home-spun tooling to build and update initramfs files, so it’s worth also considering not installing initramfs-tools configuration by default, or providing a way to avoid installing them with make install. Again I think it would be reasonable to expect packagers for other distros to carry the configuration for that distributions initramfs management tool in the package itself.

For systemd, I know reform-tools currently ships timer units and some services and it might also be worth providing an option to avoid installing those. Alpine & Chimera Linux for example don’t use systemd (OpenRC and dinit respectively). Gentoo also can optionally use OpenRC. So we could either not install those conditionally (via a flag or something again) or perhaps adding example sysvinit style files for other init systems to the project might be an option, and they could be installed with a separate make target. I think the most likely approach initially will be just disabling installing the systemd targets in install with a specific flag and then letting packagers for other distros also package their own init units.

For NetworkManager, this is similar to systemd in that not all distros use it, so if we could avoid installing the config file by default or make it able to be disabled, that would helpful. In many distributions it’s available as an option, but it’s also common to use things like iwd or connman instead, so additional handling in packages will need to take care of that for other distributions. I suppose it’s technically possible for Debian you could switch to connman or iwd or even using wpa_supplicant directly for WiFi management, so maybe as a long term thing the tools repo could carry different powersave configs (if needed, I’m not sure if this is a specific NetworkManager workaround we’re shipping) and the package could “do the right thing”.

This is everything I can think of right now so I help this is helpful! This is definitely not meant to be a list of things which need to be fixed before it’s possible to package reform-tools for other platforms, but it’s more a list of considerations which would definitely help make the packaging simpler and the packaging metadata smaller for other distros.

2 Likes

@digitalrayne Really good writeup. A lot of the modifications you outlined would be helpful on Void too. Void is using runit and dracut. I can pretty easily package this up so it doesn’t install the stuff I don’t want, but it would be super dope if I didn’t have to do that :slight_smile:

1 Like

If you find GNU-isms in that make-file (which might totally exist because I had the GNU make manual open while writing it) then please shout. The makefile doesn’t try to do anything fancy, so I’m sure we can make it work with all flavours of make.

The reform-handbook script will not be the first script which hardcodes data-paths in /usr. Is there a common shell script snippet which NixOS would use to patch other software with that I can just copypaste instead of writing my own buggy version?

In case of reform-handbook though, the solution might be a different one than XDG_DATA_DIRS. The handbook ships HTML and PDF documentation and hence is registered with doc-base and installs a control file into /usr/share/doc-base/. How do other documentation packages with a doc-base registration make themselves find-able in NixOS or GUIX?

The link to the reform-tools gitlab is in my first post. Here is the list of 63 git tags we have so far: Tags · Reform / MNT Reform Tools · GitLab

How is that handled for other packages in these distros that ship udev rules? What is their recommended advice to upstreams for what to do?

It currently is. The makefile makes no attempt to do anything with the contents of the lpc directory.

Would it? With that line of thinking, the initramfs-specific scripts should also be provided by the Debian packaging of reform-tools and not by reform-tools itself. But I cannot think of another package that does that. Typically upstreams would carry those files. In my mind, would it not make more sense for a dracut implementation to be shipped by reform-tools as well? This would also benefit Debian because not all Debian users use initramfs-tools, we also have dracut packaged and used.

There are also multiple init systems in Debian. I know people who use sysvinit in Debian. So if somebody can contribute a sysvinit or openrc implementation, then lets add it to reform-tools. But should the makefile allow not installing them? Suppose there are scripts for more init systems in the reform-tools project, I would certainly not disable them in the Debian packaging of reform-tools. The systemd sleep config which reform-tools installs is 267 bytes. Do packagers of distros without systemd go through the trouble of avoiding installing 267 bytes of data?

But if it’s an option nevertheless, then their packages do have to install the config file or otherwise it’s not an option anymore, no? Yes, connman and iwd are packaged in Debian as well. If somebody contributes a fitting config for those, then I’ll add it to reform-tools.

I’m looking forward to receiving patches from those who do end up packaging it elsewhere.

Thank you for your review!

That would just be the systemd specific stuff then? How does void linux packaging handle other upstream packages that ship systemd units. Are those always patched out? What is the common mechanism they use? reform-tools will not be the first package which attempts installing a file into /usr/lib/systemd.

I’ll take a look to see how other packages are handled. If I recall correctly some packages which are normally systemd services have patched versions of the packages with runit style services.

I’m still pretty new to packaging my own void binaries so I don’t want to misspeak.

Void has a tool for package creation that allows for me to pull from the official repo directly, then allows me to apply patches at build time, so it’s a little better than having my own forked repo.

It’s pretty trivial for me to make all of the packages in your repo, then just install them to whatever directory I want.

Lets avoid having to patch this. Before anybody has to carry patches, lets make reform-tools in the repo flexible enough such that you don’t need to have any patches.

1 Like

Just to be even clearer about my post above, where I say “it would be reasonable”, what I specifically mean is “if you didn’t want to do this, someone reasonable would not think you were being unreasonable”. If you are open to carrying sysvinit, dinit, runit files etc for services, alternate configs for different distros to manage rebuilding kernel modules, I’m sure that would be super helpful, and you would have no shortage of MRs from folks wanting to upstream those configuration files to reduce the size of package configuration.

Yes, because it’s not about the size of the file, it’s about cluttering the system with configuration files for system services which do not exist. Would you install dracut configuration files on a Debian system you were packaging for? Typically this would be handled by only grabbing the files you need or want from an upstream project, however if you would like to handle this process with a make install as suggested in the initial post, then having a way to avoid installing configuration files for services which do not exist I think would be helpful.

It’s kind of the other way around. Typically files will be installed into the Nix or GUIX store under a package-specific directory, containing the hash of the package. Derivations/packages depending on those packages containing documentation (such as the main system derivation you boot into) will typically link the directories in the derivation to a known location (for example, under /var/nix) and then the packages (i.e. for reading manpages, in the case of manpages) which access that documentation will be patched and/or configured to also search in those directories. Additionally, those directories will typically be included in the appropriate environment variables (the XDG_ ones I mentioned previously are common). You can use /usr/share but it’s likely these paths will need to be patched for systems like NixOS or GUIX to look in the appropriate place where the installed documentation will be linked.

What do the files in the “machines” directory do? I don’t recognize them and I’m definitely not using them in my current setup.

I am open to carrying these. Support for init systems other than systemd benefit a wide array of distros including Debian. Lets see about the “no shortage of MRs” :slight_smile:

Yes. In Debian, packages typically ship the maximum configuration unless it conflicts with other parts of the system. If there were dracut files in reform-tools, I would add them to the Debian package as well even though dracut is not the default. Same with the init-system files.

I think I’ll just do it with a INSTALL_SKIP option where items to be skipped can be listed separated by commas and then in the Makefile:

COMMA = ,
ifeq (,$(filter nosystemd,$(subst $(COMMA), ,$(INSTALL_SKIP))))
    # install systemd files
endif

Okay, if you find yourself patching things in NixOS or GUIX, please send me a patch instead so that you can avoid that patching. :slight_smile:

Those files contain configuration constants for all the possible configurations of the Reform and Pocket Reform and Reform Next. The utilities in bin and sbin use these constants instead of hardcoding the right thing over and over and over again.

The install skip section solves 90% of it for me. Do you want my dracut config that I’m using for void? It’s in my repo under srcpkgs/pocket-reform-base/files

Thank you. But could you, in the future, please supply a patch? I tried creating one from your git making a few assumptions: reform-tools 1.64 (!103) · Merge requests · Reform / MNT Reform Tools · GitLab

The advantages of you sending me a patch either by mail or by merge request are:

  • you send me something that you tested
  • i will not make mistakes when crating something myself (i can sometimes not test it, like in this case)
  • you choose the git commit author of your choice
  • we can discuss your submission and review it

In this case, that dracut config only seems to be applicable for pocket reform with imx8mplus. Can you make it such that installing the dracut config can work for all platforms? That’s what I had to figure out how to do with initramfs-tools. I have no idea how to do the same with dracut. Something that might work is that you take advantage of ${bootargs} as set by u-boot. I also wonder why you have to set the bootargs at all because those should be set correctly either by u-boot itself or by boot.scr or by extlinux.conf.

Yeah, I definitely can. My bad :confused:

Yeah, this imx8mp pocket reform is the only hardware I have. I’m really new to messing with dracut and uboot and was just working on getting my system to boot and be usable. I hope to get some other hardware in the near future, but I can do some best-guess implementations in the meantime.

I’ll take a look at some other void packages that I suspect require custom dracut additions and see how they handle it. Maybe I’ll find some clues.

Ha. I was actually messing around back and forth with that last night. I’m actually pretty sure now that I don’t have to set the bootargs in there.

I would like to clarify that I’m just some dude with a pocket reform who wanted to get Void linux running on it. I’m super stoked to share my work as much as possible so I can give back to open source for once. With that said, My goal is to get the pocket reform packages in the official void repos, I just don’t want to misrepresent myself as someone who knows what they are doing :joy:

Thanks for the help/info :slight_smile: I’m looking forward to getting more involved and learning more

1 Like

@josch I just verified that the boot args in the dracut script are all superfluous except the one identifying the root partition. I think there’s a way for me to make dracut base that info off of fstab, but I’ll need to do a little digging. I tried making an account on the self-hosted gitlab so I could make a PR, but it seems like my account is awaiting approval.

Is there a way I can attach a patch file to a reply in a thread?

Here’s the patch for my commit removing those args:

https://github.com/joe-albanese/void-packages/commit/bd88d4ab985558a25e277fdcb567c56c1013c2f3.patch

Good. They should be, because u-boot should provide the ones you need.

If you tell me your username, I can ping Lukas about it in the IRC channel.

Ah this is another thing, chatting about reform related topics is a bit easier via IRC. If you are an IRC user, please consider joining MNT Reform on libera.

I have not much of a clue how this javascript driven platform works. I think patches by mail or merge requests are easier than trying to shoehorn a patch submission system into discourse.

Thanks, I applied it: reform-tools 1.64 (!103) · Merge requests · Reform / MNT Reform Tools · GitLab