How do you beta? With Jasper Wiegratz

Red Hat Enterprise Linux (RHEL) 10 has been released. It's designed to support businesses through challenges including limited resources, cloud  and AI strategies, and post-quantum security threats. RHEL 10 provides a solid foundation for modern organizations, but have you ever wondered what it takes to reach a general release? RHEL is developed openly, using open source projects and principles with every iteration. Its quality assurance is boosted by the active participation and input from industry professionals in the form of beta testing.

I sat down with Red Hat Accelerator Jasper Wiegratz to talk about what it means to beta test an operating system, what he’s learned doing it, and what he’s looking forward to in Red Hat Enterprise Linux (RHEL). Jasper is a community project maintainer and the Red Hat Certified Professional of the Year in 2023. 

Matt Micene: Tell us about yourself, Jasper. Who are you and what do you do?

Jasper Wiegratz: I’m Jasper Wiegratz from SVA System Vertrieb Alexander GmbH in Germany. I'm a solution architect, mostly focused on Red Hat technologies. I'm a consultant to the customers of SVA. I create architectures for the hybrid cloud, mostly with Red Hat OpenShift. I do have a long reaching background in containers, since Docker in 2015, but have been doing Red Hat OpenShift for the last five or six years. I started with Red Hat OpenShift 4 and I have customers from many different ecosystems. I do a lot in finance and retail, as well.

MM: How would you describe the Linux environment you currently work in?

JW: As an architect, I'm involved in a lot of different projects, but I'm doing a lot of work on Red Hat OpenShift Virtualization projects at the moment. It doesn't touch RHEL directly but we see RHEL as a workload. That will impact my usage of RHEL.

We do have a lot of VMware customers, or also Red Hat Virtualization customers, who are willing to migrate to Red Hat OpenShift Virtualization, some of them with a dual vendor strategy. And to be honest, I'm less involved in RHEL projects at the moment because we have a few real experts on RHEL ecosystems at SVA.

I'm involved with RHEL 10 for more personal reasons because I've been contributing to the upstream of RHEL 10, for Podman to be precise. 

MM: What was your first experience working with RHEL?

JW: Back in my apprenticeship in 2015. I was at an international full stack managed service provider. Most of the systems we had were based on RHEL. We had the full Red Hat ecosystem, including Red Hat Satellite 6 based on upstream Foreman.

Within the apprenticeship, I set up Identity Management. It wasn't just x86, we had PowerPC64 and systems across a range of customers. So that was my first experience with RHEL. Personally, you could say I came from the typical Debian/Ubuntu usage back then. I found the Ubuntu CDs in printed magazines, so I started with that, but I got to learn what Enterprise Linux really is during my apprenticeship. I found it to be predominant in the professional context for obvious reasons, because you would have a long lifetime of each RHEL release and customers require stability and support offerings. Eventually we upgraded everything to RHEL 8 and we kept it there for a few years. You didn't have to run for a new iteration very often.

I was very involved with containers at that time, too. I was learning professional IT during my apprenticeship but mostly with containers, and I was extremely interested when I saw the Project Atomic of Red Hat. I think it has its root in the CoreOS acquisition but I was well acquainted.with Docker, and then I saw there are new tools coming up like Podman, Skopeo and Buildah. This was very new at that time. I quickly realized there was a lot of innovation in the container sphere coming from Red Hat and RHEL, and that really got my attention. I chose to go the Podman route from then on, and that eventually got me into Red Hat OpenShift.

I remember it was late in the work day in late 2016, and I found the Project Atomic website and this greenish page where it said we have all these things like Docker, but we split it up into multiple tools. We've improved it. It was exciting. I knew at the time I wouldn't get to use it just now, but I saw that this was the future. It's being developed, and it's going to get better.

MM: What's your main motivation for beta testing an operating system? Personal curiosity, work related, something in between?

JW: I do have a little personal story here, because I did another master’s degree that I finished last year. In 2022, I had a project at university where we needed to build a network security appliance based on OpenWRT Linux with every component written in Rust. We were 12 masters students in computer science, and we had this goal to work on it for one year. And my problem to solve was to write software in Rust that would orchestrate the firewall of OpenWRT Linux such that it would implement micro segmentation in a local network.

And what I found is that you would use nftables on a modern Linux, for example in OpenWRT. The same is true for RHEL, since RHEL 8 when nftables became the default. I found myself with the challenge: How do I communicate with nftables, how do I orchestrate nftables from Rust? At that time, there wasn't a complete solution. Because I needed to work with the complete set of firewall rules, all possible combinations of firewall rules you could implement with nftables, and there was no existing SDK for Rust, so I had to create this interface. It took a few months, but I wrote a complete library called nftables-rs. I made it look nice the day before we got the grades from the professor because I wanted to have a good grade.

And that's probably the only reason why I created a README, put it on GitHub, just to tell my professor I created something for the benefit of the whole world. But that was it, and it was sitting there in public, on GitHub. A complete SDK for nftables in Rust.

A few pull requests came in from hackers, basically people at conferences building large LED pixel walls with dedicated network appliances that would accept pixels as UDP packets. Things like that happened, and then a few more months passed. I think it was the beginning of 2024, I received two or three pull requests from Red Hat and the Podman team. I then saw that the netavark project, which is the network integration stack in Podman, started to implement nftables support mostly by interfacing with my library. I kept track of that!

It actually really stressed me, because this was just a personal project and here Red Hat was downstream of what I created! I didn't see that coming. So I’ve put a lot of work into this project since then, because in addition to Red Hat, a lot of other projects have started using it. And of course, Fedora 41 uses nftables as the default for Podman, which means all the users of Fedora 41 using Podman will go through my code. I knew it had some issues, like if you had a certain preconfigured firewall, my code could misbehave at some point. Over the past four months, I've worked on making sure it's just less risky to use my code. Fedora 41, and then of course RHEL 10, have these changes. I think it's not even on the release notes because it's really under the hood. But in fact, also here in RHEL 10, Podman defaults to the nftables netavark backend. 

And that's why I'm really excited for RHEL 10. With it, I have an operating system where I think, "some of the inner workings of this are possible because I wanted to get a good grade back in uni."

MM: How do you go about thinking about and setting up infrastructure for beta testing something like an operating system?

JW: I was pleasantly surprised that I could just pick the RHEL 10 Beta in Red Hat OpenShift Virtualization. Initially, I wanted to get the ISO and then upload it, and then do the installation, but instead there was just this option for RHEL 10 Beta. It took exactly 40 seconds after I saw this until I had a login shell from RHEL 10. So that was extremely easy. I wish it would always be that easy if you want to beta test something, that it's just that accessible.

And to be honest, if you install RHEL, it's supposed to be a very simple process. It's a quick install, and then you have a running system. It's nothing fancy, it's supposed to just work and so it does. 

I'm a release note addict. So that's what I do first. Skim through the release notes and think about what's relevant for me. What sounds interesting? So when I beta tested RHEL 10, I thought about the use cases that interested me, and are probably interesting for my organization and customers. At this time professionally, I'm not that involved with RHEL now beyond supporting it as a workload in Red Hat OpenShift Virtualization. Personally, I was mostly interested in container integration. As I said, the networking layer of Podman changed a lot. There's the nftables back end, which I mentioned before. There's also the pasta network integration now, which I also really need in production.

I wanted to run a few tests on RHEL 10 with Podman to see how networking is configured, how it works, and compare that specifically with RHEL 8 and RHEL 9 (because even though there are major releases in small increments, you still see differences).

MM: Is beta testing an OS different from other software you've beta tested before?

JW: An operating system like RHEL is a very big ship, so to speak. I cannot just change a line of code and then re-build the whole thing and see how it changes in behavior. 

Testing one version of a containerized application against another, that would be a few seconds. With a full RHEL operating system, for many people who go the manual installation way, it's more like 30 minutes of going through the installer. But on Red Hat OpenShift Virtualization, it's just 40 seconds, which is a big plus. To be honest, if I can iterate fast, then it doesn't feel much different from a containerized application. It takes 40 seconds for me to spin up one VM of RHEL 10, and I can do another test with a different networking or storage setup.

Also, in the base set of applications, you don't just immediately see everything. I also look at the new GNOME desktop to see how it's configured and what it's offering me now. It's very reasonable, I would say. There's no bloat in it, which is also good.

MM: For the readers, what advice do you have? What's something you wished you knew before you started testing?

JW: My way of doing this is to read the release notes to figure out what I should look at. Then I make a plan, bring up a use case and decide how to test it.

This time around, I created a Podman container on RHEL 8, RHEL 9 and on the RHEL 10 Beta so I could compare how it's set up internally. There are big changes in those operating systems, and for the use cases I have (production servers running RHEL), it really matters. For example, I have one Internet service server, a typical container host with applications on it, and it could not see the source IPs because I was using rootless Podman on RHEL 9. It'll be different on RHEL 10, for the betterment of security. 

The other advice I have is to be able to iterate fast. I think for it to be fun, you shouldn't be doing the exhausting work. Red Hat OpenShift Virtualization can create an instance of RHEL 10 that's completely preconfigured with my own SSH key and it takes maybe 40 seconds. That's great. I did some testing on that, but I also wanted to see the installation, to see whether the Anaconda setup changed. So I also did it the manual way, just to have a basis for comparison. But I didn't need to do it more than once.

MM: Was there anything that surprised you about the beta?

JW: I'd expected to see some of the changes already in Fedora manifest in RHEL 10. I was really excited about these container-specific changes in RHEL 10. To my surprise, many of the parts I was really interested in were backported to RHEL 9. So RHEL 9 apparently moved to Podman 5 last week [with RHEL 9.5]. What a treat it was not to have to wait for RHEL 10 for that!

I'm excited because RHEL is always a compromise of stability and getting the latest features. To me, RHEL is on the conservative side, but even so the iteration cycle from RHEL 8 to 9 and to 10 has gotten faster. This is good for the customer because, especially with my use cases around containers, there's a lot of movement in Podman and container technology. If you're feeling stuck on an old RHEL release, then you don't have to worry because there's a new RHEL release very soon, and there's a lot to look forward to.

Thanks to Jasper for sharing with us his approach to a RHEL beta. It's thanks to the hard work and scrutiny of community members like Jasper, and many others, that has helped improve and influence RHEL releases.