800+ IT
News
als RSS Feed abonnieren📚 CentOS Blog: CentOS Stream is Continuous Delivery
💡 Newskategorie: Unix Server
🔗 Quelle: blog.centos.org
Continuous Delivery 101: Do the hard things continuously,
so that they become easy.
From the outside, it may appear that the way we build RHEL (and thus the CentOS Linux content) hasn’t changed in a decade. But beneath the covers, we’re pulling off a monumental transformation of how we develop RHEL without impacting our customers.
I've told this story at various conferences, but the announcements about CentOS Linux 8 and CentOS Stream have provided the impetus to tell the story here.
Three years ago, several of us working in RHEL Engineering had an idea: what if we applied modern development practices to RHEL such as continuous integration, continuous delivery, predictable release cadence … paired with open source development practices like release early release often, pull requests, forking, and code review.
Obvious, no? … No.
The Linux distribution is the grand challenge of
continuous integration and delivery.
What drew me into open source has always been this integration challenge. There is an infinite sea of uncoordinated projects. It really is an amazing example of evolution. If you squint your eyes like so, you can just about see the strange organisms, the mutations, the microcosms, and the natural selection all happening before you.
Over the last 20 years, I’ve contributed to over a hundred different projects. My contributions focused on making projects function seamlessly together so the user would have a coherent experience.
The Cockpit project is the most visible example of this. We connected about 95 Linux APIs and components, each developed separately, and released on different schedules in over 10 different distros, into a coherent user experience, delivering stable releases every other week for six years and counting.
If Linux is the grand challenge of continuous integration and delivery, then I saw RHEL as the unparalleled absolute: take ten thousand uncoordinated projects, thousands of contributors, add additional structure (like kABI) and additional guarantees (like 10 + 3 years of hardware enablement), integrate them constantly, and deliver a stable release every single day.
With dreamy (well, watery) eyes, we called such an effort “Always Ready RHEL”.
The effort started painstakingly onboarding the thousands of packages into continuous integration. It shocked many that we didn’t already have CI for all RHEL components back in 2017. But if it was easy, it would have happened much earlier.
Today, any update that goes into RHEL has to pass continuous integration gating before landing in our nightly compose, which runs automated tests for that component. Then, each change needs to be explicitly verified to a RHEL quality (mostly by Quality Engineering) before it can land in the RHEL nightly builds.
The “Always Ready RHEL” effort now continues with continuous delivery, which you now know as CentOS Stream: the RHEL nightly composes are already delivered in CentOS Stream. The whole point of continuous delivery is to make each release as stable as the one before. We’re delivering daily.
Are we done? … No.
To the untrained eye, CentOS Stream is
already as stable as RHEL.
But the challenge here is unparalleled, and RHEL engineers carry awareness of that. The way the different teams do their work integrating RHEL is as diverse as the upstream communities themselves. Yet because so many people are iterating together toward different aspects of this goal, I’m convinced we can make Continuous Delivery a reality..
Here’s how the flow of delivery looks for 8 and 9:
You can see the Fedora releases on the left. And the chart illustrates how CentOS Stream is synonymous with the work on RHEL X.Y releases. Technically speaking, CentOS Stream and RHEL updates are two binary packages built from the same source. An update will be published to CentOS Stream if and only if it is published to the RHEL nightly builds. Thus the RHEL nightly builds are the CentOS Stream updates you get. Once we branch from Fedora, our development gets into a stride where each change is integrated cleanly on top of everything that went before. An update is pushed to CentOS Stream if and only if it is published to the unreleased minor update of RHEL. RHEL customers later see each of these as a RHEL Errata update.
Each of these changes, whether bug fixes or features, is tested via automated tests and verified by Quality Engineering processes before landing in CentOS Stream.
The only work not directly and immediately visible in Stream is the work we do on the already-released RHEL minor versions themselves (indicated as “errata” in the diagram). Often this work is done under NDA, are embargoed, or are backports of changes already in CentOS Stream.
CentOS Stream intends to be as stable as RHEL,
It’s fundamental to continuous delivery.
But hey, even the RHEL-released product is not completely stable. Back in July, a RHEL (andCentOS) fix for the “boot hole” vulnerabilities ended up being far worse than the CVE itself: it caused many systems not to boot. Oh, man.
As a result, we’re not only investing time in reworking upstream components, but also adapting our process to ensure that this cannot happen again. Rinse, repeat.
While I wasn’t part of the decision to EOL CentOS Linux 8, I’m committed to putting my effort toward pulling off CentOS Stream. Doubly so, because it makes RHEL be Open Source: Where we can work together with an entire ecosystem on this exciting continuous integration and delivery challenge.
Open sourcing a product is hard, yet we’ve made amazing progress. So far we’ve aligned the RHEL development process with Fedora, placed all the actual sources of RHEL in one readable place, enabled contributors to open a pull request against any part of RHEL, released early and often ...
And this is just the start. There are hundreds of people working toward this CentOS Stream change, all while not missing a beat delivering the RHEL releases you’ve come to expect.
CentOS Stream is the stable and reliable
continuous delivery of RHEL