Migration between Linux distributions within the same family¶

There may be circumstances where it's desirable to migrate from one Linux distribution to another. An example would be systems that are currently deployed with CentOS Stream 8, wishing to migrate to AlmaLinux 8 or Red Hat Enterprise Linux 8.

Migration between families of the same version (eg: '8' or '9') is possible and mostly entails changing repositories and upgrading (or downgrading) packages.

Please be aware that migration may entail package downgrades, which could mean that some packages will be installed with older versions and may require some configuration changes.

Please find below the instructions for your particular setup:

Migration for unmanaged, standalone or locmap-managed machines¶

As root, run the following commands on the machine to migrate:

The first step is to install the CERN version of the 'release' RPM for the new distribution. The CERN version ensures that your system will utilise the CERN repository infrastructure.

Important: Please take a moment to confirm which command to run (depending on your current distribution, and the distribution that you wish to migrate to

Migrate from	Migrate to	Command(s) to run
CentOS Stream	Red Hat Enterprise Linux	`dnf remove epel-next-release` followed by `dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/rhel/$releasever/CERN/$basearch/' swap centos-stream-release redhat-release --nogpgcheck`
CentOS Stream	AlmaLinux	`dnf remove epel-next-release` followed by `dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/alma/$releasever/CERN/$basearch/' swap centos-stream-release almalinux-release --nogpgcheck`
Rocky Linux	Red Hat Enterprise Linux	`rpm -e --nodeps rocky-repos` followed by `dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/rhel/$releasever/CERN/$basearch/' swap rocky-release redhat-release --nogpgcheck`
Rocky Linux	AlmaLinux	`rpm -e --nodeps rocky-repos` followed by `dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/alma/$releasever/CERN/$basearch/' swap rocky-release almalinux-release --nogpgcheck`
Red Hat Enterprise Linux	AlmaLinux	`:> /etc/dnf/protected.d/redhat-release.conf` followed by `:> /etc/yum.repos.d/redhat.repo` followed by `dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/alma/$releasever/CERN/$basearch/' swap redhat-release almalinux-release --nogpgcheck`
AlmaLinux	Red Hat Enterprise Linux	`rpm -e --nodeps almalinux-repos` followed by `dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/rhel/$releasever/CERN/$basearch/' swap almalinux-release redhat-release --nogpgcheck`

The commands above use --nogpgcheck. This is to avoid a chicken/egg problem if the host you are migrating from does not already have "CERN" gpgkeys installed

If you are migrating from a non-CERN version of Stream, you will likely need to also run rpm -e --nodeps centos-stream-repos to avoid transaction conflicts on the above dnf swap command. This step is NOT required if you are using a 'CERN' version of CentOS Stream

Once the 'new' release RPM has been installed from the above command, it's time to run 'distro-sync' to apply the new distribution content to your system.

Please note that after this initial step, some repositories (such as the openafs ones, if used) may be inaccessible due to a missing DNF variable, which is why those repositories need to be disabled in the subsequent step below (this means you may need to disable other repos if necessary).

Step 1 below will change some repository configuration files, so the second distro-sync in step 2 is necessary to upgrade packages coming from those repositories as well.

The reboot is necessary to use the new kernel and it's associated modules.

Please follow these commands:

dnf distro-sync --allowerasing --disablerepo=openafs*
dnf distro-sync

If you are migrating from CentOS Stream 8 to RHEL8 (and your system is EFI), you must also rebuild the grub configuration. Failing to do this will result in your system being unable to boot. If your target is RHEL9 or AlmaLinux (8 or 9), you do not need to perform this step

grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

You may now reboot the system

If you encounter a 'transaction error' - please refer to the Troubleshooting section below

Migration for CERN IT Puppet-managed machines (VMs or Ironic Bare Metal)¶

Before starting, it is strongly recommended that you create a new machine with a the desired target distribution image in your hostgroup. This will allow you to verify that your Puppet catalog compiles correctly, that all your dependencies are met and that all functional tests for your service pass. Doing this will significantly improve your chances of a smooth migration.

Once that is complete, you have two migration options:

Create or Rebuild machines¶

You can use ai-bs to create new machines in your hostgroup:

$ ai-bs --rhel9 -g foohostgroup fooname.cern.ch

Or you can use ai-rebuild to rebuild your existing machines:

$ ai-rebuild --rhel9 fooname.cern.ch

In-place migration¶

Migration with no reinstall for CERN IT Puppet-managed machines was enabled by the changes in CRM-4443. Here are the steps to follow:

Remove your machine(s) from production, you will be rebooting and your service will be impacted.
In Hiera, set base::migrate::target to either rhel or almalinux for the machine/hostgroup you wish to upgrade.

Run puppet agent -t on your machine(s) twice. This could be done with mco like this:

$ mco puppet runonce --batch 100 --batch-sleep 60 -t 5 -T lxsoft -F hostgroup_0=lxsoft
$ mco puppet status  --batch 100 --batch-sleep 60 -t 5 -T lxsoft -F hostgroup_0=lxsoft (repeat until all machines are "Currently idling")
$ mco puppet runonce --batch 100 --batch-sleep 60 -t 5 -T lxsoft -F hostgroup_0=lxsoft

Run systemctl start --wait dnf-distro-sync.service. With mco, this would be:

$ mco shell run "systemctl start --wait dnf-distro-sync.service" --batch 100 --batch-sleep 60 -t 5 -T lxsoft -F hostgroup_0=lxsoft

If you are migrating from CentOS Stream 8 to RHEL8 (and your system is EFI), you must also rebuild the grub configuration. Failing to do this will result in your system being unable to boot. If your target is RHEL9 or AlmaLinux (8 or 9), you do not need to perform this step

grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

You may now reboot the system and once you're happy, add your machine(s) back into production.

The first Puppet run is needed so Facter reports this machine as the 'new' operating system to the Puppet servers, the second Puppet run is the one that actually configures the new repositories. Both of them must complete before running distro-sync.

Migration for non-IT Puppet-managed machines¶

Administrators of non-IT Puppet managed machines can look at the changes introduced in CRM-4443 for inspiration on what they need to do. In short, here is what is needed:

Puppet module changes¶

You will need to change your Puppet modules to account for the new distribution and new sets of repos. Detailed instructions are outside the scope of this guide, but you can use the following merge requests as inspiration:

Client changes¶

On the machines to upgrade:

Replace the contents of /etc/redhat-release with AlmaLinux Release 9.1 (Lime Lynx) or Red Hat Enterprise Linux release 9.1 (Plow) This will "trick" Facter into reporting that machine as a the new target operating system.
Run puppet agent -t twice.
Run dnf swap (refer to the table listed at the top of this page), otherwise the distro-sync won't be able to succeed.
Run dnf distro-sync
Reboot.

Troubleshooting a failed migration¶

In some circumstances the migration process from 'old' to 'new' may 'fail' due to package conflicts.

If this occurs you should analyse what package has the conflict, remove the package and then reattempt the distro-sync command.

A current example of this is if you are migrating from CentOS Stream 9 to either AlmaLinux or Red Hat Enterprise Linux and have python3-perf installed.

In this case, the RPM transaction test will fail similar to:

Error: Transaction test error:
  file /usr/lib64/python3.9/site-packages/perf-0.1-py3.9.egg-info from install of python3-perf-5.14.0-162.6.1.el9_1.x86_64 conflicts with file from package python3-perf-5.14.0-229.el9.x86_64

The solution in this example would be to:

dnf remove python3-perf
dnf distro-sync --allowerasing