Cobbler with CentOS 7 failure to boot/kickstart

Over the past week I've been working on building out an instance of Cobbler and testing some of the provisioning that it is able to do. One of the operating systems that I wanted to deploy is CentOS 7.

After I imported the system into cobbler, it correctly showed up in the pxelinux boot menu and it would happily load the kernel and the initrd, however after initial bootup it would throw the following error message:

dracut-initqueue[867]: Warning: Could not boot.
dracut-initqueue[867]: Warning: /dev/root does not exist

         Starting Dracut Emergency Shell...
Warning: /dev/root does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view the system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report

After that it gives you a root shell.

Some Google searching led me to an mailing list post for Cobbler where someone mentioned that adding ksdevice=link to the Cobbler profile allowed the system to boot without issues.

However before I just implement a change I want to know why that fixes the issue, so I searched Google for "kickstart ksdevice" and found Red Hat's documentation on starting a kickstart. Searching that page for "ksdevice" led me to this section:

ksdevice=<device>

The installation program uses this network device to connect to the network. You can specify the device in one of five ways:

• the device name of the interface, for example, eth0
• the MAC address of the interface, for example, 00:12:34:56:78:9a
• the keyword link, which specifies the first interface with its link in the up state
• the keyword bootif, which uses the MAC address that pxelinux set in the BOOTIF variable. Set IPAPPEND 2 in your pxelinux.cfg file to have pxelinux set the BOOTIF variable.
• the keyword ibft, which uses the MAC address of the interface specified by iBFT

For example, consider a system connected to an NFS server through the eth1 device. To perform a kickstart installation on this system using a kickstart file from the NFS server, you would use the command ks=nfs:<server>:/<path> ksdevice=eth1 at the boot: prompt.

While ksdevice=link would work for some of the machines I am deploying, it wouldn't work for most since they have multiple interfaces and each one of those interfaces would have link, what I really wanted was ksdevice=bootif, which is the most sensible default.

So I modified the profile with ksdevice=link just to test, and that worked without issues, so then I modified the profile and added ksdevice=link, and this failed.

I figured I should check the pxelinux.cfg/default file that Cobbler generates upon issuing a cobbler sync and verify that ksdevice=bootif is actually listed correctly.

What I found was this:

LABEL CentOS-7.1-x86_64
        kernel /images/CentOS-7.1-x86_64/vmlinuz
        MENU LABEL CentOS-7.1-x86_64
        append initrd=/images/CentOS-7.1-x86_64/initrd.img ksdevice=${net0/mac} lang=  kssendmac text  ks=http://10.10.10.1/cblr/svc/op/ks/profile/CentOS-7.1-x86_64
        ipappend 2

This has a ksdevice=${net0/mac} which is not what I had put in the profile, overwriting ksdevice in the profile with ksdevice=link did correctly put that into the pxelinux.cfg/default file, so Cobbler was overwriting my change somehow.

A quick search of ${net0/mac} led me to a page about gPXE commandline items that contained the same variable. At which point I remembered that in Cobbler you set up your profile to be gPXE enabled or not. The default when you import an image is to enable gPXE support.

cobbler profile report  --name=CentOS-7.1-x86_64

Name                           : CentOS-7.1-x86_64
TFTP Boot Files                : {}
Comment                        : 
DHCP Tag                       : default
Distribution                   : CentOS-7.1-x86_64
Enable gPXE?                   : True
Enable PXE Menu?               : 1
[...]

So let's modify the profile to disable gPXE support:

cobbler profile edit --name=CentOS-7.1-x86_64 --enable-gpxe=False
cobbler sync

Verify that the change was made:

cobbler profile report  --name=CentOS-7.1-x86_64

[...]
Enable gPXE?                   : False
[...]

Then let's take a look at our pxelinux.cfg/default file and make sure that it looks correct:

LABEL CentOS-7.1-x86_64
        kernel /images/CentOS-7.1-x86_64/vmlinuz
        MENU LABEL CentOS-7.1-x86_64
        append initrd=/images/CentOS-7.1-x86_64/initrd.img ksdevice=bootif lang=  kssendmac text  ks=http://10.10.10.1/cblr/svc/op/ks/profile/CentOS-7.1-x86_64
        ipappend 2

This time our ksdevice is correctly set. Upon rebooting my PXE booted server it picked up the correct interface, made a DHCP request and kickstarted the server using the provided kickstart file, and installation completed successfully.

So unless you chain-boot gPXE from pxelinux by default, make sure that your profiles are not set to be gPXE enabled if you want to use them directly from the pxelinux menu.

While researching more about this article, I found a blog post by Vlad Ionescu about PXE installing RHEL 7 from Cobbler where he suggests disabling ksdevice entirely and adding an extra inst.repo variable to the kernel command line, however on older versions of CentOS 7 and Red Hat Enterprise Linux 7 there is a bug report that shows that an empty ksdevice could cause anaconda to crash, and setting a manual inst.repo for every profile seems like overkill when just disabling gPXE for the profile also solves the problem.