Linux Kernel Building: What Can You say to Your Boss

Thierry Moreau

Document Number C004694

2008/10/30

Copyright (C) 2008 CONNOTECH Experts-conseils inc.

This document is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Specifications subject to change without notice.

Table of contents

1. Introduction

2. Kernel Building Activity Planning

2.1 Justification

3. Linux Kernel Configuration Overview

3.1 Background

3.2 Semantic Organization of Kernel Parameters

3.3 Dependencies and Configuration as Saved To a File

3.3.1 When Things Go Wrong

3.3.2 Where to Look for Target System Information

3.3.2.1 [[ PCI ... ]]

3.3.2.2 [[ hard disk subsystem ...]]

4. Configuration Contents

4.1 CPU and Core Hardware

4.2 File Systems

4.3 Networking

4.4 Peripheral Buses

4.5 Peripheral Drivers

4.6 Power Management

4.7 Miscellaneous Parameters

5. References

5.1 Normative Rerefences

5.2 Informative Rerefences

Document Revision History

C-Number

Date

Explanation

C004694

Current version

C004676

2008/09/28

Initial release

C004694

2008/10/30

Minor document editing change for on-line distribution.

1. Introduction

Suppose you need to rebuild the Linux kernel for some reason, but this activity does not readily fit your job description. Perhaps this document is for you. You will find in here:

So, this document is not exactly about kernel development work, but more specifically about mere provisioning of a compiled, bootable, and reliable Linux kernel for a purpose.

2. Kernel Building Activity Planning

2.1 Justification

Why do we need to build the Linux kernel? A quick and dirty answer is, "because it's broken," and a more polite one is "because the compiled kernels we have currently do not fit our reqirements." Until a car breaks down, there is no need to look under the hood. Building the kernel is like opening the hood in the case of a car breakdown with a small tool box and limited knowledge of car mechanics, with the expectation that some ad-hoc diagnostic and a quick fix might allow to resume the journey. Nowadays, engine troubleshooting requires sophisticated electronics, likely to be controlled by an embedded Linux kernel, so the analogy could be circular.

But shouldn't the issue be referred to some seasoned Linux kernel experts? First of all, Linux grew from a project led by a small team of volunteers to a significant chunk of the IT industry, so there ought to be a reasonable path down the learning curve. Thus, you and I should be among the target audience for existing HOWTO and FAQ documents that explain the building process and make it feasible for computer specialists. As a matter of fact, the Linux kernel is easily rebuilt once the kernel configuration is established. The catch is thus to determine the kernel configuration suitable for the task at hand, to validate this configuration, and to record the changes for later reference, troubleshooting, and possibly further kernel configuration changes. So, the in-house versus outsoursing decision should be based on the extent and sophistication of kernel configuration changes needed for the new kernel.

As a manager, I do not feel comfortable about developing in-house expertise merely because it's feasible and/or popular. How can you reassure me? In many cases, the specialization of an organization's activities are reflected in the type of kernel configuration changes that justify a new kernel. Thus, the in-house option is better suited whenever the kernel configuration changes can be traced to some specific aspects of an organization's mission. Also, the in-house documentation of kernel configuration and building processes is important for various reasons, notably process reproducibility and traceability. In this respect, the present document is intended as a starting point.

3. Linux Kernel Configuration Overview

3.1 Background

The Linux kernel source base is over 20,000 files spread over about 1,200 directories. The software tools required to make the binary kernel image are relatively simple. The README file for the kernel sources is typically located at file:///usr/src/linux/README. This is definitely a recommended reading, but as a computer specialist interested in kernel building "for some reason" having nothing to do with kernel development, we were looking for kernel configuration guidance, preferably starting at a high level introduction and then pointers to primary sources of relevant information.

The kernel is only a small portion of Linux distributions, which also include a large number of software packages. The term "GNU/Linux" is advocated by the Free Software Foundation in reference to the relative importance of GNU software packages in Linux distributions. Generally, Linux distributions come with a few kernel binary images suitable for most hardware configurations targeted by the distribution. The kernel source is typically included as well, but most Linux uses do not need them.

The CRUX distribution is one that relies on the kernel sources for the installation, and thus requires to build the kernel binary image. The installation procedure is documented in the "Handbook for CRUX 2.4" available at http://crux.nu/Main/Handbook2-4. In this document, the kernel build process is a mere three commands:

make menuconfig
make all
make modules_install

The tricky portion is the first command during which the configuration parameters has to be input by the user. There are variations of this command, e.g. make config, make xconfig, or make gconfig; they are differentiated by the user interaction mechanism. None of these seem to show the kernel parameters semantic organization, basically a hierarchy structure with specific dependencies on some parameters. The end-product of make config is a flat file named .config with simple parameter names and values. An example is located at file:///usr/src/linux-obj/x86_64/default/.config (the directory name x86_64 refers to a CPU architecture as will be explained shortly).

3.2 Semantic Organization of Kernel Parameters

The Linux kernel is said to be portable across various computer system architectures. In the source files, say for a CPU architecture x86_64, the file file:///usr/src/linux/arch/x86_64/Kconfig will be relevant for the kernel configuration. Indeed, the selection of the CPU architecture is the first and perhaps the foremost kernel configuration element. In the case of the CRUX distribution above, a single architecture is supported, i.e. the mainstream i386.

The portable source code for various kernel features are not systematically available for every cpu architectures. For instance, the PCI bus support would be absent from the mainframe s390 architecture. It should be noted that the very configuration dialog rules, such as the ones implied by the architecture-specific Kconfig file, might be the single and last integrity check for some kernel compatibility limitations, e.g. before a kernel crash reveals an unsupported configuration mix. In other words, kernel configuration time validations are essential to the kernel integrity, and the .config file should never be edited manually.

The files Kconfig and the file names with the format Kconfig.* in various kernel source directories hold the kernel configuration rules in a user-friendly textual form, with contextual help text interspersed with the formal rules. This special-purpose format is documented in the file file:///usr/src/linux/Documentation/kbuild/kconfig-language.txt. While the user performs the interactive data entry session driven by make config or one of its variants, it is highly recommended to look at the files Kconfig or Kconfig.* in a parallel session. This recommendation allows a better understanding of configuration parameter context, meaning, and dependencies in a way not available otherwise. Notably, the parallel session shows

3.3 Dependencies and Configuration as Saved To a File

The parameter dependencies may hide other parameter values from the configuration session. When un-hiding dependent parameters, the configuration session seems to revert to the last settings before the parameters disappeared from the user view. (This applies to the make menuconfig variant - it has not been verified with other types of configuration sessions.) However, these last settings are not saved in the .config file or a user-named file for saving the configuration, and hidden parameter values are lost through a save then load cycle.

The above observation is perhaps most dramatic for kernel network support, i.e. the CONFIG_NET parameter documented at the start of file file:///usr/src/linux/net/Kconfig. To obfuscate things a little bit more, the CONFIG_NET parameter value is often determined by a double indirection through the parameters CONFIG_SCSI_NETLINK and CONFIG_SCSI_FC_ATTRS. The latter two parameters are referring to IP transport support over point-to-point Fiberchannel (also known as IEEE1394) connections.

3.3.1 When Things Go Wrong

Kernel building debugging is not easy. When something goes wrong, diagnostic tools are not readily available to non experts. Fortunately, the quality of the kernel software seems excellent, despite sheer complexity due to the large set of configuration options and supported target system hardware.

When facing a kernel-build-and-install problem, the poor computer specialist faces the challenge of system integration diagnostic. Anything might cause the observed symptoms. In this document, we do not delve into the technical details or provide a checklist. Instead, we just suggest a mental process for a high-level divide and conquer approach to diagnostic.

There are three or four broad categories of "root causes". If your boss is on your back too pervasively while you spend your time on this, say that you follow a "root cause analysis" strategy - that's a notion rooted in both engineering and management fields.

The high level system integration troubleshooting activity calls for an open-minded consideration of every possible causes, at once without taking anything for granted and forgiving for uncertainty. E.g. "we are pretty sure that the SCSI adapter is compatible with the well known manufacturer X, but we may revisit this assumption later on if the current diagnostic effort increases our confidence in another integration aspect without solving the difficulty." System integration troublechooting is not the time for finger pointing and a liability avoidance attitude is counter productive.

Preferably, two broad categories of root causes would fit the divide and conquer approach to kernel problem diagnostic, but we are stuck with three or four.

Configuration options incompatibilities
This seems to be the foremost area of troubles: kernel configuration compatibility with the target system hardware, or compatibility issues among configuration elements (beyond or in spite of the configuration rules programmed in the Kconfig files), or the like. In this case, we should search on-line documentation about the kernel features that might cause the problem, including mailing list archives and specialized wikis in which installation issues may be reported. If the kernel installation problem fits in this category, some configuration parameter change will change the observed symptoms, and advance the installation one step, i.e. either to successful completion or to another installation issue (in which case your boss might not readily see much of an advance).
Build and install process flaw
A possible cause for a kernel installation bug is an oversight in the detailed procedure that turns a kernel configuration file, kernel source code, and other scripts and parameter elements into a bootable media that is (hopefully) invoked when the target computer system is reset. This category of possible root cause addresses the assurance level about what exactly is being tried when the system is booted.
Wrong assumptions about the target system
Yet another possible cause for a kernel installation bug is a wrong assumption about the target system. The personal computer technology offers many hooks for the software to probe the system components hardware and firmware configuration, but there is still room for errors. Notably, the peripheral controller integrated circuits are referred to by various names and trademarks, creating confusion. Fortunately, the Linux kernel source base seems to be good at remaining compatible with the actual hardware configuration detected at boot time.
Inadequate methodology
Your boss may legitimately wonder about the very idea of being stalled with a Linux kernel installation bug as an accepted step in the kernel build and install process. Does it make sense to have so many integration pitfalls validated together during the kernel boot sequence? As a matter of fact, there is a refined Linux kernel installation methodology, by the technical name initrd, that provides a two-phase boot process in which the intermediate phase allows kernel modules to be selectively tried against the target system hardware. That's how the leading Linux distributions adapt the kernel to the diversified hardware configurations of their target market. However, the kernel build process using the initrd mechanism is more invoved than a kernel build without it, so the "more adequate" methodology for kernel installation troubleshooting has a stepper learning curve. In other words, the non-use of the initrd mechanism is a possible root cause for the kernel build and install inefficiency, but not for the the kernel installation bug itself.

3.3.2 Where to Look for Target System Information

Most likely, the target system for the Linux kernel will be compatible with at least some bootable media for a Linux variant. In this common case, it is possible to execute shell commands that query the target system. The installation procedures for most Linux distributions are indeed based on a Linux system booted from a CD or DVD distribution, and scripts that adjust installation parameters from local queries about the target system characteristics.

Once assigned the task of rebuilding the kernel, a non-expert computer specialist has to gather various information about the target system, but we saw no convenient guidance document about the native Linux facilities and commands for this. In the following document subsections, we try to help with some hints.

3.3.2.1 [[ PCI ... ]]

Do the Linux command lspi for information about the PCI bus subsystem.

3.3.2.2 [[ hard disk subsystem ...]]

Have a look at the file proc/partitions for low-level information about existing disk partitions.

4. Configuration Contents

In this section, we record the configuration contents for a specific Linux kernel build operation, with a high level documentation perspective.

The intended Linux installation is for a Laptop computer, i.e. a Lenovo Thinkpad T61, that came with a factory installed Suse distribution. The reference kernel configuration for this is either the file file:///usr/src-suse/linux-2.6.16.54-0.2.3-obj/i386/default/.config or file:///usr/src-suse/linux-2.6.16.54-0.2.3-obj/i386/smp/.config. A special driver has been written for the Thinkpad portable computer family, see the web site for this driver development project at http://ibm-acpi.sourceforge.net/ and this documentation file shipped with a recent version of the kernel: file:///usr/src-crux/linux-2.6.23.9/Documentation/thinkpad-acpi.txt.

The Linux distribution to be installed is CRUX version 2.4, see this distribution web site at http://crux.nu/Main/HomePage. The purpose is to get a controlled Linux installation for C++ software development, where free software packages can be installed from source with greater chances of troubleshooting difficulties with the development tools. It is deemed easier to understand what is going on with an installation not inflated with too many software missions and interrelated packages. The software development is targeted at web server systems, so a controlled Linux installation is envisioned also for the runtime environment where a portable system makes no sense. The CRUX distribution has a lightweight software package installation approach, which is considered a plus when installation from sources should systematically remain as an option.

As a special [(and out of scope)] note, the target software development is ultimately protected by strong end-to-end cryptographic mechanisms, to which a well managed corporate proxy firewall should logically deny access unless specifically registered (otherwise the proxy would authorize encrypted tunnels with no control of any kind on the traffic). Given this observation, the laptop system is a convenient demonstration equipment for demos staged within the corporate premises - if you can't reach the demo server on the public Internet for good reasons, let the presentator bring a demo server with him.

Here is a list of mailing list archive entries, and other Internet public information, in no particular order, that were visited to overcome a difficulty during the kernel build and install process.

http://bbs.archlinux.org/viewtopic.php?id=52968
Page title: Arch Linux Forums / Building a kernel without initrd. I am stuck :). Representative discussion of difficulties similar to the one we encountered.
http://kmandla.wordpress.com/2008/06/08/howto-16-second-boot-on-550mhz-celeron-with-crux/
Page title: Howto: 16-second boot on 550Mhz Celeron with Crux « Motho ke motho ka botho. An interesting overview of CRUX installation details, including kernel configuration.
http://fixunix.com/kernel/516804-cant-boot-2-6-26-rt1.html
Page title: Can't boot 2.6.26-rt1 - Unix Linux Forum. A kernel installation bug discussion in which the initrd mechanism is being discussed, including about relevance and pitfalls.
http://www.thinkwiki.org/wiki/How_to_configure_and_use_libata_SATA_/_PATA_drivers
Page title: How to configure and use libata SATA / PATA drivers - ThinkWiki. This wiki entry was very useful in our kernel configuration effort, as a matter of both relevance and technical quality.
http://ibm-acpi.sourceforge.net/
Page title: ibm-acpi - IBM ThinkPad ACPI Extras Driver. This is the project home page for a specialized Linux kernel driver of specific relevance to our kernel environment.

4.1 CPU and Core Hardware

The architecture is i386. A Linux installation with a more powerful architecture suitable for high volume server operations is a deferred activity. In the short term, the universal support of the i386 architecture by software developers is a good thing. The Crux distribution is foremostly focused on this architecture as well. The starting point for kernel configuration rules in this context is the file file:///usr/src-crux/linux-2.6.23.9/arch/i386/Kconfig.

The architecture definition refinements are conservatively defined, including CONFIG_SMP, CONFIG_X86_GENERICARCH (CONFIG_X86_PC would be more specific). However the CONFIG_MCORE2 is specific to the target system processor as verified by /proc/cpuinfo, and hence CONFIG_X86_GENERIC is not set.

CONFIG_HIGHMEM4G is selected since the target system has 2GB physical memory.

CONFIG_IRQBALANCE is not selected although it looks attractive in performance hungry configuration. The reference configuration omits it as well.

4.2 File Systems

We deliberately make reasoned decisions about the file system by the kernel, instead of accepting defaults or mimicking the reference configuration.

CONFIG_LBD is omitted (block devices > 2TB).

The I/O scheduler is fixed to CONFIG_IOSCHED_DEADLINE.

Besides the above two very basic parameters, the file system configuration seems straightforward from the suggested defaults. We merely removed ReiserFS support and added CIFS support.

4.3 Networking

No IP transport over Fiberchannel over SCSI, so the CONFIG_NET value is user-defined.

No IRDA port on the target system hardware, CONFIG_IRDA is appropriately omitted.

The overall approach is to have a fairly restricted network kernel support configuration. This includes:

CONFIG_SYN_COOKIES is set because a server system might have to defend itself against denial of service attacks irrespective of the cryptographic controls for legitimate remote party authentication and access control.

4.4 Peripheral Buses

The integrated "chipset" used in the target system hardware, and the motherboard, should be used as a documentation base for refined settings in this area. In the absence of complete documents, peripheral buses parameters are set according to the reference configuration. Few, if any, parameter names seem to be introduced after the reference kernel version.

4.5 Peripheral Drivers

Memory Technology Device (i.e. all sorts of flash and other small size memry devices) support options are liberally configured in the kernel, and not as modules. In the case of the mapping drivers for chip access, the reference configuration was used as a base, notably for physical address mapping, but BIOS rewrite support was removed. These drivers, except for the memory-map and PCI drivers, are left as kernel modules. Some of the self-contained MTD drivers just looked ugly and/or not applicable to the system herdware and were omitted. CONFIG_MTD_NAND is omitted because the paramater help is just too confusing! OneNAND support is also omitted.

Parallel port support is omitted, being present in the hardware system only through a docking station which is not available these days.

The reference configuration seems to rely on the SATA driver inserted in the kernel (not as a module) with the parameter CONFIG_SATA_INTEL_COMBINED. As of the CRUX 2.4 kernel version, the CONFIG_SATA_INTEL_COMBINED was removed as a quite inefficient and annoying SATA support through other bus techniques. This was reported in a specialized kernel development mailing list at http://www.uwsg.iu.edu/hypermail/linux/kernel/0703.1/0736.html. The replacement SATA driver parameter should be CONFIG_ATA_PIIX (ICH8 is present in the target system hardware). Most other SATA controllers have their respective parameters omitted.

[...]

4.6 Power Management

Power management parameters are set according to the reference configuration. Few, if any, parameter names seem to be introduced after the reference kernel version.

4.7 Miscellaneous Parameters

For sake of determinism and reliability, we omit CONFIG_MODULE_FORCE_UNLOAD and CONFIG_MODVERSIONS. This is to be understood in a context where kernel rebuilding is realistic, instead of patching such as allowed by CONFIG_MODVERSIONS.

Removed all kernel hacking options, but kept kernel instumentation.

The security options, cryptographic API, and library routines are all omitted. In the case of the cryptographic API, the omission of IPSEC from the network configuration allows a complete omission.

The firmware driver options are set neither in the kernel nor as loadable modules, e.g. CONFIG_EFI, which anyway depends on ACPI that is not set for SMP systems.

Somehow surprisingly, CONFIG_KEXEC is selected in the reference configuration. This looks like a tricky feature, and omitted.

CONFIG_RELOCATABLE seems a parameter name newer than the reference configuration. We omit it, but only because it is qualified as EXPERIMENTAL in the help text.

5. References

5.1 Normative Rerefences

5.2 Informative Rerefences