verify-bugs-and-bisect-regressions.rst

Temporarily disabling solutions like Secure Boot is another way to make your own
Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup
utility; the required steps vary a lot between machines and therefore cannot be
described here.

On mainstream x86 Linux distributions there is a third and universal option:
disable all Secure Boot restrictions for your Linux environment. You can
initiate this process by running ``mokutil --disable-validation``; this will
tell you to create a one-time password, which is safe to write down. Now
restart; right after your BIOS performed all self-tests the bootloader Shim will
show a blue box with a message 'Press any key to perform MOK management'. Hit
some key before the countdown exposes, which will open a menu. Choose 'Change
Secure Boot state'. Shim's 'MokManager' will now ask you to enter three
randomly chosen characters from the one-time password specified earlier. Once
you provided them, confirm you really want to disable the validation.
Afterwards, permit MokManager to reboot the machine.

[:ref:`back to step-by-step guide <secureboot_bissbs>`]

.. _bootworking_bisref:

Boot the last kernel that was working
-------------------------------------

  *Boot into the last working kernel and briefly recheck if the feature that
  regressed really works.* [:ref:`...<bootworking_bissbs>`]

This will make later steps that cover creating and trimming the configuration do
the right thing.

[:ref:`back to step-by-step guide <bootworking_bissbs>`]

.. _diskspace_bisref:

Space requirements
------------------

  *Ensure to have enough free space for building Linux.*
  [:ref:`... <diskspace_bissbs>`]

The numbers mentioned are rough estimates with a big extra charge to be on the
safe side, so often you will need less.

If you have space constraints, be sure to hay attention to the :ref:`step about
debug symbols' <debugsymbols_bissbs>` and its :ref:`accompanying reference
section' <debugsymbols_bisref>`, as disabling then will reduce the consumed disk
space by quite a few gigabytes.

[:ref:`back to step-by-step guide <diskspace_bissbs>`]

.. _rangecheck_bisref:

Bisection range
---------------

  *Determine the kernel versions considered 'good' and 'bad' throughout this
  guide.* [:ref:`...<rangecheck_bissbs>`]

Establishing the range of commits to be checked is mostly straightforward,
except when a regression occurred when switching from a release of one stable
series to a release of a later series (e.g. from 6.0.11 to 6.1.4). In that case
Git will need some hand holding, as there is no straight line of descent.

That's because with the release of 6.0 mainline carried on to 6.1 while the
stable series 6.0.y branched to the side. It's therefore theoretically possible
that the issue you face with 6.1.4 only worked in 6.0.11, as it was fixed by a
commit that went into one of the 6.0.y releases, but never hit mainline or the
6.1.y series. Thankfully that normally should not happen due to the way the
stable/longterm maintainers maintain the code. It's thus pretty safe to assume
6.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel
will be built and tested in the segment '2' of this guide; Git would force you
to do this as well, if you tried bisecting between 6.0.11 and 6.1.13.

[:ref:`back to step-by-step guide <rangecheck_bissbs>`]

.. _buildrequires_bisref:

Install build requirements
--------------------------

  *Install all software required to build a Linux kernel.*
  [:ref:`...<buildrequires_bissbs>`]

The kernel is pretty stand-alone, but besides tools like the compiler you will
sometimes need a few libraries to build one. How to install everything needed
depends on your Linux distribution and the configuration of the kernel you are
about to build.

Here are a few examples what you typically need on some mainstream
distributions:

* Arch Linux and derivatives::

    sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \
      pahole perl zlib ncurses qt6-base

* Debian, Ubuntu, and derivatives::

    sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \
      libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \
      libncurses-dev qt6-base-dev g++

* Fedora and derivatives::

    sudo dnf install binutils \
      /usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \
      /usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction}

* openSUSE and derivatives::

    sudo zypper install bc binutils bison dwarves flex gcc git \
      kernel-install-tools libelf-devel make modutils openssl openssl-devel \
      perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel

These commands install a few packages that are often, but not always needed. You
for example might want to skip installing the development headers for ncurses,
which you will only need in case you later might want to adjust the kernel build
configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit
the headers of Qt6 is you do not plan to adjust the .config using 'xconfig'.

You furthermore might need additional libraries and their development headers
for tasks not covered in this guide -- for example when building utilities from
the kernel's tools/ directory.

[:ref:`back to step-by-step guide <buildrequires_bissbs>`]

.. _sources_bisref:

Download the sources using Git
------------------------------

  *Retrieve the Linux mainline sources.*
  [:ref:`...<sources_bissbs>`]

The step-by-step guide outlines how to download the Linux sources using a full
Git clone of Linus' mainline repository. There is nothing more to say about
that -- but there are two alternatives ways to retrieve the sources that might
work better for you:

 * If you have an unreliable internet connection, consider
   :ref:`using a 'Git bundle'<sources_bundle_bisref>`.

 * If downloading the complete repository would take too long or requires too
   much storage space, consider :ref:`using a 'shallow
   clone'<sources_shallow_bisref>`.

.. _sources_bundle_bisref:

Downloading Linux mainline sources using a bundle
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Use the following commands to retrieve the Linux mainline sources using a
bundle::

    wget -c \
      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
    git clone --no-checkout clone.bundle ~/linux/
    cd ~/linux/
    git remote remove origin
    git remote add mainline \
      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
    git fetch mainline
    git remote add -t master stable \
      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

In case the 'wget' command fails, just re-execute it, it will pick up where
it left off.

[:ref:`back to step-by-step guide <sources_bissbs>`]
[:ref:`back to section intro <sources_bisref>`]

.. _sources_shallow_bisref:

Downloading Linux mainline sources using a shallow clone
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

First, execute the following command to retrieve the latest mainline codebase::

    git clone -o mainline --no-checkout --depth 1 -b master \
      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
    cd ~/linux/
    git remote add -t master stable \
      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

Now deepen your clone's history to the second predecessor of the mainline
release of your 'good' version. In case the latter are 6.0 or 6.0.11, 5.19 would
be the first predecessor and 5.18 the second -- hence deepen the history up to
that version::

    git fetch --shallow-exclude=v5.18 mainline

Afterwards add the stable Git repository as remote and all required stable
branches as explained in the step-by-step guide.

Note, shallow clones have a few peculiar characteristics:

 * For bisections the history needs to be deepened a few mainline versions
   farther than it seems necessary, as explained above already. That's because
   Git otherwise will be unable to revert or describe most of the commits within
   a range (say 6.1..6.2), as they are internally based on earlier kernels
   releases (like 6.0-rc2 or 5.19-rc3).

 * This document in most places uses ``git fetch`` with ``--shallow-exclude=``
   to specify the earliest version you care about (or to be precise: its git
   tag). You alternatively can use the parameter ``--shallow-since=`` to specify
   an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to
   define the depth of the history you want to download. When using them while
   bisecting mainline, ensure to deepen the history to at least 7 months before
   the release of the mainline release your 'good' kernel is based on.

 * Be warned, when deepening your clone you might encounter an error like
   'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'.
   In that case run ``git repack -d`` and try again.

[:ref:`back to step-by-step guide <sources_bissbs>`]
[:ref:`back to section intro <sources_bisref>`]

.. _oldconfig_bisref:

Start defining the build configuration for your kernel
------------------------------------------------------

  *Start preparing a kernel build configuration (the '.config' file).*
  [:ref:`... <oldconfig_bissbs>`]

*Note, this is the first of multiple steps in this guide that create or modify
build artifacts. The commands used in this guide store them right in the source
tree to keep things simple. In case you prefer storing the build artifacts
separately, create a directory like '~/linux-builddir/' and add the parameter
``O=~/linux-builddir/`` to all make calls used throughout this guide. You will
have to point other commands there as well -- among them the ``./scripts/config
[...]`` commands, which will require ``--file ~/linux-builddir/.config`` to
locate the right build configuration.*

Two things can easily go wrong when creating a .config file as advised:

 * The oldconfig target will use a .config file from your build directory, if
   one is already present there (e.g. '~/linux/.config'). That's totally fine if
   that's what you intend (see next step), but in all other cases you want to
   delete it. This for example is important in case you followed this guide
   further, but due to problems come back here to redo the configuration from
   scratch.

 * Sometimes olddefconfig is unable to locate the .config file for your running
   kernel and will use defaults, as briefly outlined in the guide. In that case
   check if your distribution ships the configuration somewhere and manually put
   it in the right place (e.g. '~/linux/.config') if it does. On distributions
   where /proc/config.gz exists this can be achieved using this command::

     zcat /proc/config.gz > .config

   Once you put it there, run ``make olddefconfig`` again to adjust it to the
   needs of the kernel about to be built.

Note, the olddefconfig target will set any undefined build options to their
default value. If you prefer to set such configuration options manually, use
``make oldconfig`` instead. Then for each undefined configuration option you
will be asked how to proceed; in case you are unsure what to answer, simply hit
'enter' to apply the default value. Note though that for bisections you normally
want to go with the defaults, as you otherwise might enable a new feature that
causes a problem looking like regressions (for example due to security
restrictions).

Occasionally odd things happen when trying to use a config file prepared for one
kernel (say 6.1) on an older mainline release -- especially if it is much older
(say 5.15). That's one of the reasons why the previous step in the guide told
you to boot the kernel where everything works. If you manually add a .config
file you thus want to ensure it's from the working kernel and not from a one
that shows the regression.

In case you want to build kernels for another machine, locate its kernel build
configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy
that file to the build machine and store it as ~/linux/.config; afterwards run
``make olddefconfig`` to adjust it.

[:ref:`back to step-by-step guide <oldconfig_bissbs>`]

.. _localmodconfig_bisref:

Trim the build configuration for your kernel
--------------------------------------------

  *Disable any kernel modules apparently superfluous for your setup.*
  [:ref:`... <localmodconfig_bissbs>`]

As explained briefly in the step-by-step guide already: with localmodconfig it
can easily happen that your self-built kernels will lack modules for tasks you
did not perform at least once before utilizing this make target. That happens
when a task requires kernel modules which are only autoloaded when you execute
it for the first time. So when you never performed that task since starting your
kernel the modules will not have been loaded -- and from localmodonfig's point
of view look superfluous, which thus disables them to reduce the amount of code
to be compiled.

You can try to avoid this by performing typical tasks that often will autoload
additional kernel modules: start a VM, establish VPN connections, loop-mount a
CD/DVD ISO, mount network shares (CIFS, NFS, ...), and connect all external
devices (2FA keys, headsets, webcams, ...) as well as storage devices with file
systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, ...). But it
is hard to think of everything that might be needed -- even kernel developers
often forget one thing or another at this point.

Do not let that risk bother you, especially when compiling a kernel only for
testing purposes: everything typically crucial will be there. And if you forget
something important you can turn on a missing feature manually later and quickly
run the commands again to compile and install a kernel that has everything you
need.

But if you plan to build and use self-built kernels regularly, you might want to
reduce the risk by recording which modules your system loads over the course of
a few weeks. You can automate this with `modprobed-db
<https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to
point localmodconfig to the list of modules modprobed-db noticed being used::

  yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig

That parameter also allows you to build trimmed kernels for another machine in
case you copied a suitable .config over to use as base (see previous step). Just
run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to
your build's host home directory. Then run these commands instead of the one the
step-by-step guide mentions::

  yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig

[:ref:`back to step-by-step guide <localmodconfig_bissbs>`]

.. _tagging_bisref:

Tag the kernels about to be build
---------------------------------

  *Ensure all the kernels you will build are clearly identifiable using a
  special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`]

This allows you to differentiate your distribution's kernels from those created
during this process, as the file or directories for the latter will contain
'-local' in the name; it also helps picking the right entry in the boot menu and
not lose track of you kernels, as their version numbers will look slightly
confusing during the bisection.

[:ref:`back to step-by-step guide <tagging_bissbs>`]

.. _debugsymbols_bisref:

Decide to enable or disable debug symbols
-----------------------------------------

  *Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`]

Having debug symbols available can be important when your kernel throws a
'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be
able to find the exact place where the problem occurred in the code. But
collecting and embedding the needed debug information takes time and consumes
quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel
trimmed with localmodconfig consumed around 5 Gigabyte of space with debug
symbols, but less than 1 when they were disabled. The resulting kernel image and
modules are bigger as well, which increases storage requirements for /boot/ and
load times.

In case you want a small kernel and are unlikely to decode a stack trace later,
you thus might want to disable debug symbols to avoid those downsides. If it
later turns out that you need them, just enable them as shown and rebuild the
kernel.

You on the other hand definitely want to enable them for this process, if there
is a decent chance that you need to decode a stack trace later. The section
'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst
explains this process in more detail.

[:ref:`back to step-by-step guide <debugsymbols_bissbs>`]

.. _configmods_bisref:

Adjust build configuration
--------------------------

  *Check if you may want or need to adjust some other kernel configuration
  options:*

Depending on your needs you at this point might want or have to adjust some
kernel configuration options.

.. _configmods_distros_bisref:

Distro specific adjustments
~~~~~~~~~~~~~~~~~~~~~~~~~~~

  *Are you running* [:ref:`... <configmods_bissbs>`]

The following sections help you to avoid build problems that are known to occur
when following this guide on a few commodity distributions.

**Debian:**

 * Remove a stale reference to a certificate file that would cause your build to
   fail::

    ./scripts/config --set-str SYSTEM_TRUSTED_KEYS ''

   Alternatively, download the needed certificate and make that configuration
   option point to it, as `the Debian handbook explains in more detail
   <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_
   -- or generate your own, as explained in
   Documentation/admin-guide/module-signing.rst.

[:ref:`back to step-by-step guide <configmods_bissbs>`]

.. _configmods_individual_bisref:

Individual adjustments
~~~~~~~~~~~~~~~~~~~~~~

  *If you want to influence the other aspects of the configuration, do so
  now.* [:ref:`... <configmods_bissbs>`]

At this point you can use a command like ``make menuconfig`` or ``make nconfig``
to enable or disable certain features using a text-based user interface; to use
a graphical configuration utility, run ``make xconfig`` instead. Both of them
require development libraries from toolkits they are rely on (ncurses
respectively Qt5 or Qt6); an error message will tell you if something required
is missing.

[:ref:`back to step-by-step guide <configmods_bissbs>`]

.. _saveconfig_bisref:

Put the .config file aside
--------------------------

  *Reprocess the .config after the latest changes and store it in a safe place.*
  [:ref:`... <saveconfig_bissbs>`]

Put the .config you prepared aside, as you want to copy it back to the build
directory every time  during this guide before you start building another
kernel. That's because going back and forth between different versions can alter
.config files in odd ways; those occasionally cause side effects that could
confuse testing or in some cases render the result of your bisection
meaningless.

[:ref:`back to step-by-step guide <saveconfig_bissbs>`]

.. _introlatestcheck_bisref:

Try to reproduce the regression
-----------------------------------------

  *Verify the regression is not caused by some .config change and check if it
  still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`]

For some readers it might seem unnecessary to check the latest codebase at this
point, especially if you did that already with a kernel prepared by your
distributor or face a regression within a stable/longterm series. But it's
highly recommended for these reasons:

* You will run into any problems caused by your setup before you actually begin
  a bisection. That will make it a lot easier to differentiate between 'this
  most likely is some problem in my setup' and 'this change needs to be skipped
  during the bisection, as the kernel sources at that stage contain an unrelated
  problem that causes building or booting to fail'.

* These steps will rule out if your problem is caused by some change in the
  build configuration between the 'working' and the 'broken' kernel. This for
  example can happen when your distributor enabled an additional security
  feature in the newer kernel which was disabled or not yet supported by the
  older kernel. That security feature might get into the way of something you
  do -- in which case your problem from the perspective of the Linux kernel
  upstream developers is not a regression, as
  Documentation/admin-guide/reporting-regressions.rst explains in more detail.
  You thus would waste your time if you'd try to bisect this.

* If the cause for your regression was already fixed in the latest mainline
  codebase, you'd perform the bisection for nothing. This holds true for a
  regression you encountered with a stable/longterm release as well, as they are
  often caused by problems in mainline changes that were backported -- in which
  case the problem will have to be fixed in mainline first. Maybe it already was
  fixed there and the fix is already in the process of being backported.

* For regressions within a stable/longterm series it's furthermore crucial to
  know if the issue is specific to that series or also happens in the mainline
  kernel, as the report needs to be sent to different people:

  * Regressions specific to a stable/longterm series are the stable team's
    responsibility; mainline Linux developers might or might not care.

  * Regressions also happening in mainline are something the regular Linux
    developers and maintainers have to handle; the stable team does not care
    and does not need to be involved in the report, they just should be told
    to backport the fix once it's ready.

  Your report might be ignored if you send it to the wrong party -- and even
  when you get a reply there is a decent chance that developers tell you to
  evaluate   which of the two cases it is before they take a closer look.

[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]

.. _checkoutmaster_bisref:

Check out the latest Linux codebase
-----------------------------------

  *Check out the latest Linux codebase.*
  [:ref:`... <introlatestcheck_bissbs>`]

In case you later want to recheck if an ever newer codebase might fix the
problem, remember to run that ``git fetch --shallow-exclude [...]`` command
again mentioned earlier to update your local Git repository.

[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]

.. _build_bisref:

Build your kernel
-----------------

  *Build the image and the modules of your first kernel using the config file
  you prepared.* [:ref:`... <build_bissbs>`]

A lot can go wrong at this stage, but the instructions below will help you help
yourself. Another subsection explains how to directly package your kernel up as
deb, rpm or tar file.

Dealing with build errors
~~~~~~~~~~~~~~~~~~~~~~~~~

When a build error occurs, it might be caused by some aspect of your machine's
setup that often can be fixed quickly; other times though the problem lies in
the code and can only be fixed by a developer. A close examination of the
failure messages coupled with some research on the internet will often tell you
which of the two it is. To perform such investigation, restart the build
process like this::

  make V=1

The ``V=1`` activates verbose output, which might be needed to see the actual
error. To make it easier to spot, this command also omits the ``-j $(nproc
--all)`` used earlier to utilize every CPU core in the system for the job -- but
this parallelism also results in some clutter when failures occur.

After a few seconds the build process should run into the error again. Now try
to find the most crucial line describing the problem. Then search the internet
for the most important and non-generic section of that line (say 4 to 8 words);
avoid or remove anything that looks remotely system-specific, like your username
or local path names like ``/home/username/linux/``. First try your regular
internet search engine with that string, afterwards search Linux kernel mailing
lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_.

This most of the time will find something that will explain what is wrong; quite
often one of the hits will provide a solution for your problem, too. If you
do not find anything that matches your problem, try again from a different angle
by modifying your search terms or using another line from the error messages.

In the end, most issues you run into have likely been encountered and
reported by others already. That includes issues where the cause is not your
system, but lies in the code. If you run into one of those, you might thus find a
solution (e.g. a patch) or workaround for your issue, too.

Package your kernel up
~~~~~~~~~~~~~~~~~~~~~~

The step-by-step guide uses the default make targets (e.g. 'bzImage' and
'modules' on x86) to build the image and the modules of your kernel, which later
steps of the guide then install. You instead can also directly build everything
and directly package it up by using one of the following targets:

 * ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package

 * ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package

 * ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball

This is just a selection of available make targets for this purpose, see
``make help`` for others. You can also use these targets after running
``make -j $(nproc --all)``, as they will pick up everything already built.

If you employ the targets to generate deb or rpm packages, ignore the
step-by-step guide's instructions on installing and removing your kernel;
instead install and remove the packages using the package utility for the format
(e.g. dpkg and rpm) or a package management utility build on top of them (apt,
aptitude, dnf/yum, zypper, ...). Be aware that the packages generated using
these two make targets are designed to work on various distributions utilizing
those formats, they thus will sometimes behave differently than your
distribution's kernel packages.

[:ref:`back to step-by-step guide <build_bissbs>`]

.. _install_bisref:

Put the kernel in place
-----------------------

  *Install the kernel you just built.* [:ref:`... <install_bissbs>`]

What you need to do after executing the command in the step-by-step guide
depends on the existence and the implementation of ``/sbin/installkernel``
executable on your distribution.

If installkernel is found, the kernel's build system will delegate the actual
installation of your kernel image to this executable, which then performs some
or all of these tasks:

 * On almost all Linux distributions installkernel will store your kernel's
   image in /boot/, usually as '/boot/vmlinuz-<kernelrelease_id>'; often it will
   put a 'System.map-<kernelrelease_id>' alongside it.

 * On most distributions installkernel will then generate an 'initramfs'
   (sometimes also called 'initrd'), which usually are stored as
   '/boot/initramfs-<kernelrelease_id>.img' or
   '/boot/initrd-<kernelrelease_id>'. Commodity distributions rely on this file
   for booting, hence ensure to execute the make target 'modules_install' first,
   as your distribution's initramfs generator otherwise will be unable to find
   the modules that go into the image.

 * On some distributions installkernel will then add an entry for your kernel
   to your bootloader's configuration.

You have to take care of some or all of the tasks yourself, if your
distribution lacks a installkernel script or does only handle part of them.
Consult the distribution's documentation for details. If in doubt, install the
kernel manually::

   sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)
   sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)

Now generate your initramfs using the tools your distribution provides for this
process. Afterwards add your kernel to your bootloader configuration and reboot.

[:ref:`back to step-by-step guide <install_bissbs>`]

.. _storagespace_bisref:

Storage requirements per kernel
-------------------------------

  *Check how much storage space the kernel, its modules, and other related files
  like the initramfs consume.* [:ref:`... <storagespace_bissbs>`]

The kernels built during a bisection consume quite a bit of space in /boot/ and
/lib/modules/, especially if you enabled debug symbols. That makes it easy to
fill up volumes during a bisection -- and due to that even kernels which used to
work earlier might fail to boot. To prevent that you will need to know how much
space each installed kernel typically requires.

Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in
the guide will match all files needed to boot your kernel -- but neither the
path nor the naming scheme are mandatory. On some distributions you thus will
need to look in different places.

[:ref:`back to step-by-step guide <storagespace_bissbs>`]

.. _tainted_bisref:

Check if your newly built kernel considers itself 'tainted'
-----------------------------------------------------------

  *Check if the kernel marked itself as 'tainted'.*
  [:ref:`... <tainted_bissbs>`]

Linux marks itself as tainted when something happens that potentially leads to
follow-up errors that look totally unrelated. That is why developers might
ignore or react scantly to reports from tainted kernels -- unless of course the
kernel set the flag right when the reported bug occurred.

That's why you want check why a kernel is tainted as explained in
Documentation/admin-guide/tainted-kernels.rst; doing so is also in your own
interest, as your testing might be flawed otherwise.

[:ref:`back to step-by-step guide <tainted_bissbs>`]

.. _recheckbroken_bisref:

Check the kernel built from a recent mainline codebase
------------------------------------------------------

  *Verify if your bug occurs with the newly built kernel.*
  [:ref:`... <recheckbroken_bissbs>`]

There are a couple of reasons why your bug or regression might not show up with
the kernel you built from the latest codebase. These are the most frequent:

* The bug was fixed meanwhile.

* What you suspected to be a regression was caused by a change in the build
  configuration the provider of your kernel carried out.

* Your problem might be a race condition that does not show up with your kernel;
  the trimmed build configuration, a different setting for debug symbols, the
  compiler used, and various other things can cause this.

* In case you encountered the regression with a stable/longterm kernel it might
  be a problem that is specific to that series; the next step in this guide will
  check this.

[:ref:`back to step-by-step guide <recheckbroken_bissbs>`]

.. _recheckstablebroken_bisref:

Check the kernel built from the latest stable/longterm codebase
---------------------------------------------------------------

  *Are you facing a regression within a stable/longterm release, but failed to
  reproduce it with the kernel you just built using the latest mainline sources?
  Then check if the latest codebase for the particular series might already fix
  the problem.* [:ref:`... <recheckstablebroken_bissbs>`]

If this kernel does not show the regression either, there most likely is no need
for a bisection.

[:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`]

.. _introworkingcheck_bisref:

Ensure the 'good' version is really working well
------------------------------------------------

  *Check if the kernels you build work fine.*
  [:ref:`... <introworkingcheck_bissbs>`]

This section will reestablish a known working base. Skipping it might be
appealing, but is usually a bad idea, as it does something important:

It will ensure the .config file you prepared earlier actually works as expected.
That is in your own interest, as trimming the configuration is not foolproof --
and you might be building and testing ten or more kernels for nothing before
starting to suspect something might be wrong with the build configuration.

That alone is reason enough to spend the time on this, but not the only reason.

Many readers of this guide normally run kernels that are patched, use add-on
modules, or both. Those kernels thus are not considered 'vanilla' -- therefore
it's possible that the thing that regressed might never have worked in vanilla
builds of the 'good' version in the first place.

There is a third reason for those that noticed a regression between
stable/longterm kernels of different series (e.g. 6.0.13..6.1.5): it will
ensure the kernel version you assumed to be 'good' earlier in the process (e.g.
6.0) actually is working.

[:ref:`back to step-by-step guide <introworkingcheck_bissbs>`]

.. _recheckworking_bisref:

Build your own version of the 'good' kernel
-------------------------------------------

  *Build your own variant of the working kernel and check if the feature that
  regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`]

In case the feature that broke with newer kernels does not work with your first
self-built kernel, find and resolve the cause before moving on. There are a
multitude of reasons why this might happen. Some ideas where to look:

* Check the taint status and the output of ``dmesg``, maybe something unrelated
  went wrong.

* Maybe localmodconfig did something odd and disabled the module required to
  test the feature? Then you might want to recreate a .config file based on the
  one from the last working kernel and skip trimming it down; manually disabling
  some features in the .config might work as well to reduce the build time.

* Maybe it's not a kernel regression and something that is caused by some fluke,
  a broken initramfs (also known as initrd), new firmware files, or an updated
  userland software?

* Maybe it was a feature added to your distributor's kernel which vanilla Linux
  at that point never supported?

Note, if you found and fixed problems with the .config file, you want to use it
to build another kernel from the latest codebase, as your earlier tests with
mainline and the latest version from an affected stable/longterm series were most
likely flawed.

[:ref:`back to step-by-step guide <recheckworking_bissbs>`]

.. _bisectstart_bisref:

Start the bisection
-------------------

  *Start the bisection and tell Git about the versions earlier established as
  'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`]

This will start the bisection process; the last of the commands will make Git
check out a commit round about half-way between the 'good' and the 'bad' changes
for you to test.

[:ref:`back to step-by-step guide <bisectstart_bissbs>`]

.. _bisectbuild_bisref:

Build a kernel from the bisection point
---------------------------------------

  *Build, install, and boot a kernel from the code Git checked out using the
  same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`]

There are two things worth of note here:

* Occasionally building the kernel will fail or it might not boot due some
  problem in the code at the bisection point. In that case run this command::

    git bisect skip

  Git will then check out another commit nearby which with a bit of luck should
  work better. Afterwards restart executing this step.

* Those slightly odd looking version identifiers can happen during bisections,
  because the Linux kernel subsystems prepare their changes for a new mainline
  release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus
  base them on a somewhat earlier point like 6.1-rc1 or even 6.0 -- and then
  get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This
  leads to those slightly odd looking version identifiers coming up during
  bisections.

[:ref:`back to step-by-step guide <bisectbuild_bissbs>`]

.. _bisecttest_bisref:

Bisection checkpoint
--------------------

  *Check if the feature that regressed works in the kernel you just built.*
  [:ref:`... <bisecttest_bissbs>`]

Ensure what you tell Git is accurate: getting it wrong just one time will bring
the rest of the bisection totally off course, hence all testing after that point
will be for nothing.

[:ref:`back to step-by-step guide <bisecttest_bissbs>`]

.. _bisectlog_bisref:

Put the bisection log away
--------------------------

  *Store Git's bisection log and the current .config file in a safe place.*
  [:ref:`... <bisectlog_bissbs>`]

As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will
render the end result of a bisection useless. In that case you'd normally have
to restart the bisection from scratch. The log can prevent that, as it might
allow someone to point out where a bisection likely went sideways -- and then
instead of testing ten or more kernels you might only have to build a few to
resolve things.

The .config file is put aside, as there is a decent chance that developers might
ask for it after you report the regression.

[:ref:`back to step-by-step guide <bisectlog_bissbs>`]

.. _revert_bisref:

Try reverting the culprit
-------------------------

  *Try reverting the culprit on top of the latest codebase to see if this fixes
  your regression.* [:ref:`... <revert_bissbs>`]

This is an optional step, but whenever possible one you should try: there is a
decent chance that developers will ask you to perform this step when you bring
the bisection result up. So give it a try, you are in the flow already, building
one more kernel shouldn't be a big deal at this point.

The step-by-step guide covers everything relevant already except one slightly
rare thing: did you bisected a regression that also happened with mainline using
a stable/longterm series, but Git failed to revert the commit in mainline? Then
try to revert the culprit in the affected stable/longterm series -- and if that
succeeds, test that kernel version instead.

[:ref:`back to step-by-step guide <revert_bissbs>`]


Supplementary tasks: cleanup during and after the bisection
-----------------------------------------------------------

.. _makeroom_bisref:

Cleaning up during the bisection
--------------------------------

  *To remove one of the kernels you installed, look up its 'kernelrelease'
  identifier.* [:ref:`... <makeroom_bissbs>`]

The kernels you install during this process are easy to remove later, as its
parts are only stored in two places and clearly identifiable. You thus do not
need to worry to mess up your machine when you install a kernel manually (and
thus bypass your distribution's packaging system): all parts of your kernels are
relatively easy to remove later.

One of the two places is a directory in /lib/modules/, which holds the modules
for each installed kernel. This directory is named after the kernel's release
identifier; hence, to remove all modules for one of the kernels you built,
simply remove its modules directory in /lib/modules/.

The other place is /boot/, where typically two up to five files will be placed
during installation of a kernel. All of them usually contain the release name in
their file name, but how many files and their exact names depend somewhat on
your distribution's installkernel executable and its initramfs generator. On
some distributions the ``kernel-install remove...`` command mentioned in the
step-by-step guide will delete all of these files for you while also removing
the menu entry for the kernel from your bootloader configuration. On others you
have to take care of these two tasks yourself. The following command should
interactively remove the three main files of a kernel with the release name
'6.0-rc1-local-gcafec0cacaca0'::

  rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0

Afterwards check for other files in /boot/ that have
'6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well.
Now remove the boot entry for the kernel from your bootloader's configuration;
the steps to do that vary quite a bit between Linux distributions.

Note, be careful with wildcards like '*' when deleting files or directories
for kernels manually: you might accidentally remove files of a 6.0.11 kernel
when all you want is to remove 6.0 or 6.0.1.

[:ref:`back to step-by-step guide <makeroom_bissbs>`]

Cleaning up after the bisection
-------------------------------

.. _finishingtouch_bisref:

  *Once you have finished the bisection, do not immediately remove anything
  you set up, as you might need a few things again.*
  [:ref:`... <finishingtouch_bissbs>`]

When you are really short of storage space removing the kernels as described in
the step-by-step guide might not free as much space as you would like. In that
case consider running ``rm -rf ~/linux/*`` as well now. This will remove the
build artifacts and the Linux sources, but will leave the Git repository
(~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the
sources back.

Removing the repository as well would likely be unwise at this point: there is a
decent chance developers will ask you to build another kernel to perform
additional tests. This is often required to debug an issue or check proposed
fixes. Before doing so you want to run the ``git fetch mainline`` command again
followed by ``git checkout mainline/master`` to bring your clone up to date and
checkout the latest codebase. Then apply the patch using ``git apply
<filename>`` or ``git am <filename>`` and build yet another kernel using the
familiar commands.

Additional tests are also the reason why you want to keep the
~/kernel-config-working file around for a few weeks.

[:ref:`back to step-by-step guide <finishingtouch_bissbs>`]


Additional reading material
===========================

Further sources
---------------

* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
  `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
  in the Git documentation.
* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
  from kernel developer Nathan Chancellor.
* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.

..
   end-of-content
..
   This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
   you spot a typo or small mistake, feel free to let him know directly and
   he'll fix it. You are free to do the same in a mostly informal way if you
   want to contribute changes to the text -- but for copyright reasons please CC
   linux-doc@vger.kernel.org and 'sign-off' your contribution as
   Documentation/process/submitting-patches.rst explains in the section 'Sign
   your work - the Developer's Certificate of Origin'.
..
   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
   of the file. If you want to distribute this text under CC-BY-4.0 only,
   please use 'The Linux kernel development community' for author attribution
   and link this as source:
   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst

..
   Note: Only the content of this RST file as found in the Linux kernel sources
   is available under CC-BY-4.0, as versions of this text that were processed
   (for example by the kernel's build system) might contain content taken from
   files which use a more restrictive license.