verify-bugs-and-bisect-regressions.rst

[:ref:`back to step-by-step guide <bisectlog_bissbs>`]

.. _revert_bisref:

Try reverting the culprit
~~~~~~~~~~~~~~~~~~~~~~~~~

  *Try reverting the culprit on top of the latest codebase to see if this fixes
  your regression.* [:ref:`... <revert_bissbs>`]

This is an optional step, but whenever possible one you should try: there is a
decent chance that developers will ask you to perform this step when you bring
the bisection result up. So give it a try, you are in the flow already, building
one more kernel shouldn't be a big deal at this point.

The step-by-step guide covers everything relevant already except one slightly
rare thing: did you bisected a regression that also happened with mainline using
a stable/longterm series, but Git failed to revert the commit in mainline? Then
try to revert the culprit in the affected stable/longterm series -- and if that
succeeds, test that kernel version instead.

[:ref:`back to step-by-step guide <revert_bissbs>`]

Cleanup steps during and after following this guide
---------------------------------------------------

  *During and after following this guide you might want or need to remove some
  of the kernels you installed.* [:ref:`... <introclosure_bissbs>`]

The steps in this section describe clean-up procedures.

[:ref:`back to step-by-step guide <introclosure_bissbs>`].

.. _makeroom_bisref:

Cleaning up during the bisection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  *To remove one of the kernels you installed, look up its 'kernelrelease'
  identifier.* [:ref:`... <makeroom_bissbs>`]

The kernels you install during this process are easy to remove later, as its
parts are only stored in two places and clearly identifiable. You thus do not
need to worry to mess up your machine when you install a kernel manually (and
thus bypass your distribution's packaging system): all parts of your kernels are
relatively easy to remove later.

One of the two places is a directory in /lib/modules/, which holds the modules
for each installed kernel. This directory is named after the kernel's release
identifier; hence, to remove all modules for one of the kernels you built,
simply remove its modules directory in /lib/modules/.

The other place is /boot/, where typically two up to five files will be placed
during installation of a kernel. All of them usually contain the release name in
their file name, but how many files and their exact names depend somewhat on
your distribution's installkernel executable and its initramfs generator. On
some distributions the ``kernel-install remove...`` command mentioned in the
step-by-step guide will delete all of these files for you while also removing
the menu entry for the kernel from your bootloader configuration. On others you
have to take care of these two tasks yourself. The following command should
interactively remove the three main files of a kernel with the release name
'6.0-rc1-local-gcafec0cacaca0'::

  rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0

Afterwards check for other files in /boot/ that have
'6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well.
Now remove the boot entry for the kernel from your bootloader's configuration;
the steps to do that vary quite a bit between Linux distributions.

Note, be careful with wildcards like '*' when deleting files or directories
for kernels manually: you might accidentally remove files of a 6.0.13 kernel
when all you want is to remove 6.0 or 6.0.1.

[:ref:`back to step-by-step guide <makeroom_bissbs>`]

Cleaning up after the bisection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. _finishingtouch_bisref:

  *Once you have finished the bisection, do not immediately remove anything
  you set up, as you might need a few things again.*
  [:ref:`... <finishingtouch_bissbs>`]

When you are really short of storage space removing the kernels as described in
the step-by-step guide might not free as much space as you would like. In that
case consider running ``rm -rf ~/linux/*`` as well now. This will remove the
build artifacts and the Linux sources, but will leave the Git repository
(~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the
sources back.

Removing the repository as well would likely be unwise at this point: there
is a decent chance developers will ask you to build another kernel to
perform additional tests -- like testing a debug patch or a proposed fix.
Details on how to perform those can be found in the section :ref:`Optional
tasks: test reverts, patches, or later versions <introoptional_bissbs>`.

Additional tests are also the reason why you want to keep the
~/kernel-config-working file around for a few weeks.

[:ref:`back to step-by-step guide <finishingtouch_bissbs>`]

.. _introoptional_bisref:

Test reverts, patches, or later versions
----------------------------------------

  *While or after reporting a bug, you might want or potentially will be asked
  to test reverts, patches, proposed fixes, or other versions.*
  [:ref:`... <introoptional_bissbs>`]

All the commands used in this section should be pretty straight forward, so
there is not much to add except one thing: when setting a kernel tag as
instructed, ensure it is not much longer than the one used in the example, as
problems will arise if the kernelrelease identifier exceeds 63 characters.

[:ref:`back to step-by-step guide <introoptional_bissbs>`].


Additional information
======================

.. _buildhost_bis:

Build kernels on a different machine
------------------------------------

To compile kernels on another system, slightly alter the step-by-step guide's
instructions:

* Start following the guide on the machine where you want to install and test
  the kernels later.

* After executing ':ref:`Boot into the working kernel and briefly use the
  apparently broken feature <bootworking_bissbs>`', save the list of loaded
  modules to a file using ``lsmod > ~/test-machine-lsmod``. Then locate the
  build configuration for the running kernel (see ':ref:`Start defining the
  build configuration for your kernel <oldconfig_bisref>`' for hints on where
  to find it) and store it as '~/test-machine-config-working'. Transfer both
  files to the home directory of your build host.

* Continue the guide on the build host (e.g. with ':ref:`Ensure to have enough
  free space for building [...] <diskspace_bissbs>`').

* When you reach ':ref:`Start preparing a kernel build configuration[...]
  <oldconfig_bissbs>`': before running ``make olddefconfig`` for the first time,
  execute the following command to base your configuration on the one from the
  test machine's 'working' kernel::

    cp ~/test-machine-config-working ~/linux/.config

* During the next step to ':ref:`disable any apparently superfluous kernel
  modules <localmodconfig_bissbs>`' use the following command instead::

    yes '' | make localmodconfig LSMOD=~/lsmod_foo-machine localmodconfig

* Continue the guide, but ignore the instructions outlining how to compile,
  install, and reboot into a kernel every time they come up. Instead build
  like this::

    cp ~/kernel-config-working .config
    make olddefconfig &&
    make -j $(nproc --all) targz-pkg

  This will generate a gzipped tar file whose name is printed in the last
  line shown; for example, a kernel with the kernelrelease identifier
  '6.0.0-rc1-local-g928a87efa423' built for x86 machines usually will
  be stored as '~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz'.

  Copy that file to your test machine's home directory.

* Switch to the test machine to check if you have enough space to hold another
  kernel. Then extract the file you transferred::

    sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C /

  Afterwards :ref:`generate the initramfs and add the kernel to your boot
  loader's configuration <install_bisref>`; on some distributions the following
  command will take care of both these tasks::

    sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6.0.0-rc1-local-g928a87efa423

  Now reboot and ensure you started the intended kernel.

This approach even works when building for another architecture: just install
cross-compilers and add the appropriate parameters to every invocation of make
(e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]``).

Additional reading material
---------------------------

* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
  `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
  in the Git documentation.
* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
  from kernel developer Nathan Chancellor.
* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.

..
   end-of-content
..
   This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
   you spot a typo or small mistake, feel free to let him know directly and
   he'll fix it. You are free to do the same in a mostly informal way if you
   want to contribute changes to the text -- but for copyright reasons please CC
   linux-doc@vger.kernel.org and 'sign-off' your contribution as
   Documentation/process/submitting-patches.rst explains in the section 'Sign
   your work - the Developer's Certificate of Origin'.
..
   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
   of the file. If you want to distribute this text under CC-BY-4.0 only,
   please use 'The Linux kernel development community' for author attribution
   and link this as source:
   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst

..
   Note: Only the content of this RST file as found in the Linux kernel sources
   is available under CC-BY-4.0, as versions of this text that were processed
   (for example by the kernel's build system) might contain content taken from
   files which use a more restrictive license.