Commit 5c3d1d0a ("kbuild: add a tool to list files ignored by git")
added a new tool, scripts/list-gitignored. My intention was to create
source packages without cleaning the source tree, without relying on git.

Linus strongly objected to it, and suggested using 'git archive' instead.
[1] [2] [3]

This commit goes in that direction - Remove scripts/list-gitignored.c
and rewrites Makefiles and scripts to use 'git archive' for building
Debian and RPM source packages. It also makes 'make perf-tar*-src-pkg'
use 'git archive' again.

Going forward, building source packages is only possible in a git-managed
tree. Building binary packages does not require git.

[1]: https://lore.kernel.org/lkml/CAHk-=wi49sMaC7vY1yMagk7eqLK=1jHeHQ=yZ_k45P=xBccnmA@mail.gmail.com/
[2]: https://lore.kernel.org/lkml/CAHk-=wh5AixGsLeT0qH2oZHKq0FLUTbyTw4qY921L=PwYgoGVw@mail.gmail.com/
[3]: https://lore.kernel.org/lkml/CAHk-=wgM-W6Fu==EoAVCabxyX8eYBz9kNC88-tm9ExRQwA79UQ@mail.gmail.com/



Fixes: 5c3d1d0a ("kbuild: add a tool to list files ignored by git")
Fixes: e0ca1674 ("kbuild: make perf-tar*-src-pkg work without relying on git")
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Prepare to add more files to the source RPM.

Also, fix the build error when KCONFIG_CONFIG is set:
  error: Bad file: ./.config: No such file or directory

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

That commit required the use of filechk_kernel.release for the
kernelrelease Makefile target. It is currently only being set when
KBUILD_EXTMOD is not set. Make sure it is set in that case as well.

Fixes: 1cb86b6c ("kbuild: save overridden KERNELRELEASE in include/config/kernel.release")
Signed-off-by: Tzafrir Cohen <nvidia@cohens.org.il>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

In short, the motivation of this commit is to build a source package
without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before
creating a source tarball. Otherwise build artifacts such as *.o,
*.a, etc. would be included in the tarball. Yet, the tarball ends up
containing several garbage files since 'make clean' does not clean
everything.

Cleaning the tree every time is annoying since it makes the incremental
build impossible. It is desirable to create a source tarball without
cleaning the tree.

In fact, there are some ways to achieve this.

The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses
it, but I do not like it because it works only when the source tree is
managed by git, and all files you want in the tarball must be committed
in advance.

I want to make it work without relying on git. We can do this.

Files that are ignored by git are generated files, so should be excluded
from the source tarball. We can list them out by parsing the .gitignore
files. Of course, .gitignore does not cover all the cases, but it works
well enough.

tar(1) claims to support it:

  --exclude-vcs-ignores

    Exclude files that match patterns read from VCS-specific ignore files.
    Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.

The best scenario would be to use 'tar --exclude-vcs-ignores', but this
option does not work. --exclude-vcs-ignore does not understand any of
the negation (!), preceding slash, following slash, etc.. So, this option
is just useless.

Hence, I wrote this gitignore parser. The previous version [1], written
in Python, was so slow. This version is implemented in C, so it works
much faster.

I imported the code from git (commit: 23c56f7bd5f1), so we get the same
result.

This tool traverses the source tree, parsing all .gitignore files, and
prints file paths that are ignored by git.

The output is similar to 'git ls-files --ignored --directory --others
--exclude-per-directory=.gitignore', except

  [1] Not sorted
  [2] No trailing slash for directories

[2] is intentional because tar's --exclude-from option cannot handle
trailing slashes.

[How to test this tool]

  $ git clean -dfx
  $ make -s -j$(nproc) defconfig all                       # or allmodconifg or whatever
  $ git archive -o ../linux1.tar --prefix=./ HEAD
  $ tar tf ../linux1.tar | LANG=C sort > ../file-list1     # files emitted by 'git archive'
  $ make scripts_package
    HOSTCC  scripts/list-gitignored
  $ scripts/list-gitignored  --prefix=./ -o ../exclude-list
  $ tar cf ../linux2.tar --exclude-from=../exclude-list .
  $ tar tf ../linux2.tar | LANG=C sort > ../file-list2     # files emitted by 'tar'
  $ diff  ../file-list1 ../file-list2 | grep -E '^(<|>)'
  < ./Documentation/devicetree/bindings/.yamllint
  < ./drivers/clk/.kunitconfig
  < ./drivers/gpu/drm/tests/.kunitconfig
  < ./drivers/hid/.kunitconfig
  < ./fs/ext4/.kunitconfig
  < ./fs/fat/.kunitconfig
  < ./kernel/kcsan/.kunitconfig
  < ./lib/kunit/.kunitconfig
  < ./mm/kfence/.kunitconfig
  < ./tools/testing/selftests/arm64/tags/
  < ./tools/testing/selftests/arm64/tags/.gitignore
  < ./tools/testing/selftests/arm64/tags/Makefile
  < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
  < ./tools/testing/selftests/arm64/tags/tags_test.c
  < ./tools/testing/selftests/kvm/.gitignore
  < ./tools/testing/selftests/kvm/Makefile
  < ./tools/testing/selftests/kvm/config
  < ./tools/testing/selftests/kvm/settings

The source tarball contains most of files that are tracked by git. You
see some diffs, but it is just because some .gitignore files are wrong.

  $ git ls-files -i -c --exclude-per-directory=.gitignore
  Documentation/devicetree/bindings/.yamllint
  drivers/clk/.kunitconfig
  drivers/gpu/drm/tests/.kunitconfig
  drivers/hid/.kunitconfig
  fs/ext4/.kunitconfig
  fs/fat/.kunitconfig
  kernel/kcsan/.kunitconfig
  lib/kunit/.kunitconfig
  mm/kfence/.kunitconfig
  tools/testing/selftests/arm64/tags/.gitignore
  tools/testing/selftests/arm64/tags/Makefile
  tools/testing/selftests/arm64/tags/run_tags_test.sh
  tools/testing/selftests/arm64/tags/tags_test.c
  tools/testing/selftests/kvm/.gitignore
  tools/testing/selftests/kvm/Makefile
  tools/testing/selftests/kvm/config
  tools/testing/selftests/kvm/settings

[1]: https://lore.kernel.org/all/20230128173843.765212-1-masahiroy@kernel.org/



Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

rust-project.json is the configuration file used by rust-analyzer.
As it is a configuration file and it is not needed to build external
modules, it should be delete by make clean. So, delete rust-project.json
when running make clean.

Link: https://github.com/Rust-for-Linux/linux/issues/939


Suggested-by: Björn Roy Baron <bjorn3_gh@protonmail.com>
Signed-off-by: Maíra Canal <mcanal@igalia.com>
Reviewed-by: Finn Behrens <fin@nyantec.com>
Acked-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>

I added $(srctree)/ to some included Makefiles in the following commits:

 - 3204a7fb ("kbuild: prefix $(srctree)/ to some included Makefiles")
 - d8285639 ("kbuild: do not require sub-make for separate output tree builds")

They were a preparation for removing --include-dir flag.

I have never thought --include-dir useful. Rather, it _is_ harmful.

For example, run the following commands:

  $ make -s ARCH=x86 mrproper defconfig
  $ make ARCH=arm O=foo dtbs
  make[1]: Entering directory '/tmp/linux/foo'
    HOSTCC  scripts/basic/fixdep
  Error: kernelrelease not valid - run 'make prepare' to update it
    UPD     include/config/kernel.release
  make[1]: Leaving directory '/tmp/linux/foo'

The first command configures the source tree for x86. The next command
tries to build ARM device trees in the separate foo/ directory - this
must stop because the directory foo/ has not been configured yet.

However, due to --include-dir=$(abs_srctree), the top Makefile includes
the wrong include/config/auto.conf from the source tree and continues
building. Kbuild traverses the directory tree, but of course it does
not work correctly. The Error message is also pointless - 'make prepare'
does not help at all for fixing the issue.

This commit fixes more arch Makefile, and finally removes --include-dir
from the top Makefile.

There are more breakages under drivers/, but I do not volunteer to fix
them all. I just moved --include-dir to drivers/Makefile.

With this commit, the second command will stop with a sensible message.

  $ make -s ARCH=x86 mrproper defconfig
  $ make ARCH=arm O=foo dtbs
  make[1]: Entering directory '/tmp/linux/foo'
    SYNC    include/config/auto.conf.cmd
  ***
  *** The source tree is not clean, please run 'make ARCH=arm mrproper'
  *** in /tmp/linux
  ***
  make[2]: *** [../Makefile:646: outputmakefile] Error 1
  /tmp/linux/Makefile:770: include/config/auto.conf.cmd: No such file or directory
  make[1]: *** [/tmp/linux/Makefile:793: include/config/auto.conf.cmd] Error 2
  make[1]: Leaving directory '/tmp/linux/foo'
  make: *** [Makefile:226: __sub-make] Error 2

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Add missing underscore in CONFIG_DEBUG_INFO_BTF_MODULES.

Fixes: f73edc89 ("kbuild: unify two modpost invocations")
Signed-off-by: Carlos Llamas <cmllamas@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

After commit 8d9acfce ("kbuild: Stop using '-Qunused-arguments' with
clang"), the PowerPC vDSO shows the following error with clang-13 and
older when CONFIG_INIT_STACK_ALL_ZERO is enabled:

  clang: error: argument unused during compilation: '-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang' [-Werror,-Wunused-command-line-argument]

clang-14 added a change to make sure this flag never triggers
-Wunused-command-line-argument, so it is fixed with newer releases. For
older releases that the kernel still supports building with, just filter
out this flag, as has been done for other flags.

Fixes: f0a42fba ("powerpc/vdso: Improve linker flags")
Fixes: 8d9acfce ("kbuild: Stop using '-Qunused-arguments' with clang")
Link: https://github.com/llvm/llvm-project/commit/ca6d5813d17598cd180995fb3bdfca00f364475f


Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

${KERNELRELEASE} is used as a part of the installation path.
(INSTALL_DTBS_PATH, MODLIB, etc.)

When KERNELRELEASE is overridden from the command line, it should be
saved in include/config/kernel.release, so that it will be consistently
used for the installation steps.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Print $(KERNELVERSION) in setlocalversion so that the callers get
simpler.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

This option masks all unused command line argument warnings, which can
hide potential issues, such as an architecture Makefile adding an
unsupported flag to KBUILD_AFLAGS or KBUILD_CFLAGS, which will cause all
as-option and cc-options to silently fail due to -Werror with no
indication as to why in the main kernel build.

Remove this flag so that warnings of this nature can be caught early and
obviously in a build.

Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Tested-by: Linux Kernel Functional Testing <lkft@linaro.org>
Tested-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Reuse the standard naming schema for temporary files also for temporary
directories.

Such a directory will be used by the kheaders generation.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
Tested-by: Nicolas Schier <nicolas@fjasle.eu>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

scripts/ is a better place to generate files used treewide.

With target.json moved to scripts/, you do not need to add target.json
to no-clean-files or MRPROPER_FILES.

'make clean' does not visit scripts/, but 'make mrproper' does.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Miguel Ojeda <ojeda@kernel.org>
Tested-by: Miguel Ojeda <ojeda@kernel.org>

The top .gitignore comments about how to detect files breaking
.gitignore rules, but people rarely care about it.

Add a new W=1 warning to detect files that are tracked but ignored by
git. If git is not installed or the source tree is not tracked by git
at all, this script does not print anything.

Running it on v6.2-rc1 detected the following:

  $ make W=1 misc-check
  Documentation/devicetree/bindings/.yamllint: warning: ignored by one of the .gitignore files
  drivers/clk/.kunitconfig: warning: ignored by one of the .gitignore files
  drivers/gpu/drm/tests/.kunitconfig: warning: ignored by one of the .gitignore files
  drivers/hid/.kunitconfig: warning: ignored by one of the .gitignore files
  fs/ext4/.kunitconfig: warning: ignored by one of the .gitignore files
  fs/fat/.kunitconfig: warning: ignored by one of the .gitignore files
  kernel/kcsan/.kunitconfig: warning: ignored by one of the .gitignore files
  lib/kunit/.kunitconfig: warning: ignored by one of the .gitignore files
  mm/kfence/.kunitconfig: warning: ignored by one of the .gitignore files
  tools/testing/selftests/arm64/tags/.gitignore: warning: ignored by one of the .gitignore files
  tools/testing/selftests/arm64/tags/Makefile: warning: ignored by one of the .gitignore files
  tools/testing/selftests/arm64/tags/run_tags_test.sh: warning: ignored by one of the .gitignore files
  tools/testing/selftests/arm64/tags/tags_test.c: warning: ignored by one of the .gitignore files

These are ignored by the '.*' or 'tags' in the top .gitignore, but
there is no rule to negate it.

You might be tempted to do 'git add -f' but I want to have the real
issue fixed (by fixing a .gitignore, or by renaming files, etc.).

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

The top Makefile sets KBUILD_VERBOSE to 0 by default, it looks weird
now because V=1 and V=2 can be OR'ed as V=12. The default should be
empty.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

Commit a6de553d ("kbuild: Allow to combine multiple W= levels")
supported W=123 to enable all the extra warning groups.

I think a similar idea is applicable to the V= option.

  V=1 echos the whole command
  V=2 prints the reason for rebuilding

These are orthogonal, and can be enabled at the same time.

This commit supports V=12 to enable both of them.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Nicolas Schier <nicolas@fjasle.eu>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

"make V=1" prints the whole command instead of the short log, but I
think it is nicer to print both so that you can easily spot the build
rule of your interest.

This commit changes V=1 to print the short log (the line starts with
'#'), followed by the full log.

In parallel builds, the short/full logs from the same build rule may
be interspersed. If you want to avoid it, please add -Otarget option.
Kbuild will never set it by default because Make would buffer the logs
and lose the escape sequences. (Modern compilers print warnings and
errors in color, but only when they write to a terminal.)

This is also a preparation for supporting V=12 because V=2 appends the
reason for rebuilding to the short log.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Nicolas Schier <nicolas@fjasle.eu>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

Factor out $(findstring s,...).

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

When CONFIG_DEBUG_INFO_BTF_MODULES=y, running 'make modules'
in the clean kernel tree will get the following error.

  $ grep CONFIG_DEBUG_INFO_BTF_MODULES .config
  CONFIG_DEBUG_INFO_BTF_MODULES=y
  $ make -s clean
  $ make modules
    [snip]
    AR      vmlinux.a
  ar: ./built-in.a: No such file or directory
  make: *** [Makefile:1241: vmlinux.a] Error 1

'modules' depends on 'vmlinux', but builtin objects are not built.

Define KBUILD_BUILTIN.

Fixes: f73edc89 ("kbuild: unify two modpost invocations")
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Nathan Chancellor reports that $(NM) emits an error message when
GNU Make 4.4 is used to build the ARM zImage.

  $ make-4.4 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- O=build defconfig zImage
    [snip]
    LD      vmlinux
    NM      System.map
    SORTTAB vmlinux
    OBJCOPY arch/arm/boot/Image
    Kernel: arch/arm/boot/Image is ready
  arm-linux-gnueabi-nm: 'arch/arm/boot/compressed/../../../../vmlinux': No such file
  /bin/sh: 1: arithmetic expression: expecting primary: " "
    LDS     arch/arm/boot/compressed/vmlinux.lds
    AS      arch/arm/boot/compressed/head.o
    GZIP    arch/arm/boot/compressed/piggy_data
    AS      arch/arm/boot/compressed/piggy.o
    CC      arch/arm/boot/compressed/misc.o

This occurs since GNU Make commit 98da874c4303 ("[SV 10593] Export
variables to $(shell ...) commands"), and the O= option is needed to
reproduce it. The generated zImage is correct despite the error message.

As the commit description of 98da874c4303 [1] says, exported variables
are passed down to $(shell ) functions, which means exported recursive
variables might be expanded earlier than before, in the parse stage.

The following test code demonstrates the change for GNU Make 4.4.

[Test Makefile]

  $(shell echo hello > foo)
  export foo = $(shell cat bar/../foo)
  $(shell mkdir bar)

  all:
          @echo $(foo)

[GNU Make 4.3]

  $ rm -rf bar; make-4.3
  hello

[GNU Make 4.4]

  $ rm -rf bar; make-4.4
  cat: bar/../foo: No such file or directory
  hello

The 'foo' is a resursively expanded (i.e. lazily expanded) variable.

GNU Make 4.3 expands 'foo' just before running the recipe '@echo $(foo)',
at this point, the directory 'bar' exists.

GNU Make 4.4 expands 'foo' to evaluate $(shell mkdir bar) because it is
exported. At this point, the directory 'bar' does not exit yet. The cat
command cannot resolve the bar/../foo path, hence the error message.

Let's get back to the kernel Makefile.

In arch/arm/boot/compressed/Makefile, KBSS_SZ is referenced by
LDFLAGS_vmlinux, which is recursive and also exported by the top
Makefile.

GNU Make 4.3 expands KBSS_SZ just before running the recipes, so no
error message.

GNU Make 4.4 expands KBSS_SZ in the parse stage, where the directory
arm/arm/boot/compressed does not exit yet. When compiled with O=,
the output directory is created by $(shell mkdir -p $(obj-dirs))
in scripts/Makefile.build.

There are two ways to fix this particular issue:

 - change "$(obj)/../../../../vmlinux" in KBSS_SZ to "vmlinux"
 - unexport LDFLAGS_vmlinux

This commit takes the latter course because it is what I originally
intended.

Commit 3ec8a5b3 ("kbuild: do not export LDFLAGS_vmlinux")
unexported LDFLAGS_vmlinux.

Commit 5d4aeffb ("kbuild: rebuild .vmlinux.export.o when its
prerequisite is updated") accidentally exported it again.

We can clean up arch/arm/boot/compressed/Makefile later.

[1]: https://git.savannah.gnu.org/cgit/make.git/commit/?id=98da874c43035a490cdca81331724f233a3d0c9a

Link: https://lore.kernel.org/all/Y7i8+EjwdnhHtlrr@dev-arch.thelio-3990X/


Fixes: 5d4aeffb ("kbuild: rebuild .vmlinux.export.o when its prerequisite is updated")
Reported-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
Tested-by: Nathan Chancellor <nathan@kernel.org>

The single *.ko build is broken since commit f65a4868 ("kbuild:
change module.order to list *.o instead of *.ko").

Fixes: f65a4868 ("kbuild: change module.order to list *.o instead of *.ko")
Reported-by: Marc Kleine-Budde <mkl@pengutronix.de>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Marc Kleine-Budde <mkl@pengutronix.de>

This was previously alphabetically sorted. Sort it again.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Miguel Ojeda <ojeda@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>

While not documented, schema checks for single dtb targets mostly work
already by setting 'CHECK_DTBS=1'. However, the dependencies are not
handled and it only works if 'make dt_bindings_check' was run first and
generated processed-schema.json. In addition, changing a binding file
doesn't cause the schema to be rebuilt and dtb to be revalidated.

Making this work turns out to be simple. Whenever CHECK_DTBS is set,
make 'dt_binding_check' a 'dtbs_prepare' dependency.

I reimplemented here what Masahiro had originally come up with a while
back.

Suggested-by: Masahiro Yamada <masahiroy@kernel.org>
Acked-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Tested-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Tested-by: Marek Vasut <marex@denx.de>
Link: https://lore.kernel.org/r/20221220013233.2890335-1-robh@kernel.org


Signed-off-by: Rob Herring <robh@kernel.org>

Documentation/process/changes.rst notes the minimal GNU Make version,
but it is not checked anywhere.

We could check $(MAKE_VERSION), but another simple way is to check
$(.FEATURES) since the feature list always grows.

GNU Make 3.81 expands $(.FEATURES) to:
  target-specific order-only second-expansion else-if archives jobserver check-symlink

GNU Make 3.82 expands $(.FEATURES) to:
  target-specific order-only second-expansion else-if shortest-stem undefine archives jobserver check-symlink

To ensure Make >= 3.82, you can check either 'shortest-stem' or
'undefine'.

This way is not always possible. For example, Make 4.0 through 4.2 have
the same set of $(.FEATURES). At that point, we will need to come up
with a different approach.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

scripts/Makefile.build replaces the suffix .o with .ko, then
scripts/Makefile.modpost calls the sed command to change .ko back
to the original .o suffix.

Instead of converting the suffixes back-and-forth, store the .o paths
in modules.order, and replace it with .ko in 'make modules_install'.

This avoids the unneeded sed command.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Luis Chamberlain <mcgrof@kernel.org>