Python retired in 2020, and some distributions do not provide the
'python' command any more.

As in commit 51839e29 ("scripts: switch explicitly to Python 3"),
we need to use more specific 'python3' to invoke scripts even if they
are written in a way compatible with both Python 2 and 3.

This commit removes the variable 'PYTHON', and switches the existing
users to 'PYTHON3'.

BTW, PEP 394 (https://www.python.org/dev/peps/pep-0394/

) is a helpful
material.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Section 3.11 was incorrectly called 3.9, fix it.

Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

There is no explanation about subdir-y.

Let's document it.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>

The difference between extra-y and always-y is obscure.

Basically, Kbuild builds targets listed in extra-y and always-y in
visited Makefiles without relying on any dependency.

The difference is that extra-y is used to list the targets needed for
vmlinux whereas always-y is used to list the targets that must be always
built irrespective of final targets.

Kbuild skips extra-y when it is building only modules (i.e.
'make modules'). This is the long-standing behavior since extra-y was
introduced in 2003, and it is explained in that commit log [1].

For clarification, this is the extra-y vs always-y table:

                  extra-y    always-y
  'make'             y          y
  'make vmlinux'     y          y
  'make modules'     n          y

Kbuild skips extra-y also when building external modules since obviously
it never builds vmlinux.

Unfortunately, extra-y is wrongly used in many places of upstream code,
and even in external modules.

Using extra-y in external module Makefiles is wrong. What you should
use is probably always-y or 'targets'.

The current documentation for extra-y is misleading. I rewrote it, and
moved it to the section 3.7.

always-y is not documented anywhere. I added.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/commit/?id=f94e5fd7e5d09a56a60670a9bb211a791654bba8



Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>

The if_changed macro is currently explained in the section
"Commands useful for building a boot image", but the use of
if_changed is not limited to the boot image.

It is often used together with custom rules. Let's split it as a
separate section, and insert it after the "Custom Rules" section.

I slightly reworded the explanation, re-numbered to fill the <deleted>
section, and also fixed the broken indentation of the Note: part.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The two sections "3.10 Special Rules" and "7.8 Custom kbuild commands"
are related because you must understand both of them when you write
custom rules.

Actually I do not understand the policy about what to go into
"3 The kbuild files" and what into "7 Architecture Makefile".

This commit reworks the custom rule explanation as follows:

 - Merged "7.8 Custom kbuild commands" into "3.10 Special Rules".

 - Reword "Special Rules" to "Custom Rules" for consistency.

 - Update the example for kecho because the blackfin Makefile
   does not exist any more.

 - Replace the example for cmd_<command> with a simpler one.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Fix stale information:

 - Fix the section number in the reference from 6.4 to 7.4.

 - Remove init-y and net-y. They were removed by commit 23febe37
   ("kbuild: merge init-y into core-y") and commit 95fb6317
   ("kbuild: merge net-y and virt-y into drivers-y"), respectively.

 - Update the example because arch/sparc64/Makefile does not exit.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Precisely speaking, the arch directory is specified by $(SRCARCH),
not $(ARCH).

In old days, $(ARCH) actually matched to the arch directory because
32-bit and 64-bit were supported as separate architectures.

Most architectures (except arm/arm64) were unified like follows:

    arch/i386, arch/x86_64    ->  arch/x86
    arch/sh, arch/sh64        ->  arch/sh
    arch/sparc, arch/sparc64  ->  arch/sparc

To not break the user interface, commit 6752ed90 ("Kbuild: allow
arch/xxx to use a different source path") introduced SRCARCH to point
to the arch directory, still allowing to pass in the former ARCH=i386
or ARCH=x86_64.

Update the documents for preciseness, and add the explanation of SRCARCH.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>

This line was written in 2003. Now we have much more Makefiles.

The number of Makefiles is not important. The point is we have a
Makefile in (almost) every directory.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

During an investigation to fix up the execute bits of scripts in the
kernel repository, Andrew Morton and Kees Cook pointed out that the
execute bit should not matter, and that build scripts cannot rely on that.
Kees could not point to any documentation, though.

Masahiro Yamada explained the convention of setting execute bits to make
it easier for manual script invocation.

Provide some basic documentation how the build shall invoke scripts, such
that the execute bits do not matter, and acknowledge that execute bits are
useful nonetheless.

This serves as reference for further clean-up patches in the future.

Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Ujjwal Kumar <ujjwalkumar0501@gmail.com>
Cc: Lukas Bulwahn <lukas.bulwahn@gmail.com>
Link: https://lore.kernel.org/lkml/20200830174409.c24c3f67addcce0cea9a9d4c@linux-foundation.org/
Link: https://lore.kernel.org/lkml/202008271102.FEB906C88@keescook/
Link: https://lore.kernel.org/linux-kbuild/CAK7LNAQdrvMkDA6ApDJCGr+5db8SiPo=G+p8EiOvnnGvEN80gA@mail.gmail.com/
Link: https://lkml.kernel.org/r/20201001075723.24246-1-lukas.bulwahn@gmail.com


Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

This is a general cleanup of kbuild/makefiles.rst:

* Use "Chapter" for major heading references and use "section" for
  the next-level heading references, for consistency.
* Section 3.8 was deleted long ago.
* Drop the ending ':' in section names in the contents list.
* Correct some section numbering references.
* Correct verb agreement typo.
* Fix run-on sentence punctuation.

Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

To build host programs, you need to add the program names to 'hostprogs'
to use the necessary build rule, but it is not enough to build them
because there is no dependency.

There are two types of host programs: built as the prerequisite of
another (e.g. gen_crc32table in lib/Makefile), or always built when
Kbuild visits the Makefile (e.g. genksyms in scripts/genksyms/Makefile).

The latter is typical in Makefiles under scripts/, which contains host
programs globally used during the kernel build. To build them, you need
to add them to both 'hostprogs' and 'always-y'.

This commit adds hostprogs-always-y as a shorthand.

The same applies to user programs. net/bpfilter/Makefile builds
bpfilter_umh on demand, hence always-y is unneeded. In contrast,
programs under samples/ are added to both 'userprogs' and 'always-y'
so they are always built when Kbuild visits the Makefiles.

userprogs-always-y works as a shorthand.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Acked-by: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>

CFLAGS_REMOVE_<file>.o filters out flags when compiling a particular
object, but there is no convenient way to do that for every object in
a directory.

Add ccflags-remove-y and asflags-remove-y to make it easily.

Use ccflags-remove-y to clean up some Makefiles.

The add/remove order works as follows:

 [1] KBUILD_CFLAGS specifies compiler flags used globally

 [2] ccflags-y adds compiler flags for all objects in the
     current Makefile

 [3] ccflags-remove-y removes compiler flags for all objects in the
     current Makefile (New feature)

 [4] CFLAGS_<file> adds compiler flags per file.

 [5] CFLAGS_REMOVE_<file> removes compiler flags per file.

Having [3] before [4] allows us to remove flags from most (but not all)
objects in the current Makefile.

For example, kernel/trace/Makefile removes $(CC_FLAGS_FTRACE)
from all objects in the directory, then adds it back to
trace_selftest_dynamic.o and CFLAGS_trace_kprobe_selftest.o

The same applies to lib/livepatch/Makefile.

Please note ccflags-remove-y has no effect to the sub-directories.
In contrast, the previous notation got rid of compiler flags also from
all the sub-directories.

The following are not affected because they have no sub-directories:

  arch/arm/boot/compressed/
  arch/powerpc/xmon/
  arch/sh/
  kernel/trace/

However, lib/ has several sub-directories.

To keep the behavior, I added ccflags-remove-y to all Makefiles
in subdirectories of lib/, except the following:

  lib/vdso/Makefile        - Kbuild does not descend into this Makefile
  lib/raid/test/Makefile   - This is not used for the kernel build

I think commit 2464a609 ("ftrace: do not trace library functions")
excluded too much. In the next commit, I will remove ccflags-remove-y
from the sub-directories of lib/.

Suggested-by: Sami Tolvanen <samitolvanen@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Acked-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
Acked-by: Brendan Higgins <brendanhiggins@google.com> (KUnit)
Tested-by: Anders Roxell <anders.roxell@linaro.org>

Commit d503ac53 ("kbuild: rename LDFLAGS to KBUILD_LDFLAGS") missed
to update the documentation.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Kbuild now supports the syntax 'userprogs' to compile userspace
programs for the same architecture as the kernel.

Insert the section '5 Userspace Program support' to explain it.

I copy-pasted '4 Host Program support' and fixed it up.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Acked-by: Sam Ravnborg <sam@ravnborg.org>

Make it consistent with the other sections.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

All the files in Documentation/kbuild/ were converted to reST.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

obj-* needs a trailing slash for a directory, but subdir-* does not.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

This sentence does not make sense in the section about mandatory-y.

This seems to be a copy-paste mistake of commit fcc8487d ("uapi:
export all headers under uapi directories").

The correct description would be "The convention is to list one
mandatory-y per line ...".

I just removed it instead of fixing it. If such information is needed,
it could be commented in include/asm-generic/Kbuild and
include/uapi/asm-generic/Kbuild.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

In old days, the "host-progs" syntax was used for specifying host
programs. It was renamed to the current "hostprogs-y" in 2004.

It is typically useful in scripts/Makefile because it allows Kbuild to
selectively compile host programs based on the kernel configuration.

This commit renames like follows:

  always       ->  always-y
  hostprogs-y  ->  hostprogs

So, scripts/Makefile will look like this:

  always-$(CONFIG_BUILD_BIN2C) += ...
  always-$(CONFIG_KALLSYMS)    += ...
      ...
  hostprogs := $(always-y) $(always-m)

I think this makes more sense because a host program is always a host
program, irrespective of the kernel configuration. We want to specify
which ones to compile by CONFIG options, so always-y will be handier.

The "always", "hostprogs-y", "hostprogs-m" will be kept for backward
compatibility for a while.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The difference between "always" and "extra-y" is that the targets
listed in $(always) are always built, whereas the ones in $(extra-y)
are built only when KBUILD_BUILTIN is set.

So, "make modules" does not build the targets in $(extra-y).

vmlinux.lds is only needed for linking vmlinux. So, adding it to extra-y
is more correct. In fact, arch/x86/kernel/Makefile does this.

Fix the example code.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Kbuild descends into a directory by either 'y' or 'm', but there is an
important difference.

Kbuild combines the built-in objects into built-in.a in each directory.
The built-in.a in the directory visited by obj-y is merged into the
built-in.a in the parent directory. This merge happens recursively
when Kbuild is ascending back towards the top directory, then built-in
objects are linked into vmlinux eventually. This works properly only
when the Makefile specifying obj-y is reachable by the chain of obj-y.

On the other hand, Kbuild does not take built-in.a from the directory
visited by obj-m. This it, all the objects in that directory are
supposed to be modular. If Kbuild descends into a directory by obj-m,
but the Makefile in the sub-directory specifies obj-y, those objects
are just left orphan.

The current statement "Kbuild only uses this information to decide that
it needs to visit the directory" is misleading. Clarify the difference.

Reported-by: Johan Hovold <johan@kernel.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Johan Hovold <johan@kernel.org>

There are both positive and negative options about this feature.
At first, I thought it was a good idea, but actually Linus stated a
negative opinion (https://lkml.org/lkml/2019/9/29/227

). I admit it
is ugly and annoying.

The baseline I'd like to keep is the compile-test of uapi headers.
(Otherwise, kernel developers have no way to ensure the correctness
of the exported headers.)

I will maintain a small build rule in usr/include/Makefile.
Remove the other header test functionality.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Commit 40df759e ("kbuild: Fix build with binutils <= 2.19")
introduced ar-option and KBUILD_ARFLAGS to deal with old binutils.

According to Documentation/process/changes.rst, the current minimal
supported version of binutils is 2.21 so you can assume the 'D' option
is always supported. Not only GNU ar but also llvm-ar supports it.

With the 'D' option hard-coded, there is no more user of ar-option
or KBUILD_ARFLAGS.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Tested-by: Nick Desaulniers <ndesaulniers@google.com>

These flags were added by commit 61754c18 ("kbuild: Allow arch
Makefiles to override {cpp,ld,c}flags") to allow ARC to override -O2.

We did not see any other usage after all. Now that ARC switched to
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE_O3, there is no more user of
these variables.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

The only the difference between clean-files and clean-dirs is the -r
option passed to the 'rm' command.

You can always pass -r, and then remove the clean-dirs syntax.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Commit 055efab3 ("kbuild: drop support for cc-ldoption") correctly
removed the cc-ldoption from Documentation/kbuild/makefiles.txt, but
commit cd238eff ("docs: kbuild: convert docs to ReST and rename
to *.rst") revived it. I guess it was a rebase mistake.

Remove it again.

Fixes: cd238eff ("docs: kbuild: convert docs to ReST and rename to *.rst")
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

I see the following warnings when I open this document with a ReST
viewer, retext:

/home/masahiro/ref/linux/Documentation/kbuild/makefiles.rst:1142: (WARNING/2) Inline emphasis start-string without end-string.
/home/masahiro/ref/linux/Documentation/kbuild/makefiles.rst:1152: (WARNING/2) Inline emphasis start-string without end-string.
/home/masahiro/ref/linux/Documentation/kbuild/makefiles.rst:1154: (WARNING/2) Inline emphasis start-string without end-string.

These hunks were added by commit e846f0dc ("kbuild: add support
for ensuring headers are self-contained") and commit 1e21cbfa
("kbuild: support header-test-pattern-y"), respectively. They were
written not for ReST but for the plain text, and merged via the
kbuild tree.

In the same development cycle, this document was converted to ReST
by commit cd238eff ("docs: kbuild: convert docs to ReST and rename
to *.rst"), and merged via the doc sub-system.

Merging them together into Linus' tree resulted in the current situation.

To fix the syntax, surround the asterisks with back-quotes, and
use :: for the code sample.

Fixes: 39ceda5c ("Merge tag 'kbuild-v5.3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild"

)
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Currently, the timestamp of module linker scripts are not checked.
Add them to the dependency of modules so they are correctly rebuilt.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

These three variables are not intended to be tweaked by users.
Move them from kbuild.rst to makefiles.rst.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

'make clean' descends into ./Kbuild, but does not clean anything
since everything is added to no-clean-files.

There is no need to descend to ./Kbuild in the first place.
We can drop the no-clean-files assignment.

With this, there is no more user of no-clean-files. I will keep it
for a while to see whether a new user will appear.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

The assembler files in the kernel are *.S instead of *.s, so they must
be preprocessed. Since 'as' of GNU binutils is not able to preprocess,
we always use $(CC) as an assembler driver.

$(AS) is almost unused in Kbuild. As of v5.2, there is just one place
that directly invokes $(AS).

  $ git grep -e '$(AS)' -e '${AS}' -e '$AS' -e '$(AS:' -e '${AS:' -- :^Documentation
  drivers/net/wan/Makefile:  AS68K = $(AS)

The documentation about *_AFLAGS* sounds like the flags were passed
to $(AS). This is somewhat misleading.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Nathan Chancellor <natechancellor@gmail.com>

The tag ".. include" should be replaced by ".. literalinclude" at
issues.rst, otherwise it causes TeX to crash due to excessive usage
of stack with Sphinx 2.0.

While here, solve a few minor issues at the kbuild book output by
adding extra blank lines.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>

The kbuild documentation clearly shows that the documents
there are written at different times: some use markdown,
some use their own peculiar logic to split sections.

Convert everything to ReST without affecting too much
the author's style and avoiding adding uneeded markups.

The conversion is actually:
  - add blank lines and identation in order to identify paragraphs;
  - fix tables markups;
  - add some lists markups;
  - mark literal blocks;
  - adjust title markups.

At its new index.rst, let's add a :orphan: while this is not linked to
the main index.rst file, in order to avoid build warnings.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

If you want to see if your linker supports a certain flag, then ask the
linker directly with ld-option (not the compiler with cc-ldoption).
Checking for linker flag support is an antipattern that complicates the
usage of various linkers other than bfd via -fuse-ld={bfd|gold|lld}.

Cc: clang-built-linux@googlegroups.com
Suggested-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Currently, every arch/*/include/uapi/asm/Kbuild explicitly includes
the common Kbuild.asm file. Factor out the duplicated include directives
to scripts/Makefile.asm-generic so that no architecture would opt out
of the mandatory-y mechanism.

um is not forced to include mandatory-y since it is a very exceptional
case which does not support UAPI.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

There is no more direct user of this macro; it is only used by
cc-ifversion.

Calling this macro is not efficient since it invokes the compiler to
get the compiler version. CONFIG_GCC_VERSION is already calculated in
the Kconfig stage, so Makefile can reuse it.

Here is a note about the slight difference between cc-version and
CONFIG_GCC_VERSION:

When using Clang, cc-version is evaluated to '0402' because Clang
defines __GNUC__ and __GNUC__MINOR__, and looks like GCC 4.2 in the
version point of view. On the other hand, CONFIG_GCC_VERSION=0
when $(CC) is clang.

There are currently two users of cc-ifversion:
  arch/mips/loongson64/Platform
  arch/powerpc/Makefile

They are not affected by this change.

The format of cc-version is <major><minor>, while CONFIG_GCC_VERSION
<major><minor><patch>. I adjusted cc-ifversion for the difference of
the number of digits.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

The symbol table in the final archive is unneeded; the linker does not
require the symbol table after the --whole-archive option. Every object
file in the archive is included in the link anyway.

Pass thin archives from subdirectories directly to the linker, and
remove the final archiving step.

Fix up the document and comments as well.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Nicholas Piggin <npiggin@gmail.com>

Some time ago, Sam pointed out a certain degree of overwrap between
generic-y and mandatory-y. (https://lkml.org/lkml/2017/7/10/121

)

I tweaked the meaning of mandatory-y a little bit; now it defines the
minimum set of ASM headers that all architectures must have.

If arch does not have specific implementation of a mandatory header,
Kbuild will let it fallback to the asm-generic one by automatically
generating a wrapper. This will allow to drop lots of redundant
generic-y defines.

Previously, "mandatory" was used in the context of UAPI, but I guess
this can be extended to kernel space ASM headers.

Suggested-by: Sam Ravnborg <sam@ravnborg.org>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Sam Ravnborg <sam@ravnborg.org>

The last user of cc-fullversion was removed by commit f2910f0e
("powerpc: remove old GCC version checks").

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>