Skip to content
Snippets Groups Projects
  1. Dec 24, 2020
    • Sumera Priyadarsini's avatar
      scripts: coccicheck: Correct usage of make coccicheck · d8f6e5c6
      Sumera Priyadarsini authored
      
      The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the
      error:
              ./scripts/coccicheck: line 65: -1: shift count out of range
      
      This happens because every time the C variable is specified,
      the shell arguments need to be "shifted" in order to take only
      the last argument, which is the C file to test. These shell arguments
      mostly comprise flags that have been set in the Makefile. However,
      when coccicheck is specified in the make command as a rule, the
      number of shell arguments is zero, thus passing the invalid value -1
      to the shift command, resulting in an error.
      
      Modify coccicheck to print correct usage of make coccicheck so as to
      avoid the error.
      
      Signed-off-by: default avatarSumera Priyadarsini <sylphrenadin@gmail.com>
      Signed-off-by: default avatarJulia Lawall <Julia.Lawall@inria.fr>
      d8f6e5c6
  2. Dec 12, 2020
  3. Oct 12, 2020
  4. Sep 27, 2020
    • Sumera Priyadarsini's avatar
      scripts: coccicheck: Change default value for parallelism · e16a7c47
      Sumera Priyadarsini authored
      
      By default, coccicheck utilizes all available threads to implement
      parallelisation. However, when all available threads are used,
      a decrease in performance is noted. The elapsed time is  minimum
      when at most one thread per core is used.
      
      For example, on benchmarking the semantic patch kfree.cocci for
      usb/serial using hyperfine, the outputs obtained for J=5 and J=2
      are 1.32 and 1.90 times faster than those for J=10 and J=9
      respectively for two separate runs. For the larger drivers/staging
      directory, minimium elapsed time is obtained for J=3 which is 1.86
      times faster than that for J=12. The optimal J value does not
      exceed 6 in any of the test runs. The benchmarks are run on a machine
      with 6 cores, with 2 threads per core, i.e, 12 hyperthreads in all.
      
      To improve performance, modify coccicheck to use at most only
      one thread per core by default.
      
      Signed-off-by: default avatarSumera Priyadarsini <sylphrenadin@gmail.com>
      Signed-off-by: default avatarJulia Lawall <Julia.Lawall@inria.fr>
      e16a7c47
  5. Sep 13, 2020
  6. Aug 21, 2020
  7. Aug 13, 2018
  8. Feb 01, 2018
  9. Nov 23, 2017
    • Masahiro Yamada's avatar
      coccinelle: fix parallel build with CHECK=scripts/coccicheck · d7059ca0
      Masahiro Yamada authored
      
      The command "make -j8 C=1 CHECK=scripts/coccicheck" produces
      lots of "coccicheck failed" error messages.
      
      Julia Lawall explained the Coccinelle behavior as follows:
      "The problem on the Coccinelle side is that it uses a subdirectory
      with the name of the semantic patch to store standard output and
      standard error for the different threads.  I didn't want to use a
      name with the pid, so that one could easily find this information
      while Coccinelle is running.  Normally the subdirectory is cleaned
      up when Coccinelle completes, so there is only one of them at a time.
      Maybe it is best to just add the pid.  There is the risk that these
      subdirectories will accumulate if Coccinelle crashes in a way such
      that they don't get cleaned up, but Coccinelle could print a warning
      if it detects this case, rather than failing."
      
      When scripts/coccicheck is used as CHECK tool and -j option is given
      to Make, the whole of build process runs in parallel.  So, multiple
      processes try to get access to the same subdirectory.
      
      I notice spatch creates the subdirectory only when it runs in parallel
      (i.e. --jobs <N> is given and <N> is greater than 1).
      
      Setting NPROC=1 is a reasonable solution; spatch does not create the
      subdirectory.  Besides, ONLINE=1 mode takes a single file input for
      each spatch invocation, so there is no reason to parallelize it in
      the first place.
      
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Acked-by: default avatarJulia Lawall <Julia.Lawall@lip6.fr>
      d7059ca0
  10. Nov 14, 2017
  11. Nov 02, 2017
    • Greg Kroah-Hartman's avatar
      License cleanup: add SPDX GPL-2.0 license identifier to files with no license · b2441318
      Greg Kroah-Hartman authored
      
      Many source files in the tree are missing licensing information, which
      makes it harder for compliance tools to determine the correct license.
      
      By default all files without license information are under the default
      license of the kernel, which is GPL version 2.
      
      Update the files which contain no license information with the 'GPL-2.0'
      SPDX license identifier.  The SPDX identifier is a legally binding
      shorthand, which can be used instead of the full boiler plate text.
      
      This patch is based on work done by Thomas Gleixner and Kate Stewart and
      Philippe Ombredanne.
      
      How this work was done:
      
      Patches were generated and checked against linux-4.14-rc6 for a subset of
      the use cases:
       - file had no licensing information it it.
       - file was a */uapi/* one with no licensing information in it,
       - file was a */uapi/* one with existing licensing information,
      
      Further patches will be generated in subsequent months to fix up cases
      where non-standard license headers were used, and references to license
      had to be inferred by heuristics based on keywords.
      
      The analysis to determine which SPDX License Identifier to be applied to
      a file was done in a spreadsheet of side by side results from of the
      output of two independent scanners (ScanCode & Windriver) producing SPDX
      tag:value files created by Philippe Ombredanne.  Philippe prepared the
      base worksheet, and did an initial spot review of a few 1000 files.
      
      The 4.13 kernel was the starting point of the analysis with 60,537 files
      assessed.  Kate Stewart did a file by file comparison of the scanner
      results in the spreadsheet to determine which SPDX license identifier(s)
      to be applied to the file. She confirmed any determination that was not
      immediately clear with lawyers working with the Linux Foundation.
      
      Criteria used to select files for SPDX license identifier tagging was:
       - Files considered eligible had to be source code files.
       - Make and config files were included as candidates if they contained >5
         lines of source
       - File already had some variant of a license header in it (even if <5
         lines).
      
      All documentation files were explicitly excluded.
      
      The following heuristics were used to determine which SPDX license
      identifiers to apply.
      
       - when both scanners couldn't find any license traces, file was
         considered to have no license information in it, and the top level
         COPYING file license applied.
      
         For non */uapi/* files that summary was:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|-------
         GPL-2.0                                              11139
      
         and resulted in the first patch in this series.
      
         If that file was a */uapi/* path one, it was "GPL-2.0 WITH
         Linux-syscall-note" otherwise it was "GPL-2.0".  Results of that was:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|-------
         GPL-2.0 WITH Linux-syscall-note                        930
      
         and resulted in the second patch in this series.
      
       - if a file had some form of licensing information in it, and was one
         of the */uapi/* ones, it was denoted with the Linux-syscall-note if
         any GPL family license was found in the file or had no licensing in
         it (per prior point).  Results summary:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|------
         GPL-2.0 WITH Linux-syscall-note                       270
         GPL-2.0+ WITH Linux-syscall-note                      169
         ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause)    21
         ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)    17
         LGPL-2.1+ WITH Linux-syscall-note                      15
         GPL-1.0+ WITH Linux-syscall-note                       14
         ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause)    5
         LGPL-2.0+ WITH Linux-syscall-note                       4
         LGPL-2.1 WITH Linux-syscall-note                        3
         ((GPL-2.0 WITH Linux-syscall-note) OR MIT)              3
         ((GPL-2.0 WITH Linux-syscall-note) AND MIT)             1
      
         and that resulted in the third patch in this series.
      
       - when the two scanners agreed on the detected license(s), that became
         the concluded license(s).
      
       - when there was disagreement between the two scanners (one detected a
         license but the other didn't, or they both detected different
         licenses) a manual inspection of the file occurred.
      
       - In most cases a manual inspection of the information in the file
         resulted in a clear resolution of the license that should apply (and
         which scanner probably needed to revisit its heuristics).
      
       - When it was not immediately clear, the license identifier was
         confirmed with lawyers working with the Linux Foundation.
      
       - If there was any question as to the appropriate license identifier,
         the file was flagged for further research and to be revisited later
         in time.
      
      In total, over 70 hours of logged manual review was done on the
      spreadsheet to determine the SPDX license identifiers to apply to the
      source files by Kate, Philippe, Thomas and, in some cases, confirmation
      by lawyers working with the Linux Foundation.
      
      Kate also obtained a third independent scan of the 4.13 code base from
      FOSSology, and compared selected files where the other two scanners
      disagreed against that SPDX file, to see if there was new insights.  The
      Windriver scanner is based on an older version of FOSSology in part, so
      they are related.
      
      Thomas did random spot checks in about 500 files from the spreadsheets
      for the uapi headers and agreed with SPDX license identifier in the
      files he inspected. For the non-uapi files Thomas did random spot checks
      in about 15000 files.
      
      In initial set of patches against 4.14-rc6, 3 files were found to have
      copy/paste license identifier errors, and have been fixed to reflect the
      correct identifier.
      
      Additionally Philippe spent 10 hours this week doing a detailed manual
      inspection and review of the 12,461 patched files from the initial patch
      version early this week with:
       - a full scancode scan run, collecting the matched texts, detected
         license ids and scores
       - reviewing anything where there was a license detected (about 500+
         files) to ensure that the applied SPDX license was correct
       - reviewing anything where there was no detection but the patch license
         was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
         SPDX license was correct
      
      This produced a worksheet with 20 files needing minor correction.  This
      worksheet was then exported into 3 different .csv files for the
      different types of files to be modified.
      
      These .csv files were then reviewed by Greg.  Thomas wrote a script to
      parse the csv files and add the proper SPDX tag to the file, in the
      format that the file expected.  This script was further refined by Greg
      based on the output to detect more types of files automatically and to
      distinguish between header and source .c files (which need different
      comment types.)  Finally Greg ran the script using the .csv files to
      generate the patches.
      
      Reviewed-by: default avatarKate Stewart <kstewart@linuxfoundation.org>
      Reviewed-by: default avatarPhilippe Ombredanne <pombredanne@nexb.com>
      Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
      b2441318
  12. Oct 11, 2016
  13. Jul 22, 2016
    • Luis R. Rodriguez's avatar
      coccicheck: refer to Documentation/coccinelle.txt and wiki · c100d537
      Luis R. Rodriguez authored
      Refer to the Documentation/coccinelle.txt and supplemental documentation
      on the wiki:
      
      https://bottest.wiki.kernel.org/coccicheck
      
      
      
      This page shall always refer to the linux-next iteration of scripts/coccicheck.
      
      v4: only refer to the wiki as supplemental documentation, and also
          update Documentation/coccinelle.txt.
      
      Signed-off-by: default avatarLuis R. Rodriguez <mcgrof@kernel.org>
      Acked-by: default avatarNicolas Palix <nicolas.palix@imag.fr>
      Signed-off-by: default avatarMichal Marek <mmarek@suse.com>
      c100d537
    • Luis R. Rodriguez's avatar
      coccicheck: add support for requring a coccinelle version · a9e064c0
      Luis R. Rodriguez authored
      
      Enable Coccinelle SmPL patches to require a specific version of
      Coccinelle. In the event that the version does not match we just
      inform the user, if the user asked to go through all SmPL patches
      we just inform them of the need for a new version of coccinelle for
      the SmPL patch and continue on with the rest.
      
      This uses the simple kernel scripts/ld-version.sh to create a weight
      on the version provided by spatch. The -dirty attribute is ignored if
      supplied, the benefit of scripts/ld-version.sh is it has a long history
      and well tested.
      
      While at it, document the // Options stuff as well.
      
      v4: Document // Options and // Requires as well on
          Documentation/coccinelle.txt.
      
      Signed-off-by: default avatarLuis R. Rodriguez <mcgrof@kernel.org>
      Acked-by: default avatarNicolas Palix <nicolas.palix@imag.fr>
      Signed-off-by: default avatarMichal Marek <mmarek@suse.com>
      a9e064c0
    • Luis R. Rodriguez's avatar
      coccicheck: replace --very-quiet with --quiet when debugging · 5c384dba
      Luis R. Rodriguez authored
      
      When debugging (using --profile or --show-trying) you want to
      avoid supressing output,  use --quiet instead. While at it, extend
      documentation for SPFLAGS use.
      
      For instance one can use:
      
      $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
      $ make coccicheck DEBUG_FILE="poo.err" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c
      
      Expand Documentation/coccinelle.txt as well.
      
      v4: expand Documentation/coccinelle.txt
      v3: rebased, resolve conflicts, expand Documentation/coccinelle.txt
      v2: use egrep instead of the *"=--option"* check, this doesn't work for
          disjunctions.
      
      Signed-off-by: default avatarLuis R. Rodriguez <mcgrof@kernel.org>
      Acked-by: default avatarJulia Lawall <julia.lawall@lip6.fr>
      Signed-off-by: default avatarMichal Marek <mmarek@suse.com>
      5c384dba
    • Luis R. Rodriguez's avatar
      coccicheck: add support for DEBUG_FILE · be1fa900
      Luis R. Rodriguez authored
      
      Enable to capture stderr via a DEBUG_FILE variable passed to
      coccicheck. You can now do:
      
      $ rm -f cocci.err
      $ export COCCI=scripts/coccinelle/free/kfree.cocci
      $ make coccicheck MODE=report DEBUG_FILE=cocci.err
      ...
      $ cat cocci.err
      
      This will be come more useful once we add support to
      use more things which would go into stderr, such as
      profiling. That will be done separately in another
      commit.
      
      Expand Documentation/coccinelle.txt with details.
      
      Signed-off-by: default avatarLuis R. Rodriguez <mcgrof@kernel.org>
      Acked-by: default avatarNicolas Palix <nicolas.palix@imag.fr>
      Signed-off-by: default avatarMichal Marek <mmarek@suse.com>
      be1fa900
    • Luis R. Rodriguez's avatar
      coccicheck: enable parmap support · c930a1b2
      Luis R. Rodriguez authored
      
      Coccinelle has had parmap support since 1.0.2, this means
      it supports --jobs, enabling built-in multithreaded functionality,
      instead of needing one to script it out. Just look for --jobs
      in the help output to determine if this is supported and use it
      only if your number of processors detected is > 1.
      
      If parmap is enabled also enable the load balancing to be dynamic, so
      that if a thread finishes early we keep feeding it.
      
      stderr is currently sent to /dev/null, addressing a way to capture
      that will be addressed next.
      
      If --jobs is not supported we fallback to the old mechanism.
      We expect to deprecate the old mechanism as soon as we can get
      confirmation all users are ready.
      
      While at it propagate back into the shell script any coccinelle error
      code. When used in serialized mode where all cocci files are run this
      also stops processing if an error has occured. This lets us handle some
      errors in coccinelle cocci files and if they bail out we should inspect
      the errors. This will be more useful later to help annotate coccinelle
      version dependency requirements. This will let you run only SmPL files
      that your system supports.
      
      Extend Documentation/coccinelle.txt as well.
      
      As a small example, prior to this change, on an 8-core system:
      
      Before:
      
      $ export COCCI=scripts/coccinelle/free/kfree.cocci
      $ time make coccicheck MODE=report
      ...
      
      real    29m14.912s
      user    103m1.796s
      sys     0m4.464s
      
      After:
      
      real    16m22.435s
      user    128m30.060s
      sys     0m2.712s
      
      v4:
      
      o expand Documentation/coccinelle.txt to reflect parmap support info
      o update commit log to reflect what we actually do now with stderr
      o split out DEBUG_FILE use into another patch
      o detect number of CPUs and if its 1 then skip parmap support,
        note that if you still support parmap, but have 1 CPU you will
        also go through the new branches, so the old complex multithreaded process
        is skipped as well.
      
      v3:
      
      o move USE_JOBS to avoid being overriden
      
      v2:
      
      o redirect coccinelle stderr to /dev/null by default and
        only if DEBUG_FILE is used do we pass it to a file
      o fix typo of paramap/parmap
      
      Signed-off-by: default avatarLuis R. Rodriguez <mcgrof@kernel.org>
      Acked-by: default avatarNicolas Palix <nicolas.palix@imag.fr>
      Signed-off-by: default avatarMichal Marek <mmarek@suse.com>
      c930a1b2
    • Luis R. Rodriguez's avatar
      coccicheck: make SPFLAGS more useful · 8e826ad5
      Luis R. Rodriguez authored
      
      SPFLAGS is set early, it means that any heuristics done on
      coccicheck cannot be overridden currently. Move SPFLAGS
      after OPTIONS and set this at the end. This lets you override
      any heuristics as coccinelle treats conflicts by only listening
      to the last option that makes sense.
      
      v3: this patch was added in the v3 series
      v4: Update Documentation/coccinelle.txt explaining how
          SPFLAGS works as well.
      
      Signed-off-by: default avatarLuis R. Rodriguez <mcgrof@kernel.org>
      Acked-by: default avatarNicolas Palix <nicolas.palix@imag.fr>
      Signed-off-by: default avatarMichal Marek <mmarek@suse.com>
      8e826ad5
    • Luis R. Rodriguez's avatar
      coccicheck: move spatch binary check up · 13d94865
      Luis R. Rodriguez authored
      
      This has no functional changes. This is being done
      to enable us to later use spatch binary for some
      flag checking for certain features early on.
      
      Signed-off-by: default avatarLuis R. Rodriguez <mcgrof@kernel.org>
      Acked-by: default avatarNicolas Palix <nicolas.palix@imag.fr>
      Signed-off-by: default avatarMichal Marek <mmarek@suse.com>
      13d94865
  14. Jun 20, 2016
  15. May 19, 2016
  16. Oct 26, 2015
  17. Jul 03, 2013
  18. Jun 14, 2013
  19. Apr 08, 2013
  20. Feb 22, 2013
  21. Oct 11, 2012
  22. Feb 24, 2012
  23. Jan 26, 2012
  24. Jan 14, 2012
  25. Oct 27, 2010
Loading